Microflora prediction and petroleum pollution remediation method based on machine learning

Abstract

The invention discloses a flora prediction and petroleum pollution remediation method based on machine learning. The method comprises the following steps: collecting multiple groups of experimental data of a diesel oil pollution sample treated by a microbial agent, extracting environmental factors, microbial community characteristics and target response variables to construct a training data set after missing value processing, abnormal value detection and standardized pretreatment, and importing the data set into a preset machine learning model to obtain a training result; carrying out feature learning, classification training and hyper-parameter optimization by adopting a GridSearchCV method in combination with 10-fold cross validation; evaluating the correlation between a target response variable classification result and the features through a multivariable Pearson's correlation matrix, and constructing an optimal test set; and finally, selecting an optimal prediction model according to a preset index. According to the method, the model training quality is improved through data preprocessing and correlation analysis, efficient flora prediction and algorithm application evaluation are achieved by means of multiple machine learning algorithms, scientific support is provided for petroleum pollution remediation, and remediation accuracy and efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of ecological restoration, and more specifically, to a method for microbial community prediction and oil pollution remediation based on machine learning. Background Technology

[0002] As a fundamental energy source and chemical raw material for modern industrial society, petroleum and its derivatives inevitably cause numerous environmental pollution incidents during their extraction, transportation, processing, and use. According to the International Energy Agency (IEA), more than a thousand oil pollution incidents occur globally each year due to oil spills, accidental discharges, and industrial wastewater discharges, severely impacting the stability and sustainability of soil, groundwater, and marine ecosystems (IEA, 2022). Petroleum pollutants not only contain high concentrations of alkanes, aromatic hydrocarbons, and colloidal substances, but also large amounts of polycyclic aromatic hydrocarbons (PAHs) and benzene compounds, which have carcinogenic, teratogenic, and mutagenic effects, seriously threatening human health and ecological security (Zhang et al., 2021).

[0003] Traditional physicochemical remediation methods, such as incineration, leaching, and chemical oxidation, while possessing certain removal efficiencies, generally suffer from high energy consumption, high costs, and a significant risk of secondary pollution, and are difficult to achieve in-situ remediation and ecological restoration of contaminated sites. Microbial remediation technology, on the other hand, has gained widespread recognition and application in the environmental remediation field due to its advantages of low cost, ease of operation, and strong environmental compatibility. However, a major challenge currently facing microbial remediation technology lies in the chaotic and untargeted market for microbial agents. A wide variety of microbial agents are available on the market, but their effectiveness varies greatly, failing to meet the remediation needs under different environmental conditions. This chaotic situation has long plagued the remediation industry, and an innovative solution is urgently needed.

[0004] Environmental differences are a key factor contributing to variations in the effectiveness of microbial agents. Although pollutant degradation during environmental remediation is influenced by various environmental factors, specifically designed microbial agents often exhibit superior pollutant degradation capabilities. Currently, microbial-based remediation technologies are not widely adopted, microbial agents lack specificity, and often show better results in laboratory settings than in actual field applications. Furthermore, testing is costly and time-consuming, making it unsuitable for large-scale remediation of contaminated sites. Therefore, providing a scalable and replicable method for constructing targeted microbial agents based on machine learning to address the differences in hydrogeological conditions and pollutant distribution characteristics of organically contaminated sites is a crucial issue in ecological remediation. Summary of the Invention

[0005] To address the issue of poor adaptability of microbial agents for contaminated sites, this invention aims to provide a method for constructing microbial agents using machine learning combined with amplicon sequencing. Based on on-site hydrogeological conditions and pollutant characteristics, this method can quickly predict the pollutant degradation function of compound microbial agents, thereby determining whether the microbial agent formulation is suitable for a specific contaminated site and providing a reference for the formulation of microbial remediation solutions.

[0006] The first aspect of this invention provides a method for microbial community prediction and oil pollution remediation based on machine learning, comprising: S1: Collect experimental data of multiple groups of diesel pollution samples treated with microbial agents, and perform missing value processing, outlier detection and data standardization preprocessing on the experimental data; S2: Extract environmental factors, microbial community characteristics, and target response variables from experimental data and form a training dataset; S3: The training dataset is divided into training and test sets and imported into various preset machine learning models for feature learning and classification training. Hyperparameter optimization is performed using the GridSearchCV method combined with 10-fold cross-validation for data training. S4: Introduce a multivariate Pearson correlation matrix of environmental factors and microbial community characteristics from the training dataset to evaluate the classification results and correlations of the target response variables, screen out variables with high and low correlations, and set the preferred test set; S5: Import the second test set into various preset machine learning models for training, evaluation, and secondary optimization, and select the prediction model through preset model indicators.

[0007] In this solution, S1 specifically refers to: Experimental data include environmental factors, microbial community characteristics, and the target variable Above70; Environmental factors include diesel, pH, Tm, and DO parameters; Microbial community characteristics were determined by measuring the relative abundance data of multiple genera using 16S rRNA gene amplicon sequencing, which was used to characterize the microbial community composition of different groups. The target variable Above70 is a binary categorical variable, indicating whether the microbial agent achieves a total petroleum hydrocarbon degradation efficiency of ≥70%.

[0008] In this scheme, S1 further includes: Missing and outlier analysis was performed on the experimental data, and numerical imputation was performed using the mean and median; Analysis of relative abundance characteristics of microorganisms Based on the characteristics of the microbial community, key variables with sensitivity and extreme values ​​in relative abundance were screened out. Box plots were drawn for the key variables and outliers were identified. After removing outliers, the outlier data were supplemented according to the distribution characteristics. In the experimental data, all numerical variables were standardized using Z-score.

[0009] In this scheme, S2 specifically refers to: Environmental factors, microbial community characteristics, and target response variables were extracted from the experimental data. Variables were then filtered, retaining only those features relevant to model building, including multiple columns of continuous microbial relative abundance data, key environmental factors, and target response variables used for supervised learning in the training set. The training dataset was obtained based on the extraction and filtering of experimental data; The training dataset is divided into training and test sets using a 75%:25% ratio.

[0010] In this solution, S3 specifically refers to: A variety of preset machine learning models are available, including linear models, Bayesian models, single-tree models, forest models, boosting models, bagging models, SVM models, discriminant analysis models, Gaussian processes, and lazy learning models. All preset machine learning models are based on the Python 3.10 environment and use the scikit-learn and XGBoost libraries for learning and training. Hyperparameter optimization was performed using the GridSearchCV method and 10-fold cross-validation for data training. Classification performance was evaluated by comprehensively assessing accuracy, precision, recall, F1-score, and AUC-ROC.

[0011] In this solution, S4 specifically refers to: From the training dataset, multiple variables were set for pollution indicators, environmental factors, and microbial community indicators, and the Pearson correlation index between the variables was calculated comprehensively. A correlation matrix among multiple variables was constructed based on the Pearson correlation index; In the correlation matrix, variables that are highly correlated with and lowly correlated with the total petroleum hydrocarbon (TPH) variable are selected and labeled as the first variable and the second variable, respectively. The test set is set based on the first variable. The process is as follows: in the training dataset, analyze the range of variation of the first variable, and set multiple continuously varying test values ​​within the range to ensure that the multiple test values ​​cover the range of variation. Based on each test value, set the corresponding test data and classification results to form the first test set. The second test set is set based on the second variable.

[0012] In this solution, S5 specifically refers to: The second test set was imported into various preset machine learning models for training, evaluation, and secondary optimization. We evaluate multiple preset model indicators by weighting and select the best prediction model.

[0013] In this scheme, the prediction model can be one or more models.

[0014] A second aspect of the present invention also provides a machine learning-based microbial community prediction and oil pollution remediation system. The system includes a memory and a processor. The memory includes a machine learning-based microbial community prediction and oil pollution remediation program. When executed by the processor, the machine learning-based microbial community prediction and oil pollution remediation program performs the following steps: S1: Collect experimental data of multiple groups of diesel pollution samples treated with microbial agents, and perform missing value processing, outlier detection and data standardization preprocessing on the experimental data; S2: Extract environmental factors, microbial community characteristics, and target response variables from experimental data and form a training dataset; S3: The training dataset is divided into training and test sets and imported into various preset machine learning models for feature learning and classification training. Hyperparameter optimization is performed using the GridSearchCV method combined with 10-fold cross-validation for data training. S4: Introduce a multivariate Pearson correlation matrix of environmental factors and microbial community characteristics from the training dataset to evaluate the classification results and correlations of the target response variables, screen out variables with high and low correlations, and set the preferred test set; S5: Import the second test set into various preset machine learning models for training, evaluation, and secondary optimization, and select the prediction model through preset model indicators.

[0015] A third aspect of the present invention also provides a computer-readable storage medium comprising a machine learning-based microbial community prediction and oil pollution remediation program, wherein when the machine learning-based microbial community prediction and oil pollution remediation program is executed by a processor, it implements the steps of the machine learning-based microbial community prediction and oil pollution remediation method as described in any of the preceding claims.

[0016] To address the issue of poor adaptability of microbial agents for contaminated sites, this invention aims to provide a method for constructing microbial agents using machine learning combined with amplicon sequencing. Based on on-site hydrogeological conditions and pollutant characteristics, this method can quickly predict the pollutant degradation function of compound microbial agents, thereby determining whether the microbial agent formulation is suitable for a specific contaminated site and providing a reference for the formulation of microbial remediation solutions. Attached Figure Description

[0017] Figure 1 A flowchart of a machine learning-based microbial community prediction and oil pollution remediation method of the present invention is shown. Figure 2A flowchart illustrating the correlation matrix of oil pollutants, environmental factors, and microbial communities according to the present invention is shown. Figure 3 The average ROC curves of various models of this invention are shown; Figure 4 The following diagram illustrates the performance metrics of various models of the present invention; Figure 5 A block diagram of a microbial community prediction and oil pollution remediation system based on machine learning is shown in this invention. Detailed Implementation

[0018] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0019] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0020] Figure 1 The flowchart of a machine learning-based microbial community prediction and oil pollution remediation method of the present invention is shown.

[0021] like Figure 1 As shown, the first aspect of this invention provides a method for microbial community prediction and oil pollution remediation based on machine learning, comprising: S1: Collect experimental data of multiple groups of diesel pollution samples treated with microbial agents, and perform missing value processing, outlier detection and data standardization preprocessing on the experimental data; S2: Extract environmental factors, microbial community characteristics, and target response variables from experimental data and form a training dataset; S3: The training dataset is divided into training and test sets and imported into various preset machine learning models for feature learning and classification training. Hyperparameter optimization is performed using the GridSearchCV method and 10-fold cross-validation for data training. S4: Introduce a multivariate Pearson correlation matrix of environmental factors and microbial community characteristics from the training dataset to evaluate the classification results and correlations of the target response variables, screen out variables with high and low correlations, and set the preferred test set; S5: Import the second test set into various preset machine learning models for training, evaluation, and secondary optimization, and select the prediction model through preset model indicators.

[0022] According to an embodiment of the present invention, S1 specifically includes: Experimental data include environmental factors, microbial community characteristics, and the target variable Above70; Environmental factors include diesel, pH, Tm, and DO parameters; Microbial community characteristics were determined by measuring the relative abundance data of multiple genera using 16S rRNA gene amplicon sequencing, which was used to characterize the microbial community composition of different groups. The target variable Above70 is a binary categorical variable, indicating whether the microbial agent achieves a total petroleum hydrocarbon degradation efficiency of ≥70%.

[0023] In this embodiment of the invention, experimental data from 75 sets of microbial agents treating diesel contaminated samples were collected, divided into a training set (train.csv) and a test set (test.csv). All experiments were conducted under laboratory conditions to ensure the comparability and repeatability of the data. Each sample set contains the following information: Environmental Variables: The data also includes information on several environmental factors and pollutants, including pollution categories, used to describe external environmental conditions and potential pollution gradients. Diesel: Initial diesel concentration (μg / L); pH: Acidity / alkalinity of the culture system; Tm: Culture temperature (°C); DO: Dissolved oxygen concentration (mg / L).

[0024] Microbial Community Features: Based on 16S rRNA gene amplicon sequencing, relative abundance data of 32 bacterial genera were obtained (e.g., Pseudomonas , Acinetobacter , Flavobacterium , Gordonia (etc.), used to characterize the microbial community composition of different groups.

[0025] Target Variable Above70: A binary categorical variable indicating whether the microbial agent achieves a total petroleum hydrocarbon (TPH) degradation efficiency of ≥70% (1 for yes, 0 for no).

[0026] Meanwhile, in addition to the aforementioned input features, the training set also includes target response variables (labels) for supervising the construction of the learning model, used for model training, parameter tuning, and performance evaluation; the test set only contains input features, without label information, and is used for independent verification of the model's generalization performance and final prediction output. By using both the training and test sets simultaneously, this invention can construct, optimize, and evaluate prediction models based on microbial community data, providing a reliable data foundation for subsequent environmental change analysis and functional interpretation.

[0027] In data processing, the `info()` and `sample()` functions can be used to quickly and roughly understand the data type of variables (i.e., qualitative and quantitative). The variables above70 are either outcome or dependent variables (labels), and are binary nominal data types. 1 indicates that the degradation rate of total petroleum hydrocarbons in diesel by the microbial agent is ≥70%, and 0 indicates that the degradation rate is <70%. All other variables are potential predictor variables or independent variables. The `diesel` variable represents the initial concentration of pollutants, in micrograms per liter. Other variables represent the relative abundance of microbial strains.

[0028] In the formation of the training dataset, categorical data can be converted into dummy variables and x and y variables can be defined; one-hot encoding is performed using sklearn (Scikit-learn library) and pandas functions. In this step, x (independent / feature / explanation / prediction / etc.) and y (dependent / target / outcome / response / etc.) variables are defined for data modeling.

[0029] To avoid overfitting the model to the data during training, which would reduce its generalization ability to unseen data, this study first uses the `train_test_split` function provided by scikit-learn to divide the training data into training and test sets in a 75%:25% ratio. It is important to emphasize that the model can only access samples in the training set during training, while the test set is completely independent and invisible during training. This effectively prevents the model from obtaining inflated performance by "memorizing" answers, thus ensuring the objectivity and reliability of the evaluation results. In the subsequent model performance comparison section, this study further employs scikit-learn's cross-validation method. By repeatedly training and evaluating the model on multiple different training / validation subsets, cross-validation provides more robust performance estimates and helps reduce the random bias introduced by a single data split.

[0030] This invention constructs multiple predictive models to analyze the changing characteristics of microbial composition under different environmental conditions and evaluate their indicative ability to target response variables. The overall process, from raw data import to predictive model output, covers key steps such as data processing, feature construction, statistical analysis, and machine learning modeling. First, raw microbial sequencing data and their corresponding environmental factor information from multiple sources are collected and imported, integrating data from different batches or formats to form a data matrix (dataset) with a unified structure. Subsequently, all variables are systematically screened and processed, including missing value detection and imputation, filtering of low abundance and low frequency variables, variable standardization, factor variable encoding (such as one-hot encoding), and feature derivation to ensure the quality and consistency of the input data. After data preparation, basic statistical analysis and visualization methods (such as abundance distribution, correlation analysis, principal component analysis, etc.) are used to explore the basic laws governing the microbial community structure characteristics and environmental variables. Based on the preprocessed feature set, various machine learning algorithms (such as random forest, gradient boosting tree, support vector machine, etc.) are used for model training, and the optimal model is selected through cross-validation, hyperparameter optimization, and performance evaluation metrics (such as R², RMSE, AUC, etc.). Finally, the constructed prediction model was used to predict the output of the response variable, and the influence of the microbial community on the target variable was explained by combining the model's importance analysis and feature contribution assessment, providing a basis for subsequent mechanism inference and ecological function analysis.

[0031] It is worth mentioning that traditional microbial agents exhibit significant functional instability in practical applications. Their degradation efficiency is affected by a variety of factors, including: microbial community composition and functional redundancy; environmental physicochemical factors (such as pH, temperature, dissolved oxygen, and pollutant concentration); interactions between microorganisms (such as competition, synergy, and inhibition); and pollutant type and bioavailability.

[0032] Traditional experimental methods rely on extensive indoor culture and degradation experiments, which are time-consuming, costly, and have poor reproducibility, making it difficult to meet the needs of rapid screening and precise application.

[0033] This invention constructs multiple machine learning algorithms to analyze and form a dataset from a single collection of multi-sample experimental data. It evaluates the characteristics and capabilities of different MLA (machine learning algorithms) for predicting the degradation of petroleum hydrocarbons and the multi-factor, multi-variable correlations of different microbial community characteristics with the current pollution status of petroleum hydrocarbons. Simultaneously, it optimizes the results of different MLAs by combining correlation matrices, effectively selecting the MLA prediction model suitable for the current state. This model is then applied to the actual microbial community degradation of petroleum hydrocarbons and petroleum pollution prediction, identifying efficient microbial agent solutions for remediation. This has significant environmental remediation implications and realizes an efficient, low-cost, and scalable method for predicting the function of microbial agents, guiding the scientific formulation and precise application of microbial agents.

[0034] According to an embodiment of the present invention, S1 further includes: Missing and outlier analysis was performed on the experimental data, and numerical imputation was performed using the mean and median; Analysis of relative abundance characteristics of microorganisms Based on the characteristics of the microbial community, key variables with sensitivity and extreme values ​​in relative abundance were screened out. Box plots were drawn for the key variables and outliers were identified. After removing outliers, the outlier data were supplemented according to the distribution characteristics. In the experimental data, all numerical variables were standardized using Z-score.

[0035] It should be noted that, to ensure the stability and interpretability of model training, this invention performs systematic preprocessing on the raw data to ensure that the data quality meets the requirements of subsequent analysis. The checks include variable type identification (e.g., distinguishing between continuous and categorical variables), statistical analysis of missing and outlier proportions, evaluation of variable numerical range and distribution characteristics, and detection of potential outliers. For variables with missing values ​​in the training set, a differentiated imputation strategy is adopted based on data type and variable characteristics. For conventional numerical features, the mean or median is used for imputation to reduce the impact of missing values ​​on the statistical distribution. For biologically sensitive microbial relative abundance features that may have extreme values, outliers are identified by drawing box plots, and after removing obvious anomalies, robust imputation values ​​are selected based on distribution characteristics to avoid amplifying bias. For categorical variables, the mode is used for imputation; if a missing category has potential indicative significance, a separate "missing" category is set to retain information. After handling missing values, all variables undergo a completeness check again to confirm that there are no unusable records, ensuring that the data can directly enter the subsequent feature engineering, statistical analysis, and model building processes. Specifically, this includes: missing value handling, outlier detection, and outlier screening for continuous variables using boxplots and Z-score standardization. To eliminate the influence of different units of measurement on model training, this invention performs Z-score standardization on all numerical variables, as shown in the following formula: ; Where x is the original value, μ is the mean, and σ is the standard deviation.

[0036] According to an embodiment of the present invention, S2 specifically includes: Environmental factors, microbial community characteristics, and target response variables were extracted from the experimental data. Variables were then filtered, retaining only those features relevant to model building, including multiple columns of continuous microbial relative abundance data, key environmental factors, and target response variables used for supervised learning in the training set. The training dataset was obtained based on the extraction and filtering of experimental data; The training dataset is divided into training and test sets using a 75%:25% ratio.

[0037] Here, for the original data (original experimental data), fields with no real information content, such as pure ordinal variables or redundant label columns, can be removed simultaneously to avoid noise interference. Subsequently, a differentiated coding strategy is adopted for the categorical variables involved in the dataset based on their properties: for variables with a large number of categories but clear biological significance, one-hot encoding is used to maintain their information structure; for variables with a small number of categories, integer labels are used for direct mapping; for variables that are presented numerically but are essentially interval classifications, discretization is performed before encoding to avoid erroneous continuity assumptions. All coding rules are applied simultaneously to both the training and test sets to ensure consistency of model input. For continuous variables, especially microbial abundance features with a large distribution span, this invention uses normalization or standardization methods to adjust different variables to a comparable scale range, reducing the impact of dimensional differences on the model training process and improving the model's convergence efficiency and predictive stability.

[0038] This invention does not perform feature dimensionality reduction or screening, and retains all 37 features (32 bacterial genera + 5 environmental variables + total abundance) to fully preserve microbial community information.

[0039] It should be noted that Above70 is the target response variable.

[0040] The training dataset example is as follows: Training set:

[0041] Test set:

[0042] According to an embodiment of the present invention, S3 specifically includes: A variety of preset machine learning models are available, including linear models, Bayesian models, single-tree models, forest models, boosting models, bagging models, SVM models, discriminant analysis models, Gaussian processes, and lazy learning models. All preset machine learning models are based on the Python 3.10 environment and use the scikit-learn and XGBoost libraries for learning and training. Hyperparameter optimization was performed using the GridSearchCV method and 10-fold cross-validation for data training. Classification performance was evaluated by comprehensively assessing accuracy, precision, recall, F1-score, and AUC-ROC.

[0043] It should be noted that this invention compares the performance of seven mainstream machine learning algorithms in binary classification tasks, covering linear models, tree models, and ensemble learning methods. The principles of each algorithm are briefly described below:

[0044]

[0045] Here, all models are implemented using Python 3.10 and the scikit-learn and XGBoost libraries. The dataset is split into 75% training set and 25% test set using the `train_test_split` method, with a random seed of 42 to ensure reproducibility.

[0046] Hyperparameter optimization was performed using the GridSearchCV method with 10-fold cross-validation. The main parameters to be tuned are as follows:

[0047]

[0048]

[0049]

[0050] According to an embodiment of the present invention, S4 specifically includes: From the training dataset, multiple variables were set for pollution indicators, environmental factors, and microbial community indicators, and the Pearson correlation index between the variables was calculated comprehensively. A correlation matrix among multiple variables was constructed based on the Pearson correlation index; In the correlation matrix, variables that are highly correlated with and lowly correlated with the total petroleum hydrocarbon (TPH) variable are selected and labeled as the first variable and the second variable, respectively. The test set is set based on the first variable. The process is as follows: in the training dataset, analyze the range of variation of the first variable, and set multiple continuously varying test values ​​within the range to ensure that the multiple test values ​​cover the range of variation. Based on each test value, set the corresponding test data and classification results to form the first test set. The second test set is set based on the second variable.

[0051] Here, high correlation is defined as a correlation index above 0.6, and low correlation is defined as a correlation index below 0.2. Correlation has both positive and negative dimensions. During classification training, positive and negative correlation attributes can be introduced to define classification results and set up a test set. The preferred test sets include the first and second test sets.

[0052] The process involves setting multiple continuously varying test values ​​within an interval to ensure that these values ​​cover the range of variation. For each test value, corresponding test data is set. Specifically, within the interval of variation of the first variable, N data points can be evenly divided, with consistent changes between any two adjacent data points. These N data points serve as test values. Further, based on the test data, other variables (excluding the target response variable and the first variable) are randomly selected from the training dataset and combined with the test values ​​to form a set of test data. Multiple sets of test data are formed based on multiple test values, resulting in a first test set. In the first test set, since it is a test set for highly correlated variables, the change in total petroleum hydrocarbon (TPH) is relatively large, and the prediction rate for TPH degradation efficiency should be greater than 70%. Corresponding classification results are then set for training.

[0053] For the second test set, the predicted rate of total petroleum hydrocarbon (TPH) degradation efficiency should be less than 70%.

[0054] Here, the Pearson correlation matrix is ​​used to set up an optimal test set, which can reduce the overfitting prediction degree of various MLA models for multimicrobial agents and multiple environmental factors, improve the prediction ability of multimicrobial agents for petroleum hydrocarbon decomposition in complex environments, avoid the situation where the predicted values ​​deviate from the actual value trend, effectively improve the correlation between the actual prediction data and the actual environment, and improve the practical application ability of the model.

[0055] Figure 2 A flowchart illustrating the correlation matrix of oil pollutants, environmental factors, and microbial communities of this invention is shown.

[0056] To explore the potential correlations among pollution indicators, environmental factors, and microbial communities, this invention constructs a Pearson correlation matrix among the variables and conducts a systematic analysis of it.

[0057] like Figure 2 As shown, there is a very high positive correlation between diesel and total petroleum hydrocarbons (TPH) (r=0.93), indicating that their trends are consistent with environmental sample changes and can be considered a strong collinear combination. In subsequent machine learning modeling, a choice between the two could be considered to reduce redundant features. Furthermore, samples labeled Above70 show a moderate negative correlation with both diesel and TPH (r=-0.32 and -0.48), suggesting that high-concentration samples actually have lower oil content, which may reflect differences in different types of pollution or pollutant treatment stages.

[0058] The correlation between pH and various pollutants was less than 0.2, indicating that pH can be considered an independent background factor with limited impact on pollutant abundance. Temperature (Tm) showed a weak positive correlation with diesel fuel and TPH (r≈0.3–0.4), suggesting that the content of oil pollutants increases slightly with increasing temperature. The dissolved oxygen (DO) column had many missing values, making accurate correlation assessment impossible; these values ​​need to be removed or processed in subsequent analyses.

[0059] The correlation between oil pollutants (diesel / TPH) and microbial communities showed significant differences: the genera with the most significant positive correlations were Sphingomonas (r=0.44 / 0.35), Staphylococcus (r=0.31 / 0.27), and Bacillus (r=0.28 / 0.14), suggesting that these bacteria may play an important role as functional groups for petroleum hydrocarbon degradation in polluted environments. The genera with the most significant negative correlations included Acinetobacter (r=-0.48 / -0.37), Dietzia (r=-0.45 / -0.43), and Pseudomonas (r≈-0.02, close to neutral). This negative correlation does not necessarily indicate inhibition, but may suggest that the relative abundance of these microorganisms decreases in high oil concentration environments, or that their ecological niche is more adapted to low oil environments.

[0060] Microbial correlation analysis revealed highly positive correlations (|r|>0.7) in genera pairs including Aeromonas-Pseudonocardia (r=0.92) and Sphingomonas-Bacillus (r=0.73), suggesting potential synergistic or functionally complementary relationships. Highly negative correlations (|r|>0.7) in genera pairs included Sphingomonas-Total_abundance (r=-0.74) and Bacillus-Total_abundance (r=-0.69), reflecting a "dilution effect" where the proportion of specific genera decreases as total abundance increases, potentially stemming from competition or niche exclusion within the community.

[0061] Figure 2 The Pearson correlations between diesel fuel, total petroleum hydrocarbons (TPH), environmental factors (pH, temperature Tm, dissolved oxygen DO), and the relative abundance of microbial genera are presented. Red indicates a positive correlation, and blue indicates a negative correlation; the darker the color, the stronger the correlation. The results show that diesel fuel is highly positively correlated with TPH (r=0.93); Sphingomonas, Staphylococcus, and Bacillus are positively correlated with oils and are potential degradation indicator genera; Acinetobacter and Dietzia are negatively correlated; positive and negative correlation pairs exist among microorganisms, reflecting potential synergistic or competitive relationships.

[0062] When choosing from multiple MLAs, manual selection is often limited by MLA models familiar to experts. Traditional techniques for learning MLAs of microbial community characteristics and environmental factors often use factors such as recall and precision for evaluation, but lack evaluation of the classification results of correlation analysis, which leads to a certain degree of model overfitting. However, using the correlation matrix for secondary testing and evaluation can eliminate a certain degree of overfitting in the prediction model and optimize the prediction of actual petroleum hydrocarbon degradation.

[0063] According to an embodiment of the present invention, S5 specifically includes: The second test set was imported into various preset machine learning models for training, evaluation, and secondary optimization. We evaluate multiple preset model indicators by weighting and select the best prediction model.

[0064] The prediction model can use one or more models. Specifically, one or more models with the best performance can be selected for multiple predictions. For example, the first two MLA models can be selected and combined with preset model indicators for prediction weighting. The prediction value of the model with better performance can be predicted with higher weights.

[0065] It should be noted that, in order to comprehensively evaluate model performance, this invention uses the following four types of preset model indicators to evaluate the model's predictive ability: Accuracy: The proportion of correctly classified samples out of the total sample. Precision: The proportion of samples that were predicted to be positive but were actually positive. Recall: The proportion of samples that were actually positive that were correctly predicted; F1-score: The harmonic mean of precision and recall, which comprehensively reflects the model's performance; AUC-ROC: Area under the receiver operating characteristic curve, which measures the model's discriminative ability.

[0066] Weighted evaluations are calculated using the same weight values ​​by default.

[0067] Figure 3 The average ROC curves of various models of this invention are shown.

[0068] Furthermore, this invention performed 10-fold Shuffle cross-validation on 22 commonly used machine learning algorithms (60% training set, 30% test set, and 10% left blank), and used Accuracy, Precision, Recall, F1-score, and AUC-ROC as evaluation metrics. Figure 3 Provide the average ROC curves for all models.

[0069] Figure 3 In the figure, each ROC curve is the average result calculated on the complete training set after 10-fold cross-validation. The horizontal axis represents the false positive rate (FPR), and the vertical axis represents the true positive rate (TPR); the gray dashed line represents the random classification baseline (AUC=0.5). The legend on the right shows the AUC values ​​of each model, arranged in descending order of the curves. Most ensemble learning methods (AdaBoost, ExtraTrees, GradientBoosting, RandomForest, Bagging, DecisionTree, ExtraTree, QuadraticDiscriminantAnalysis, XGBoost) and Gaussian process classifiers all achieved an AUC=1.00, demonstrating perfect discriminative ability. KNeighbors and NuSVC (AUC≈0.98–0.97) showed near-perfect performance, but still had a small number of false positives. Traditional linear models (LogisticRegressionCV, LinearSVC, SGDClassifier, PassiveAggressiveClassifier, Perceptron) and some Bayesian and SVC models have AUCs ranging from 0.81 to 0.88, which is moderate performance.

[0070] The features used to distinguish the effects of microbial agents on diesel fuel performance prediction (Above70) in the dataset exhibit strong separability: 10 models achieved an average AUC of 1.000, demonstrating "perfect discrimination." This group includes all tree / ensemble methods (AdaBoost, Bagging, ExtraTrees, GradientBoosting, RandomForest, DecisionTree, ExtraTree), Gradient Boosting Tree (XGBoost), Quadratic Discriminant Analysis (QDA), and Gaussian Process Classifier (GPC). KNeighbors and NuSVC achieved AUCs of 0.980 and 0.972, respectively, approaching perfect discrimination but still exhibiting a small number of misclassifications. Traditional linear models (LogisticRegressionCV, LinearSVC, SGDClassifier, PassiveAggressiveClassifier, Perceptron) performed moderately, with AUCs between 0.81 and 0.88, significantly outperforming the random baseline (0.5) but lower than nonlinear / ensemble algorithms. The worst performance was BernoulliNB (AUC=0.786), indicating that its discrete independence assumption does not match the characteristic distribution of this data.

[0071] In a preferred embodiment, the features of the dataset are highly correlated with the labels, and tree models, ensemble models, and boosting algorithms can fully capture feature interactions and nonlinear structures, thus achieving the best performance; while linear models are limited by the linear assumption and their overall performance is slightly lower, but still significantly better than the random baseline.

[0072] Figure 4 The following diagram illustrates the performance metrics of various models of the present invention; In the optimal prediction model selection for S5, to evaluate the model's generalization stability, performance metrics on the training and test sets were compared, such as... Figure 4 As shown, the bar lengths represent the corresponding accuracy rates: in models with AUC=1, the training set accuracy generally reaches 100%. However, performance on the test set varies: ensemble methods such as ExtraTrees, RandomForest, AdaBoost, and XGBoost maintain high test accuracy (≥88%), demonstrating good generalization ability; while individual models such as DecisionTree and QDA have test accuracy dropping to 52–72%, with 3σ fluctuations ranging from 0.29 to 0.37, suggesting significant overfitting or sensitivity to training samples. Although KNeighbors has a near-perfect AUC, its test accuracy is 88.9%, with 3σ=0.27, indicating that it is quite sensitive to noise and boundary samples under the current parameter settings (k=5), requiring further parameter tuning or the use of distance weighting, standardization, or other measures to improve robustness. LogisticRegressionCV has the smallest gap between training and testing (approximately 5%), benefiting from built-in regularization and cross-validation hyperparameter selection, demonstrating the best generalization stability.

[0073] Figure 4 This paper presents the classification accuracy of 22 classic machine learning algorithms under the same dataset and evaluation process. Models are ranked from highest to lowest accuracy to facilitate comparison of performance differences. It can be seen that instance-based and kernel-based models (such as KNeighborsClassifier and NuSVC) perform best, with an accuracy close to 0.90; gradient boosting methods (GradientBoostingClassifier, AdaBoostClassifier) ​​and some tree models maintain high accuracy; linear models (LinearSVC, LogisticRegressionCV, SGDClassifier) ​​are at a moderate level; while Naive Bayes and some discriminative models have relatively low accuracy. These results demonstrate that, for the dataset of this invention, nonlinear and ensemble learning methods have significant advantages in feature discrimination.

[0074] In summary, although several models performed well on average ROC, the examination of training / test differences indicates that overfitting and model stability issues must be taken seriously, especially for single decision trees or discrimination methods with untuned parameters.

[0075] In most machine learning studies, AUC reflects a model's overall ranking ability across different thresholds, but it doesn't reveal the model's actual classification performance at a specific decision threshold. Therefore, relying solely on AUC can lead to a one-sided evaluation of model performance. Precision, Recall, and F1 scores reveal the model's true classification performance at actual thresholds and the sources of misclassification, thus ensuring the scientific rigor of model evaluation and application.

[0076] For all models with AUC=1, their Precision=Recall=F1=1 on the test set indicates that they not only have excellent ranking performance but also achieve error-free classification at a given threshold. This is consistent with... Figure 3 The ROC curves are closely aligned with the top left corner. The linear model's precision is between 0.73 and 0.79, recall between 0.63 and 0.73, and F1 between 0.56 and 0.73, indicating that misclassifications are mainly concentrated on boundary samples. The rise in the ROC curve in the FPR≈0.2–0.4 range corroborates the lower recall / precision of the linear model, suggesting that these models struggle to handle complex discriminative surfaces caused by sample boundaries or nonlinear interactions. Therefore, a comprehensive consideration of threshold selection and cost-sensitive learning strategies is necessary, rather than relying solely on AUC as the sole evaluation metric.

[0077] This invention clarifies the application prospects, interpretability analysis directions, and subsequent data quality and verification strategies of high-performance models by verifying the adaptability differences of various models in petroleum hydrocarbon pollution data discrimination tasks. It has important reference value for building robust microbial agent prediction models and provides a more accurate multi-microbial community and multi-agent formulation adaptation scheme for subsequent complex and multi-scenario petroleum pollution remediation, thereby achieving the purpose of pollution prediction simulation and efficient remediation.

[0078] According to an embodiment of the present invention, it further includes: Sample data were collected at multiple time periods in an oil remediation area, and corresponding second experimental data were obtained at multiple time periods. Using a predictive model, environmental factors and microbial community characteristics of the second experimental data were used as basic variables. The predictive model was used to predict and analyze the total petroleum hydrocarbon degradation efficiency P1 at different time periods. Serialize the multiple time periods P1 to obtain the first sequence; The actual total petroleum hydrocarbon degradation efficiency P2 over multiple time periods was evaluated based on the data from the second experiment. The multiple time segments P1 are serialized to obtain the second sequence; Linear regression fitting is used to fit the first sequence and the second sequence linearly, and the deviation of the linear coefficients after fitting is calculated. Based on the fitting results, linear predictions for the prediction period are performed on the first sequence and the second sequence respectively, and the obtained prediction data are subjected to linear deviation analysis. The effectiveness of the prediction model for total petroleum hydrocarbon degradation efficiency was evaluated through linear coefficient and linear deviation analysis.

[0079] In this embodiment, the total petroleum hydrocarbon degradation efficiency P for each time period is calculated based on the rate of change of petroleum hydrocarbon content between the current time period and the previous and middle time periods. While machine learning-based Above70 variable prediction generally achieves an accuracy rate of over 80%, numerical prediction of total petroleum hydrocarbon degradation efficiency may contain some errors. In multi-time period and rock remediation, there may be a need for precise analysis of the degradation rate. Therefore, it is necessary to quickly evaluate the accuracy or effectiveness of the model's degradation rate prediction to make further effective remediation decisions. The effectiveness of the total petroleum hydrocarbon degradation efficiency prediction can be evaluated through the aforementioned linear analysis to assess the effectiveness of the prediction model's results.

[0080] Linear regression fitting can be performed using the function y=kx+b, and the fitting coefficients are compared using the k value. The prediction period can be evaluated using the predicted values ​​of the next three periods. Furthermore, the effectiveness of the prediction model's degradation rate can be analyzed by comparing the predictive linear correlation between the two sequences, so as to provide precise auxiliary decision-making for microbial contamination remediation.

[0081] Figure 5 A block diagram of a microbial community prediction and oil pollution remediation system based on machine learning is shown in this invention.

[0082] A second aspect of the present invention also provides a machine learning-based microbial community prediction and oil pollution remediation system. The system includes a memory and a processor. The memory includes a machine learning-based microbial community prediction and oil pollution remediation program. When executed by the processor, the machine learning-based microbial community prediction and oil pollution remediation program performs the following steps: S1: Collect experimental data of multiple groups of diesel pollution samples treated with microbial agents, and perform missing value processing, outlier detection and data standardization preprocessing on the experimental data; S2: Extract environmental factors, microbial community characteristics, and target response variables from experimental data and form a training dataset; S3: The training dataset is divided into training and test sets and imported into various preset machine learning models for feature learning and classification training. Hyperparameter optimization is performed using the GridSearchCV method combined with 10-fold cross-validation for data training. S4: Introduce a multivariate Pearson correlation matrix of environmental factors and microbial community characteristics from the training dataset to evaluate the classification results and correlations of the target response variables, screen out variables with high and low correlations, and set the preferred test set; S5: Import the second test set into various preset machine learning models for training, evaluation, and secondary optimization, and select the prediction model through preset model indicators.

[0083] When the above system is running, it can realize all the steps of the above machine learning-based microbial community prediction and oil pollution remediation method.

[0084] A third aspect of the present invention also provides a computer-readable storage medium comprising a machine learning-based microbial community prediction and oil pollution remediation program, wherein when the machine learning-based microbial community prediction and oil pollution remediation program is executed by a processor, it implements the steps of the machine learning-based microbial community prediction and oil pollution remediation method as described in any of the preceding claims.

[0085] This invention discloses a machine learning-based method for microbial community prediction and oil pollution remediation. Multiple sets of experimental data from diesel-contaminated samples treated with microbial agents are collected. After missing value processing, outlier detection, and standardization preprocessing, environmental factors, microbial community characteristics, and target response variables are extracted to construct a training dataset. This dataset is then imported into a pre-defined machine learning model. The GridSearchCV method combined with 10-fold cross-validation is used for feature learning, classification training, and hyperparameter optimization. The correlation between the classification results of the target response variable and the aforementioned features is evaluated using a multivariate Pearson correlation matrix, and an optimal test set is constructed. Finally, the optimal prediction model is selected based on pre-defined indicators. This method improves model training quality through data preprocessing and correlation analysis, and achieves efficient microbial community prediction and algorithm application evaluation using various machine learning algorithms, providing scientific support for oil pollution remediation and improving the accuracy and efficiency of remediation.

[0086] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application can be generated, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic cable, data subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital universal optical disc), or a semiconductor medium (e.g., solid-state drive). In the various embodiments of this application, unless otherwise specified or logically conflicting, the terminology and / or descriptions between different embodiments are consistent and can be mutually referenced. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0087] In the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. In the textual description of the embodiments of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. In this application, "first," "second," and various numerical designations are only for the convenience of description and are not used to limit the scope of the embodiments of this application. For example, they are used to distinguish different messages, rather than to describe a specific order or sequence.

[0088] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The order of the process numbers does not imply the order of execution; the execution order of each process should be determined by its function and internal logic.

[0089] Finally, it should be noted that the above description is only a specific implementation of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the protection scope of this application.

Claims

1. A method for microbial community prediction and oil pollution remediation based on machine learning, characterized in that, include: S1: Collect experimental data of multiple groups of diesel pollution samples treated with microbial agents, and perform missing value processing, outlier detection and data standardization preprocessing on the experimental data; S2: Extract environmental factors, microbial community characteristics, and target response variables from experimental data and form a training dataset; S3: The training dataset is divided into training and test sets and imported into various preset machine learning models for feature learning and classification training. Hyperparameter optimization is performed using the GridSearchCV method combined with 10-fold cross-validation for data training. S4: Introduce a multivariate Pearson correlation matrix of environmental factors and microbial community characteristics from the training dataset to evaluate the classification results and correlations of the target response variables, screen out variables with high and low correlations, and set the preferred test set; S5: Import the second test set into various preset machine learning models for training, evaluation, and secondary optimization, and select the prediction model through preset model indicators.

2. The method for microbial community prediction and oil pollution remediation based on machine learning according to claim 1, characterized in that, Specifically, S1 is: Experimental data include environmental factors, microbial community characteristics, and the target variable Above70; Environmental factors include diesel, pH, Tm, and DO parameters; Microbial community characteristics were determined by measuring the relative abundance data of multiple genera using 16S rRNA gene amplicon sequencing, which was used to characterize the microbial community composition of different groups. The target variable Above70 is a binary categorical variable, indicating whether the microbial agent achieves a total petroleum hydrocarbon degradation efficiency of ≥70%.

3. The method for microbial community prediction and oil pollution remediation based on machine learning according to claim 1, characterized in that, S1 further includes: Missing and outlier analysis was performed on the experimental data, and numerical imputation was performed using the mean and median; Analysis of the relative abundance characteristics of microorganisms Based on the characteristics of the microbial community, key variables with sensitivity and extreme values ​​in relative abundance were screened out. Box plots were drawn for the key variables and outliers were identified. After removing outliers, the outlier data were supplemented according to the distribution characteristics. In the experimental data, all numerical variables were standardized using Z-score.

4. The method for microbial community prediction and oil pollution remediation based on machine learning according to claim 1, characterized in that, Specifically, S2 is: Environmental factors, microbial community characteristics, and target response variables were extracted from the experimental data. Variables were then filtered, retaining only those features relevant to model building, including multiple columns of continuous microbial relative abundance data, key environmental factors, and target response variables used for supervised learning in the training set. The training dataset was obtained based on the extraction and filtering of experimental data; The training dataset is divided into training and test sets by a ratio of 75%:25%.

5. The method for microbial community prediction and oil pollution remediation based on machine learning according to claim 1, characterized in that, Specifically, S3 is: A variety of preset machine learning models are available, including linear models, Bayesian models, single-tree models, forest models, boosting models, bagging models, SVM models, discriminant analysis models, Gaussian processes, and lazy learning models. All preset machine learning models are based on the Python 3.10 environment and use the scikit-learn and XGBoost libraries for learning and training. Hyperparameter optimization was performed using the GridSearchCV method and 10-fold cross-validation for data training. Classification performance was evaluated by comprehensively assessing accuracy, precision, recall, F1-score, and AUC-ROC.

6. The method for microbial community prediction and oil pollution remediation based on machine learning according to claim 1, characterized in that, Specifically, S4 is: From the training dataset, multiple variables were set for pollution indicators, environmental factors, and microbial community indicators, and the Pearson correlation index between the variables was calculated comprehensively. A correlation matrix among multiple variables was constructed based on the Pearson correlation index; In the correlation matrix, variables that are highly correlated with and lowly correlated with the total petroleum hydrocarbon (TPH) variable are selected and labeled as the first variable and the second variable, respectively. The test set is set based on the first variable. The process is as follows: in the training dataset, analyze the range of variation of the first variable, and set multiple continuously varying test values ​​within the range to ensure that the multiple test values ​​cover the range of variation. Based on each test value, set the corresponding test data and classification results to form the first test set. The second test set is set based on the second variable.

7. The method for microbial community prediction and oil pollution remediation based on machine learning according to claim 1, characterized in that, Specifically, S5 is: The second test set was imported into various preset machine learning models for training, evaluation, and secondary optimization. We evaluate multiple preset model indicators by weighting and select the best prediction model.

8. The method for microbial community prediction and oil pollution remediation based on machine learning according to claim 1, characterized in that, The prediction model may employ one or more models.

9. A microbial community prediction and oil pollution remediation system based on machine learning, characterized in that, The system includes a memory and a processor. The memory includes a machine learning-based microbial community prediction and oil pollution remediation program. When the processor executes the machine learning-based microbial community prediction and oil pollution remediation program, it performs the following steps: S1: Collect experimental data of multiple groups of diesel pollution samples treated with microbial agents, and perform missing value processing, outlier detection and data standardization preprocessing on the experimental data; S2: Extract environmental factors, microbial community characteristics, and target response variables from experimental data and form a training dataset; S3: The training dataset is divided into training and test sets and imported into various preset machine learning models for feature learning and classification training. Hyperparameter optimization is performed using the GridSearchCV method combined with 10-fold cross-validation for data training. S4: Introduce a multivariate Pearson correlation matrix of environmental factors and microbial community characteristics from the training dataset to evaluate the classification results and correlations of the target response variables, screen out variables with high and low correlations, and set the preferred test set; S5: Import the second test set into various preset machine learning models for training, evaluation, and secondary optimization, and select the prediction model through preset model indicators.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a machine learning-based microbial community prediction and oil pollution remediation program, which, when executed by a processor, implements the steps of the machine learning-based microbial community prediction and oil pollution remediation method as described in any one of claims 1 to 8.

Images