Microbial high-dimensional data processing and key genus identification method based on machine learning

Abstract

The application relates to a microorganism high-dimensional data processing and key genus identification method based on machine learning, and belongs to the technical field of data processing. The data processing method comprises the following steps: acquiring microorganism high-dimensional data; performing functional division on the microorganism high-dimensional data to obtain multiple type functions, wherein each type function comprises multiple functional factors; selecting concentrated factors from all the functional factors; respectively calculating the correlation values of each concentrated factor and the microorganism application quality under each type function, determining the extraction ratio of each type function based on the correlation values, and extracting the microorganism genera of each type function according to the extraction ratio; and performing screening on the microorganism high-dimensional data based on the extracted microorganism genera to obtain microorganism reduced-dimensional data. The application can effectively reduce the dimension of the microorganism high-dimensional data, and can also guarantee the representativeness and comprehensiveness of subsequent key microorganism genus identification.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a microorganism high-dimensional data processing and key genus identification method based on machine learning. BACKGROUND

[0002] Microbial fermentation refers to the process of using microorganisms to convert raw materials into products required by humans through specific metabolic pathways under suitable conditions. Based on this, microorganisms can be widely used in food, medicine, industry and agriculture, environmental protection and many other fields. For example, using microbial fermentation engineering to degrade plastics, treat wastewater and waste gas, produce ethanol, food and various enzyme preparations, and prevent pests.

[0003] For different fields of microbial application environment, the microorganism genera suitable for them are also different, and the screening results of key microorganism genera will play a key role in the application effect of microorganisms. Therefore, for a certain application environment, adding what kind of exogenous microorganism is an important step in the application of microorganisms.

[0004] In recent years, machine learning technology has been increasingly widely used in the field of microbial application. Machine learning models can determine key indicators of microorganism genera. However, current research mainly focuses on physicochemical indicators, and lacks modeling analysis of microbial data. One difficulty in identifying key microorganism genera using machine learning models is the high-dimensional nature of microbial data. The number of features (microorganism number at genus level) is greater than the number of samples, which will cause model overfitting and local optimal solution in the training process, resulting in low model accuracy.

[0005] Therefore, it is necessary to preprocess high-dimensional data, and generally adopt data dimension reduction to reduce the number of features. However, the current data dimension reduction method has the following problems:

[0006] (1) Features have ambiguity: the features after dimension reduction are composite features, which do not have practical significance, although they can be used for modeling, but it is difficult to apply to the later screening of key microorganisms;

[0007] (2) Weight is not specific: similar to the PCA method, different features can be sorted by calculating the weight. This unsupervised learning method uses spatial distance, and cannot be associated with microbial application quality to determine the optimal parameters;

[0008] (3) The result is not universal: the core of modeling is to screen the core microorganisms that can represent the changes of most microorganisms in the composting process. For some less abundant microorganisms with certain key functions, they are easily ignored in the feature selection process. SUMMARY

[0009] To at least solve one of the above technical problems, the present application provides a microorganism high-dimensional data processing and key genus identification method based on machine learning.

[0010] In a first aspect, the present application provides a microorganism high-dimensional data processing method, which adopts the following technical solution:

[0011] A microorganism high-dimensional data processing method comprises:

[0012] Obtaining microorganism high-dimensional data;

[0013] Functionally dividing the microorganism high-dimensional data to obtain multiple type functions, wherein each type function comprises multiple functional factors;

[0014] Selecting concentrated factors from all functional factors;

[0015] Calculating the correlation values of each concentrated factor and microorganism application quality under each type function respectively, determining the extraction ratio of each type function based on the correlation values, and extracting microorganism genera of each type function according to the extraction ratio;

[0016] Filtering the microorganism high-dimensional data based on the extracted microorganism genera to obtain microorganism reduced-dimensional data.

[0017] Optionally, the functionally dividing the microorganism high-dimensional data to obtain multiple type functions comprises:

[0018] Summarizing all microorganism high-dimensional data into a file;

[0019] Retaining microorganism communities at the genus level, combining and adding the same microorganism genera, and removing microorganism genera with a preset OTU number;

[0020] Splitting the file into a species abundance information file and a species classification annotation file;

[0021] Running a FAPROTAX script under a Python environment to functionally divide the species abundance information file and the species classification annotation file to obtain the multiple type functions.

[0022] Optionally, the selecting concentrated factors from all functional factors comprises:

[0023] Using a factor analysis method to calculate the eigenvalues of each functional factor under each type function respectively, selecting functional factors corresponding to eigenvalues greater than a first preset threshold as the concentrated factors.

[0024] Optionally, the method further comprises:

[0025] If the preset condition is met, the step of calculating the eigenvalue of each functional factor under each type of function by using the factor analysis method, and selecting the functional factor corresponding to the eigenvalue greater than the first preset threshold value as the concentration factor is performed; wherein the preset condition includes detecting that the matrix is not a unit matrix by Bartlett's spherical test, and detecting that the correlation values between variables are all greater than a second preset threshold value by KMO test.

[0026] Optionally, the determining the extraction ratio of each type of function based on the correlation value includes:

[0027] calculating the average correlation value of each type of function based on the correlation value between all concentration factors and the microbial application quality under each type of function;

[0028] calculating the correlation proportion of each type of function based on the average correlation value of each type of function, approximately estimating the correlation proportion of each type of function to obtain the extraction ratio of each type of function; and / or, the extracting the microbial genus of each type of function according to the extraction ratio includes:

[0029] determining the sub-extraction number corresponding to each type of function based on the preset total extraction number and the extraction ratio;

[0030] calculating the relative abundance of each microbial genus under each type of function, and sorting all relative abundances of each microbial genus under each type of function to generate a relative abundance sequence of each type of function;

[0031] extracting the microbial genus of each type of function based on the sub-extraction number and the relative abundance sequence of each type of function.

[0032] In a second aspect, the application provides a key microbial genus identification method based on machine learning, which adopts the following technical solution:

[0033] A key microbial genus identification method based on machine learning, which obtains microbial dimension reduction data by using the aforementioned microbial high-dimensional data processing method, including:

[0034] dividing the microbial dimension reduction data into a training set and a test set;

[0035] taking K-nearest neighbor model, random forest model and decision tree model as basic models, and taking linear regression model as a secondary model to construct a Stacking model, taking the relative abundance of microorganisms at the door, class and genus levels as input features of the Stacking model, and taking the quality score of microbial application as the predicted label of the Stacking model;

[0036] The training set and the test set are used to train the Stacking model to obtain a trained Stacking model; and the trained Stacking model is subjected to an explanatory analysis to evaluate the importance of each input feature to the predicted label, thereby obtaining key microbial genera.

[0037] In a third aspect, the present application provides a microbial high-dimensional data processing device, which adopts the technical scheme as follows:

[0038] A microbial high-dimensional data processing device comprises:

[0039] An acquisition module is configured to acquire microbial high-dimensional data.

[0040] A function division module is configured to divide the microbial high-dimensional data into multiple type functions, wherein each type function comprises multiple functional factors.

[0041] A concentration module is configured to select concentration factors from all functional factors.

[0042] An extraction module is configured to calculate the correlation values between each concentration factor and microbial application quality under each type function, determine the extraction ratio of each type function based on the correlation values, and extract microbial genera of each type function according to the extraction ratio.

[0043] A screening module is configured to screen the microbial high-dimensional data based on the extracted microbial genera, thereby obtaining microbial reduced-dimensional data.

[0044] In a fourth aspect, the present application provides a key microbial genus identification device based on machine learning, which adopts the technical scheme as follows:

[0045] A key microbial genus identification device based on machine learning uses the microbial reduced-dimensional data obtained by the microbial high-dimensional data processing method, and comprises:

[0046] A data set division module is configured to divide the microbial reduced-dimensional data into a training set and a test set.

[0047] A model construction module is configured to use a K-nearest neighbor model, a random forest model and a decision tree model as a basic model, use a linear regression model as a secondary model, construct a Stacking model, use the relative abundance of microorganisms at the door, eye and genus levels as input features of the Stacking model, and use the quality score of microbial application as a predicted label of the Stacking model.

[0048] A model explanation module is configured to perform an explanatory analysis on the trained Stacking model, evaluate the importance of each input feature to a predicted label, and obtain key microbial genera.

[0049] In a fifth aspect, the present application provides an electronic device, which adopts the following technical solution:

[0050] An electronic device includes a memory and a processor; the memory stores a computer program capable of being loaded and executed by the processor to perform the method of any one of the first aspect or the second aspect.

[0051] In a sixth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0052] A computer-readable storage medium stores a computer program capable of being loaded and executed by a processor to perform the method of any one of the first aspect or the second aspect.

[0053] By adopting the above technical solution, in view of the influence of high-dimensional microbial data on machine learning modeling, a high-dimensional microbial data preprocessing method based on function division-feature condensation-information extraction (from Function, Feature to Message, FFM) is established. First, the functions are classified according to the functions to ensure that the microorganisms with different functions are screened out, then the factors are condensed to reduce the number of features and ensure the integrity of the original sample information, and then the correlation between the features and the microbial application quality is analyzed, and the microbial genera are screened according to the correlation. Both the dimensionality of high-dimensional microbial data and the representativeness and comprehensiveness of subsequent key microbial genus identification can be reduced. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 FIG. 1 is a flowchart of a high-dimensional microbial data processing method according to an embodiment of the present application.

[0055] Figure 2 a, 2d, 2g, and 2j are, in sequence, characteristic value curve graphs of each functional factor in the carbon cycle function, the nitrogen cycle function, the enteropathogenic bacteria class function, and the other class function according to an embodiment of the present application.

[0056] Figure 2 b, 2e, 2h, and 2k are, in sequence, bar graphs of the variance contribution rate and the cumulative variance contribution rate of each functional factor in the carbon cycle function, the nitrogen cycle function, the enteropathogenic bacteria class function, and the other class function according to an embodiment of the present application.

[0057] Figure 2 c, 2f, 2i, and 2l are, in sequence, rotation component matrix graphs of the carbon cycle function, the nitrogen cycle function, the enteropathogenic bacteria class function, and the other class function according to an embodiment of the present application.

[0058] Figure 3 a, 3b, 3c, 3d are bar charts of the correlation of carbon cycle function, nitrogen cycle function, intestinal pathogenic bacteria function, other function and compost quality in the embodiment of the application.

[0059] Figure 4 is a process schematic diagram of a key microorganism genus identification method based on machine learning in the embodiment of the application.

[0060] Figure 5 is a model training schematic diagram in the embodiment of the application.

[0061] Figure 6 a is a bar chart of the prediction accuracy of the Stacking model at the level of door, goal and genus in the embodiment of the application.

[0062] Figure 6 b, 6c, 6d are bar charts of the identification of the importance of the microbial community characteristics in the embodiment of the application.

[0063] Figure 7 is a structural block diagram of a microbial high-dimensional data processing device in the embodiment of the application.

[0064] Figure 8 is a structural block diagram of a key microorganism genus identification device based on machine learning in the embodiment of the application.

[0065] Figure 9 is a structural block diagram of an electronic device in the embodiment of the application. DETAILED DESCRIPTION

[0066] To make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application.

[0067] Figure 1 is a process schematic diagram of a microbial high-dimensional data processing method provided in the embodiment. As shown in the figure, Figure 1 the main process of the method is described as follows (steps S101-S105):

[0068] Step S101, obtaining microbial high-dimensional data;

[0069] Step S102, functionally dividing the microbial high-dimensional data to obtain multiple types of functions, wherein each type of function includes multiple functional factors;

[0070] Step S103, selecting a concentrated factor from all functional factors;

[0071] Step S104, respectively calculate the correlation value of each concentration factor and the microbial application quality under each type of function, determine the extraction ratio of each type of function based on the correlation value, and extract the microbial genus of each type of function according to the extraction ratio;

[0072] Step S105, filtering the high-dimensional data of microorganisms based on the extracted microbial genus, and obtaining the reduced-dimensional data of microorganisms.

[0073] In this embodiment, the microbial application refers to adding exogenous microorganisms into certain engineering applications (such as food processing, organic solid waste, wastewater and waste treatment, pest control, plastic degradation, etc.) based on the principle of microbial fermentation, so as to achieve the corresponding treatment effect, which is closely related to the quality of microbial application.

[0074] Among them, the high-dimensional data of microorganisms can be obtained by literature retrieval. Specifically, the following steps are included:

[0075] (1) Retrieving literature based on preset search terms to obtain microbial data;

[0076] The search terms related to microbial application are set in advance, for example, for the microbial application of kitchen waste aerobic compost, all the literature recorded in the China National Knowledge Infrastructure (CNKI) database, Web of science database and other databases within a certain time can be searched, and the English and Chinese papers on kitchen waste aerobic compost are searched with “kitchen waste compost”, “food waste compost”, “kitchen waste”, “domestic waste”, “perishable waste” and “compost” as keywords, and then the papers are manually screened, and finally the corresponding microbial data is obtained from the screened papers.

[0077] (2) Find the sequence number information corresponding to the microbial data, obtain the microbial original information based on the sequence number information, process the microbial original information, and generate the high-dimensional data of microorganisms with OTU table.

[0078] Specifically, it is checked whether the microbial data has sequence number information uploaded on the National Center for Biotechnology Information (NCBI), the original information of each sample in FASTQ format is obtained according to the sequence number, and then the DATA2 method is used for quality filtering, trimming, denoising, merging and sequence chimerism processing, and finally the OTU table is obtained.

[0079] In some embodiments, for step S102, all microbial high-dimensional data are first summarized into a file; the microbial community at the genus level is retained, the same microbial genera are added together, and the microbial genera with a preset number of OTUs are removed; the file is split into a species abundance information file (OTU abundance table.txt) and a species classification annotation file (Taxonomy annotation of OTUs.txt); and the FAPROTAX script is run under the Python environment to perform functional classification on the species abundance information file and the species classification annotation file, thereby obtaining multiple types of functions.

[0080] In the formula, the preset value can be set to 0, and the microbial genera with 0 OTUs are removed, that is, the microbial genera that do not exist in most samples are removed, thereby greatly reducing the number of microbial genera.

[0081] FAPROTAX is a method for predicting the functions of microbial communities. In the Python programming language, the OTU classification table based on 16S rRNA can be used to annotate and predict the functions of the community. Compared with other function prediction methods, FAPROTAX is more suitable for environmental samples.

[0082] Taking kitchen waste aerobic composting as an example, 38 functions are identified and annotated by FAPROTAX, and the identified functions basically undertake the main processes of the kitchen waste aerobic composting system. The 38 identified functions are mainly divided into four categories: carbon cycle function, nitrogen cycle function, intestinal / pathogenic bacteria function, and other functions. Among them, the carbon cycle function has 13 functional factors, including fermentation, aerobic chemical heterotrophy, hydrocarbon degradation, and organic matter decomposition related functions; the nitrogen cycle function has 11 functional factors, including nitrate and nitrite ammonification, respiration, denitrification, nitrogen fixation, and absorption; the intestinal / pathogenic bacteria function has 6 functional factors, including parasitism, symbiosis, and pathogenic bacteria reduction; and the other function has 8 functional factors, including autotrophy, heterotrophy, and sulfate respiration.

[0083] In some embodiments, for step S103, the eigenvalues of each functional factor under each type of function can be calculated by using the factor analysis method, and the functional factors corresponding to the eigenvalues greater than a first preset threshold value are selected as the concentration factors.

[0084] The factor analysis (Factor analysis) method is used for data dimension reduction, and the purpose is to use fewer factors to include as much microbial information as possible. However, whether the factor analysis method can be used also needs to meet the following preset conditions:

[0085] (1) Bartlett's sphericity test is used to test the correlation between variables in the matrix, to determine whether it is a unit matrix, that is, whether each variable is independent, if not independent, it indicates that it is suitable for factor analysis.

[0086] (2) KMO test is used to test the correlation and partial correlation between variables, the value is between 0 and 1, the closer to 1, the stronger the correlation, the weaker the partial correlation, when the value is greater than the second preset threshold, for example, 0.5, it is considered that all variables are suitable for using factor analysis method.

[0087] The above preset conditions are indispensable and must be met at the same time to implement the factor analysis method to reduce the dimension of microbial data.

[0088] After the above test, the characteristic values of each functional factor under different types of functions are output by the existing factor analysis software, and the appropriate number of factors in each type of function is selected.

[0089] Taking kitchen waste aerobic composting as an example, factors with characteristic values greater than 1 can be selected as concentration factors for subsequent research, such as Figure 2 a, 2d, 2g, 2j, 5 functional factors are selected for subsequent analysis in carbon cycle, 4 functional factors are selected for subsequent analysis in nitrogen cycle, 3 functional factors are selected for subsequent analysis of enteric pathogenic bacteria, and 3 functional factors are selected for subsequent analysis of other types.

[0090] After the optimal number of factors is selected, the variance contribution rate (Variance contribution rate, VCR) and the cumulative variance contribution rate (Cumulative variance contribution rate, CVCR) are calculated, and the results are shown in Figure 2 b, 2e, 2h, 2k. In the process of factor analysis, the variance explanation rate is used to reflect the explanation degree of a factor to the original variable, and the cumulative variance contribution rate represents the total amount of information extracted from the original sample by all selected factors. Generally, the higher the value, the more it can represent the original data sample. After factor selection, the carbon cycle, nitrogen cycle, enteric pathogenic bacteria and other types of information retain 87.77%, 89.21%, 83.15% and 85.21% respectively, which to some extent reflects that the number of factors selected in each type of functional data is reasonable, and as much data information as possible can be used for subsequent analysis.

[0091] As shown in Figure 2c, 2f, 2i, 2l, the use of rotation component matrix reflects the relationship between specific functions and selected factors, generally considered when the value (factor loading coefficient) is greater than 0.4, the component factor has a corresponding relationship with the type of function, that is, the component factor can represent the data information of the type of function. For carbon cycle related functions, the 5 screened component factors cover different functions to varying degrees. Unlike carbon cycle, the first component of nitrogen cycle contains more functional information. For carbon-nitrogen related functions, generally a certain function only has a strong corresponding relationship with a certain concentration factor, while intestinal pathogenic bacteria and other classes have a large difference in function, and a function item corresponds to multiple factors and the corresponding relationship is not strong. Among the intestinal pathogenic bacteria and other classes of functions, the first component covers the most functional information, and other factors only supplement the functional information. It is worth noting that when the factor loading coefficient of a certain function item is below 0.4, it may have a strong corresponding relationship with other factors that are not selected, and is removed during factor concentration screening, which also indicates that these functions cannot represent the overall sample information.

[0092] Optionally, in step S104, the correlation value of each concentration factor and the microbial application quality is calculated by the Pearson correlation analysis method.

[0093] In some embodiments, for the extraction ratio determination in step S104, the average correlation value of each type of function is calculated based on the correlation value of all concentration factors and the microbial application quality under each type of function, which roughly reflects the relationship between the functional data of this type and the microbial application quality. Then, the correlation proportion of each type of function is calculated based on the average correlation value of each type of function, and the correlation proportions of various types of functions are approximately estimated to obtain the extraction ratio of each type of function.

[0094] Among them, for any type of function, the correlation proportion is obtained by adding the average correlation values of all types of functions as the denominator, taking the average correlation value of the type of function as the numerator, and multiplying by 100%.

[0095] In some embodiments, the microbial genera of each type of function can be extracted in the following manner:

[0096] First, based on the preset total extraction number and extraction ratio, the number of sub-extractions corresponding to each type of function is determined; then the relative abundance of each microbial genus under each type of function is calculated, and all the relative abundances of each microbial genus under each type of function are sorted to generate a relative abundance sequence of each type of function; and then based on the number of sub-extractions of each type of function and the relative abundance sequence, the microbial genera of each type of function are extracted.

[0097] The relative abundance sequence can be an ascending sequence or a descending sequence, and the embodiment is not limited specifically. For any type of function, if the number of sub-extraction is N, whether the relative abundance sequence is ascending or descending, the N microorganism genera with the largest relative abundance are selected. That is, the proportion of the flora is high in the type function with high correlation, and the proportion of the flora is low in the type function with low correlation.

[0098] The following takes the aerobic composting of kitchen waste as an example to illustrate step S104.

[0099] After determining the concentration factors of different types of functions, the Pearson correlation of each concentration factor with the compost quality is calculated, and the correlation values are averaged to roughly reflect the relationship between the function data of this type and the compost quality. As shown in FIG. 3, the average correlation of the composite factors of each type of function is 0.38, 0.31, 0.20 and 0.07. It can be seen that these composite factors have no strong correlation with the compost quality, because in the dimension reduction process of factor analysis, the correlation of the internal data is reduced to retain the main factors with more information, so that the composite factors can be used to describe the data information from multiple angles, and the data correlation in a certain aspect is weaker. Further calculation of the correlation proportion of each type of function shows that the correlation proportions of carbon cycle, nitrogen cycle, enteropathogenic bacteria and other types are 39.76%, 32.40%, 20.72% and 7.12% respectively. The cumulative proportion of carbon and nitrogen function related data in the original data is more than 70%, because in the function division process, these two types of function types are the most retained, and the relative abundance of the corresponding microorganisms is higher, which further illustrates the important role of carbon and nitrogen cycle microorganisms in the composting system.

[0100] Approximate estimation of the correlation proportion of each type of function shows that the extraction proportions of carbon cycle, nitrogen cycle, enteropathogenic bacteria and other types of functions are about 4:3:2:1. About 50 microorganism genera are selected for machine learning modeling research, 20 carbon cycle function microorganisms, 15 nitrogen cycle function microorganisms, 10 enteropathogenic bacteria and 5 other types of microorganisms are selected under the corresponding functions. The selected microorganisms are sorted together, and the repeated microorganism genera are removed. Finally, the original sample of thousands of microorganism genera is compressed to 38.

[0101] In view of the influence of high-dimensional microbial data on machine learning modeling, the embodiment provides a microbial high-dimensional data preprocessing method based on the FFM framework, which realizes effective data dimension reduction, solves the influence of high-dimensional features on machine learning modeling, and ensures the representativeness and comprehensiveness of the selected microorganisms.

[0102] After the high-dimensional microbial data is preprocessed by the above microbial high-dimensional data processing method, subsequent key microbial genus identification is performed according to the microbial dimension reduction data obtained by preprocessing.

[0103] Figure 4 A flowchart of a key microbial genus identification method based on machine learning is provided for this embodiment. As shown in Figure 4 The microbial dimension reduction data is obtained using the aforementioned microbial high-dimensional data processing method, and the main process of the method is described as follows (steps S401-S404):

[0104] Step S401, the microbial dimension reduction data is divided into a training set and a test set;

[0105] Step S402, taking K-Nearest Neighbor model, Random Forest model and Decision Tree model as the basic model, and taking Linear Regression model as the secondary model, a Stacking model is constructed, and the relative abundance of microorganisms at the door, goal and genus level is taken as the input feature of the Stacking model, and the quality score of the microbial application is taken as the predicted label of the Stacking model;

[0106] Step S403, the training set and the test set are used to train the Stacking model, and a trained Stacking model is obtained;

[0107] Step S404, the trained Stacking model is subjected to explanatory analysis to evaluate the importance of each input feature to the predicted label, and the key microbial genus is obtained.

[0108] In this embodiment, all data is divided into a training set and a test set, for example, 70% of the microbial dimension reduction data is taken as the training set for training the machine learning model, and the remaining 30% of the microbial dimension reduction data is taken as the test set. In order to verify whether the model construction is successful, the trained model is used to predict the remaining 30% of the test set data, and if it still performs well on the test set, it indicates that the constructed model has good prediction ability for unfamiliar data.

[0109] In order to improve the generalization performance of the model and reduce the overfitting of the model, the cross-validation method is used to further divide the training data into a training set and a validation set. Considering that the modeling data is not much, 5-fold cross-validation can be used.

[0110] The model composed of heterogeneous base learners can be coupled and called a stacking model. The model is based on an integrated model of different types of algorithms, which can fully exert the advantages of different models and combine the output information of each model into a model with stronger performance. As a stacking model, the implementation process is as follows: first, a plurality of models are trained on the training set to obtain a plurality of sets of prediction results, which are called meta features, and the original model is predicted to obtain the meta features of the test set; then the meta features obtained by the training set and the test set are input into a new model, and the prediction result is obtained based on the algorithm principle of the new model, and since different training data are used before and after, overfitting can be prevented to a certain extent.

[0111] In this embodiment, the construction process of the model can be performed by relying on the third-party machine learning library Scikit-Learn of Python, and the hyperparameter tuning can be implemented by using the Param grid function library. The linear regression model parameters are according to the system default values, the K nearest neighbor model adjusts the hyperparameter n_neighbors, the decision tree model adjusts the hyperparameters max_depth, min_samples_leaf and min_samples_split, and the random forest model adjusts the hyperparameter n_estimaters. The model training process is as shown in Figure 5 .

[0112] Taking aerobic composting as an example, the quality of composting is reflected by calculating the comprehensive score of T value (the ratio of carbon-nitrogen ratio at a certain composting period to the initial carbon-nitrogen ratio) and GI value (germination index of plant seeds), that is, the prediction label of the model, and the calculation formula is as follows:

[0113] (1) The calculation formula of T value score is:

[0114] x score1 =1-T i ;

[0115]

[0116] Among them, Ti is the T value of the i th sample at a certain period, x1 ' is the score after normalization to [0, 100].

[0117] (2) The calculation formula of GI score is:

[0118] x score2 =GIi;

[0119]

[0120] Among them, GIi is the GI value of the i th sample at a certain period, x2 'The score after normalization to [0, 100] by using the normalization method.

[0121] (3) The overall score of the compost sample is calculated as follows:

[0122] x score = x1 ' + x2 '

[0123]

[0124] wherein x score is the comprehensive score of the i-th sample at a certain period, and x ' is the score after normalization to [0, 100] by using the normalization method.

[0125] Since the machine learning is a black box model, after the model is constructed, an explanatory method needs to be further introduced to evaluate the influence of each input feature on the label. Two methods, SHAP and PDA, can be used for explanatory analysis.

[0126] The SHAP analysis method (SHapley Additive exPlanations) realizes global and local explanations of the label by calculating the marginal contribution of each feature to the model output. The global explanation gives the comprehensive influence of the feature on the result, and the local explanation gives the influence of the change of each sample under the feature on the result. The SHAP absolute value mean of each input feature for the microbial application quality is calculated, and the numerical value is sorted according to the size to reflect the importance of each microbial genus to the microbial application quality. The calculation formula is as follows:

[0127]

[0128] wherein g(x) is the model to be explained, f(x) is the model prediction value, is the prediction average value, is the prediction value of the corresponding sample, also called Shapley value. The positive and negative of the Shapley value represent the positive or negative influence of the sample point on the result. The calculation formula of the Shapley value is as follows:

[0129]

[0130] wherein, is the Shapley value of the i-th sample, F is the set of all features, S represents the subset of other features except the i-th sample, f S∪{I} and f S respectively represent the prediction values given by the model after inputting the i-th feature and S.

[0131] ​PDA is a global model interpretability method that reflects the specific impact of changes in feature values ​​on label values. It includes one-dimensional feature change analysis and two-dimensional feature comprehensive impact analysis, with results presented in the form of a Partial Dependency Graph (PDP). The function is defined as follows:

[0132] f s (xs)=E xc [f(xs,xc)]=∫f(xs,xc)dP(xc);

[0133] Here, xs represents the features to be plotted, and xc represents other feature variables used by the model besides xs. Together, they constitute the total feature space x. It is important to note that xs and xc should not have a strong correlation; otherwise, errors may occur in the partial dependency plot results.

[0134] The following explanation uses aerobic composting of kitchen waste as an example to illustrate step S404.

[0135] First, the predictive power of microbial data at different taxonomic levels was studied, such as... Figure 6 As shown in a, the prediction accuracy (Ri) at different classification levels... 2 The accuracy (R²) of the training and test sets at the gate classification level remains around 0.8. 2 The accuracy rates for the training and test sets at the order level were 0.76±0.09 and 0.73±0.10, respectively; the accuracy rates for the training and test sets at the genus level were 0.84±0.07 and 0.81±0.08, respectively; and the accuracy rates for the training and test sets at the genus level were 0.81±0.07 and 0.78±0.04, respectively. It can be seen that the accuracy of the microbial taxonomic datasets is lower than that of the physicochemical data models, a phenomenon observed in most studies. Generally, the relative abundance of microbial data varies considerably and differs significantly between studies, resulting in poor accuracy and generalization ability of the model during the modeling process. However, overall, the constructed model can effectively reflect compost quality through changes in the microbial community, and is therefore suitable for further feature analysis.

[0136] like Figure 6 As shown in b, at the phylum level, the three most important phyla affecting compost quality are Proteobacteria (5.75), Firmicutes (5.50), and Actinobacteria (4.56). This is mainly because these three phyla are the most abundant phyla during composting, covering most of the screened microbial genera. Furthermore, these phyla have been reported in many studies to have a high capacity for organic matter degradation, thereby promoting the maturation and stabilization of compost and having a strong relationship with compost quality.

[0137] like Figure 6As shown in c, at the microbial level, important species include Xanthomonadales (4.16), Lactobacillales (3.81), and Rhodocyclales (2.87). Since Lactobacillales has a high average relative abundance during composting, combined with the above analysis, it may have a significant negative impact on compost quality. Unlike Lactobacillales, Xanthomonadales has a lower relative abundance during composting (average 0.32%). As a proteobacteria species, its important role in the composting system is presumably manifested in material degradation and nitrogen cycling. Taking Stenotrophomonas as an example, FAPROTAX analysis revealed that it possesses multiple carbon and nitrogen cycling-related functions, including chemoheterotrophy and nitrate / nitrite respiration reduction.

[0138] like Figure 6 As shown in d, at the genus level, the important microorganisms include Bacillus (3.84), Acinetobacter (2.24), Thermobacillus (1.95), Pseudomonas (1.72), Psychrobacter (1.42), and Thermobifida (1.24). As mentioned earlier, Bacillus, Pseudomonas, and Thermobifida have high relative abundances in composting, and they are either involved in material degradation or nitrogen cycling in composting, thus affecting compost quality, hence their high values ​​in the importance analysis. Acinetobacter typically uses nitrate / nitrite as an electron acceptor to promote N2O emissions through respiration. Information on Thermobacillus and Psychrobacter is limited. Based on FAPROTAX functional analysis, Thermobacillus possesses chemoheterotrophic and xylan-decomposing functions, while Psychrobacter possesses chemoheterotrophic functions. Both are carbon cycling microorganisms in composting, suggesting their important role in substrate decomposition and material maturation.

[0139] The three most important phyla affecting the quality of microbial application are Firmicutes, Actinobacteria and Proteobacteria by calculating the average SHAP absolute value at each classification level. The important bacterial orders include Xanthomonadales, Lactobacillales and Rhodocyclale, and the corresponding key microbial genera are Bacillus, Acinetobacter, Thermobacillus, Pseudomonas, Psychrobacter and Thermobifida.

[0140] It should be noted that, Figure 6 b to Figure 6 Microorganisms with a relative abundance of less than 0.2% in d are classified as other functional categories.

[0141] In some embodiments, the coefficient of determination (R 2 ), root mean square error (RMSE) and mean absolute error (MAE) are used to evaluate the prediction of the model. Root mean square error (RMSE) and mean absolute error (MAE) are used to evaluate the difference between predicted values and true values, and both values are lower the better. The coefficient of determination (R2) is the proportion of information fitted by the model to the true information, and the closer to 1 the better. The above evaluation methods are calculated as follows:

[0142]

[0143] Where n is the number of samples, h(xi) is the predicted value, yi is the true value, is the sample mean.

[0144] Figure 7 A structural block diagram of a microorganism high-dimensional data processing device 700 provided by the embodiments of the present application is shown in FIG. 7. As shown in FIG. 7, the device 700 mainly includes: Figure 7

[0145] An acquisition module 701 is configured to acquire microorganism high-dimensional data.

[0146] A functional division module 702 is configured to divide the microorganism high-dimensional data into multiple types of functions, wherein each type of function includes multiple functional factors.

[0147] A concentration module 703 is configured to select concentration factors from all functional factors.

[0148] ​The extraction module 704 is configured to calculate a correlation value of each condensed factor under each type of function and the microbial application quality respectively, determine an extraction ratio of each type of function based on the correlation value, and extract the microbial genera of each type of function according to the extraction ratio.

[0149] In some embodiments, the function division module 702 is specifically configured to: aggregate all the microbial high-dimensional data into a file; retain the microbial community at the genus level, combine and add the same microbial genera, and remove the microbial genera with a preset OTU number; split the file into a species abundance information file and a species classification annotation file; and run a FAPROTAX script in a Python environment to perform function division on the species abundance information file and the species classification annotation file, to obtain multiple types of functions.

[0150] In some embodiments, the condensing module 703 is specifically configured to: calculate eigenvalues of each functional factor under each type of function by using a factor analysis method, and select a functional factor corresponding to an eigenvalue greater than a first preset threshold as a condensed factor.

[0151] In some embodiments, the device 700 further includes:

[0152] The execution module is configured to perform the steps of calculating the eigenvalues of each functional factor under each type of function by using the factor analysis method, and selecting the functional factor corresponding to the eigenvalue greater than the first preset threshold as the condensed factor, if a preset condition is met; and the preset condition includes that the matrix is not a unit matrix detected by a Bartlett's sphericity test, and the correlation values between the variables are all greater than a second preset threshold detected by a KMO test.

[0153] In some embodiments, the extraction module 704 includes:

[0154] The extraction ratio calculation module is configured to calculate an average correlation value of each type of function based on the correlation values of all the condensed factors and the microbial application quality under each type of function respectively, calculate a correlation proportion of each type of function based on the average correlation value of each type of function respectively, and approximately estimate the correlation proportions of the types of functions to obtain the extraction ratio of each type of function; and / or the microbial genus extraction module is configured to determine a sub-extraction number corresponding to each type of function based on a preset total extraction number and the extraction ratio, calculate the relative abundance of each microbial genus under each type of function, sort all the relative abundances of each microbial genus under each type of function to generate a relative abundance sequence of each type of function, and extract the microbial genera of each type of function based on the sub-extraction number and the relative abundance sequence of each type of function.

[0155] The various variations and specific examples in the method provided by the embodiments of the present application are also applicable to the microorganism high-dimensional data processing device provided by the embodiments of the present application. Through the foregoing detailed description of the microorganism high-dimensional data processing method, a person skilled in the art can clearly understand the implementation method of the microorganism high-dimensional data processing device in the embodiments. In order to make the description simple, the implementation method of the microorganism high-dimensional data processing device in the embodiments will not be described in detail here.

[0156] Figure 8 A structural block diagram of a key microorganism genus identification device 800 based on machine learning provided by the embodiments of the present application is shown. The microorganism dimension reduction data is obtained by using the foregoing microorganism high-dimensional data processing method, and the device 800 mainly includes: Figure 8

[0157] A data set division module 801 is configured to divide the microorganism dimension reduction data into a training set and a test set.

[0158] A model construction module 802 is configured to take a K-nearest neighbor model, a random forest model and a decision tree model as a basic model, take a linear regression model as a secondary model, construct a Stacking model, and take the relative abundance of microorganisms at the door, eye and genus levels as input features of the Stacking model and take the quality score of the microorganism application as a predicted label of the Stacking model.

[0159] A model training module 803 is configured to train the Stacking model by using the training set and the test set to obtain a trained Stacking model.

[0160] A model interpretation module 804 is configured to perform interpretive analysis on the trained Stacking model, evaluate the importance of each input feature on the predicted label, and obtain a key microorganism genus.

[0161] ​The various function modules in the embodiments of the present application can be integrated together to form a stand-alone unit, for example, integrated in a processing unit, or each module can be physically present separately, or two or more modules can be integrated to form a stand-alone unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit. When the functions are realized in the form of a software function module and sold or used as a stand-alone product, the software function module can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that make contributions to the prior art or parts of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing an electronic device (which can be a personal computer, a server or a network device, etc.) or a processor to execute all or part of the steps of the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and various media that can store program codes.

[0162] The various variations and specific examples in the method provided by the embodiments of the present application are also applicable to the machine learning-based key microorganism genus identification device provided by the embodiments of the present application. Through the foregoing detailed description of the machine learning-based key microorganism genus identification method, those skilled in the art can clearly understand the implementation method of the machine learning-based key microorganism genus identification device in the embodiments of the present application. For the sake of brevity of the description, it will not be described in detail here.

[0163] Figure 9 A structural block diagram of an electronic device 900 provided by the embodiments of the present application is shown in FIG. 9. As shown in FIG. 9, the electronic device 900 includes a memory 901, a processor 902, and a communication bus 903; the memory 901 and the processor 902 are connected through the communication bus 903. Figure 9

[0164] The memory 901 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 901 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating device, instructions for at least one function, and instructions for implementing the microorganism high-dimensional data processing method and / or the machine learning-based key microorganism genus identification method provided by the above embodiments, etc.; the data storage area can store data involved in the microorganism high-dimensional data processing method and / or the machine learning-based key microorganism genus identification method provided by the above embodiments, etc.

[0165] ​The processor 902 can include one or more processing cores. The processor 902 performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 901, calling data stored in the memory 901. The processor 902 can be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that, for different devices, the electronic device for implementing the functions of the processor 902 described above can also be other, and the embodiments of the present application are not limited specifically.

[0166] The communication bus 903 can include a path for transmitting information between the above-mentioned components. The communication bus 903 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus 903 can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 9 Only one double-headed arrow is used in the figure, but it does not mean that there is only one bus or one type of bus. Figure 9 The electronic device shown is only an example and should not limit the functions and use range of the embodiments of the present application.

[0167] The embodiments of the present application provide a computer readable storage medium, which stores a computer program capable of being loaded and executed by a processor to perform the microorganism high-dimensional data processing method and / or the key microorganism genus identification method based on machine learning provided by the above-mentioned embodiments.

[0168] In this embodiment, the computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof. Specifically, the computer readable storage medium can be a portable computer diskette, a hard disk, a USB (Universal Serial Bus) flash drive, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, and any combination thereof.

[0169] The computer program in this embodiment contains program codes for executing the method shown in the above Figure 1 , Figure 2 The computer program can be downloaded to the respective computing / processing device from a computer readable storage medium or an external computer or an external storage device through a network (for example, the Internet, a local area network, a wide area network, and / or a wireless network). The computer program can be executed entirely on the user computer or as a separate software package.

[0170] In the embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented by other means. For example, the above-described device embodiments are only illustrative, for example, the division of modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed entities can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0171] In addition, it should be understood that the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. The terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device.

[0172] The above merely provides preferred embodiments of the present application, and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modifications, equivalent replacements, improvements, etc. made within the principles and technical scope of the present application shall fall into the scope of the present application.

Claims

1. A method for processing high-dimensional microbial data, characterized in that, include: Obtain high-dimensional microbial data; The high-dimensional microbial data is functionally divided to obtain multiple types of functions, wherein each type of function includes multiple functional factors; Select condensation factors from all functional factors; Calculate the correlation values ​​between each concentration factor and the quality of microbial application under each type of function, determine the extraction ratio of each type of function based on the correlation values, and extract the microbial genera of each type of function according to the extraction ratio; Based on the extracted microbial genera, the high-dimensional data of the microorganisms is filtered to obtain microbial dimensionality-reduced data; The process of determining the extraction ratio of each type of function based on the correlation value includes: Based on the correlation values ​​between all concentration factors and microbial application quality under each of the aforementioned functional types, the average correlation value for each functional type is calculated. Calculate the relevance percentage of each type of function based on the average relevance value of each type of function; An approximate estimate of the relevance ratio of each type of function is made to obtain the extraction ratio of each type of function; and / or, The extraction of various functional microbial genera according to the extraction ratio includes: Based on the preset total number of extractions and the extraction ratio, determine the number of sub-extractions corresponding to each type of function; Calculate the relative abundance of each microbial genera under each of the aforementioned functional types, and sort all the relative abundances of each microbial genera under each of the aforementioned functional types to generate a relative abundance sequence for each of the aforementioned functional types. Based on the number of sub-extractions and relative abundance sequences of each type of function, microbial genera of each type of function are extracted.

2. The method according to claim 1, characterized in that, The functional division of the high-dimensional microbial data yields multiple functional types, including: All high-dimensional microbial data are compiled into a single file; Preserve the microbial community at the genus level, merge and add up the same microbial genera, and remove microbial genera with a preset OTU number; The file was split into a species abundance information file and a species classification annotation file; The FAPROTAX script is run in a Python environment to perform functional division of the species abundance information file and the species classification annotation file, resulting in the multiple types of functions.

3. The method according to claim 1 or 2, characterized in that, The selection of condensation factors from all functional factors includes: The eigenvalues ​​of each functional factor under each type of function are calculated using factor analysis. The functional factors corresponding to eigenvalues ​​greater than a first preset threshold are selected as the condensation factors.

4. The method according to claim 3, characterized in that, Also includes: If the preset conditions are met, the step of using factor analysis to calculate the feature values ​​of each functional factor under each type of function, and selecting the functional factor corresponding to the feature value greater than the first preset threshold as the condensed factor is executed. The preset conditions include detecting that the matrix is ​​not an identity matrix using Bartlett's test of sphericity, and detecting that the correlation values ​​between the variables are all greater than a second preset threshold using KMO.

5. A method for identifying key microbial genera based on machine learning, using the microbial high-dimensional data processing method described in claim 1 to obtain microbial dimensionality-reduced data, characterized in that, include: The microbial dimensionality reduction data was divided into a training set and a test set. Using K-nearest neighbor model, random forest model and decision tree model as the basic model, and linear regression model as the secondary model, a Stacking model is constructed. The relative abundance of microorganisms at the phylum, order and genus levels are used as the input features of the Stacking model, and the quality score of microbial application is used as the prediction label of the Stacking model. The Stacking model is trained using the training set and the test set to obtain a trained Stacking model; Interpretive analysis was performed on the trained Stacking model to evaluate the importance of each input feature to the predicted label, and the key microbial genera were obtained.

6. A microbial high-dimensional data processing device, characterized in that, include: The acquisition module is used to acquire high-dimensional microbial data; The functional partitioning module is used to partition the high-dimensional microbial data into multiple types of functions, wherein each type of function includes multiple functional factors. The concentration module is used to select a concentration factor from all functional factors; The extraction module is used to calculate the correlation value between each concentration factor and the microbial application quality under each type of function, determine the extraction ratio of each type of function based on the correlation value, and extract the microbial genera of each type of function according to the extraction ratio; The filtering module is used to filter the high-dimensional microbial data based on the extracted microbial genera to obtain microbial dimensionality-reduced data. The process of determining the extraction ratio of each type of function based on the correlation value includes: Based on the correlation values ​​between all concentration factors and microbial application quality under each of the aforementioned functional types, the average correlation value for each functional type is calculated. Calculate the relevance percentage of each type of function based on the average relevance value of each type of function; An approximate estimate of the relevance ratio of each type of function is made to obtain the extraction ratio of each type of function; and / or, The extraction of various functional microbial genera according to the extraction ratio includes: Based on the preset total number of extractions and the extraction ratio, determine the number of sub-extractions corresponding to each type of function; Calculate the relative abundance of each microbial genera under each of the aforementioned functional types, and sort all the relative abundances of each microbial genera under each of the aforementioned functional types to generate a relative abundance sequence for each of the aforementioned functional types. Based on the number of sub-extractions and relative abundance sequences of each type of function, microbial genera of each type of function are extracted.

7. A key microbial genus identification device based on machine learning, which uses the microbial high-dimensional data processing method of claim 1 to obtain microbial dimensionality-reduced data, characterized in that, include: The dataset partitioning module is used to divide the microbial dimensionality reduction data into training and testing sets; The model building module is used to construct a Stacking model based on the K-nearest neighbor model, random forest model, and decision tree model, and with the linear regression model as the secondary model. The relative abundance of microorganisms at the phylum, order, and genus levels are used as the input features of the Stacking model, and the quality score of microbial application is used as the prediction label of the Stacking model. The model training module is used to train the Stacking model using the training set and the test set to obtain a trained Stacking model. The model interpretation module is used to perform interpretive analysis on the trained Stacking model, evaluate the importance of each input feature to the predicted label, and obtain the key microbial genera.

8. An electronic device, characterized in that, It includes a memory and a processor; the memory stores a computer program that can be loaded by the processor and executed as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 5.

Images