Distributed automatic machine learning method based on collaborative filtering

Abstract

The invention relates to a distributed automatic machine learning method based on collaborative filtering, and belongs to the field of automatic machine learning. The method comprises the following steps: searching first N models for each data set in a meta-knowledge base, storing the models in a Hash dictionary, and calculating weights of meta feature subsets of the data sets in the meta-knowledge base; according to the collaborative filtering-based distributed AutoML framework CF-DAML, firstly, meta-features of a new data set are calculated through a model recommendation module, an appropriate model is recommended for the new data set, then, the CF-DAML uses a distributed model training system DSTM to train the recommended model on a training set of the new data set within a specified time limit, the trained model is evaluated on a verification set of the new data set, and finally, the CF-DAML is subjected to CF-DAML. And finally, the CF-DAML adopts a selective stacking integration system MSSE to integrate several high-performance models to predict labels for a test set of the new data set. According to the method, the time complexity is greatly reduced on the basis of ensuring that the classification accuracy is improved.

Description

Technical Field

[0001] The present invention belongs to the field of automatic machine learning, and in particular relates to a distributed automatic machine learning method based on collaborative filtering. Background Art

[0002] Machine learning has been successfully applied in various fields, such as finance, industry, healthcare, biology, and quantum physics. With the advancement of artificial intelligence, machine learning will be more widely used. However, completing a machine learning task requires a lot of parameter adjustment, which brings inconvenience to developers. The goal of automatic machine learning (AutoML) is to make these decisions in a data-driven, objective, and automated way: users only need to provide data, and the AutoML system will automatically determine the method that is most suitable for this specific field application. Therefore, AutoML makes the most advanced machine learning methods available to field scientists who are interested in machine learning applications but do not have the resources to understand the technology behind them in detail.

[0003] There are two main approaches to the research on the AutoML framework: one is to combine the hyperparameter optimization algorithm with repeated iterative optimization to explore model pipelines with better performance; the other is to build a good meta-knowledge base in the offline stage, and use it to pre-train a proxy model, and directly use the proxy model to predict the performance of the machine learning pipeline in the meta-knowledge base in the online stage.

[0004] Most AutoML frameworks based on hyperparameter optimization regard it as an algorithm selection and hyperparameter optimization problem, that is, by training a large number of models on the new data set training set, including switching algorithms and tuning hyperparameters, the algorithm and its hyperparameters with the smallest loss on the validation set are finally selected. Representative works of this type of framework include Auto-WEKA, Auto-sklearn, TPOT, etc. Auto-WEKA implements AutoML by automating the learners and feature selectors in the WEKA machine learning package; however, WEKA is a GPL-licensed Java library that was not written with scalability in mind. Auto-sklearn is an efficient AutoML system based on scikit-learn. Compared with Auto-WEKA, Auto-sklearn uses meta-learning to hot-start Bayesian optimization and combines the searched machine learning pipeline for ensemble learning. It is suitable for low-dimensional hyperparameter configuration. In high-dimensional hyperparameter space, the performance of Bayesian optimization is similar to that of random search. TPOT is an open source genetic algorithm-based AutoML system that optimizes a series of feature preprocessors and machine learning models with the goal of maximizing classification accuracy in supervised classification tasks, exploring more complex machine learning pipelines at the cost of higher time and space complexity.

[0005] The AutoML framework based on collaborative filtering regards the problem of predicting the performance of machine learning pipelines on new datasets as a collaborative filtering problem and solves it through probabilistic matrix decomposition or other similar techniques. Representative works include Oboe and Tensor-Oboe. The former implements AutoML based on matrix decomposition: in the offline stage, Oboe trains classifiers on a large number of datasets and forms a cross-validation error matrix, and then fits a low-rank model that can best predict the cross-validation error to learn the low-dimensional feature vectors of machine learning pipelines and datasets; in the online stage, Oboe runs a set of fast and useful algorithms and uses their cross-validation errors to infer potential feature vectors and recommend suitable pipelines for new datasets. The latter uses low-rank tensor decomposition as a proxy model for pipeline search to achieve better performance with fewer parameters. These frameworks can achieve the best solution in a short time, which means that they have low complexity. The disadvantage is that matrix decomposition may not fully utilize the performance matrix and cannot obtain the nonlinear characteristics of the dataset and machine learning pipeline, resulting in inaccurate recommendation results of such methods. Summary of the invention

[0006] 1. Technical issues to be resolved

[0007] The technical problem to be solved by the present invention is how to provide a distributed automatic machine learning method based on collaborative filtering to solve the problems of long time consumption and low accuracy of the existing AutoML framework based on hyperparameter optimization.

[0008] (II) Technical solution

[0009] In order to solve the above technical problems, the present invention proposes a distributed automatic machine learning method based on collaborative filtering, which includes:

[0010] S1, offline phase: search for the top N models for each dataset in the meta-knowledge base and store them in a hash dictionary, where the key is the dataset ID and the value is the top N models that perform well on the dataset; and calculate the weights of the meta-feature subsets of the dataset in the meta-knowledge base;

[0011] S2. Online stage: CF-DAML, a distributed AutoML framework based on collaborative filtering, first calculates the meta-features of the new dataset and recommends a suitable model for it through the model recommendation module. Then, CF-DAML uses the distributed model training system DSTM to train the recommended model on the training set of the new dataset within the specified time limit and evaluates the trained model on its validation set. Finally, CF-DAML uses the selective stacking integration system MSSE to integrate several high-performance models to predict labels for the test set of the new dataset.

[0012] (III) Beneficial effects

[0013] The present invention proposes a distributed automatic machine learning method based on collaborative filtering, which adopts a distributed AutoML framework based on collaborative filtering and is named CF-DAML. This method defines the AutoML problem as a coupling problem between memory-based collaborative filtering and selective ensemble learning for the first time, getting rid of the time-consuming hyperparameter optimization module that required a lot of work in the past, and greatly reducing the time complexity while ensuring the improvement of classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is the overall flow chart of the present invention;

[0015] Figure 2 is the trend of the loss function (the negative value of the objective function) changing with each weight;

[0016] Figure 3 Flowchart of the distributed model training system;

[0017] Figure 4 Integrated system workflow diagram for selective stacking;

[0018] Figure 5 Schematic diagram of the actual running time for each framework. DETAILED DESCRIPTION

[0019] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0020] The purpose of the present invention is to provide a distributed automatic machine learning method based on collaborative filtering, reduce the development difficulty of machine learning, and provide the industry with a convenient, accurate and efficient automatic machine learning framework.

[0021] This paper proposes a distributed automatic machine learning method based on collaborative filtering, which adopts a distributed AutoML framework based on collaborative filtering and is named CF-DAML. This method defines the AutoML problem as a coupling problem between memory-based collaborative filtering and selective ensemble learning for the first time, getting rid of the time-consuming hyperparameter optimization module that required a lot of work in the past, and greatly reducing the time complexity while ensuring the improvement of classification accuracy.

[0022] In the model recommendation module of CF-DAML, the weighted L1 norm (W1-norm) proposed in the present invention takes into account the different importance of different meta-features to the recommendation results, and improves the common L1 norm similarity calculation method that has been widely used in the past. In CF-DAML's distributed model training system DSTM, the selection of its base model is based on three conditions: availability (candidate models that CF-DAML can collect within the time limit), good performance (the classification accuracy on the validation set is higher than the average level of candidate models) and diversity (models are as independent as possible); in CF-DAML's selective stacking integration system MSSE, compared with previous work that requires the integration of all candidate models, MSSE can ensure the efficiency of integrated learning.

[0023] A distributed automatic machine learning method based on collaborative filtering, the steps are as follows:

[0024] S1. Offline stage: Search for the top N models (N=50 in this stage) for each dataset in the meta-knowledge base and store them in a hash dictionary, where the key is: the ID of the dataset and the value is: the top N models that perform well on the dataset; and calculate the weights of the meta-feature subsets of the dataset in the meta-knowledge base.

[0025] S2. Online stage: CF-DAML, a distributed AutoML framework based on collaborative filtering, first calculates the meta-features of the new dataset through the model recommendation module and recommends a suitable model for it. Then, CF-DAML uses the distributed model training system DSTM to train the recommended model on the training set of the new dataset within the specified time limit and evaluates the trained model on its validation set. Finally, CF-DAML uses the selective stacking ensemble system MSSE to integrate several high-performance models to predict labels for the test set of the new dataset.

[0026] The meta-knowledge base described in step S1 includes three sub-databases:

[0027] Subdatabase 1 contains a two-dimensional matrix X with 42000 rows and 553 columns. Each element x ij Represents the balanced accuracy of the i-th model on the j-th dataset, where i=1,2,...,42000; j=1,2,...,553. This library has 553 classification task datasets from OpenML, each of which contains no more than 10,000 sample points. Each model includes a dataset preprocessing method and a classifier with specific hyperparameters. The data preprocessing methods in this library include polynomial processing and principal component analysis (PCA), and the classifier comes from the scikit-learn function package.

[0028] Subdatabase 2 contains a two-dimensional matrix Y with 553 rows and 46 columns. Each element yij Represents the jth meta-feature of the i-th dataset, where i = 1, 2, ..., 553; j = 1, 2, ..., 46. Meta-features are divided into three categories: features of the dataset itself, such as the number of samples, the number of categories, and the number of features; (iconic) features related to machine learning models, such as the classification accuracy of the benchmark decision tree on the dataset; and indicators related to information theory, such as inter-class entropy.

[0029] Subdatabase 3 contains a dictionary Z. Each entry z i Represents the algorithm name and hyperparameter value of the i-th model, corresponding to the model in sub-database 1, where i = 1, 2, ..., 42000.

[0030] In the offline stage, sub-database 1 is operated to obtain sub-database 4: a two-dimensional matrix T with 553 rows and 50 columns, where t ij represents the number of the jth model that performs best on the i-th dataset, where i = 1, 2, ..., 553; j = 1, 2, ..., 50. The final sub-databases used in CF-DAML are 2, 3, and 4. For details on the classifiers and hyperparameter configurations in the databases, see Tables 1 and 2.

[0031] Table 1 Preprocessor and its hyperparameter configuration

[0032]

[0033]

[0034] Table 2 Classifiers and their hyperparameter configurations

[0035]

[0036]

[0037]

[0038] Table 3 Definition and classification results of meta-feature set

[0039]

[0040]

[0041]

[0042] The weight of the meta-feature subset described in step S1 is iterated by the Bayesian optimization algorithm. First, information-rich meta-features are selected from the 46 meta-features of the data set in the meta-knowledge base, and then the selected meta-features are clustered and grouped to obtain the meta-feature subset; a black-box function relationship between the objective function and the meta-feature subset is constructed, and the weight of the meta-feature subset is learned by Bayesian optimization until convergence is reached; finally, the similarity between the data sets is calculated by W1-norm.

[0043] The clustering of meta-features includes the following steps: studying the correlation between two meta-features in the meta-feature set through the distance correlation coefficient, eliminating meta-features with missing values ​​and meta-features with the same meaning, and clustering the remaining 30 meta-features into 8 categories of meta-feature subsets according to their physical meanings, as shown in Table 3.

[0044] The black box function relationship between the target function and the meta-feature subset is constructed, and the weight of the meta-feature subset is learned by using Bayesian optimization. The following steps are included:

[0045] S11. In order to emphasize the importance of different meta-feature subsets, the two datasets D i ,D j The distance is expressed as

[0046]

[0047] in Denotes the dataset D i ,D j The 1-norm of the k-th class feature subset, ω k N represents the weight of the k-th meta-feature set, K=8 represents 8 meta-feature subsets. k represents the number of meta-features contained in the k-th meta-feature set, Denotes the dataset D i The sth meta-feature in the kth meta-feature subset.

[0048] Two datasets D i ,D j The similarity is expressed as

[0049]

[0050] S12, how to determine the weight is an optimization problem, the present invention uses Bayesian optimization to do it. It should be emphasized that the weight training is completed offline. After the training is completed, the optimal weight is directly embedded in the model recommendation module of CF-DAML. When new data is input into the software, the weight is directly used for modeling without retraining the weight.

[0051] For data set D in sub-database 2 iUse the above distance formula to calculate the k nearest neighbor dataset ND of the dataset i ={D i,1 ,D i,2 ,...,D i,k}, D i The top-N models of each neighbor of are recommended to D i There are k*N models in total. These models are in D i The classification accuracy on does not require training and can be directly queried from sub-database 2.

[0052] Let dataset D i The score is

[0053]

[0054] Recommended to D i k*N models in the dataset D i The sum of the classification accuracy of Denotes the dataset D i The t-th model of the n-th neighbor in the dataset D i The classification accuracy on .

[0055] Taking the average score of all datasets in sub-database 2, we can get the objective function

[0056]

[0057] Where B=553 represents the number of training data sets in the meta-knowledge base.

[0058] It can be seen that the weights are recommended to the dataset D by influencing i The neighbors of D i Score i , which ultimately affects the objective function. However, the weight vector W = [w 1 ,w 2 ,...,w 8 ] cannot be explicitly related to the objective function, so the gradient descent method, Lagrange extremum method, etc. will fail. F(W) can be regarded as a black box function of the weight vector W, and the iterative update of this weight will be solved using the heuristic method - Bayesian optimization.

[0059] In the simulation experiment, the Hyperopt library in Python was used for Bayesian optimization. The experimental settings are as follows: the TPE search algorithm is used, 10,000 iterations are performed, and the range of each weight is defined as a continuous interval [0,10]. The final total loss is -0.62427, indicating that the best F is 0.62427; the final weights are [4.28, 1.09, 4.61, 8.35, 0.52, 6.06, 2.97, 1.74]. Figure 2 The trend of the loss function (the negative value of the objective function) changing with the partial weights is shown. It is not difficult to see that w4 is negatively correlated with the loss function, and w5 is positively correlated with the loss function.

[0060] Among them, the specific steps of recommending a suitable model for the new data set described in step S2 are: first, obtain the meta-features of the new data set, then calculate the k nearest neighbor data set of the data set based on W1-norm, and recommend the top N models with the best performance on each neighbor to the target data set.

[0061] The distributed model training system process described in step S2 is as follows: Figure 3 As shown, the steps are as follows:

[0062] S211. First, CF-DAML removes duplicate models from k*N initial models (LMs) to obtain intermediate models IMs.

[0063] S212, CF-DAML then turns on the multi-process mode to train IMs simultaneously. Since the models in the meta-knowledge base are not complex, the distributed model training system can be deployed on the CPU core of a single computer. All recommended models receive the same training set and validation set of the new dataset, and all CPU cores run in parallel. Each CPU core trains one model at a time and receives a new task (training the next model) after completing the current model training.

[0064] S213. If some models cannot be trained due to timeout or runtime errors, CF-DAML will discard these models and only collect the successfully trained candidate models CMs and their predicted labels on the validation set.

[0065] The selective stacking integrated system workflow in step S2 is as follows: Figure 4 As shown, the steps are as follows:

[0066] S221. After training CMs, MSSE selects high-performance models HMs from CMs and base models BMs from HMs. Finally, MSSE stacks BMs and trains them in a distributed manner. The principle of the distributed computing method is also multi-process, and the steps are shown in S212.

[0067] The above BMs should be as accurate and diverse as possible to build an integrated model with strong generalization ability. In order to meet the requirements of accuracy and diversity, the present invention adopts a multi-round screening principle.

[0068] Accuracy screening CF-DAML first obtains CMs along with their classification accuracy and label predictions on the validation set; then, CF-DAML selects models with classification accuracy above the average to obtain HMs.

[0069] Diversity Screening CF-DAML performs a greedy loop-based filtering algorithm to select a set of HMs so that each model is independent of each other. First, we use the divergence metric DM to define the diversity of two models. DM measures the proportion of inconsistent prediction results between models. The larger the DM value, the greater the diversity.

[0070] For binary or multi-classification tasks, assume that the label predictions generated by two models i and j for the same sample are

[0071] h i =[p i,1 ,p i,2 ,...,p i,n ]

[0072] h j =[p j,1 ,p j,2 ,...,p j,n ]

[0073] Then there is

[0074]

[0075] Where I(p i,k =p j,k ) is an indicator function, if p i,k =p j,k The value is 1, otherwise the value is 0.

[0076] In this algorithm, HMs are first sorted in descending order according to their classification accuracy on the validation set, and a bin R is defined to store the models to be deleted from HMs, and a list B is defined to store BMs. Then, a loop is looped through each model clf in HMs i :If clf i is saved in R, it will be ignored; otherwise, it is stored in B. Then, from clf j Start inner loop (j=i+1,i+2,...,n, where n is the number of HMs); if DM(clf i ,clf j )<τ(threshold), then clf iand clf j Very similar, and clf j Put it into R, because clf i Maybe better than clf j has higher classification accuracy. Finally, the models in List B are obtained as BMs. After many experiments, we determined the best This means that there are no identical models between BMs. As τ increases, the number of BMs decreases rapidly, resulting in poor ensemble results. The loop filtering algorithm is shown in Table 3.5.

[0077] It can be shown that BMs are independent of each other: Must have DM (clf i ,clf j )<τ, because all clf k (k≠i and DM(clf i ,clf k )≥τ) has been put into R.

[0078] Since each base model has been trained on the training set of the new dataset, there is no need to train these models again.

[0079] S222. In the training phase, in order to avoid overfitting, the new data set is re-divided into a training set and a validation set, and the output class probability and related sample labels of each base model on the training set are combined into a meta-training set for training the meta-classifier of the stacked ensemble learning (Stack). Then, the validation set is used to observe the performance of the ensemble model.

[0080] S223. In the testing phase, after the output class probabilities of each base model on the test set are combined into a meta-test set, the prediction of the meta-classifier on the meta-test set is the final result.

[0081] First, it should be noted that BMs are independent of each other, which provides opportunities for distributed learning. Finally, based on comparative experiments with LightGBM, multilayer perceptron, and logistic regression, the random forest classifier was selected as the meta-classifier. Random forest can effectively acquire the characteristics of the dataset and avoid overfitting due to its randomness. In addition, it can run quickly on large datasets due to its parallelism. The most important hyperparameter in the random forest classifier is the number of estimators. After many experiments, we set this hyperparameter to 200, while the other hyperparameters of the random forest classifier were set to the default values.

[0082] Embodiment 1:

[0083] The present invention is a distributed automatic machine learning method based on collaborative filtering.

[0084] Figure 1 The overall framework of CF-DAML is shown. In the offline stage, the top N models (N=50 in this stage) are searched for each dataset in the meta-knowledge base and stored in a hash dictionary (key: dataset ID, value: top N models that perform well on the dataset); and the weights of the meta-features of the dataset in the meta-knowledge base are calculated. In the online stage, CF-DAML first calculates the meta-features of the new dataset and recommends a suitable model for it. Then, CF-DAML uses a distributed model training system to train the recommended model on the training set within the specified time limit, and evaluates the trained model on the validation set. Finally, CF-DAML uses a selective stacking ensemble system to integrate several high-performance models to predict labels for the test set.

[0085] A distributed AutoML framework based on memory collaborative filtering. The steps are as follows:

[0086] (1) Offline stage: Search for the top N models (N = 50 in this stage) for each dataset in the meta-knowledge base and store them in a hash dictionary (key: dataset ID, value: top N models that perform well on the dataset); and calculate the weights of the meta-features of the dataset in the meta-knowledge base.

[0087] (2) Online phase: CF-DAML first calculates the meta-features of a new dataset and recommends a suitable model for it. Then, CF-DAML uses a distributed model training system to train the recommended model on the training set within a specified time limit and evaluates the trained model on the validation set. Finally, CF-DAML uses a selective stacking ensemble system to integrate several high-performance models to predict labels for the test set.

[0088] (1.1) The meta-knowledge base described in step (1) includes three sub-databases:

[0089] Subdatabase 1 contains a two-dimensional matrix X with 42000 rows and 553 columns. Each element x ij Represents the balanced accuracy of the i-th model on the j-th dataset. This library has 553 classification task datasets from OpenML, each of which contains no more than 10,000 sample points. Each model includes a dataset preprocessing method and a classifier with specific hyperparameters. The data preprocessing methods in this library include polynomial processing and principal component analysis (PCA), and the classifier comes from the scikit-learn function package.

[0090] Subdatabase 2 contains a two-dimensional matrix Y with 553 rows and 46 columns. Each element y ijRepresents the jth meta-feature of the ith dataset. Meta-features are divided into three categories: features of the dataset itself, such as the number of samples, the number of categories, and the number of features; model-related (signature) features, such as the classification accuracy of the benchmark decision tree on the dataset; and indicators related to information theory, such as inter-class entropy.

[0091] Subdatabase 3 contains a dictionary Z. Each entry z i Represents the algorithm name and hyperparameter values ​​of the i-th model, corresponding to the model in sub-database 1.

[0092] In the offline stage, sub-database 1 is operated to obtain sub-database 4: a two-dimensional matrix T with 553 rows and 50 columns, where t ij represents the ID of the jth model that performs best on the ith dataset. The final sub-databases used in CF-DAML are 2, 3, and 4. See Tables 1 and 2 for details on the classifiers and hyperparameter configurations in the databases.

[0093] (1.2) The weights of the meta-features described in step (1) are iterated through the Bayesian optimization algorithm. First, information-rich meta-features are selected from the 46 meta-features of the training data set in the meta-knowledge base, and then the selected meta-features are clustered and grouped; a black-box function relationship between the objective function and the meta-feature subset is constructed, and the weights of the meta-feature subsets are learned using Bayesian optimization, and the weights are assigned to each subset when convergence is reached; finally, the similarity between the data sets is calculated using W1-norm.

[0094] The clustering of the meta-features described in (1.2) includes the following steps: studying the correlation between each pair of variables in the meta-feature set through the distance correlation coefficient, eliminating the meta-features with missing values ​​and the meta-features with the same meaning, and clustering the remaining 30 meta-features into 8 subsets according to their physical meanings, as shown in Table 3.

[0095] The Bayesian optimization algorithm described in (1.2) is used to learn the weights of the meta-feature subsets. It includes the following steps:

[0096] (1.2.1) In order to emphasize the importance of different meta-feature subsets, the two datasets D i ,D j The distance is expressed as

[0097]

[0098] in Denotes the dataset D i ,D j The 1-norm of the k-th class feature set, ω k N represents the weight of the k-th meta-feature set, and S=8 represents 8 meta-feature subsets. krepresents the number of meta-features contained in the k-th meta-feature set, Denotes the dataset D i The sth metadata in the kth class meta-feature set.

[0099] Two datasets D i ,D j The similarity is expressed as

[0100]

[0101] (1.2.2) How to determine the weight is an optimization problem, and the present invention uses Bayesian optimization to do it. It should be emphasized that the training of weights is completed offline. After the training is completed, the optimal weights are directly embedded in the recommendation system module of CF-DAML. When new data is input into the software, the weights are directly used for modeling without retraining the weights.

[0102] For data set D in sub-database 2 i Use the above distance formula to calculate the k nearest neighbor dataset ND of the dataset i ={D i,1 ,D i,2 ,...,D i,k}, D i The top-N models of each neighbor of are recommended to D i There are k*N models in total. These models are in D i The classification accuracy on does not require training and can be directly queried from sub-database 2.

[0103] Let dataset D i The score is

[0104]

[0105] Recommended to D i k*N models in the dataset D i The sum of the classification accuracy of Denotes the dataset D i The t-th model of the n-th neighbor in the dataset D i The classification accuracy on .

[0106] Taking the average score of all datasets in sub-database 2, we can get the objective function

[0107]

[0108] Where B=550 represents the number of training data sets in the meta-knowledge base.

[0109] It can be seen that the weights are recommended to the dataset D by influencing i The neighbors of Di Score i , which ultimately affects the objective function. However, the explicit relationship between the weight W and the objective function cannot be obtained, so the gradient descent method, Lagrange extremum method, etc. will fail. F(W) can be regarded as a black box function of the weight W, and the iterative update of this weight will be solved using the heuristic method - Bayesian optimization.

[0110] In the simulation experiment, the Hyperopt library in Python was used for Bayesian optimization. The experimental settings are as follows: the TPE search algorithm is used, 10,000 iterations are performed, and the range of each weight is defined as a continuous interval [0,10]. The final total loss is -0.62427, indicating that the best F is 0.62427; the final weights are [4.28, 1.09, 4.61, 8.35, 0.52, 6.06, 2.97, 1.74]. Figure 2 The trend of the loss function (the negative value of the objective function) changing with the partial weights is shown. It is not difficult to see that w4 is negatively correlated with the loss function, and w5 is positively correlated with the loss function.

[0111] (2.1) The specific steps of recommending a suitable model for a new dataset as described in step (2) are as follows: first, obtain the meta-features of the target dataset, then calculate the k nearest neighbor dataset of the dataset based on W1-norm, and recommend the top N models with the best performance on each neighbor to the target dataset.

[0112] (2.2) The distributed model training system process described in step (2) is as follows Figure 3 As shown, the steps are as follows:

[0113] (2.2.1) First, CF-DAML removes duplicate models from k*N initial models (LMs) to obtain intermediate models (IMs).

[0114] (2.2.2) Then, CF-DAML turns on the multi-process mode to train IMs simultaneously. Since the models in the meta-knowledge base are not complex, the distributed model training system can be deployed on the CPU core of a single computer. All recommended models receive the same target dataset training set and validation set, and all CPU cores run in parallel. Each CPU core trains one model at a time and receives a new task (training the next model) after completing the current model training.

[0115] (2.2.3) If some models cannot be trained due to timeout or runtime errors, CF-DAML will discard these models and only collect the successfully trained candidate models (CMs) and their predicted labels on the validation set.

[0116] (2.3) The workflow of the selective stacking integration system described in step (2) is as follows Figure 4 As shown, the steps are as follows:

[0117] (2.3.1) After training CMs, MSSE selects high-performance models (HMs) from CMs and base models (BMs) from HMs. Finally, MSSE stacks BMs and trains them in a distributed manner. The principle of the distributed computing method is also multi-process, and the steps are shown in (2.2.2).

[0118] (2.3.2) BMs should be as accurate and diverse as possible to build an integrated model with strong generalization ability. In order to meet the requirements of accuracy and diversity, the present invention adopts the principle of multiple rounds of screening.

[0119] Accuracy screening CF-DAML first obtains CMs along with their classification accuracy and label predictions on the validation set; then, CF-DAML selects models with accuracy above the average to obtain HMs.

[0120] Diversity Screening CF-DAML performs a greedy loop-based filtering algorithm to select a set of HMs so that each classifier is independent of each other. First, we use the divergence metric (DM) to define the diversity of two classifiers. DM measures the proportion of inconsistent prediction results between classifiers. The larger the DM value, the greater the diversity.

[0121] For binary or multi-classification tasks, suppose that the label predictions generated by two classifiers i and j for the same sample are

[0122] h i =[p i,1 ,p i,2 ,...,p i,n ]

[0123] h j =[p j,1 ,p j,2 ,...,p j,n ]

[0124] Then there is

[0125]

[0126] Where I(p i,k =p j,k ) is an indicator function, if p i,k =p j,k The value is 1, otherwise the value is 0.

[0127] In this algorithm, the HMs are first sorted in descending order according to their classification accuracy on the validation set, and a bin (R) is defined to store the models to be deleted from the HMs, and a list B is defined to store the BMs. Then, a loop is looped through each model clf in the HMs i :If clf i is saved in R, it will be ignored; otherwise, it is stored in B. Then, from clf j Start inner loop (j=i+1,i+2,...,n, where n is the number of HMs); if DM(clf i ,clf j )<τ(threshold), then clf i and clf j Very similar, and clf j Put it into R, because clf i Maybe better than clf j has higher classification accuracy. Finally, we obtained BMs (i.e., the model in List B). After many experiments, we determined the best This means that there are no identical models between BMs. As τ increases, the number of BMs decreases rapidly, resulting in poor ensemble results. The loop filtering algorithm is shown in Table 3.5.

[0128] It can be shown that BMs are independent of each other: Must have DM (clf i ,clf j )<τ, because all clf k (k≠i and DM(clf i ,clf k )≥τ) has been put into R.

[0129] Distributed Stacked Ensemble Learning Since each base model has been trained on the training set of the new dataset, there is no need to train these models again.

[0130] In the training phase, in order to avoid overfitting, the new dataset is re-divided into training set and validation set, and the output class probability and related sample labels of each base model on the training set are combined into a meta-training set for training the meta-classifier. Then, the validation set is used to observe the performance of the integrated model.

[0131] In the testing phase, after the output class probabilities of each base model on the test set are combined into a meta-test set, the prediction of the meta-classifier on the meta-test set is the final result.

[0132] First, it should be noted that BMs are independent of each other, which provides an opportunity for distributed learning. Then, cross-validation is not used in this algorithm to avoid overfitting because it would waste too much time without producing better results. Finally, based on comparative experiments with LightGBM, multilayer perceptron, and logistic regression, random forest is selected as the meta-classifier. Random forest can effectively acquire the characteristics of the dataset and avoid overfitting due to its randomness. In addition, it can run quickly on large datasets due to its parallelism. The most important hyperparameter in the random forest classifier is the number of estimators. After many experiments, we set this hyperparameter to 200, while the other hyperparameters of the random forest classifier are set to the default values.

[0133] Example 2

[0134] To test the performance of CF-DAML, 11 UCI datasets that do not overlap with sub-database 1 were selected. See Table 4 for the properties of the datasets.

[0135] Table 4 Detailed properties of the test dataset

[0136]

[0137] The present invention compares CF-DAML with Auto-sklearn, Oboe, Tensor-Oboe and TPOT, which are two of the most famous open source AutoML systems. Auto-sklearn has won the first phase of the ongoing ChaLearnAutoML challenge, and Oboe and Tensor-Oboe are recent AutoML systems based on collaborative filtering. Since Auto-sklearn, Oboe, Tensor-Oboe and TPOT have time constraint mechanisms, and the running time of CF-DAML is implicitly determined by k and N, in order to ensure fairness, k and N are reasonably adjusted to obtain a suitable running time considering the size of the data set, and this time is set as the running time limit of other frameworks. In addition to Auto-sklearn being unable to perform parallel calculations, in order to compensate for this unfairness, a sufficient memory limit (100GB) is set for Auto-sklearn when searching for models.

[0138] For CF-DAML, the optimal k is about 20 and the optimal N is about 10. We set (k, N) = (20, 10) for CF-DAML for all datasets.

[0139] CF-DAML is able to reach the best accuracy with extremely fast convergence speed: it only takes about 270 seconds to run on all 11 small datasets, and the longest time on each dataset does not exceed 60 seconds. First, the running time of other frameworks on each dataset is limited to 100 seconds, and the corresponding classification accuracy comparison is shown in Table 5. On the 11 datasets, CF-DAML performs best on 7 datasets, while Auto-sklearn, Oboe, TPOT, and Tensor-Oboe perform best on 1, 0, 1, and 2 datasets, respectively. On some datasets, Oboe and TPOT had some execution errors, and the corresponding accuracy is indicated by ' / '. Figure 5 The actual running time of each framework is shown.

[0140] Table 5 Classification accuracy of each framework under time constraints on a small dataset

[0141]

[0142]

[0143] One of the biggest advantages of CF-DAML is that it can achieve excellent classification accuracy with extremely fast convergence speed. Although CF-DAML does not perform as well as other methods on datasets 0, 5, 7, and 10, its accuracy on these four datasets is not lower. When TPOT is constrained to run for the same time as CF-DAML on dataset 0, TPOT's accuracy is only 0.897, while CF-DAML's accuracy is 0.906. Similarly, when Auto-sklearn runs for the same time as CF-DAML on dataset 5, their accuracies are 0.074 and 0.978 respectively; when Tensor-Oboe runs for the same time as CF-DAML on dataset 10, their accuracies are 0.486 and 0.492 respectively.

[0144] This paper proposes a distributed automatic machine learning method based on collaborative filtering, which adopts a distributed AutoML framework based on collaborative filtering and is named CF-DAML. This method defines the AutoML problem as a coupling problem between memory-based collaborative filtering and selective ensemble learning for the first time, getting rid of the time-consuming hyperparameter optimization module that required a lot of work in the past, and greatly reducing the time complexity while ensuring the improvement of classification accuracy.

[0145] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A distributed automatic machine learning method based on collaborative filtering, characterized in that: The method includes: S1, offline phase: search for the top N models for each dataset in the meta-knowledge base and store them in a hash dictionary, where the key is the dataset ID and the value is the top N models that perform well on the dataset; and calculate the weights of the meta-feature subsets of the dataset in the meta-knowledge base; S2. Online stage: CF-DAML, a distributed AutoML framework based on collaborative filtering, first calculates the meta-features of the new dataset and recommends a suitable model for it through the model recommendation module. Then, CF-DAML uses the distributed model training system DSTM to train the recommended model on the training set of the new dataset within the specified time limit and evaluates the trained model on its validation set. Finally, CF-DAML uses the selective stacking integration system MSSE to integrate several high-performance models to predict labels for the test set of the new dataset.

2. The distributed automatic machine learning method based on collaborative filtering according to claim 1, characterized in that: The meta-knowledge base described in step S1 includes three sub-databases: Subdatabase 1 contains a two-dimensional matrix X with 42000 rows and 553 columns; each element x ij represents the balanced accuracy of the i-th model on the j-th dataset, where i=1,2,...,42000; j=1,2,...,553; this library has 553 classification task datasets from OpenML, each dataset contains no more than 10,000 sample points; each model includes a dataset preprocessing method and a classifier with specific hyperparameters; Subdatabase 2 contains a two-dimensional matrix Y with 553 rows and 46 columns; each element y ij Represents the jth meta-feature of the i-th dataset, where i = 1, 2, ..., 553; j = 1, 2, ..., 46; meta-features are divided into three categories: features of the dataset itself, features related to the machine learning model, and indicators related to information theory; Subdatabase 3 contains a dictionary Z; each entry z i represents the algorithm name and hyperparameter value of the i-th model, corresponding to the model in sub-database 1, where i = 1, 2, ..., 42000; In the offline stage, sub-database 1 is operated to obtain sub-database 4: a two-dimensional matrix T with 553 rows and 50 columns, where t ij represents the number of the jth model with the best performance on the i-th dataset, where i = 1, 2, ..., 553; j = 1, 2, ..., 50; The final sub-databases used in CF-DAML are Sub-database 2, Sub-database 3, and Sub-database 4.

3. The distributed automatic machine learning method based on collaborative filtering as claimed in claim 2, characterized in that: The weight of the meta-feature subset described in step S1 is iterated through a Bayesian optimization algorithm, firstly selecting information-rich meta-features from the 46 meta-features of the data set in the meta-knowledge base, and then clustering and grouping the selected meta-features to obtain a meta-feature subset; A black-box function relationship between the objective function and the meta-feature subset is constructed, and the weights of the meta-feature subset are learned using Bayesian optimization until convergence is reached; finally, the similarity between data sets is calculated using W1-norm.

4. The distributed automatic machine learning method based on collaborative filtering according to claim 3, characterized in that: The clustering of meta-features includes the following steps: studying the correlation between two meta-features in a meta-feature set by using a distance correlation coefficient, eliminating meta-features with missing values ​​and meta-features with the same meaning, and clustering the remaining meta-features into 8 types of meta-feature subsets according to their physical meanings.

5. The distributed automatic machine learning method based on collaborative filtering according to claim 4, characterized in that: The method of constructing a black box function relationship between the objective function and the meta-feature subset and using Bayesian optimization to learn the weight of the meta-feature subset includes: S11. In order to emphasize the importance of different meta-feature subsets, the two datasets D i ,D j The distance is expressed as in Denotes the dataset D i ,D j The 1-norm of the k-th class feature subset, ω k represents the weight of the k-th meta-feature set, K=8 represents 8 meta-feature subsets; N k represents the number of meta-features contained in the k-th meta-feature set, Denotes the dataset D i The sth meta-feature in the kth meta-feature subset; Two datasets D i ,D j The similarity is expressed as S12. Use Bayesian optimization to learn the weights of the meta-feature subsets. The weight training is done offline. After the training is completed, the optimal weight is directly embedded in the model recommendation module of CF-DAML. When new data is input into the software, the weight is directly used for modeling without retraining the weight. For data set D in sub-database 2 i Use the above distance formula to calculate the k nearest neighbor dataset ND of the dataset i ={D i,1 ,D i,2 ,...,D i,k }, D i The top-N models of each neighbor of are recommended to D i , there are k*N models in total; Let dataset D i The score is Recommended to D i k*N models in the dataset D i The sum of the classification accuracy of Denotes the dataset D i The t-th model of the n-th neighbor in the dataset D i Classification accuracy on ; Taking the average score of all datasets in sub-database 2, we can get the objective function Where, B=553 represents the number of training data sets in the meta-knowledge base; The weights are recommended to the dataset D by influencing i The neighbors of D i Score i , which ultimately affects the objective function; F(W) is regarded as a black box function of the weight vector W, and the iterative update of this weight will be solved using a heuristic method - Bayesian optimization.

6. The distributed automatic machine learning method based on collaborative filtering according to any one of claims 1 to 5, characterized in that: In step S2, the specific steps of recommending a suitable model for the new data set are: first, the meta-features of the new data set are obtained, and then the k-nearest neighbor data set of the data set is calculated based on W1-norm, and the top N models with the best performance on each neighbor are recommended to the target data set.

7. The distributed automatic machine learning method based on collaborative filtering according to claim 6, characterized in that: In step S2, the distributed model training system process includes: S211. First, CF-DAML removes duplicate models from k*N initial models LMs to obtain intermediate models IMs; S212, then, CF-DAML turns on the multi-process mode to train IMs simultaneously; the distributed model training system is deployed on the CPU core of a single computer; all recommended models receive the same training set and validation set of the new data set, and all CPU cores run in parallel; each CPU core trains one model at a time, and receives a new task to train the next model after completing the training of the current model; S213. If some models cannot be trained due to timeout or runtime errors, CF-DAML will discard these models and only collect the successfully trained candidate models CMs and their predicted labels on the validation set.

8. The distributed automatic machine learning method based on collaborative filtering according to claim 7, characterized in that: In step S2, the selective stacking integrated system workflow includes: S221. After training CMs, MSSE selects high-performance models HMs from CMs and base models BMs from HMs. Finally, MSSE stacks BMs and trains them in a distributed manner. S222. In the training phase, in order to avoid overfitting, the new data set is re-divided into a training set and a validation set, and the output class probability and related sample labels of each base model in the training set are combined into a meta-training set for training the meta-classifier of the stacked ensemble learning Stack; then, the validation set is used to observe the performance of the ensemble model; S223. In the testing phase, after the output class probabilities of each base model on the test set are combined into a meta-test set, the prediction of the meta-classifier on the meta-test set is the final result.

9. The distributed automatic machine learning method based on collaborative filtering according to claim 8, characterized in that: In S221, when selecting the base models BMs, the principle of multiple rounds of screening is adopted: Accuracy screening: CF-DAML first obtains CMs and their classification accuracy and label predictions on the validation set; Then, CF-DAML selects models with classification accuracy higher than the average to obtain HMs; Diversity screening: CF-DAML performs a greedy loop-based filtering algorithm to select a set of HMs so that each model is independent of each other. First, the divergence metric DM is used to define the diversity of two models. DM measures the proportion of inconsistent prediction results between models. The larger the DM value, the greater the diversity. For binary or multi-classification tasks, assume that the label predictions generated by two models i and j for the same sample are h i =[p i,1 ,p i,2 ,...,p i,n ] h j =[p j,1 ,p j,2 ,...,p j,n ] Then there is Where I(p i,k =p j,k ) is an indicator function, if p i,k =p j,k The value is 1, otherwise the value is 0; In this algorithm, HMs are first sorted in descending order according to their classification accuracy on the validation set, and a bin R is defined to store the models to be deleted from HMs, and a list B is defined to store BMs; then, each model clf in HMs is looped through i :If clf i is saved in R, it will be ignored; otherwise, it is stored in B; then, from clf j Start inner loop; if DM(clf i ,clf j )<τ, τ is the threshold, then clf i and clf j Very similar, and clf j Put it into R, because clf i Maybe better than clf j with higher classification accuracy; finally, the models in List B are obtained as BMs.

10. The distributed automatic machine learning method based on collaborative filtering according to claim 8, characterized in that: The random forest classifier was selected as the meta-classifier; the number of hyperparameter estimators in the random forest classifier was set to 200.

Images