Rapid and accurate hyperparameter optimization for machine learning and deep learning models using meta-modeling

Abstract

This document is a technique to train regressor(s) to predict how effective a machine learning model (MLM) would be if trained with new hyperparameters and / or a new dataset. In embodiments, for each training dataset, a computer derives values of dataset meta-features from the dataset. For each HC comprising a hyperparameter configuration of the MLM, the computer performs: configuring the MLM based on the HC, training the MLM based on the dataset, and obtaining an empirical quality score indicating how effective the trained MLM was when configured with the HC. A performance tuple is generated that contains: the HC, the values of the dataset meta-features, the empirical quality score, and for each hyperparameter configuration: the empirical quality score of the hyperparameter configuration and / or the hyperparameter configuration itself. Based on the performance tuples, a regressor is trained to predict an estimated quality score based on a given dataset and a given HC.

Description

Technical Field

[0001] This invention relates to hyperparameter optimization of machine learning (ML) models. This paper describes techniques for training one or more regressors to predict how effective a machine learning (ML) model will be if configured with new hyperparameter values ​​and / or trained on a new dataset. Background Technology

[0002] The use of machine learning (ML), such as deep learning (DL), is rapidly spreading across industries and business units and is becoming a ubiquitous tool within some companies. Optimizing an ML model requires tuning its hyperparameters as configuration settings. During hyperparameter tuning, the ML model is repeatedly trained with different (e.g., improved) hyperparameter values ​​to explore a multidimensional configuration hyperspace. This exploration can be time- and / or space-intensive. Hyperparameter settings have a strong impact on model performance (e.g., accuracy, F1 score, etc.). Tuning a model's hyperparameters is very difficult and can be quite time-consuming, especially for large datasets.

[0003] Hyperparameter optimization of ML models is essential for many applications. This is because, in most cases, default hyperparameter values ​​do not lead to the best-performing model. During hyperparameter optimization, the ML model is trained and validated multiple times on the dataset using different hyperparameter values ​​to allow the optimizer to explore the hyperparameter space. Such exploration is time-consuming.

[0004] Unlike classic convex optimization problems, hyperparameter optimization of ML models is a unique and challenging problem for at least three reasons:

[0005] • Black-box optimization: There is usually no known analytical formula to express the performance of an ML model as a function of its hyperparameters. This means that one cannot rely on classical (e.g., convex) optimization methods to find the optimal values, and the only way to explore the configuration hyperspace is through experimentation that requires training and evaluation using a set of hyperparameter values.

[0006] • Slow training: Training a single model is time-consuming, and tuning requires retraining the model and evaluating it multiple times in each tuning session.

[0007] • Numerous hyperparameters: This is especially true in deep neural network models, which typically have a large number of hyperparameters in each layer, such as the number of layers, number of neurons, dropout, L2 regularization, and activation and optimizer parameters. Attached Figure Description

[0008] In the attached diagram:

[0009] Figure 1AIt depicts a block diagram of an example computer that trains a regressor to predict how effective a machine learning (ML) model will be if configured with new hyperparameter values ​​and / or trained on a new dataset.

[0010] Figure 1B It is a block diagram depicting an example computer that uses Hyperstar to quickly and concisely reveal some outlines of the mysterious configuration of hyperspace;

[0011] Figure 2 It is a flowchart depicting an example computer process used to train a regressor to predict how effective a machine learning (ML) model will be if configured with given hyperparameter values ​​and trained on a given dataset.

[0012] Figure 3 It is a block diagram depicting an example computer that uses a dual regressor to adapt (e.g., an extraordinary) dataset;

[0013] Figure 4 This is a block diagram illustrating a computer system on which embodiments of the present invention may be implemented;

[0014] Figure 5 It is a block diagram illustrating the basic software system that can be used to control the operation of a computing system. Detailed Implementation

[0015] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the invention. However, it will be apparent that the invention can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the invention.

[0016] General Overview

[0017] To prepare a meta-model (i.e., a meta-learning hyperparameter optimizer) for cold starts (i.e., an untrained meta-model or an unfamiliar dataset), several initial points are used to coarsely investigate the multidimensional hyperparameter space used to configure the machine learning (ML) model before exploring regions with more promising sets of hyperparameter values. To obtain each point, the ML model needs to be trained / evaluated on the dataset, which is typically time-consuming. A large number of hyperparameters leads to a high-dimensional search space for the optimizer, which can exacerbate the cold start problem: the optimizer needs even more points to better cover the high-dimensional search space.

[0018] The hyperparameter optimization process based on meta-learning includes the following three stages:

[0019] 1. Model training: Generating a meta-dataset by repeatedly training the model with the dataset;

[0020] 2. Meta-model training: Train the meta-model using the meta-dataset;

[0021] 3. Inference: Optimize model hyperparameters for the new dataset using a meta-model.

[0022] All of these stages require model training. The second stage requires training a general meta-model. In this embodiment, the online third stage requires training an additional meta-model specifically for the new dataset.

[0023] In the offline training phases 1-2, a meta-dataset needs to be collected from a corpus across the datasets. To collect metadata, the ML model is trained by varying hyperparameters across a series of values ​​for each dataset and recording the scores obtained for each different pairing of datasets and hyperparameter configurations. This involves generating a set of configuration superstars (i.e., landmarks), which are performance reference points that can be reused across multiple datasets. Superstars can be used to quickly and concisely reveal some contours of the mysterious configuration hyperspace and / or datasets. In this paper, superstars are used to make quality score predictions more flexible and accurate for the meta-learning regressor.

[0024] Meta-features are obtained for each dataset and combined with superstars, the current hyperparameter configuration, and the corresponding performance score. A regressor (i.e., a meta-model) is tuned and trained on the meta-dataset to predict performance scores for the model. The trained regressor is used in an online third stage to guide hyperparameter tuning. This is done by first extracting meta-features from the new meta-dataset and combining them with a set of (e.g., randomly generated) hyperparameters. The trained regressor predicts scores for each generated hyperparameter configuration on the new dataset, and the model is trained on the hyperparameter configuration with the highest predicted score.

[0025] The third online phase requires inference via a general meta-model for hyperparameter optimization of the ML model given a new dataset. Meta-features about the new dataset are used together with the trained meta-model to warm-start the hyperparameter optimization algorithm. The general meta-model predicts the performance score of the ML model for each of the randomly generated hyperparameter configurations. Specifically, static and dynamic meta-features are obtained (e.g., extracted) for the new dataset. Many hyperparameter configurations are generated, and each is combined with the meta-features. Some top-predicting hyperparameter configurations can be selected to evaluate the ML model using them. The runtime overhead of the prediction phase is typically negligible compared to the actual model evaluation (i.e., training and validation).

[0026] In this embodiment, for each training dataset, the computer derives the values ​​of the dataset's meta-features from the training dataset. For each hyperparameter configuration of the machine learning (ML) model, including feature point hyperparameter configurations, the computer performs the following: configures the ML model based on the hyperparameter configuration, trains the ML model based on the training dataset, and obtains an empirical quality score indicating how effectively the trained ML model is trained when the hyperparameter configuration is configured. A performance tuple is generated, containing: the hyperparameter configuration, the values ​​of the dataset's meta-features, the empirical quality score, and the empirical quality score for each feature point configuration. Based on the performance tuple, a regressor is trained to predict the estimated quality score based on the given dataset and the given hyperparameter configuration.

[0027] A regressor is general in the sense that it is trained on multiple datasets for eventual reuse with many new datasets. In an embodiment, an additional regressor is dedicated to learning and inferring new (i.e., unfamiliar) datasets and is not exposed to other datasets. In an embodiment, the general regressor can operate as a “training round” for a specialized regressor, allowing the general regressor to replace or be used in conjunction with the specialized regressor, at least until the specialized regressor has been sufficiently trained to outperform the general regressor on new datasets. In an embodiment, the collaboration of the two regressors requires confidence weighting to reflect how well the specialized regressor has been trained so far, and that the training of the specialized regressor is still ongoing. In an embodiment, the confidence weights are repeatedly adjusted to favor the specialized regressor to reflect the monotonic improvement in the performance quality of the specialized regressor during training.

[0028] 1.0 Sample Computer

[0029] Figures 1A-1B These are block diagrams that alternately depict the same example computer 100 in the embodiments. Figure 1A In this context, computer 100 trains a regressor to predict how effective a machine learning (ML) model will be if configured with new hyperparameter values ​​and / or trained on a new dataset. Computer 100 can be one or more of a rack server such as a blade server, a personal computer, a mainframe, a virtual computer, a smartphone, or other computing device.

[0030] Computer 100 can store two ML models 130 and 150 in its memory. Depending on the embodiment, ML model 130 is designed for clustering, classification, regression, anomaly detection, prediction, or dimensionality reduction (i.e., simplification). Examples of ML algorithms include decision trees, support vector machines (SVM), Bayesian networks, stochastic algorithms such as genetic algorithms (GA), and connectionist topologies such as artificial neural networks (ANN). ML implementations may rely on matrices, symbolic models, and hierarchical and / or associative data structures. Parameterized (i.e., configurable) implementations of best-in-class ML model types can be found in open-source libraries such as scikit-learn (sklearn), Google's TensorFlow for Python and C++, or Georgia Tech's MLPack for C++. Shogun is an open-source C++ ML library with adapters for several programming languages, including C#, Ruby, Lua, Java, MatLab, R, and Python.

[0031] ML Model 130 can be one of many model types. Each model type can have tunable properties (i.e., hyperparameters such as XY) that can be optimized to improve performance during training and / or inference in various ways (such as improved inference accuracy and / or reduced consumption of one or more resources such as time and / or space). Different model types have different numbers and kinds of hyperparameters.

[0032] Before training, the ML model 130 should be assigned a configuration, such as different configurations 121-122 of 180. Each configuration consists of corresponding values ​​for each of the hyperparameters X and Y. For example, for configuration 121, the value of hyperparameter X is 5.5.

[0033] Each of the hyperparameters X and Y has a series of multiple (e.g., infinitely many) values. The combination of hyperparameters X and Y represents a hyperdimensional configuration space, where each of the hyperparameters X and Y is a dimension of the configuration space.

[0034] The lifecycle of ML Model 130 has two phases. The first phase is preparatory and requires training in a laboratory setting, as shown in the figure. The second phase requires inference in a production environment, such as using real-time and / or streaming data (not shown).

[0035] During inference, ML model 130 is applied to (e.g., unfamiliar) samples that can be fed into ML model 130 as input. This causes ML model 130 to process the samples according to its internal mechanisms, which are specifically configured based on reinforcement learning during previous training. For example, if ML model 130 is a classifier, it can select one of several mutually exclusive labels (i.e., classifications) (such as hot and cold) for the samples.

[0036] During training, exploration of the configuration space can alter the performance of the ML model 130, which can result in one or more improvements and / or one or more degradations. For example, changing the values ​​of one or more hyperparameters may simultaneously lead to faster training and reduced accuracy.

[0037] Supervised training of the ML model 130 may require processing of the training corpus 170 for reinforcement learning. The training corpus 170 contains at least training datasets 111-112, which may each contain documents, records, or other data objects from which the ML model 130 can form impressions for generalization.

[0038] Datasets 111-112 each have more or less different content. For example, dataset 111 may contain color photographs, while dataset 112 may contain monochrome photographs. In this embodiment, the training corpus 170 is itself a whole dataset, and datasets 111-112 can be separated from this whole dataset into non-overlapping cross-validation folds (i.e., parts) of the whole dataset. Freely available benchmark datasets for proof of concept include the OpenML binary classification dataset.

[0039] Regardless of how similar or dissimilar the datasets 111-112 may be, some generalizations are possible such that all datasets within the training corpus 170 have their own (e.g., different) values ​​for the same set of meta-features that describe the dataset as a whole, more or less disregarding any particular item within the dataset. For example, a meta-feature might calculate how many colors are in the palette needed to represent all the photos in the dataset, which might have a small integer value if the dataset is monochromatic.

[0040] For example, meta-feature A could be the average brightness of all photos in the dataset. For instance, for dataset 111, the value of meta-feature A would be 1.1. The values ​​of meta-features can conform to the same data type, such as integers, real numbers, or categorical values, such as photo orientations with a range of values, such as portrait and landscape.

[0041] Deriving meta-features AB from a dataset such as 111-112 depends on the implementation. Examples of meta-feature derivations include counts of items (i.e., examples, samples) in the training dataset, counts of features of items in the training dataset, or statistical moments (e.g., first, second, third, mean, or variance) of the values ​​of features in the training dataset. Other meta-features may have more complex derivations, such as mutual information between a first feature in the training dataset and a classification label or second feature in the training dataset.

[0042] During training, ML model 130 is repeatedly reconfigured and retrained (e.g., simultaneously), each time using a different pairing of the training dataset 111-112 and the configuration 121-122. Because the training performance of ML model 130 depends on which configuration and which dataset, each training run achieves its own (e.g., different) performance metric, displayed as a quality score. The quality score can be accuracy, error, precision, recall, a combination of these, or other metrics indicating how well the machine learning model 130 has been trained.

[0043] Using different scores from different datasets and different hyperparameter values, performance tuple 140 can be generated based on training corpus 170 and configuration 180. When the illustrative header lines shown are ignored, for example, the first data line of corpus 170 and the first data line of configuration 180 are combined with the quality score they jointly achieve, which forms the first data line of performance tuple 140.

[0044] One approach to detecting potential quality scores for new hyperparameter values ​​is to actually configure the ML model 130 with these values ​​and actually measure the quality achieved from training the ML model 130. However, training the ML model 130 can be time-consuming and involve significant power consumption by the central processing unit (CPU). As presented in this paper, time and energy can be saved by alternatively having the regressor 150 predict the quality score of the ML model 130 without actually configuring and training the ML model 130 based on new hyperparameter values. For example, when inference is performed with a new dataset and / or a new hyperparameter configuration, the regressor 150 can predict that the ML model 130 will achieve an estimated quality score 160.

[0045] Computer 100 performs meta-learning because predictive regressor 150 learns by observing the performance of ML model 130. Performance tuple 140 can be used as a training dataset for regressor 150 to learn to predict quality scores for other datasets and / or other hyperparameter configurations. For example, each row of performance tuple 140 can be encoded as a feature vector (e.g., a portion thereof), which regressor 150 can apply to during training.

[0046] In this embodiment, the regressor 150 is a random forest that learns the correlations between quality scores, model hyperparameters, and / or dataset meta-features, or is accompanied by such a random forest. Random forests can naturally provide ML interpretability (MLX) to the model and / or dataset. For example, a random forest can reveal which hyperparameters and / or meta-features are most or least important in influencing the quality score.

[0047] In an embodiment, for a given dataset, computer 100 can discover promising hyperparameter configurations that potentially or actually achieve the best quality score by exploring the configuration hyperspace of ML model 130. For example, after regressor 150 is trained, stochastic descent (not shown) or other multidimensional optimizations can repeatedly invoke regressor 150 using (e.g., increasingly better) different hyperparameter configurations to eventually reach the optimal configuration. For example, open-source tools such as hyperopt can provide a harness for optimization by repeatedly invoking regressor 150. In an embodiment, the best or some of the best (i.e., a fixed number) of predictive (one or more) hyperparameter configurations are used to actually train ML model 130 to validate predictions and / or select empirically optimal configurations.

[0048] The termination (i.e. convergence) criterion for such exploration depends on the implementation. For example, regressor 150 may be called several times to quickly find a good configuration, or it may be called many times to laboriously find a much better configuration. Any quality scores predicted by regressor 150 and / or any hyperparameter configurations proposed by the optimization exploration may or may not be empirically validated subsequently by actually training the ML model 130.

[0049] The following example pseudocode demonstrates example hyperparameter optimization.

[0050]

[0051] 2.0 Superstar

[0052] exist Figure 1B In this study, the same computer 100 used SuperStar to quickly and concisely reveal some outlines of the mysterious configuration of hyperspace. Figure 1B In the various tabular data structures shown, cells that appear empty actually contain real values ​​that are not displayed.

[0053] While a training performance tuple containing quality score, dataset meta-features, and hyperparameter configuration provides good information, it is insufficient to describe the natural range of possible operational performance, such as the possible quality scores for other hyperparameter configurations of the same ML model and / or other datasets. In this paper, superstars are points in the configuration hyperspace that are used as feature points, such as 185, and are introduced to make quality score predictions more flexible and accurate for regressors (e.g., 150).

[0054] The training of ML model 130 can initially use feature point configuration 185, which is a small subset of configuration 180 used during training. Feature point configuration 185 is shown as a table with some empty cells. Each empty cell actually contains a corresponding identifier for the hyperparameter X or Y of the training dataset or ML model 130, where each of the feature points IL has a unique combination of such values.

[0055] The following is an example heuristic for generating individual feature points IL for feature point configuration 185. In this embodiment, feature points(one or more) are generated independently for each hyperparameter XY, while the remaining hyperparameters remain constant. In this embodiment, the remaining hyperparameters are set to their respective default values ​​or the corresponding midpoints of their respective value ranges. In this embodiment, the hyperparameters that are not kept constant are changed according to a heuristic.

[0056] For example, for hyperparameters that are not kept constant, minimum, maximum, midpoint, default value, and / or regular grid spacing may appear in feature points. For instance, if hyperparameter X has an integer range of 4-10, and hyperparameter Y defaults to 5, then some feature point configurations represented as (hyperparameter X, hyperparameter Y) could be (4,5), (4+10 / 2=7,5), and (10,5). Other feature point configurations can alternatively vary Y. Default values ​​can be artificially generated, for example, through experimentation or vendor recommendations.

[0057] Non-feature point configurations can be generated using random values, default values, grid intervals, or values ​​discovered by stateful activities such as gradients, binary search, neighborhood search, or other explorations (e.g., optimization). Here, stateful generation means that subsequent configurations can be (e.g., greedily or randomly) based on previous configurations, such as through interpolation, extrapolation, or refinement (e.g., through random descent).

[0058] Initially, training actually uses feature point configuration 185 for ML model 130 and captures the quality scores of the rightmost column shown as feature point configuration 185. After feature point configuration 185 has quality scores, training of ML model 130 can continue with other hyperparameter configurations and / or datasets, and the generation of performance tuples 140 can begin.

[0059] Performance tuple 140 can be generated as discussed above, along with some additional data padding. Each row (i.e., tuple) of performance tuple 140 is stored in performance tuple 140 as the corresponding quality score for each feature point, displayed as the feature point score IL.

[0060] After training, regressor 150 can make predictions on new datasets (such as 113) and / or new hyperparameter configurations. As shown in the figure, the input tuples used for inference can include meta-features of the new dataset 113, hyperparameters of the ML model 130, and feature point scores. For example, in a production environment, processing of the new production dataset 113 can begin by actually training the ML model 130 with the new dataset 113 and the feature point configuration IL to generate the feature point scores shown in the production input table.

[0061] In this embodiment, these configuration feature points and / or the quality scores of their implementations are encoded as features into feature vectors, which the regressor 150 can use to train or otherwise apply. Thus, a (e.g., very) low-resolution overview of the performance of the configuration hyperspace of a new dataset can be provided to the regressor 150 to improve the accuracy of the regressor's predictions.

[0062] Computer 100 can at least generate feature point configurations IL that have different sets of values ​​and are well (e.g., uniformly and extensively) spaced within a configuration hyperspace of at least hyperparameters XY. The feature point configurations XY should be representative samples that at least provide a glimpse of the overview (i.e., gradients) within the configuration hyperspace. For example, once the performance metric of the feature point configuration IL is known, some extrapolation and / or interpolation may be possible (i.e., somewhat efficient). In an embodiment, a fixed number of feature point configurations are generated. In an embodiment, the number of feature point configurations depends on how many hyperparameters (i.e., dimensions) are involved. Techniques for generating feature point configurations are discussed later in this document.

[0063] The following example pseudocode obtains the meta-dataset for training a predictive regressor.

[0064]

[0065] 3.0 Example Regressor Training Process

[0066] Figure 2 This is a flowchart depicting how effective a computer 100 in training a regressor to predict machine learning (ML) models would be if configured with given hyperparameter values ​​and trained on a given dataset. (Reference) Figures 1A-1B discuss Figure 2 .

[0067] Figure 2 The process involves training both ML models 130 and 150. In steps 201-205, the target ML model 130 is repeatedly trained with various hyperparameter configurations to collect target performance data. Step 206 trains the predictive regressor 150 based on the performance data.

[0068] In this embodiment, step 206 can be pipelined in parallel with other steps. For example, whenever other steps generate individual performance tuples or batches of tuples, these tuples or batches can be processed by step 206, with or without waiting for other steps to generate all performance tuples.

[0069] Repeat steps 201-205 for each training dataset in the training corpus. Step 201 is preparatory and yields the results as described above for... Figures 1A-1B The values ​​of the meta-features of the current training dataset under discussion. In this embodiment, step 201 occurs simultaneously for different training datasets.

[0070] Steps 202-205 are repeated for each of the many different hyperparameter configurations of ML model 130. The actual training of ML model 130 occurs during steps 202-204 using the current dataset and the current hyperparameter configuration. Within hyperparameter configuration 180 is feature point configuration 185, which is first processed by steps 202-204.

[0071] Step 202 configures the ML model 130 with the current hyperparameter configuration, and step 203 trains the ML model 130 with the current dataset. Therefore, step 204 can measure the corresponding quality score, which captures how effective (e.g., accurate) the ML model 130 is with the current hyperparameters and dataset. In an embodiment, steps 202-205 can be repeated simultaneously for each hyperparameter configuration, such as via a separate computer and / or processor core.

[0072] Step 205 generates a performance tuple based on the quality score, the current hyperparameters, the dataset, and the feature point scores discussed above. For example, performance tuple 140 is generated separately from the corresponding individual occurrence of step 205.

[0073] After repeatedly retraining the ML model 130 in steps 202-204, the ML model 130 may be retrained later in more training runs, such as when the predictor regressor 150 is finally (e.g., by hyperopt) used to further explore the configuration hyperspace of the ML model 130 to find more potentially improved hyperparameter configurations and hopefully find a new optimal configuration.

[0074] Step 206 performs meta-learning by training regressor 150 to make performance predictions about ML model 130. As discussed in this paper, feature point scores within performance tuple 140 accelerate the training of regressor 150 and increase the accuracy of regressor 150.

[0075] 4.0 Dual Regressor

[0076] Figure 3This is a block diagram depicting an example computer 300 in an embodiment. Computer 300 uses a dual regressor to adapt (e.g., an unusual) dataset. Computer 300 may be an implementation of computer 100. Figure 3 In the table data structure shown, cells that appear empty actually contain actual values ​​that are not displayed.

[0077] The shared regressor 351 is deployed in production inference mode and can be an implementation of the regressor discussed earlier. Therefore, the shared regressor 351 has been trained on many datasets and is designed to be reused to make predictions on multiple new datasets (such as 310) with or without retraining.

[0078] Because the shared regressor 351's performance training corpus (not shown) contains a large number of items, the statistics may exhibit central trends (e.g., patterns) that contribute to the reinforcement learning of the shared regressor 351. After reinforcement training, the shared regressor 351 is adequately prepared to identify a performance overview of the target ML model (not shown) handling common examples, which may more or less populate the common datasets processed by the target ML model. However, the target ML model's uncommon datasets may (e.g., accidentally) contain many outliers, which could reduce the accuracy of both the target ML model and the shared regressor 351 as follows.

[0079] Uncommon target datasets may require uncommon (e.g., undiscovered) hyperparameter values ​​for the target ML model. Hyperparameter values ​​that previously performed well may perform poorly on unusual target datasets. Therefore, the shared regressor 351 can be trained to make somewhat inaccurate predictions for unusual datasets (such as 310). The computer 300 has the following three strategies for adapting to new datasets (e.g., unusual datasets).

[0080] First, it is best to have a dedicated regressor, such as 352, which is trained only on the new dataset 310 to make specific predictions for dataset 310. For example, after being trained only on dataset 310, the dedicated regressor 352 can (e.g., by hyperopt) be used to find a better target hyperparameter configuration for dataset 310 than the shared regressor 351 can find.

[0081] Secondly, although the dedicated regressor 352 is potentially (i.e. ultimately) better than the shared regressor 351, the dedicated regressor 352 was initially untrained when the new dataset 310 arrived. The shared regressor 351, on the other hand, was fully trained. Therefore, at least initially, it is best to use the shared regressor 351 to make predictions about the new dataset 310.

[0082] In this embodiment, the predictions of regressors 351-352 are confidence-weighted. The shared regressor 351 may initially have moderate or somewhat high confidence. The untrained dedicated regressor 352 may initially have low or no confidence.

[0083] As training of the dedicated regressor 352 progresses, its confidence increases monotonically, such as according to a schedule, based on time, iterations, the performance tuples used so far during training, error, and / or other convergence criteria. In an embodiment, the regressor with the larger weight among regressors 351-352 has a greater influence on the selection of hyperparameter configuration values. In a multidimensional optimization (e.g., hyperopt) embodiment, each of regressors 351-352 ranks the proposed hyperparameter configurations as rank R1 and R2 (not shown). The rankings are then rescaled based on the weights W1 of regressor 351 and W2 of regressor 352. The final ranking of the hyperparameter configuration is R, which is R1*W1 + R2*W2. After rescaling the rankings of all other hyperparameter configurations, the top-ranked configuration is selected. In an embodiment, when the confidence of the dedicated regressor 352 exceeds a threshold or exceeds the confidence of the shared regressor 351 by a certain threshold amount, regressor 351 is no longer used to make predictions for the new dataset 310.

[0084] The third way the computer 300 adapts to the new dataset 310 involves feature point scores MN. Even though the shared regressor 351 has no training experience with the new dataset 310, the feature point scores MN more or less directly provide a rough overview of the performance of the new dataset 310. Therefore, the feature point scores MN improves the accuracy of regressors 351-352 and accelerates the training of the dedicated regressor 352.

[0085] In the illustrated embodiment, regressors 351 and 352 reuse the same feature point configuration to achieve different feature point scores. In embodiments not shown, the feature point configuration used to train regressor 352 may include a fixed number of optimal (e.g., non-feature point) configurations used for regressor 351.

[0086] The result of the dedicated regressor 352 for the new dataset 310 is that the value of the meta-feature EF is constant throughout the lifetime of the dedicated regressor 352 (i.e., training and inference), which means that the meta-feature EF has no effect on the operation of the dedicated regressor 352. Therefore, all meta-feature EFs are irrelevant to the dedicated regressor 352 and never need to be provided to the dedicated regressor 352.

[0087] In the illustrated embodiment, the dashed arrows pointing to regressors 351-352 indicate which subset (i.e., columns) of the performance characteristics are provided to which regressor. As shown, the shared regressor 351 expects all columns. As shown, the meta-feature EF is not provided to the dedicated regressor 352.

[0088] In the illustrated embodiment, the feature point scores MN are not provided to the dedicated regressor 352. Alternatively, and although not shown, the feature point scores may be interchanged for the dedicated regressor 352 such that the feature points are provided to the dedicated regressor 352 as separate performance tuples with corresponding feature point scores stored in the quality score column.

[0089] In those ways, and as shown in the figure, the dedicated regressor 352 can be applied to a performance feature vector that is narrower (i.e., fewer bytes) than the performance feature vector of the shared regressor 352.

[0090] The example pseudocode below implements meta-learning using a dedicated regressor R2, which is accompanied by a pre-trained general regressor R1.

[0091]

[0092] Hardware Overview

[0093] According to one embodiment, the techniques described herein are implemented by one or more dedicated computing devices. The dedicated computing device may be hardwired to execute these techniques, or may include digital electronic devices persistently programmed to execute the techniques, such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs), or may include one or more general-purpose hardware processors programmed to execute the techniques according to program instructions in firmware, memory, other storage devices, or combinations thereof. Such a dedicated computing device may also combine custom hardwired logic, ASICs, or FPGAs with custom programming to implement the techniques. The dedicated computing device may be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device incorporating hardwired and / or program logic to implement the techniques.

[0094] For example, Figure 4 This is a block diagram illustrating a computer system 400 on which embodiments of the present invention may be implemented. The computer system 400 includes a bus 402 or other communication mechanism for transmitting information, and a hardware processor 404 coupled to the bus 402 for processing information. The hardware processor 404 may be, for example, a general-purpose microprocessor.

[0095] Computer system 400 also includes main memory 406, such as random access memory (RAM) or other dynamic storage device, coupled to bus 402, for storing instructions and information to be executed by processor 404. Main memory 406 can also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 404. When such instructions are stored in a non-transitory storage medium accessible to processor 404, computer system 400 becomes a dedicated machine customized to perform the operations specified in the instructions.

[0096] Computer system 400 also includes a read-only memory (ROM) 408 or other static storage device coupled to bus 402 for storing static information and instructions for processor 404. Storage device 410, such as a disk, optical disk, or solid-state drive, is provided and coupled to bus 402 for storing information and instructions.

[0097] Computer system 400 may be coupled to display 412, such as a cathode ray tube (CRT), via bus 402 for displaying information to the computer user. Input device 414, including alphanumeric keys and other keys, is coupled to bus 402 for transmitting information and command selections to processor 404. Another type of user input device is cursor control 416, such as a mouse, trackball, or cursor arrow keys for transmitting directional information and command selections to processor 404 and for controlling cursor movement on display 412. This input device typically has two degrees of freedom in two axes (a first axis (e.g., x) and a second axis (e.g., y)), allowing the device to specify a position in a plane.

[0098] Computer system 400 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic. This custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic, combined with the computer system, enable or program the computer system 400 to be a dedicated machine. According to one embodiment, the techniques described herein are executed by computer system 400 in response to processor 404 executing one or more sequences of one or more instructions contained in main memory 406. Such instructions may be read into main memory 406 from another storage medium, such as storage device 410. Execution of the sequence of instructions contained in main memory 406 causes processor 404 to perform the processing steps described herein. In alternative embodiments, hardwired circuitry may be used instead of or in combination with software instructions.

[0099] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that enable a machine to operate in a specific manner. Such storage media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs, magnetic disks, or solid-state drives, such as storage device 410. Volatile media include dynamic memory, such as main memory 406. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROM, EPROM, FLASH-EPROM, NVRAM, any other memory chips, or magnetic tape cassettes.

[0100] Storage media differ from transmission media, but can be used in conjunction with them. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including wires containing bus 402. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0101] Various forms of media may be involved when carrying one or more sequences of one or more instructions to processor 404 for execution. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit them over a telephone line using a modem. A modem local to computer system 400 may receive data over the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal, and appropriate circuitry may place the data on bus 402. Bus 402 carries the data to main memory 406, from which processor 404 retrieves and executes the instructions. The instructions received in main memory 406 may optionally be stored on storage device 410 before or after execution by processor 404.

[0102] Computer system 400 also includes a communication interface 418 coupled to bus 402. Communication interface 418 provides bidirectional data communication coupling to a network link 420 connected to a local network 422. For example, communication interface 418 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem providing a data communication connection to a corresponding type of telephone line. As another example, communication interface 418 may be a local area network (LAN) card providing a data communication connection to a LAN-compatible network. A wireless link may also be implemented. In any such implementation, communication interface 418 transmits and receives electrical, electromagnetic, or optical signals carrying digital data streams representing various types of information.

[0103] Network link 420 typically provides data communication to other data devices via one or more networks. For example, network link 420 may provide a connection via local network 422 to host 424 or to a data device operated by an Internet Service Provider (ISP) 426. ISP 426 then provides data communication services via a global packet data communication network now commonly referred to as the “Internet” 428. Both local network 422 and Internet 428 use electrical, electromagnetic, or optical signals carrying digital data streams. Signals through various networks and on network link 420 and through communication interface 418 (which carry digital data to and from computer system 400) are example forms of transmission media.

[0104] Computer system 400 can send messages and receive data, including program code, through one or more networks, network links 420, and communication interfaces 418. In the Internet example, server 430 can send application request codes through the Internet 428, ISP 426, local network 422, and communication interface 418.

[0105] The received code may be executed by processor 404 upon receipt and / or stored in storage device 410 or other non-volatile storage device for later execution.

[0106] Software Overview

[0107] Figure 5 This is a block diagram of a basic software system 500 that can be used to control the operation of computing system 400. Software system 500 and its components, including their connections, relationships, and functions, are intended to be exemplary only and are not intended to limit the implementation of one or more example embodiments. Other software systems suitable for implementing one or more example embodiments may have different components, including components with different connections, relationships, and functions.

[0108] Software system 500 is provided to guide the operation of computing system 400. Software system 500, which may be stored on system memory (RAM) 406 and fixed storage devices (e.g., hard disk or flash memory) 410, includes a kernel or operating system (OS) 510.

[0109] OS 510 manages low-level aspects of computer operations, including managing process execution, memory allocation, file input and output (I / O), and device I / O. One or more applications, designated 502A, 502B, 502C…502N, can be “loaded” (e.g., transferred from fixed storage device 410 to memory 406) for execution by system 500. Applications or other software intended for use on computer system 400 can also be stored as a set of downloadable computer-executable instructions, for example, for downloading and installing from Internet locations (e.g., web servers, app stores, or other online services).

[0110] Software system 500 includes a graphical user interface (GUI) 515 for receiving user commands and data graphically (e.g., "click" or "touch gestures"). These inputs can then be acted upon by system 500 according to instructions from operating system 510 and / or (one or more) applications 502. GUI 515 is also used to display the results of operations from OS 510 and (one or more) applications 502, to which the user can provide additional input or terminate the session (e.g., log off).

[0111] OS 510 can be executed directly on the bare hardware 520 of computer system 400 (e.g., one or more processors 404). Alternatively, a hypervisor or virtual machine monitor (VMM) 530 can be inserted between the bare hardware 520 and OS 510. In this configuration, VMM 530 acts as a software “buffer” or virtualization layer between OS 510 and the bare hardware 520 of computer system 400.

[0112] VMM 530 instantiates and runs one or more virtual machine instances (“guest machines”). Each guest machine includes a “guest” operating system (such as OS 510) and one or more applications (such as application(s)502) designed to run on the guest operating system. VMM 530 presents a virtual operating platform to the guest operating system and manages the execution of the guest operating system.

[0113] In some cases, VMM 530 can allow a guest operating system to run as if it were running directly on the bare hardware 520 of computer system 500. In these cases, the same version of the guest operating system configured to run directly on the bare hardware 520 can also run on VMM 530 without modification or reconfiguration. In other words, in some cases, VMM 530 can provide complete hardware and CPU virtualization to the guest operating system.

[0114] In other cases, for efficiency, the guest operating system can be specifically designed or configured to run on the VMM 530. In these cases, the guest operating system "knows" that it is running on the virtual machine monitor. In other words, in some situations, the VMM 530 can provide paravirtualization for the guest operating system.

[0115] A computer system process involves the allocation of hardware processor time and memory (physical and / or virtual). Memory allocation is used to store instructions executed by the hardware processor, data generated by the execution of those instructions, and / or to store hardware processor state (e.g., register contents) during the time allocation when the computer system process is not running. Computer system processes run under the control of the operating system and can also run under the control of other programs executing on the computer system.

[0116] cloud computing

[0117] The term “cloud computing” is generally used in this article to describe a computing model that enables on-demand access to a shared pool of computing resources, such as computer networks, servers, software applications and services, and allows for the rapid provisioning and release of resources with minimal management effort or service provider interaction.

[0118] Cloud computing environments (sometimes referred to as cloud environments or the cloud itself) can be implemented in a wide variety of ways to best suit different needs. For example, in a public cloud environment, the underlying computing infrastructure is owned by the organization that makes its cloud services available to other organizations or the general public. In contrast, private cloud environments are typically designed for use by a single organization or within a single organization. Community clouds are designed to be shared by several organizations within a community; while hybrid clouds include two or more types of clouds (e.g., private, community, or public) bound together by data and application portability.

[0119] Typically, cloud computing models deliver some of the responsibilities that might have previously been provided by an organization's own IT departments as service layers within the cloud environment for use by consumers (whether internal or external to the organization, depending on the public / private nature of the cloud). Depending on the specific implementation, the precise definition of the components or features provided by each cloud service layer or within each cloud service layer can vary, but common examples include: Software as a Service (SaaS), where consumers use software applications running on cloud infrastructure, while the SaaS provider manages or controls the underlying cloud infrastructure and applications; Platform as a Service (PaaS), where consumers can use software programming languages ​​and development tools supported by the PaaS provider to develop, deploy, and otherwise control their own applications, while the PaaS provider manages or controls other aspects of the cloud environment (i.e., everything in the runtime execution environment); and Infrastructure as a Service (IaaS), where consumers can deploy and run arbitrary software applications and / or provide processing, storage, networking, and other basic computing resources, while the IaaS provider manages or controls the underlying physical cloud infrastructure (i.e., everything below the operating system layer). Database as a Service (DBaaS) is a service where consumers use database servers or database management systems that run on cloud infrastructure, while the DBaaS provider manages or controls the underlying cloud infrastructure and applications.

[0120] The basic computer hardware and software, as well as the cloud computing environment described above, are presented to illustrate the basic underlying computer components that can be used to implement one or more example embodiments. However, the one or more example embodiments are not necessarily limited to any particular computing environment or computing device configuration. Rather, the one or more example embodiments can be implemented in any type of system architecture or processing environment that a person skilled in the art will understand from this disclosure as capable of supporting the features and functionality of the one or more example embodiments presented herein.

[0121] Machine learning models

[0122] Machine learning models are trained using specific machine learning algorithms. Once trained, the input is applied to the machine learning model to make predictions, which in this paper may also be referred to as the predicted output or output. The attributes of the input may be called features, and the values ​​of the features may be referred to as feature values ​​in this paper.

[0123] A machine learning model comprises a model data representation or a model artifact. The model artifact includes parameter values, which will be referred to as theta (θ) values ​​in this paper, and is applied to the input by the machine learning algorithm to generate a predicted output. Training a machine learning model requires determining the theta values ​​of the model artifact. The structure and organization of the theta values ​​depend on the machine learning algorithm.

[0124] In supervised training, training data is used by a supervised training algorithm to train a machine learning model. The training data includes inputs and "known" outputs. In this embodiment, the supervised training algorithm is an iterative process. In each iteration, the machine learning algorithm applies the model artifact and the inputs to generate a predicted output. The error or variance between the predicted output and the known output is calculated using an objective function. In practice, the output of the objective function indicates the accuracy of the machine learning model based on the specific state of the model artifact during the iteration. The θ value of the model artifact is adjusted by applying an optimization algorithm based on the objective function. An example of an optimization algorithm is gradient descent. The iterations can be repeated until the desired accuracy is achieved or some other criterion is met.

[0125] In software implementation, when a machine learning model is referred to as receiving input, executing, and / or generating output or predictions, the computer system process executing the machine learning algorithm applies the model artifact to the input to generate the predicted output. The computer system process executes the machine learning algorithm by executing software configured to cause the algorithm to execute.

[0126] Machine learning (ML) excels at problems including clustering, classification, regression, anomaly detection, prediction, and dimensionality reduction (i.e., simplification). Examples of machine learning algorithms include decision trees, support vector machines (SVM), Bayesian networks, stochastic algorithms such as genetic algorithms (GA), and connectionist topologies such as artificial neural networks (ANN). Implementations of machine learning may rely on matrices, symbolic models, and hierarchical and / or associative data structures. Parameterized (i.e., configurable) implementations of best-in-class machine learning algorithms can be found in open-source libraries such as Google's TensorFlow for Python and C++, or Georgia Tech's MLPack for C++. Shogun is an open-source C++ ML library with adapters for several programming languages, including C#, Ruby, Lua, Java, MatLab, R, and Python.

[0127] Artificial Neural Networks

[0128] Artificial neural networks (ANNs) are machine learning models that model systems of neurons interconnected by directed edges at a high level. An overview of neural networks is described within the context of hierarchical feedforward neural networks. Other types of neural networks share the characteristics of the neural networks described below.

[0129] In hierarchical feedforward networks (such as multilayer perceptrons (MLPs)), each layer contains a set of neurons. Hierarchical neural networks consist of an input layer, an output layer, and one or more intermediate layers called hidden layers.

[0130] Neurons in the input and output layers are referred to as input neurons and output neurons, respectively. Neurons in the hidden or output layers may be referred to as activating neurons in this paper. Activating neurons are associated with activation functions. The input layer does not contain any activating neurons.

[0131] Starting from each neuron in the input and hidden layers, there may be one or more directed edges leading to activated neurons in subsequent hidden or output layers. Each edge is associated with a weight. The edge from a neuron to an activated neuron represents the input from the neuron to the activated neuron, adjusted by the weights.

[0132] For a given input to a neural network, each neuron in the network has an activation value. For an input neuron, the activation value is simply the input value to the input. For an activating neuron, the activation value is the output of the corresponding activation function.

[0133] Each edge from a specific neuron to an activating neuron represents the activation value of the specific neuron as the input to the activating neuron, i.e., the input to the activation function of the activating neuron, adjusted by the edge weights. Therefore, activating neurons in subsequent layers represent the activation values ​​of specific neurons as the input to the activation function of the activating neuron, adjusted by the edge weights. An activating neuron can have multiple edges pointing to it, each edge representing the activation value from the originating neuron as the input to the activation function of the activating neuron, adjusted by the edge weights.

[0134] Each activated neuron is associated with a bias. To generate the activation value of an activated neuron, an activation function is applied to the neuron, weighted by the activation value and the bias.

[0135] Illustrative data structures for neural networks

[0136] The artifacts in a neural network may include weight and bias matrices. Training a neural network involves iteratively adjusting the weight and bias matrices.

[0137] For hierarchical feedforward networks and other types of neural networks, the artifact may include one or more edge matrices W. Matrix W represents the edge from layer L-1 to layer L. Assuming that the number of neurons in layer L-1 and layer L are N[L-1] and N[L] respectively, then the dimension of matrix W is N[L-1] columns and N[L] rows.

[0138] The deviation of a specific layer L can also be stored in a matrix B with one column and N[L] rows.

[0139] Matrices W and B can be stored in RAM as vectors or arrays, or as comma-separated sets of values. When the artifact is persistently stored in persistent storage, matrices W and B can be stored as comma-separated values ​​in compressed and / or serialized form, or other suitable persistent form.

[0140] The specific inputs applied to a neural network include the value of each input neuron. A specific input can be stored as a vector. Training data includes multiple inputs, each referred to as a sample in a set of samples. Each sample includes the value of each input neuron. Samples can be stored as vectors of input values, while multiple samples can be stored as a matrix, where each row of the matrix represents a sample.

[0141] When input is applied to a neural network, activation values ​​are generated for the hidden and output layers. For each layer, the activation values ​​can be stored in a column of matrix A, which has a row for each neuron in that layer. In a vectorized approach used for training, the activation values ​​can be stored in a matrix with a column for each sample in the training data.

[0142] Training a neural network requires storing and processing additional matrices. The optimization algorithm generates a derivative matrix of the matrices used to adjust the weights W and biases B. Generating the derivative may use, and requires storing, an intermediate value matrix generated when calculating the activation values ​​for each layer.

[0143] The number of neurons and / or edges determines the size of the matrix required to implement a neural network. Fewer neurons and edges in a neural network result in a smaller matrix and a smaller amount of memory required to store it. Furthermore, a smaller number of neurons and edges reduces the computational cost required to apply or train the neural network. Fewer neurons mean fewer activation values ​​need to be computed, and / or fewer derivative values ​​need to be computed during training.

[0144] The properties of the matrix used to implement the neural network correspond to neurons and edges. A cell in matrix W represents a specific edge from layer L-1 to layer L. An activated neuron represents the activation function of the layer, including the activation function. An activated neuron in layer L corresponds to a row of weights for the edge between layers L and L-1 in matrix W, and a column of weights for the edge between layers L and L+1 in matrix W. During the execution of the neural network, neurons also correspond to one or more activation values ​​stored in matrix A for the layers and generated by the activation function.

[0145] ANNs undergo vectorization with data parallelism, leveraging vector hardware such as Single Instruction Multiple Data (SIMD) and graphics processing units (GPUs). Matrix partitioning allows for horizontal scaling, such as using Symmetric Multiprocessing (SMP), multi-core CPUs, and / or multiple coprocessors like GPUs. Feedforward computation within an ANN may occur one step per neural layer. Activations in a layer are computed based on weighted propagation of activations from previous layers, allowing values ​​to be computed sequentially for each subsequent layer, such as using corresponding iterations of a for loop. Layering imposes a non-parallelizable computational order. Therefore, network depth (i.e., the number of layers) can lead to computational latency. Deep learning requires endowing multilayer perceptrons (MLPs) with many layers. Each layer implements data abstractions, where complex abstractions (i.e., multidimensionality with multiple inputs) require multiple layers to implement cascaded processing. Matrix operations based on reusable matrices in ANN implementations and feedforward processing are readily available and parallelizable in neural network libraries such as Google's TensorFlow for Python and C++, OpenNN for C++, and the University of Copenhagen's Fast Artificial Neural Networks (FANN). These libraries also provide model training algorithms, such as backpropagation.

[0146] Backpropagation

[0147] The output of an ANN may be more or less correct. For example, an ANN that recognizes letters might mistake an "I" for an "L" because those letters have similar features. The correct output may have one or more specific values, while the actual output may have some different values. The arithmetic or geometric difference between the correct and actual outputs can be measured as error according to a loss function, such that zero represents error-free (i.e., perfectly accurate) behavior. For any edge in any layer, the difference between the correct and actual outputs is the incremental value.

[0148] Backpropagation involves distributing the error backward through the layers of the ANN to all connections within the ANN. This error propagation leads to adjustments in the edge weights, depending on the error gradient of each edge. The edge gradient is calculated by multiplying the edge's error increment by the activation value of the upstream neuron. When the gradient is negative, the edge contributes a larger amount of error to the network, and its weight should be reduced more significantly—this is negative reinforcement. When the gradient is positive, positive reinforcement increases the weight of the edge whose activation has reduced the error. The edge weights are adjusted based on the percentage of the edge's gradient. The steeper the gradient, the larger the adjustment. Not all edge weights are adjusted by the same amount. As the model continues training with additional input samples, the ANN's error should decrease. Training can stop when the error stabilizes (i.e., stops decreasing) or disappears below a threshold (i.e., approaches zero). Example mathematical formulas and techniques (including matrix operations and backpropagation) for feedforward multilayer perceptrons (MLPs) are taught in Christopher M. Bishop’s relevant reference, “Exact Computation of the Hessian Matrix for the Multilayer Perceptron”.

[0149] Model training can be supervised or unsupervised. For supervised training, the expected (i.e., correct) output is known for each example in the training set. The training set is pre-configured by assigning classification labels to each example (e.g., by a human expert). For example, the training set for optical character recognition might consist of blurry photographs of individual letters, and an expert could pre-label each photograph based on the displayed letter. Error calculation and backpropagation occur as explained above.

[0150] Unsupervised model training is more prevalent because the desired output needs to be discovered during training. Unsupervised training is often easier to implement because human experts don't need to pre-label training examples. Therefore, unsupervised training saves manpower. A natural way to achieve unsupervised training is to use an autoencoder, which is a type of ANN. An autoencoder acts as an encoder / decoder with two sets of layers. The first set of layers encodes the input examples into compressed code that needs to be learned during model training. The second set of layers decodes the compressed code to regenerate the original input examples. The two sets of layers are trained together as a combined ANN. The error is defined as the difference between the original input and the regenerated input, such as the decoded version. After sufficient training, the decoder will more or less accurately output whatever was originally input.

[0151] Autoencoders rely on compressed codes as an intermediate format for each input example. These intermediate compressed codes do not exist initially but emerge only during model training, which may be counterintuitive. Unsupervised training can achieve an intermediate encoded vocabulary based on features and distinctions with unexpected relevance. For example, which examples and labels are used during supervised training may depend on some unscientific (e.g., anecdotal) or otherwise incomplete understanding of the problem space by human experts. However, unsupervised training discovers a suitable intermediate vocabulary that is more or less entirely based on statistical trends that reliably converge to optimality with sufficient training due to internal feedback from the regenerated decoding. Autoencoder implementation and integration techniques are taught in the related U.S. patent application No. 14 / 558,700, entitled "Autoencoder Enhanced Self-Diagnostic Components for Model Monitoring". This patent application elevates supervised or unsupervised ANN models to a first class of objects that are subject to management techniques, such as monitoring and governance during model development (e.g., during training).

[0152] Random Forest

[0153] Random forests, or random decision forests, are a general learning method that constructs a set of randomly generated nodes and decision trees during the training phase. Different forest decision trees are constructed, each randomly restricted to a specific subset of the dataset's feature dimensions, such as using feature-guided aggregation (bagging). Therefore, the decision trees gain accuracy as they grow, without being forced to overfit the training data, as would happen when the decision trees are forced to learn all feature dimensions of the dataset. Predictions can be computed based on the mean (or other integrals such as soft max) of predictions from different decision trees.

[0154] Random forest hyperparameters may include: the number of trees in the forest, the maximum number of features considered in split nodes, the number of levels in each decision tree, the minimum number of data points in a leaf node, and the method of sampling data points.

[0155] In the foregoing description, embodiments of the invention have been described with reference to numerous specific details that may change with implementation. Therefore, the description and drawings are to be considered illustrative rather than restrictive. The unique and exclusive references to the scope of the invention, and what the applicant intends to be within the scope of the invention, are the literal and equivalent scope of the set of claims published from this application in the specific form of the claims, including any subsequent corrections.

Claims

1. A computer-implemented method comprising: for each training data set of a plurality of training data sets, wherein the plurality of training data sets comprises photographs: a) deriving, from the training data set, a plurality of values of a plurality of dataset meta-features; b) generating, for a machine learning, ML, model having a plurality of hyperparameters: i) a plurality of feature point hyperparameter configurations comprising a first feature point hyperparameter configuration and a second feature point hyperparameter configuration, and ii) a plurality of different hyperparameter configurations comprising the plurality of feature point hyperparameter configurations, wherein each hyperparameter configuration of the plurality of different hyperparameter configurations contains a value for each hyperparameter of the plurality of hyperparameters; and c) for each hyperparameter configuration of the plurality of different hyperparameter configurations: i) configuring the ML model with the hyperparameter configuration; ii) training the ML model when configured with the hyperparameter configuration based on the training data set; iii) obtaining an empirical quality score indicative of how effective the training of the ML model when configured with the hyperparameter configuration was; and iv) generating a performance tuple of a plurality of performance tuples, wherein the performance tuple contains: the hyperparameter configuration, the plurality of values of the plurality of dataset meta-features, and the empirical quality score; encoding each performance tuple of the plurality of performance tuples into a respective feature vector, the respective feature vector containing: an empirical quality score of the first feature point hyperparameter configuration, and an empirical quality score of the second feature point hyperparameter configuration; training, based on the feature vectors of the plurality of performance tuples, a regressor to predict, based on a given data set and a given hyperparameter configuration, an estimated quality score; deriving, from a new data set, a new plurality of values of the plurality of dataset meta-features; for each hyperparameter configuration of a second plurality of different hyperparameter configurations, predicting, by the regressor, a new estimated quality score of a first plurality of estimated quality scores, wherein the new estimated quality score is based on: the hyperparameter configuration and the new plurality of values of the plurality of dataset meta-features; selecting a particular hyperparameter configuration of the second plurality of different hyperparameter configurations having a highest estimated quality score of the first plurality of estimated quality scores; and training the ML model based on the particular hyperparameter configuration and the new data set.

2. The method of claim 1, further comprising: making, by the ML model, an inference based on the particular hyperparameter configuration and the new data set.

3. The method of claim 1, wherein the second plurality of different hyperparameter configurations comprises the plurality of feature point hyperparameter configurations.

4. The method of claim 1, wherein: selecting the particular hyperparameter configuration comprises selecting a fixed number of hyperparameter configurations having a highest estimated quality score of the first plurality of estimated quality scores; the fixed number of hyperparameter configurations is greater than one; and training the ML model based on the particular hyperparameter configuration comprises, for each hyperparameter configuration of the fixed number of hyperparameter configurations, training the ML model based on the hyperparameter configuration. ​ 5. The method of claim 4, further comprising, for each hyperparameter configuration in the second plurality of different hyperparameter configurations: training a second regressor to predict a second estimated quality score based on the fixed number of hyperparameter configurations, and predicting, by the second regressor, a second estimated quality score in the second plurality of estimated quality scores.

6. The method of claim 5, further comprising: computing a weighted average of the highest estimated quality score in the first plurality of estimated quality scores and the second estimated quality score in the second plurality of estimated quality scores with adjustable weights that monotonically increase in favor of the second estimated quality score in the second plurality of estimated quality scores.

7. The method of claim 1, wherein at least one configuration of the plurality of different hyperparameter configurations comprises: a respective randomly generated value and / or a default value for each hyperparameter in the one or more hyperparameters.

8. The method of claim 1, wherein at least one configuration in the plurality of different hyperparameter configurations is generated during a random descent.

9. The method of claim 1, wherein the plurality of dataset meta-features include, for a training dataset in the plurality of training datasets, at least one selected from the group consisting of: a count of features of examples in the training dataset, a second or third statistical moment of values of features in the training dataset, mutual information between a first feature and a label in the training dataset, and mutual information between a first feature in the training dataset and a second feature in the training dataset.

10. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause performance of the steps of the method of any of claims 1-9.

11. One or more computers configured to perform the steps of the method of any of claims 1-9.

Images