Systems and methods for automated machine learning

Abstract

To provide techniques for automatically generating machine learning models. [Solution] In one embodiment, a set of hyperparameters corresponding to a machine learning model trained on one training dataset is provided as input. The hyperparameters are iteratively selected using an algorithm such as a bandit algorithm and used to train an ML model using another training dataset. The performance of the trained ML model is evaluated for each iteration until the ML model performance exceeds a threshold. The resulting model can be used to train the resulting model. In some embodiments, the ML models are combined over iterations to improve performance.

Description

[Technical Field]

[0001] This disclosure relates in general to machine learning, and more specifically to systems and methods for automated machine learning. [Background technology]

[0002] Automated Machine Learning (AutoML) is fundamentally changing the field of machine learning by automating the process of selecting and optimizing algorithms and hyperparameters. Generally speaking, machine learning requires the use of datasets to train machine learning models and perform machine learning tasks.

[0003] Hyperparameter optimization, a key element of AutoML, requires fine-tuning various parameters that control the learning process of machine learning models. This is generally a resource-intensive task, requiring the training of numerous models to determine the most effective hyperparameters, and therefore consuming considerable computational resources. [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] Therefore, automating machine learning involves the technical challenge of automatically determining the hyperparameters and corresponding models used during the training process. [Means for solving the problem]

[0005] This disclosure relates to techniques that provide a technical solution to this problem. [Brief explanation of the drawing]

[0006] [Figure 1] This figure shows a system for performing automated machine learning according to one embodiment. [Figure 2]This figure shows a method for automated machine learning according to one embodiment. [Figure 3A] This figure shows a method for generating a set of hyperparameters according to one embodiment. [Figure 3B] This figure shows a system for performing automated machine learning according to another embodiment. [Figure 3C] This figure shows an example of generating a set of hyperparameters according to one embodiment. [Figure 3D] This figure shows boosting according to one embodiment. [Figure 3E] This is a diagram showing a decision tree according to one embodiment. [Figure 4A] This is a diagram illustrating an exemplary boost algorithm. [Figure 4B] This diagram illustrates an example of an adversarial bandit algorithm. [Figure 5] This figure shows exemplary hardware for a dedicated computing system configured in accordance with the above disclosure. [Modes for carrying out the invention]

[0007] This specification describes techniques for implementing automated machine learning. In the following description, numerous examples and specific details are provided to give a thorough understanding of several embodiments. The various embodiments defined by the claims may include, alone or in combination with other features described below, some or all of the features of these examples, and may further include modifications or equivalents of the features and concepts described herein.

[0008] Figure 1 shows a computer system 100 configured to perform an automated machine learning ("AutoML") according to one embodiment. Features and advantages of this disclosure include hyperparameter optimization that does not require training multiple models. This technique can offer significant advantages in some computing environments, particularly in terms of computational cost. It can be particularly beneficial in low-resource environments, for example, where the computational demands of traditional hyperparameter optimization methods may be impractical. As shown in Figure 1, the computer system 100 may run an AutoML software system 101, which may include, for example, one or more software components. The computer system 100 receives a set of hyperparameters 110, which may include, for example, a set of hyperparameters 111a to 111x. The set of hyperparameters generally includes one or more parameter values ​​for constructing a machine learning model during training. The set of hyperparameters may also be simply referred to as "hyperparameters" in this specification. The hyperparameters 111a to x may be associated with machine learning ("ML") models 102a to n. ML102a~n may be pre-trained using, for example, the training dataset 103 and have the same machine learning model type (e.g., a tree model). Hyperparameters 111a~x may be determined during the training of ML102a~n. In some embodiments described in more detail below, the hyperparameters 111a~x may be generated according to an algorithm that produces hyperparameters that are particularly efficient (or optimized according to the algorithm) for training ML102a~n using, for example, the training dataset 103. Thus, the hyperparameters 111a~x are configured to train the corresponding multiple machine learning models 102a~n using the training dataset 103.

[0009] Generally, hyperparameters are external constructs used by data scientists to manage the training of machine learning models. Sometimes called model hyperparameters, hyperparameters are manually set before the model is trained. In contrast, the more common "parameters" in machine learning are variables learned from the data during the training process. These parameters are used, for example, to represent fundamental relationships in the data and to make predictions for new data. On the other hand, hyperparameters are variables that are set before the training process begins. For example, in a random forest tree, the hyperparameter is the number of trees (e.g., n_estimators), which is the number of decision trees that will be created in the forest. Hyperparameters in a decision tree include, for example, criteria for splitting (e.g., Gini impurity or entropy), maximum depth, minimum sample per leaf unit, and the number of features considered for splitting. The set of hyperparameters used herein is one or more parameters of a machine learning model set before it is trained to constitute several aspects of the model behavior as it is trained.

[0010] As described above, one technical challenge in automated machine learning ("AutoML") is determining an effective set of hyperparameters to use when applying a training dataset to a particular machine learning model. Embodiments of this disclosure include solutions to this technical problem. For example, in some cases, it may be desirable to receive new training data 115 that is not included in the training data 103, and to generate a machine learning model using hyperparameters optimized around that new training data. Here, the training dataset 115 is received by the AutoML software system 101, and the new machine learning model 140 is automatically generated using hyperparameters 111a~x. The system in Figure 1 may perform several iterations as follows: Firstly, a selection algorithm software component 112 selects one of several sets of hyperparameters 111a~x. In some exemplary embodiments, as will be discussed in more detail below, the selection algorithm may be a "bandit" algorithm, such as an adversarial bandit algorithm, for selecting a particular set of hyperparameters 111a~x for each iteration. Next, the machine learning training software component ("ML training") 113 receives a machine learning model (ML) 114 and a new training dataset 115. ML114 can be an initial machine learning model of the same machine learning model type as, for example, ML102a~n, such that the hyperparameters 111a~x are relevant to the execution of ML114. The ML training block 113 trains ML114 using the training dataset 115 and the hyperparameters selected by the selection algorithm 112. The model coupler software component 120 couples the trained ML114 with a pre-trained machine learning model over multiple iterations. For example, for the first iteration, ML114 is trained and is the first trained machine learning model, so no coupling occurs.For the next iteration, the trained machine learning model (using different hyperparameters, as discussed) is trained and coupled with a pre-trained machine learning model (e.g., ML121a). For subsequent iterations, the model trained from ML training block 113 (e.g., using different hyperparameters 111a-x) is coupled with, for example, additional trained models ML121a~m.

[0011] Next, the output of the model coupler 120 is provided to the performance analyzer software component 130. The performance analyzer 130 determines the combined performance of the combined machine learning models, such as one or more machine learning models 121a~m, which are, for example, the most recently trained ML and pre-trained over previous iterations. The performance analyzer 130 may determine whether the change in the combined performance of the machine learning models meets a threshold (e.g., ΔP>Th). If the change in the combined performance of the trained models does not meet the threshold, the selection algorithm 112 is modified based on the change in combined performance (e.g., between the current performance for the current iteration and one or more previous performance iterations).

[0012] As described above, the selection algorithm 112 selects different hyperparameters for different iterations. The selection algorithm 112 may select different hyperparameters for each iteration based, for example, on modifications to the selection algorithm parameters across each iteration. Thus, the system iteratively selects hyperparameters 111a~x using the dynamic selection algorithm to train a machine learning model and form a combined machine learning model (including, for example, ML121a~m).

[0013] Ultimately, a trained result model 140 is produced. The result model 140 may be produced, for example, after reaching a stopping condition or after reaching a maximum number of iterations. For example, the result model 140 may be produced when the changes in the combined performance meet (or are equivalent to) a threshold. In some embodiments, the result model 140 may include multiple combined machine learning models that have been pre-trained over multiple iterations.

[0014] Figure 2 shows a method for automated machine learning according to one embodiment. In 201, a set of hyperparameters is received. The set of hyperparameters is configured to train multiple corresponding machine learning models using multiple training datasets. The machine learning models have the same machine learning model type. In 202, a new training dataset is received that is not included in the multiple training datasets. In 203, the method starts iterating over multiple iterations. In 204, one of the multiple hyperparameter sets is selected based on a first selection algorithm. In 205, a first machine learning model is trained using the first training dataset and the selected set of hyperparameters. The first machine learning model has the same machine learning model type as the multiple machine learning models using the hyperparameters from 201. In 206, the first machine learning model is combined with one or more machine learning models that have been pre-trained over multiple iterations. In 207, the combined performance is determined. The combined performance relates to the combined first machine learning model and one or more machine learning models that have been pre-trained over multiple iterations. At step 208, the method determines whether another iteration should be performed (e.g., whether the changes in the combined performance meet a threshold). For example, if the changes in the combined performance do not meet the threshold, the selection algorithm is modified at 209 (e.g., based on the changes in the combined performance), and the method returns to steps 203-207, where new hyperparameters are selected, the ML is trained, the ML is combined, and the combined performance is determined, based on the modified selection algorithm. However, when iterations are performed (e.g., when the changes in the combined performance meet a threshold), an resulting model is produced. The resulting model includes the first machine learning model and one or more machine learning models that have been pre-trained over multiple iterations. The trained resulting model is output at 210.

[0015] Figure 3A shows a method for generating a set of hyperparameters according to one embodiment. In some embodiments, the set of hyperparameters used to generate the machine learning model described above may be generated as follows: In 303, a set of candidate hyperparameters is determined. For example, in some embodiments, a state-of-the-art hyperparameter optimization algorithm, such as the Tree-structured Parzen Estimator method available in the Optuna library, may be used to generate the initial candidate set. This process requires a large hyperparameter search space and cross-validation for model performance evaluation. The time budget for such computations can be considerable (e.g., 10 hours per experiment). Due to the computational density of this process for generating the set of hyperparameters, this process may be performed offline, for example. This step is performed with each corresponding task T={t1,…,t N The candidate configuration C = {c1, ..., c} represents the optimal hyperparameter setup for (for example, a training task). N This results in a set of hyperparameters, also known as the}. In 304, a subset of candidate hyperparameters is formed. For example, this step may require building a reduced set or portfolio of effective hyperparameter configurations, as further described below, utilizing configurations within a regret matrix and task data. In 305, the Sum of Excess Regret (SER) is determined for the subset of hyperparameters. In 306, as described above, a subset of candidate hyperparameters (aka portfolio) with the lowest SER may be selected and used as the set of hyperparameters for iteratively generating a machine learning model. As shown in 307, the algorithm starts with an empty set of hyperparameter configurations and iterates through these steps, progressively adding configurations until a stopping criterion is met.

[0016] Figure 3B shows a system for performing automated machine learning according to another embodiment. In this example, computer system 350 includes software 359 for generating a set of hyperparameters used, for example, to generate a machine learning algorithm on computer system 300. In some embodiments, the software for generating the set of hyperparameters may be running computer system 350 offline, and AutoML software 301 may be running online on computer system 300. The hyperparameter generation software 359 may include a metadata set 351, a hyperparameter optimizer 352 for generating a candidate set of hyperparameters 353, a regret matrix computer 354, and a portfolio builder 355 that outputs a final set of hyperparameters (also known as H-params) 310. The set of hyperparameters may be received by AutoML software 301 operating, for example, as described in FIG. 1.

[0017] Figure 3C shows the generation of a set of hyperparameters according to an embodiment. Blocks 361-366 illustrate exemplary implementations of blocks 351-355 and 310, respectively.

[0018] The metadata set 361 includes training data for configuring a machine learning model to perform a task. A task refers to a specific learning problem that requires tabular data with its own dataset having distinct features (or columns) and a target variable to be predicted. For example, one task may require a dataset with numerical and categorical features for predicting a binary outcome. The meta-learning algorithm effectively learns a reduced set of generalizable hyperparameter settings that can be used in new, unseen tabular tasks by leveraging diverse tasks. Hundreds of such tasks can be considered, but only a small number (generally less than 10) of high-performance hyperparameter configurations are retained as a result of the meta-learning process. The hyperparameter optimizer 362 receives the dataset 361 and parameters including, for example, a loss metric L, a search space S, and a time budget B, and outputs a set C = {c1, …, c N} of candidate configurations representing optimal hyperparameter setups (e.g., machine learning models 102a~n) for each corresponding task T = {t1, …, t N}. For example, exemplary hyperparameters including a learning rate and a maximum depth are shown at 363. At 364, a regret matrix is constructed (e.g., R by evaluating the Cartesian product of the configuration C and the task T). One exemplary regret matrix R is as follows:

[0019]

Number

[0020] Here, L is a loss metric (bounded between 0 and 1), and c *(t) represents the optimal configuration in C for task t. In the next step, Portfolio Builder 365 leverages the data in the Regret Matrix to construct a reduced set, or portfolio, of effective hyperparameter configurations. For this purpose, a portfolio selection algorithm is used. This algorithm is a greedy method that starts with an empty set S and adds one configuration at a time incrementally. The iterative process is (1) one of several sets formed by each adding a new configuration to the current portfolio set.

[0021]

number

[0022] (2) To build,

[0023]

number

[0024] This includes calculating the excess regret sum (SER) for each candidate set within:

[0025]

number

[0026] Here, the input ∈=0.01 is the target regret, and any further improvement beyond that is ignored.

[0027] Next, the system selects candidate portfolios with the lowest SER (Tie (Difference < 10)). -5 In this case, the mean regret (MR) is used as the tiebreaker:

[0028]

number

[0029] The stopping condition is (i) min L SER ≤10 -5 , that is, the target regret has been achieved for all tasks, or (ii) an early stop trigger:

[0030]

number

[0031] Matching can be performed using [this method]. The final output is a compact portfolio of effective hyperparameter configurations that achieves a target regret of ∈=0.01 across all training tasks. If the portfolio size is excessively large, the target regret may be adjusted slightly upward, and the selection algorithm is reapplied. Examples of hyperparameter sets 110 and 310, and the output portfolio, S, i.e., the set of hyperparameters, are shown in 366.

[0032] By generating a hyperparameter set, embodiments of the present disclosure deviate from conventional methods that focus on predicting the optimal configuration from a predefined portfolio for a new task (e.g., generating a new machine learning model using a new training dataset). In one exemplary embodiment, one technique is distinguished, at least in part, from existing methods by aiming to optimize hyperparameters in a boosting model rather than pursuing a meta-feature-based prediction method. The inventors propose a simpler but potentially more effective strategy: combining configurations from a portfolio (e.g., using the entire set of hyperparameter inputs) during the training process of the boosting model.

[0033] Boosting is an ensemble machine learning technique that combines several weak learners to create a strong learner. In some embodiments, boosting works by successively training new models to correct the errors of previous models, with each model placing more emphasis on instances misclassified by previously used models. The final prediction is the sum of the predictions from the combined weak learners. Figure 3D illustrates boosting, for example, where machine learning models are combined (for example, as described above in Figures 1 and 2) to form a combined machine learning model. More specifically, a boosting algorithm iteratively combines multiple “weak learners,” i.e., simple models with limited predictive power, to form a more accurate ensemble model. A commonly used example in this category is the gradient boosting algorithm (e.g., XGBoost shown in Figure 4A). Generally, in a boosting algorithm, all weak learners share a single hyperparameter configuration. However, embodiments of this disclosure deviate from this norm by varying the hyperparameters across boosting iterations.

[0034] Various types of weak learners can be used, including decision trees, linear models, Naive Bayesian classifiers, and even simple neural networks. Figure 3E shows an exemplary decision tree according to one embodiment. A decision tree is a well-known statistical model. A series of trials are performed as the data points, each with multiple features, are traversed from the top 371 to the bottom of the tree. At each intermediate node 372a-b, the features are compared to a threshold. Depending on the result of the comparison, the data points proceed to either the left or right child node 373a-b or 373c-d. Thus, different paths are traversed depending on the data points. The traversal is complete when one of the leaf nodes is reached. Each leaf node 373a-d produces a set of instances, i.e., the set of all data points at which the traversal ended. Leaves are also associated with leaf weights, which indicate the predictions to be made for all data points within that set of instances. Decision trees are computationally efficient and can handle mixed data types, i.e., numerical (continuous or discrete) and categorical (nominal or ordinal) features. In some embodiments, decision trees are used as weak learners in boosting algorithms due to their ability to capture nonlinear relationships and interactions between features.

[0035] Embodiments of this disclosure perform offline meta-learning on a diverse set of training tasks and then employ meta-learning to create a condensed portfolio of high-performance hyperparameter configurations. However, rather than attempting to predict a single best configuration for a new task, embodiments select specific hyperparameters over an iterative process (e.g., using multi-armed bandit theory). This allows for the integration of multiple hyperparameters (aka configurations) during the training process, which can generate a boosting model, enabling a more dynamic and effective approach for model training.

[0036] The multi-armed bandit problem, a fundamental concept in decision theory and reinforcement learning, can be advantageously used to select hyperparameter sets. The problem is named after a row of slot machines in a casino (sometimes called "one-armed bandits"). Each machine (or "arm") provides a random reward from its own unique probability distribution. The task is to find a strategy that maximizes the amount of reward received over a series of plays. At the heart of multi-armed bandit theory revolves around the exploration-exploitation dilemma. Exploitation requires trying different machines to understand their reward patterns, while exploitation means sticking to the machine that appears to provide the best reward based on the current information. Therefore, embodiments of selection algorithms may have input parameters corresponding to exploration and / or exploitation.

[0037] In the current AutoML framework, the process dynamically updates the hyperparameter set used in each iteration of the training process for the boosting model (for example, as shown in Figure 2). Each “arm” represents a different hyperparameter set from the inventors’ meta-learning portfolio of hyperparameter sets. In some embodiments, the “reward” is, for example, the performance improvement against a validation segment of the dataset applied during performance analysis. In some embodiments, some boosting iterations may be excluded from the hyperparameters and optimized separately through an early stopping technique, for example, by monitoring the model’s performance against a validation set during training and stopping the training process when performance stops improving or begins to deteriorate. In some embodiments, a maximum number of boosting iterations may be set to avoid excessively long training times.

[0038] The multi-armed bandit problem has several commonly used variations, each tailored to different scenarios and challenges. Applications to hyperparameter selection can correspond to non-stationary reward systems where rewards generally decrease with each training iteration. For example, in non-stationary systems, the probability distribution associated with the reward for each movement may change over time as the boosting process progresses. For instance, one hyperparameter configuration might provide the best performance improvement at one stage, while another configuration might yield better results at a later stage. Furthermore, the inventors do not assume any particular form for the reward distribution of each hyperparameter configuration. Therefore, some embodiments may utilize adversarial multi-armed bandit algorithms (adversarial bandit algorithms as defined herein) in which rewards are controlled by an adversary whose reward distribution may change unpredictably, reflecting the current dynamic and unpredictable training environment.

[0039] The goal of the adversarial bandit problem is often expressed as minimizing regret, which is the difference between the reward that could have been obtained by consistently selecting the best arm in hindsight and the reward actually received. Useful adversarial bandit algorithms should be selected to be robust to changes in the reward distribution. They must be resilient to potentially misleading reward patterns created by adversaries, while also adapting quickly to new information. Adversarial bandit algorithms that satisfy the design constraints of the target system are known to those skilled in the art.

[0040] In one embodiment, the EXP3 adversarial bandit algorithm, which represents the "exponential weights for exploration and exploitation", can be used. EXP3 is a strategy specifically designed for the adversarial bandit problem. The EXP3 algorithm is shown in 4B. EXP3 balances exploration and exploitation through a probability distribution that favors the arm with better historical rewards while still allowing exploration. Its exponential update rule aims to minimize regret and enables rapid adaptation to the changing reward distribution. EXP3 guarantees a regret bound that is sub-linear in the number of rounds, which implies that the average regret per round approaches zero as the number of rounds increases. This property can be important to ensure that the performance of the algorithm improves over time even when an adversary is present.

[0041] The EXP3 algorithm, as described above, is an example of an adversarial algorithm where the parameters of the algorithm can be changed based on the performance of the model. In this example, the EXP3 algorithm requires two input parameters, namely, η and γ. η is the learning rate parameter and controls the step size in updating the weights of the arms. A larger η means faster learning. γ is the exploration parameter and controls the amount of exploration the algorithm performs. A larger γ means more exploration.

[0042] In one embodiment, updating the selection algorithm for each iteration is the weight w of the algorithm iThis may include updating the following. The following exemplary iterations show the update weights of the EXP3 algorithm, but do not show the input parameters η and γ. The following example shows 10 consecutive iterations in which the first arm consistently outperforms the second arm. The following EXP3 parameters are used: η=1, γ=0.1, K=2, and T=10. Across the 10 iterations, the reward for arm 0 is [0.2, 0.2, 0.2, 0.1, 0.07, 0.05, 0.05, 0.05, 0.02, 0.02] and the reward for arm 1 is [0.1, 0.05, 0.01, 0.01, 0.005, 0.005, 0.001, 0.001, 0.001, 0.001]. The probabilities, selections, rewards, and updated weights for each iteration are as follows:

[0043] Iteration 1: Probability: [0.5 0.5], Selected Move: 0, Reward: 0.2, Updated Weight: [1.22140276 1.]

[0044] Iteration 2: Probability: [0.5448506 0.4551494], Selected Move: 1, Reward: 0.05, Updated Weight: [1.22140276 1.05646351]

[0045] Iteration 3: Probability: [0.53258429 0.46741571], Selected Move: 0, Reward: 0.2, Updated Weight: [1.47368152 1.05646351]

[0046] Iteration 4: Probability: [0.57420448 0.42579552], Selected Move: 0, Reward: 0.1, Updated Weight: [1.60775806 1.05646351]

[0047] Iteration 5: Probability: [0.59311633 0.40688367], Selected Move: 0, Reward: 0.07, Updated Weight: [1.70548759 1.05646351]

[0048] Iteration 6: Probability: [0.60574439 0.39425561], Selected Move: 0, Reward: 0.05, Updated Weight: [1.77734838 1.05646351]

[0049] Iteration 7: Probability: [0.61447414 0.38552586], Selected Move: 1, Reward: 0.001, Updated Weight: [1.77734838 1.05783456]

[0050] Iteration 8: Probability: [0.61420117 0.38579883], Selected Move: 0, Reward: 0.05, Updated Weight: [1.85118478 1.05783456]

[0051] Iteration 9: Probability: [0.62272438 0.37727562], Selected Move: 0, Reward: 0.02, Updated Weight: [1.88115194 1.05783456]

[0052] Iteration 10: Probability: [0.62606142 0.37393858], Selected Move: 1, Reward: 0.001, Updated Weight: [1.88115194 1.05924995]

[0053] Cumulative reward (final performance): 0.7420000000000002

[0054] It should be understood that numerous variations of the EXP3 algorithm or other algorithms can be used as selection algorithms. Furthermore, in various embodiments known to those skilled in the art, strategies may be used to select and dynamically update the search parameter γ, the learning rate η, or other selection algorithm parameters. Thus, this disclosure is adaptable and not limited to any particular bandit algorithm. However, although this disclosure can be applied to algorithms not specifically designed for adversarial bandit problems, some applications may benefit from using algorithms that achieve sublinear regret bounds.

[0055] Figure 5 shows the hardware of a dedicated computing system 500 configured in accordance with the above disclosure. The following hardware description is illustrative only. It should be understood that various computer topologies may be used to implement the above techniques. An exemplary computer system 510 is shown in Figure 5. The computer system 510 includes a bus 505 or other communication mechanism for communicating information and one or more processors 501 coupled to the bus 505 for processing information. The computer system 510 also includes a memory 502 coupled to the bus 505 for storing information and instructions executed by the processors 501, including, for example, information and instructions for implementing some of the techniques described above. The memory 502 may be used to store programs executed by the processors 501. Possible implementations of the memory 502 may be, but are not limited to, random access memory (RAM), read-only memory (ROM), or both. A storage device 503 is also provided for storing information and instructions. Common forms of storage devices include, for example, hard drives, magnetic disks, optical disks, CD-ROMs, DVDs, solid-state disks, flash memory or other non-volatile memory, USB memory cards, or any other electronic storage medium from which a computer can read. Storage device 503 may include, for example, source code, binary code, or software files for performing the techniques described above. Both storage device 503 and memory 502 are examples of non-temporary computer-readable storage media (also known as storage media).

[0056] In some systems, the computer system 510 may be coupled via bus 505 to a display 512 for displaying information to the computer user. Input devices 511 such as a keyboard, touchscreen, and / or mouse are coupled to bus 505 to communicate information and command selections from the user to the processor 501. The combination of these components enables the user to communicate with the system. In some systems, bus 505 may represent, for example, multiple dedicated buses for coupling various components of the computer together.

[0057] The computer system 510 also includes a network interface 504 coupled to the bus 505. The network interface 504 can provide two-way data communication between the computer system 510 and a local network 520. The network 520 may represent one or more network technologies, such as Ethernet, a local wireless network (e.g., Wi-Fi), or a cellular network. The network interface 504 may be, for example, a wireless or wired connection. The computer system 510 can send and receive information through the network interface 504 over a wired or wireless local area network, intranet, or cellular network to the Internet 530, for example. In some embodiments, a front-end (e.g., a browser) may access data and features on a back-end software system that may reside, for example, on multiple different on-premises hardware servers 531 or over the network 520 (e.g., Ethernet or the Internet) on servers 532-543. One or more of the servers 532-534 may reside, for example, in a cloud computing environment.

[0058] Further examples Each of the following non-limiting features in the examples below may stand alone or be combined with one or more of the other features in the examples below in various substitutions or combinations. In various embodiments, this disclosure may be implemented as a system, method, or computer-readable medium.

[0059] Embodiments of this disclosure may include systems, methods, or computer-readable media. In one embodiment, the disclosure includes a computer system comprising at least one processor and at least one non-temporary computer-readable medium (e.g., memory) storing computer-executable instructions, which, when executed by at least one processor, cause the computer system to carry out methods described herein and in the following examples. In another embodiment, the disclosure includes a non-temporary computer-readable medium storing computer-executable instructions, which, when executed by at least one processor, cause the computer system to carry out methods described herein and in the following examples.

[0060] In one embodiment, the Disclosure provides a method comprising the steps of: receiving a set of hyperparameters configured in a computer system to train a set of corresponding machine learning models using a set of training datasets, wherein the set of machine learning models have the same machine learning model type; receiving a first training dataset not included in the set of training datasets; and over a set of iterations, (1) selecting one of the set of hyperparameters based on a first selection algorithm; (2) training a first machine learning model using the first training dataset and the selected set of hyperparameters, wherein the first machine learning model has the same machine learning model type; (3) combining the first machine learning model with one or more machine learning models pre-trained over a set of iterations; (4) determining the combined performance of the combined first machine learning model and one or more machine learning models pre-trained over a set of iterations; and (5) determining whether a change in the combined performance meets a threshold, wherein if a change in the combined performance meets a threshold, a result model is produced, the result model comprising the first machine learning model and one or more machine learning models pre-trained over a set of iterations.

[0061] If the changes in the combined performance do not meet the threshold, (6) the step of modifying the selection algorithm based on the changes in the combined performance is performed. Includes methods.

[0062] In one embodiment, multiple machine learning models and result models are generated sequentially by adding each specific trained machine learning model to a pre-trained machine learning model.

[0063] In one embodiment, the resulting model is generated using a boosting algorithm.

[0064] In one embodiment, each of the multiple machine learning models and each first machine learning model across multiple iterations includes a weak learner model.

[0065] In one embodiment, multiple weak learner models are combined to form a result model, and this result model has higher performance than the individual weak learner models.

[0066] In one embodiment, the multiple machine learning models, the first machine learning model, and the result model are tree-type machine learning models.

[0067] In one embodiment, the multiple machine learning models, the first machine learning model, and the result model are all decision tree type machine learning models.

[0068] In one embodiment, the step of modifying the selection algorithm based on changes in combined performance includes the step of modifying one or more parameters of the selection algorithm.

[0069] In one embodiment, the step of modifying the selection algorithm based on a change in combined performance includes a step of modifying the selection algorithm based solely on an increase in combined performance, and the iteration stops when the change in combined performance is reduced.

[0070] In one embodiment, the selection algorithm is an adversarial bandit algorithm.

[0071] In one embodiment, the adversarial bandit algorithm is the EXP3 algorithm.

[0072] In one embodiment, the method further includes the step of generating a set of hyperparameters configured to train a set of corresponding machine learning models using a set of training datasets, the generating step of determining a set of candidate sets of hyperparameters for training a set of machine learning models using a set of training datasets; determining a set of regret values ​​for each of the candidate sets of hyperparameters to be used to train the set of data models; forming a set of subsets of the candidate sets of hyperparameters based on the regret values; determining an excess regret sum (SER) for each subset of the candidate sets of hyperparameters; and selecting a subset of the candidate sets of hyperparameters having the lowest SER as a set of multiple hyperparameters.

[0073] The above description illustrates various embodiments, along with examples of how aspects of several embodiments may be implemented. The above examples and embodiments should not be considered as sole embodiments, but are presented to demonstrate the flexibility and advantages of several embodiments as defined by the appended claims. Based on the above disclosure and the appended claims, other configurations, embodiments, implementations, and equivalents may be adopted without departing from the scope defined by the claims. [Explanation of Symbols]

[0074] 100 Computer Systems 101 AutoML Software System 102a~n Machine Learning Models, ML 103 Training dataset, training data 110 hyperparameter set, hyperparameter set 111a~x Hyperparameters 112 Selection algorithm software components, selection algorithm 113 Machine Learning Training Software Components, ML Training Blocks 114 Machine Learning Models, ML 115 New training data, training dataset 120 Model Coupler Software Components, Model Coupler 121a~m Machine Learning Models, ML 130 Performance analyzer software components, performance analyzer 140 New Machine Learning Models, Outcome Models 300 Computer Systems 301 AutoML Software 310 Final hyperparameter set, hyperparameter set 350 Computer Systems 351 metadata sets 352 Hyperparameter Optimizer 353 Hyperparameters 354 Regret Matrix Computer 355 Portfolio Builder 359 Software, Hyperparameter Generation Software 361 metadata sets 362 Hyperparameter Optimizer 363 Hyperparameters 371 Top 372a~b Intermediate Nodes 373a~d Child nodes, leaf nodes 500 Computing Systems 501 Processor 502 memory 503 Storage Devices 504 Network Interface 505 Bus 510 Computer Systems 511 Input Devices 512 displays 520 Local network, network 530 Network 531 On-premises hardware servers Servers 532-543

Claims

1. It is a method, A computer system receiving multiple sets of hyperparameters configured to train multiple corresponding machine learning models using multiple training datasets, wherein the multiple machine learning models have the same machine learning model type. The steps include receiving a first training dataset that is not included in the aforementioned plurality of training datasets, Over multiple iterations, (1) A step of selecting one of the sets of multiple hyperparameters based on a first selection algorithm, (2) A step of training a first machine learning model using the first training dataset and one of the selected sets of multiple hyperparameters, wherein the first machine learning model has the same machine learning model type, (3) The step of combining the first machine learning model with one or more machine learning models that have been pre-trained over the plurality of iterations, (4) A step of determining the combined performance of the combined first machine learning model and the one or more machine learning models that have been pre-trained over a plurality of iterations, (5) A step of determining whether the combined performance changes meet a threshold. The steps to carry out each of these Equipped with, When the changes in the combined performance satisfy the threshold, an resulting model is created, the resulting model comprising the first machine learning model and one or more machine learning models pre-trained over the multiple iterations, When the change in the combined performance does not satisfy the threshold, (6) A step of modifying the selection algorithm based on the changes in the combined performance. The method by which it is carried out.

2. The method according to claim 1, wherein the plurality of machine learning models and the result model are generated sequentially by adding each specific trained machine learning model to a pre-trained machine learning model.

3. The method according to claim 1, wherein the resulting model is generated using a boosting algorithm.

4. The method according to claim 2, wherein each of the plurality of machine learning models and each first machine learning model over the plurality of iterations comprises a weak learner model.

5. The method according to claim 4, wherein a plurality of weak learner models are combined to form the result model, and the result model has higher performance than the plurality of weak learner models.

6. The method according to claim 2, wherein the plurality of machine learning models, the first machine learning model, and the result model are tree-type machine learning models.

7. The method according to claim 6, wherein the plurality of machine learning models, the first machine learning model, and the result model are decision tree type machine learning models.

8. The method according to claim 1, wherein the step of modifying the selection algorithm based on the changes in the combined performance comprises the step of modifying one or more parameters of the selection algorithm.

9. The method according to claim 1, wherein the step of modifying the selection algorithm based on the change in the combined performance comprises a step of modifying the selection algorithm based solely on an increase in the combined performance, and the iteration stops when the change in the combined performance decreases.

10. The method according to claim 1, wherein the selection algorithm is an adversarial bandit algorithm.

11. The method according to claim 10, wherein the adversarial bandit algorithm is the EXP3 algorithm.

12. The step further comprises generating a set of multiple hyperparameters configured to train the corresponding multiple machine learning models using the multiple training datasets, wherein the generating step is The steps include determining a set of candidate hyperparameters for training the multiple machine learning models using the multiple training datasets, The steps include determining a plurality of regret values ​​for each of the candidate sets of hyperparameters used to train the plurality of data models, The steps include forming a plurality of subsets of the candidate set of hyperparameters based on the regret value, The excess regret sum (SER) for each subset of the candidate set of hyperparameters. The steps to determine, A step of selecting a subset of the candidate set of hyperparameters having the lowest SER as the set of multiple hyperparameters. The method according to claim 1, comprising:

13. A computer system, At least one processor, A non-temporary computer-readable medium for storing computer executable instructions and The computer system is provided with such that when the computer executable instruction is executed by the at least one processor, the computer system The computer system receives multiple sets of hyperparameters configured to train multiple corresponding machine learning models using multiple training datasets, wherein the multiple machine learning models have the same machine learning model type. Receiving a first training dataset that is not included in the aforementioned multiple training datasets, Over multiple iterations, (1) Selecting one of the sets of multiple hyperparameters based on a first selection algorithm, (2) Training a first machine learning model using the first training dataset and one of the selected sets of multiple hyperparameters, wherein the first machine learning model has the same machine learning model type. (3) Combining the first machine learning model with one or more machine learning models that have been pre-trained over the multiple iterations, (4) Determining the combined performance of the combined first machine learning model and the one or more machine learning models that have been pre-trained over the multiple iterations, (5) Determining whether the combined performance changes meet the threshold. To implement each of these and A method that includes this will be implemented. When the changes in the combined performance satisfy the threshold, an resulting model is created, the resulting model comprising the first machine learning model and one or more machine learning models pre-trained over the multiple iterations, When the change in the combined performance does not satisfy the threshold, (6) Modify the selection algorithm based on the changes in the combined performance. A computer system in which this is implemented.

14. The computer system according to claim 13, wherein the plurality of machine learning models and the result model are generated sequentially by adding each specific trained machine learning model to a pre-trained machine learning model.

15. The computer system according to claim 14, wherein each of the plurality of machine learning models and each first machine learning model across the plurality of iterations comprises a weak learner model, the plurality of weak learner models are combined to form the result model, and the result model has higher performance than the plurality of weak learner models.

16. The computer system according to claim 14, wherein the plurality of machine learning models, the first machine learning model, and the result model are tree-type machine learning models.

17. The computer system according to claim 13, wherein the selection algorithm is an adversarial bandit algorithm.

18. A non-temporary computer-readable medium for storing computer-executable instructions, wherein the computer-executable instructions are executed by at least one processor of a computer system. The computer system receives multiple sets of hyperparameters configured to train multiple corresponding machine learning models using multiple training datasets, wherein the multiple machine learning models have the same machine learning model type. Receiving a first training dataset that is not included in the aforementioned multiple training datasets, Over multiple iterations, (1) Selecting one of the sets of multiple hyperparameters based on a first selection algorithm, (2) Training a first machine learning model using the first training dataset and one of the selected sets of multiple hyperparameters, wherein the first machine learning model has the same machine learning model type. (3) Combining the first machine learning model with one or more machine learning models that have been pre-trained over the multiple iterations, (4) Determining the combined performance of the combined first machine learning model and the one or more machine learning models that have been pre-trained over the multiple iterations, (5) Determining whether the combined performance changes meet the threshold. To implement each of these and A method is implemented that includes, When the changes in the combined performance satisfy the threshold, an resulting model is created, the resulting model comprising the first machine learning model and one or more machine learning models pre-trained over the multiple iterations, When the change in the combined performance does not satisfy the threshold, (6) Modify the selection algorithm based on the changes in the combined performance. A non-temporary computer-readable medium on which the process is carried out.

19. The non-temporary computer-readable medium according to claim 18, wherein the plurality of machine learning models and the result model are generated sequentially by adding each specific trained machine learning model to a pre-trained machine learning model.

20. The non-temporary computer-readable medium according to claim 18, wherein the selection algorithm is an adversarial bandit algorithm.

Images