Runtime estimation of machine learning data processing pipelines

Abstract

An input for constructing a data processing pipeline may be received, the data processing pipeline being configured to implement a process for generating a machine learning model for performing a task associated with an input data set. The process may include a plurality of machine learning trials, each of the plurality of machine learning trials applying a different type of machine learning model and / or a different set of trial parameters to a training data set and / or a validation data set generated based on the input data set. The machine learning model is generated based on the results of the plurality of machine learning trials. A runtime estimate for the process of generating the machine learning model may be determined. The runtime estimate may enable allocation of a sufficient time budget for the process. Furthermore, if the runtime of the process does not exceed the available time budget, the process may be executed.

Description

Technical Field

[0001] The subject matter described herein relates generally to machine learning and, more specifically, to runtime estimation of data processing pipelines that implement machine learning models. Background Art

[0002] Machine learning models can be trained to perform a variety of cognitive tasks, including, for example, object recognition, natural language processing, information retrieval, speech recognition, classification, regression, and the like. For example, an enterprise resource planning (ERP) system may include an issue tracking system configured to generate tickets in response to errors reported via one or more phone calls, emails, short messaging service (SMS) messages, social media posts, web chats, and the like. The issue tracking system may generate tickets to include a textual description of the error associated with the ticket. Thus, to determine an appropriate response for resolving the error associated with the ticket, the enterprise resource planning system may include a machine learning model trained to perform text classification. For example, the machine learning model may be trained to determine a priority for the ticket corresponding to the severity of the error based at least on the textual description of the error. Summary of the Invention

[0003] Systems, methods, and articles of manufacture, including computer program products, for estimating the runtime of a data processing pipeline that implements a machine learning model are provided. In one aspect, a system comprising at least one data processor and at least one memory is provided. The at least one memory may store instructions that, when executed by the at least one data processor, cause operations. The operations may include: receiving one or more inputs from a client device for constructing a data processing pipeline, the data processing pipeline configured to implement a process for generating a machine learning model for performing a task associated with an input data set, the process comprising a plurality of machine learning trials, each of the plurality of machine learning trials applying a different type of machine learning model and / or a different set of trial parameters to a training data set, and the machine learning model being generated based at least on results of the plurality of machine learning trials; determining a first runtime for the process of generating the machine learning model; and executing the process to generate the machine learning model based at least on the first runtime not exceeding an available time budget.

[0004] In some variations, one or more features disclosed herein, including the following, may optionally be included in any feasible combination. The first runtime may include non-computation time associated with provisioning a compute instance and commencing execution of a plurality of machine learning trials. The non-computation time may be determined at least by applying a linear regression model configured to determine the non-computation time based at least on the size of a training dataset and / or a validation dataset.

[0005] In some variations, the first runtime may include computation time associated with training the machine learning model, evaluating the performance of the machine learning model, and outputting the trained machine learning model. The computation time may be calculated by applying a computation time model configured to provide a probability distribution of runtimes over different sets of trial parameters.

[0006] In some embodiments, the computational time model may include a machine learning model, a hybrid model, a neural network, an isotonic regression model, an instance-based model, a regularized model, a decision tree, a random forest, a Bayesian model, a clustering model, an association model, a dimensionality reduction model, and / or an ensemble model generated by executing a data processing pipeline.

[0007] In some variations, the computation time model may be trained based at least on timing information collected from one or more test trials and / or user trials.

[0008] In some variations, training of the computational time model may prioritize more recent timing information and / or timing information from user trials.

[0009] In some variations, timing information may be collected at one or more checkpoints. The timing information may include the amount of time that elapsed between two or more consecutive checkpoints.

[0010] In some variations, the first runtime may be determined based at least on an expected amount of machine learning experiments required to generate the machine learning model.

[0011] In some variations, the expected amount of machine learning experiments can be determined as the amount of machine learning experiments performed to generate a machine learning model against one or more benchmark datasets. The one or more benchmark datasets can cover different use cases and have input datasets with different characteristics.

[0012] In some variations, the first runtime may be generated based at least on the amount of multiple machine learning models executed in parallel.

[0013] In some variations, a user interface at the client device may be updated to display the first runtime.An indication may be received from the client device to allocate a time budget corresponding to the first runtime.

[0014] In some variations, a second run time for a first machine learning trial included in the plurality of machine learning trials may be determined based at least on a first trial parameter set associated with the first machine learning trial. A third run time for a second machine learning trial included in the plurality of machine learning trials may be determined based at least on a second trial parameter set associated with the second machine learning trial. The first machine learning trial may be executed, but the second machine learning trial may not be executed, based at least on the second run time not exceeding an available time budget and the third run time exceeding the available time budget.

[0015] In some variations, the first and second sets of experimental parameters may be generated by sampling from a uniform distribution and / or applying information-based optimization techniques.

[0016] In some variations, execution of the first machine learning experiment may include applying a first machine learning model having a first set of experiment parameters to a training dataset and a validation dataset.

[0017] In some variations, a third machine learning experiment can be performed by at least applying the first machine learning model with the second set of trial parameters, the second machine learning model with the first set of trial parameters, or the second machine learning model with the third set of trial parameters to the training dataset and the validation dataset.

[0018] In some variations, a data processing pipeline may include an orchestrator node, a preparer node, and an executor node. The preparer node may be configured to generate, based at least on an input dataset, a training dataset for training a machine learning model to perform a task and a validation dataset for evaluating the performance of the machine learning model trained to perform the task. The executor node may be configured to execute multiple machine learning experiments. The orchestrator node may be configured to identify, based at least on the results of the multiple machine learning experiments, a machine learning model for performing the task associated with the input dataset.

[0019] In some variations, a machine learning model for performing a task may be generated by at least executing a data processing pipeline based on a corresponding graphical representation of the data processing pipeline, the data processing pipeline comprising an orchestrator node, a preparer node, and an executor node.

[0020] In some variations, the machine learning model may include a neural network, a regression model, an instance-based model, a regularized model, a decision tree, a random forest, a Bayesian model, a clustering model, an association model, a dimensionality reduction model, and / or an ensemble model.

[0021] In another aspect, a method for estimating the runtime of a data processing pipeline implementing a machine learning model is provided. The method may include: receiving one or more inputs for constructing a data processing pipeline from a client device, the data processing pipeline configured to implement a process for generating a machine learning model for performing a task associated with an input data set, the process comprising a plurality of machine learning trials, each of the plurality of machine learning trials applying a different type of machine learning model and / or a different set of trial parameters to a training data set, and the machine learning model being generated based at least on results of the plurality of machine learning trials; determining a first runtime of the process for generating the machine learning model; and executing the process to generate the machine learning model based at least on the first runtime not exceeding an available time budget.

[0022] In another aspect, a computer program product comprising a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium may include program code that causes operations when executed by at least one data processor. The operations may include: receiving one or more inputs from a client device for constructing a data processing pipeline, the data processing pipeline configured to implement a process for generating a machine learning model for performing a task associated with an input data set, the process comprising a plurality of machine learning trials, each of the plurality of machine learning trials applying a different type of machine learning model and / or a different set of trial parameters to a training data set, and the machine learning model being generated based at least on results of the plurality of machine learning trials; determining a first run time for the process of generating the machine learning model; and executing the process to generate the machine learning model based at least on the first run time not exceeding an available time budget.

[0023] The implementation of the current theme may include methods consistent with the description provided herein and articles including tangible machine-readable media that can be operated to cause one or more machines (e.g., computers, etc.) to cause one or more operations of implementing the described features. Similarly, a computer system is also described that may include one or more processors and one or more memories coupled to the one or more processors. The memory that may include a non-transitory computer-readable or machine-readable storage medium may include, encode, store, etc., one or more programs that cause one or more processors to perform one or more operations described herein. The computer-implemented method consistent with one or more implementations of the current theme may be implemented by one or more data processors residing in a single computing system or multiple computing systems. Such multiple computing systems can be interconnected and can exchange data and / or commands or other instructions via one or more connections (including, for example, connections via a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, etc.)), via a direct connection between one or more of the multiple computing systems, etc.

[0024] The details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features and advantages of the subject matter described herein will be apparent from the description and drawings, and from the claims. Although certain features of the presently disclosed subject matter are described for illustrative purposes in relation to machine learning data processing pipelines, it should be readily understood that these features are not intended to be limiting. The claims appended to this disclosure are intended to define the scope of the protected subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate certain aspects of the subject matter disclosed herein and, together with the description, help explain some of the principles associated with the disclosed implementations.

[0026] Figure 1 depicts a system diagram illustrating a data processing pipeline generator system according to some example embodiments;

[0027] Figure 2A depicts a schematic diagram illustrating an example of a data processing pipeline having modular pipeline elements, according to some example embodiments;

[0028] Figure 2B depicts an example of a graph representing a data processing pipeline according to some example embodiments;

[0029] Figure 2C depicts an example of operator nodes forming a data processing pipeline configured to generate a machine learning model, according to some example embodiments;

[0030] Figure 3A depicts a block diagram illustrating an example communication flow between operator nodes forming a data processing pipeline configured to generate a machine learning model, according to some example embodiments;

[0031] Figure 3B depicts a flowchart illustrating an example of data processing operations performed by executor nodes forming a data processing pipeline configured to generate a machine learning model, according to some example embodiments;

[0032] Figure 3C depicts an example of a user interface according to some example embodiments;

[0033] Figure 3D depicts another example of a user interface according to some example embodiments;

[0034] Figure 4 depicts a timing diagram illustrating an example of a process for generating a machine learning model trained to perform a task, according to some example embodiments;

[0035] Figure 5 depicts a schematic diagram illustrating an example of deployment of a data processing pipeline implementing a machine learning model, according to some example embodiments;

[0036] Figure 6A depicts a schematic diagram illustrating various components of runtime, according to some example embodiments;

[0037] Figure 6B depicts a timing diagram illustrating an example of an extended message exchange according to some example embodiments;

[0038] Figure 7A depicts a flowchart illustrating a process for generating a data processing pipeline configured to generate a machine learning model, according to some example embodiments;

[0039] Figure 7B depicts a flowchart illustrating an example of a process for estimating the runtime of a machine learning data processing pipeline, according to some example embodiments; and

[0040] Figure 8 Depicted is a block diagram illustrating a computing system, according to some example embodiments.

[0041] Where practical, like reference numerals indicate like structures, features or elements. DETAILED DESCRIPTION

[0042] A data processing pipeline can include a series of operations for processing data, including collecting and / or manipulating data, such as unusually large and / or complex data sets known as "big data." A data processing pipeline can be graphically represented as a plurality of operator nodes interconnected by one or more directed edges of a directed graph. Each operator node can correspond to a data processing operation performed on the data passing through the operator node. At the same time, a directed edge interconnecting two operator nodes can indicate the flow of data between the data processing operations corresponding to each operator node. Thus, a graph (such as a directed graph) representing a data processing pipeline can be constructed by interconnecting at least a series of operator nodes using one or more directed edges.

[0043] A graph representing a data processing pipeline can be constructed to include one or more operator nodes configured to generate a machine learning model trained to perform a task. For example, the graph can be constructed to include an orchestrator node, one or more preparer nodes, and one or more executor nodes. The orchestrator node can be configured to coordinate the operations of the preparer node and the one or more executor nodes. For example, each preparer node can be configured to generate a training dataset and a validation dataset based at least on an input dataset associated with a task. At the same time, each executor node can be configured to apply different types of machine learning models and / or different parameter sets to the training dataset and validation dataset generated by the preparer node. The orchestrator node can be configured to determine a machine learning model including model parameters and hyperparameter sets for performing a task associated with the input dataset based at least on the performance of different types of machine learning models and / or different parameter sets. Therefore, a machine learning model trained to perform a task can be generated at least by executing a data processing pipeline including an orchestrator node, one or more preparer nodes, and one or more executor nodes.

[0044] A data processing pipeline that implements a machine learning model can be deployed on different cloud computing platforms, each of which has a different timing definition. For example, factors that affect the runtime of a data processing pipeline (including storage capacity, technical limitations, response time, etc.) may be different on different cloud computing platforms. Even on the same cloud computing platform, these factors may evolve over time due to changes in individual user provisioning (including, for example, storage type, data center, etc.). Changes in timing definitions across different cloud computing platforms may result in deviations in runtime estimates for the data processing pipeline. However, accurate and consistent runtime estimates may be necessary to establish reliable and effective time budgets. For example, accurate and consistent runtime estimates may enable allocation of a sufficient time budget to execute the amount of machine learning experiments required to determine a machine learning model including a set of model parameters and hyperparameters for executing a task. Alternatively and / or additionally, accurate and consistent runtime estimates may be required to determine whether one or more machine learning experiments can be executed without exceeding the remaining time budget.

[0045] In some example embodiments, a runtime estimator may be configured to determine a runtime estimate for executing one or more machine learning trials for a data processing pipeline, each of the one or more machine learning trials comprising a different type of machine learning model and / or a different set of trial parameters. For example, the runtime estimator may define checkpoints for collecting timing information to avoid inconsistencies introduced by different timing definitions at different cloud computing platforms. Additionally, the runtime estimator may be configured to generate a time budget estimate for a process based at least on the runtime estimate, in which the one or more machine learning trials are executed to generate a machine learning model comprising a set of model parameters and hyperparameters for performing a task. Alternatively and / or additionally, the runtime estimator may identify one or more machine learning trials that can be executed without exceeding an available time budget based at least on the runtime estimates for the individual machine learning trials included in the process.

[0046] Figure 1 Depicted is a system diagram illustrating a data processing pipeline generator system 100 according to some example embodiments. Figure 1 , the data processing pipeline generator system 100 may include a pipeline engine 110. Figure 1As shown, the pipeline engine 110 and the client 120 can be communicatively coupled via a network 130. The client 120 can be a processor-based device, including, for example, a smartphone, a tablet computer, a wearable device, a virtual assistant, an Internet of Things (IoT) appliance, etc. Meanwhile, the network 130 can be any wired and / or wireless network, including, for example, a public land mobile network (PLMN), a wide area network (WAN), a local area network (LAN), a virtual local area network (VLAN), the Internet, etc.

[0047] The client 120 can interact with the pipeline engine 110 to generate one or more data processing pipelines. Figure 1 As shown, the pipeline engine 110 can be associated with a user interface 150, and the user interface 150 is configured to receive one or more inputs from a user 125 at a client 120 for editing a graph representing a data processing pipeline, and output the progress and / or results of executing the data processing pipeline to the user 125 at the client 120. In some example embodiments, the one or more inputs can edit the graph representing the data processing pipeline to include one or more operator nodes, the one or more operator nodes being configured to generate a machine learning model trained to perform a task associated with the input dataset based on at least an input dataset. For example, the graph representing the data processing pipeline can be edited to include an orchestrator node, one or more preparer nodes, and one or more executor nodes.

[0048] In some example embodiments, the orchestrator node may be configured to coordinate the operation of one or more executor nodes. At the same time, each executor node may be configured to apply different types of machine learning models to a training dataset and a validation dataset generated by one or more preparer nodes based on an input dataset. In addition, the orchestrator node may be configured to identify a machine learning model for performing a task associated with the input dataset based at least on the performance of different types of machine learning models. Thus, a machine learning model trained to perform a task may be generated by at least executing a data processing pipeline comprising an orchestrator node, one or more preparer nodes, and one or more executor nodes. For example, a data processing pipeline may be executed to generate a machine learning model that is trained to perform cognitive tasks such as object recognition, natural language processing, information retrieval, speech recognition, classification, and / or regression. The machine learning model may be any type of machine learning model, including, for example, a neural network, a regression model, an instance-based model, a regularized model, a decision tree, a random forest, a Bayesian model, a clustering model, an association model, a dimensionality reduction model, an ensemble model, and the like.

[0049] To further illustrate, Figure 2ADepicted is a schematic diagram illustrating an example of a data processing pipeline with modular pipeline elements according to some example embodiments. Figure 2A As shown, the data processing pipeline can include different combinations of elements for data preparation, feature engineering, feature selection, model training, integration, etc. Each element of the data processing pipeline can be associated with one or more hyperparameters. The machine learning model used to perform the task associated with the input data set can be identified at least by evaluating the performance of the data processing pipeline across different combinations of pipeline elements and hyperparameters. For example, the executor node can be configured to execute one or more machine learning experiments, each of the one or more machine learning experiments corresponding to a different combination of pipeline elements and hyperparameters. In addition, the orchestrator node can identify the machine learning model used to perform the task associated with the input data set based at least on the performance of the one or more machine learning experiments.

[0050] Figure 2B Depicted is an example of a diagram representing a data processing pipeline 200 according to some example embodiments. Figure 2B The data processing pipeline 200 may include a plurality of operator nodes, including, for example, a first operator node 210 a, a second operator node 210 b, a third operator node 210 c, a fourth operator node 210 d, a fifth operator node 210 e, etc. Each of the first operator node 210 a, the second operator node 210 b, the third operator node 210 c, the fourth operator node 210 d, and / or the fifth operator node 210 e may correspond to a data processing operation performed on data passing through the operator node.

[0051] also, Figure 2B Show the first operator node 210a, the second operator node 210b, the 3rd operator node 210c, the 4th operator node 210d and / or the 5th operator node 210e via one or more directed edges interconnection.Directed edges can indicate the data flow between the data processing operation corresponding to the operator node by directed edges interconnection.For example, the first side 220a can interconnect the first operator node 210a and the 4th operator node 210d, to indicate at least the output of the data processing operation corresponding to the first operator node 210a is provided as the input of the data processing operation corresponding to the 4th operator node 210d.Alternatively and / or additionally, the second side 220b of the interconnection second operator node 210b and the 4th operator node 210d can indicate the output of the data processing operation corresponding to the 4th operator node 210d can be provided as the input of the data processing operation corresponding to the second operator node 210b.

[0052] In some example embodiments, a data processing pipeline may be constructed to include one or more specific operator nodes to implement a machine learning model trained to perform cognitive tasks such as, for example, object recognition, natural language processing, information retrieval, speech recognition, classification, and / or regression. Figure 2C Depicted are examples of operator nodes forming a data processing pipeline 250 configured to generate a machine learning model, according to some example embodiments. Figure 2C As shown, data processing pipeline 250 can be configured to include orchestrator node 230 and one or more preparer nodes, such as preparer node 240. Furthermore, data processing pipeline 250 can be configured to include one or more executor nodes, including, for example, executor node 280. Alternatively and / or additionally, data processing pipeline 250 can be configured to include one or more auxiliary operator nodes, including, for example, start node 260, user interface node 270, and graphical terminator node 290. Start node 260 can receive an initial configuration to generate a machine learning model, for example, specified by user 125 at client 120. Simultaneously, user interface node 270 can be configured to generate and / or update user interface 150 to display the progress of executing data processing pipeline 200 at client 120. Graphical terminator node 290 can be called to terminate execution of data processing pipeline 250. It should be understood that data processing pipeline 250 can implement a process in which one or more machine learning experiments are executed to generate a machine learning model for performing a task.

[0053] In some example embodiments, the preparer node 240 may be configured to validate and preprocess an input data set received, for example, from a client 120. In addition, the preparer node 240 may be configured to generate a training data set and a validation data set based on at least the input data set. For example, the input data set may include text associated with one or more errors reported to an issue tracking system. The preparer node 240 may validate the input data set and, in response to identifying one or more errors present in the input data set, terminate additional processing of the input data set. When validating the input data set, the preparer node 240 may preprocess the input data set, including by removing invalid rows and / or columns of data from the input data set and encoding any text included in the input data set. The preparer node 240 may split the validated and preprocessed input data set into a training data set for training a machine learning model to perform text classification and a validation data set for evaluating the performance of the trained machine learning model in performing text classification.

[0054] In some example embodiments, the executor node 280 may be configured to execute one or more machine learning experiments based on the training dataset and / or validation dataset generated by the preparer node 240. Each machine learning experiment may include applying a machine learning model having a particular experiment parameter set to the training dataset and / or validation dataset. The experiment parameter set may include one or more parameters of the machine learning model, such as, for example, initial weights applied to the machine learning model prior to training. In addition, the experiment parameter set may include one or more hyperparameters of the machine learning model, including, for example, a learning rate (e.g., step size) of a neural network, a value of a constant k in a k-nearest neighbors algorithm, a cost and sigma associated with a support vector machine, and the like.

[0055] The executor node 280 can be configured to perform a series of machine learning experiments, each of which includes a different type of machine learning model and / or a different set of experiment parameters. For example, the executor node 280 can perform a first machine learning experiment by at least applying a first machine learning model with a first set of experiment parameters to a training dataset and / or a validation dataset. The executor node 280 can also perform a second machine learning experiment by at least applying a first machine learning model with a second set of experiment parameters or a second machine learning model with a third set of experiment parameters to the training dataset and / or the validation dataset. As used herein, a "process" can refer to a test process or sequence that includes one or more machine learning experiments. A process can be executed by the executor node 280 to generate a machine learning model that includes a set of model parameters and hyperparameters for performing a task.

[0056] In some example embodiments, orchestrator node 230 may be configured to coordinate operations of preparer node 240 and executor node 280. Orchestrator node 230 may respond to receiving an initial configuration for implementing a machine learning model from start node 260 by at least triggering generation of a training dataset and a validation dataset at preparer node 240. The initial configuration may include, for example, a task type (e.g., classification, regression, etc.), a target column (e.g., a column in the training dataset corresponding to a ground-truth label), a target metric (e.g., accuracy), column names and / or types in the training dataset and the validation dataset, a computational resource budget (e.g., maximum execution time, etc.), a path to an output directory, a path to the training dataset, a path to the validation dataset, and the like.

[0057] Upon receiving an indication from preparer node 240 that preparer node 240 has generated a training dataset and a validation dataset, orchestrator node 230 may determine a machine learning model, including a set of model parameters and hyperparameters, for performing a task associated with the input dataset (e.g., classifying text associated with an issue tracking system, etc.). For example, orchestrator node 230 may determine the machine learning model, including a set of model parameters and hyperparameters, for performing the task associated with the input dataset by at least triggering execution of one or more machine learning experiments at executor node 280, each of the one or more machine learning experiments including a different type of machine learning model and / or a different set of experiment parameters. Orchestrator node 230 may trigger, at executor node 280, an amount of machine learning experiments consistent with the computational resource budget set forth in the initial configuration for implementing the machine learning model. For example, orchestrator node 230 may trigger additional machine learning experiments based at least on the amount of remaining computational resources sufficient to support execution of the additional machine learning experiments.

[0058] In some example embodiments, a set of experimental parameters for a machine learning experiment may be randomly selected from a hyperparameter space, which includes parameters that govern the configuration of the data processing pipeline 250 and the configuration of each node within the data processing pipeline 250. A machine learning model including a set of model parameters and hyperparameters for performing a task associated with an input data set may be identified by applying an information-based optimization technique (e.g., Bayesian optimization, etc.), starting with a random set of experimental parameters and then combining the corresponding results to identify the region of the hyperparameter space that is most likely to include the model parameters and hyperparameter sets associated with the machine learning model identified as being best suited for the task associated with the input data set within the available time budget. With each successive update, the sampling of the hyperparameter space may transition from uniform sampling to information-weighted sampling using a Bayesian (or other) approach.

[0059] Executor node 280 may execute a first machine learning experiment including a first machine learning model having a first experiment parameter set and a second machine learning experiment including the first machine learning model having a second experiment parameter set or the second machine learning model having a third experiment parameter set. Thus, orchestrator node 230 may select a machine learning model including a set of model parameters and hyperparameters for performing a task associated with an input data set based at least on the respective performances of the first machine learning model having the first experiment parameter set, the first machine learning model having the second experiment parameter set, and / or the second machine learning model having the third experiment parameter set. Upon selecting a machine learning model including a set of model parameters and hyperparameters for performing a task associated with the input data set, orchestrator node 230 may terminate execution of data processing pipeline 250, including by sending a message to graph terminator node 290 to terminate execution of data processing pipeline 250.

[0060] As described above, executor node 280 can sequentially execute the first machine learning experiment and the second machine learning experiment. However, it should be understood that data processing pipeline 250 can be configured to include multiple executor nodes, and orchestrator node 230 can coordinate the operations of multiple executor nodes to execute multiple machine learning experiments in parallel. Furthermore, data processing pipeline 250 can be configured to include multiple preparer nodes, and orchestrator node 230 can coordinate the operations of multiple preparer nodes to generate input datasets and validation datasets in parallel.

[0061] In some example embodiments, orchestrator node 230, preparer node 240, and executor node 280 may communicate via one or more messages. However, these messages may exclude intermediate data, such as, for example, training and validation datasets generated by preparer node 240. Instead, orchestrator node 230, preparer node 240, and executor node 280 may exchange intermediate data via shared persistence 115 accessible to orchestrator node 230, preparer node 240, and executor node 280. For example, orchestrator node 230 may store at least a portion of an initial configuration including an input dataset associated with a specified task in shared persistence 115. Similarly, preparer node 240 may store training and validation datasets generated based on the input dataset in shared persistence 115. Instead of directly sending the input dataset, the training dataset, and the validation dataset, orchestrator node 230, preparer node 240, and executor node 280 may exchange the input dataset, the training dataset, and / or the validation dataset by at least sending messages indicating the availability of the input dataset, the training dataset, and / or the validation dataset in shared persistence 115. Thus, preparer node 240 may access shared persistence 115 to retrieve the input dataset in order to generate the training dataset and the validation dataset, and executor node 280 may access shared persistence 115 to retrieve the training dataset and the validation dataset for use during the execution of one or more machine learning experiments.

[0062] Figure 3A Depicted is a block diagram illustrating an example communication flow between operator nodes forming a data processing pipeline 250 configured to generate a machine learning model, according to some example embodiments. Figure 3A , the orchestrator node 230, the preparer node 240 and the executor node 280 can exchange intermediate data by at least accessing the experiment persistence 300, which can form Figure 1 As shown in the shared persistence 115 portion. Figure 3AAs shown, user 125 at client 120 can interact with user interface 150 to specify an initial configuration of a machine learning model via application programming interface 310, the machine learning model performing tasks such as, for example, object recognition, natural language processing, information retrieval, speech recognition, classification, and / or regression. In response to receiving the initial configuration of the machine learning model, orchestrator node 230 can trigger the generation of a training dataset at preparer node 240 to train the machine learning model to perform the specified task, and trigger a validation dataset to evaluate the performance of the trained machine learning model performing the specified task. Preparer node 240 can store the training dataset and the validation dataset in experiment persistence 300. In addition, preparer node 240 can send a first message to orchestrator node 230 to notify orchestrator node 230 of the availability of the training dataset and the validation dataset.

[0063] In response to the first message from preparer node 240, orchestrator node 230 may send a second message to executor node 280 that triggers the execution of one or more machine learning experiments. For example, executor node 280 may respond to the second message by at least executing a first machine learning experiment including a first machine learning model having a first experiment parameter set and / or a second machine learning experiment including a first machine learning model having a second experiment parameter set or a second machine learning model having a third experiment parameter set. Executor node 280 may also store results of the machine learning experiments, for example, corresponding to the respective performances of the first machine learning model having the first experiment parameter set, the first machine learning model having the second experiment parameter set, and / or the second machine learning model having the third experiment parameter set, in experiment persistence 300. To identify a machine learning model including a set of model parameters and hyperparameters for performing a specified task, orchestrator node 230 may access at least experiment persistence 300 to evaluate the results of the machine learning experiments, for example, relative to a target metric specified by user 125 as part of the initial configuration of the machine learning model.

[0064] A data processing pipeline 250, including orchestrator nodes 230, preparer nodes 240, and executor nodes 280, can be executed to perform feature extraction, feature preprocessing, and training of a machine learning model. For example, feature extraction can be performed to generate numerical features based on data from one or more columns of an input dataset, including by encoding categorical features in each column and / or extracting values ​​from data fields. Feature preprocessing can include normalizing the values ​​occupying one or more columns in the input dataset. Thus, a machine learning model can be trained by applying the machine learning model to the numerical columns generated by feature extraction and / or feature preprocessing.

[0065] In some example embodiments, the data processing pipeline 250 can be configured to dynamically adjust based on metrics and / or configuration of the input dataset. Furthermore, the data processing pipeline 250 can be dynamically adjusted based on one or more previous processing operations in the data processing pipeline 250. In this way, some operations can be omitted from the data processing pipeline 250 to reduce operational costs, minimize training time, and increase the accuracy of the resulting machine learning model. For example, whether the executor node 280 performs feature selection may depend on the number of available features. That is, if more than a threshold number of features are available, the executor node 250 may perform feature selection. The executor node 280 may also avoid performing any additional processing on the embedded columns to avoid distorting the embedding space. In the event that the input dataset does not include any columns containing text data, the executor node 280 may omit any text encoding, thereby reducing the hyperparameter space. Furthermore, the executor node 280 may exclude one or more columns from the input dataset that are determined to have information values ​​below a threshold, such as columns with a target cross-entropy below a threshold.

[0066] In some example embodiments, the quantization of a hyperparameter can be adjusted based on a metric of the input dataset, including, for example, the number of columns and / or the number of unique values ​​on a column containing a particular data type. As used herein, "quantization" of a hyperparameter can refer to the discrete values ​​that a hyperparameter can take during each machine learning trial. For example, if a hyperparameter column sampling rate produces substantially the same results at 10% and 12%, the hyperparameter column sampling rate can be varied in increments of 20% for each machine learning trial.

[0067] Figure 3B A flowchart illustrating an example of data processing operations performed by an executor node 280 forming a data processing pipeline 250 configured to generate a machine learning model is depicted, according to some example embodiments. In some example embodiments, executor node 280 may respond to a second message from orchestrator node 230 by at least accessing shared persistence 115 to retrieve a training dataset and / or a validation dataset generated by preparer node 240. Executor node 280 may perform a series of data processing operations on the training dataset and / or the validation dataset, each of the series of data processing operations applying a different transformation to the training dataset and / or the validation dataset. Figure 3B As shown, the executor node 280 can perform data processing operations, such as column selection operations, feature selection operations, text encoding operations, category encoding operations, imputation operations, normalization operations, classification operations, etc.

[0068] In some example embodiments, an executor node 280 executing a single machine learning experiment may generate a corresponding candidate machine learning model with a specific set of parameters and / or hyperparameters. Executor node 280 may store the candidate machine learning model in shared persistence 115 (e.g., experiment persistence 300). In addition, executor node 280 may send the results of the machine learning experiment to the orchestrator node, which may correspond to the performance of the candidate machine learning model operating on a validation dataset. For example, executor node 280 may store the results of the machine learning experiment in shared persistence 115 (e.g., experiment persistence 300) so that orchestrator node 230 can access shared persistence 115 (e.g., experiment persistence 300) to retrieve the results of the machine learning experiment. As described above, orchestrator node 230 may access shared persistence 115 (e.g., experiment persistence 300) to evaluate the results of one or more machine learning experiments and generate a machine learning model that includes model parameters and hyperparameter sets for performing the task specified by user 125 on client 120.

[0069] Figure 3C-3D An example of a user interface 150 is depicted in accordance with some example embodiments. Figure 3C-3D As shown, the user interface 150 can be updated to display the progress and results of one or more machine learning experiments at the client 120. For example, the user interface 150 can be updated to display the model accuracy, calibration curve, confusion matrix, the importance of each feature (e.g., the relevance of each column in the training data set for the machine learning model), etc. at the client 120. Figure 3C In the example of user interface 150 shown, the progress and results associated with multiple types of machine learning models can be sorted to identify one or more machine learning models with the best results. Figure 3D Depicted is an example of a user interface 150 displaying the progress and results of a single type of machine learning model (e.g., an XGBoost classifier).

[0070] Figure 4 Depicted is a timing diagram illustrating a process 400 for generating a machine learning model trained to perform a task, according to some example embodiments. Figure 1 、 Figure 2A-2C 、 Figure 3A-3C and Figure 4 , process 400 can be performed by the pipeline engine 110 as part of executing the data processing pipeline 250 to generate a machine learning model having a set of parameters and / or hyperparameters for performing cognitive tasks (such as, for example, object recognition, natural language processing, information retrieval, speech recognition, classification and / or regression).

[0071] Orchestrator node 230 may receive an initial configuration 455 from client 120 for implementing a machine learning model to perform cognitive tasks (such as, for example, object recognition, natural language processing, information retrieval, speech recognition, classification, and / or regression). In response to receiving initial configuration 455 from client 120, orchestrator node 230 may trigger, at preparer node 240, generation of a training dataset for training the machine learning model to perform the task and a validation dataset for evaluating the performance of the machine learning model trained to perform the task. Figure 4 As shown, orchestrator node 230 may trigger the generation of a training dataset and a validation dataset by at least sending a first message 460a to preparer node 240. Preparer node 240 may respond to first message 460a by generating the training dataset and the validation dataset and performing one or more preparation tasks, such as embedding and / or encoding various data (e.g., textual data, numerical data, spatial data, categorical data, etc.). When preparer node 240 completes generating the training dataset and the validation dataset and the preparation tasks, preparer node 240 may send a second message 460b to orchestrator node 230 notifying orchestrator node 230 of the availability of the training dataset and the validation dataset and the results of the preparation tasks. Preparer node 240 may store the training dataset and the validation dataset in shared persistence 115 (e.g., experiment persistence 300), where they may be accessible to orchestrator node 230 and executor nodes 280. In this way, preparer node 240 may avoid sending the training dataset and the validation dataset directly to orchestrator node 230 in second message 460b.

[0072] Orchestrator node 230 may respond to second message 460b by at least triggering execution of the first machine learning experiment at executor node 280. Figure 4 As shown, orchestrator node 230 can trigger execution of a first machine learning experiment by at least sending a first experiment parameter set 470a to executor node 280. First experiment parameter set 470a can include one or more parameters of a machine learning model, such as, for example, initial weights applied to the machine learning model before training. In addition, first experiment parameter set 470a can include one or more hyperparameters of the machine learning model, including, for example, a learning rate (e.g., step size) of a neural network, a value of a constant k in a k-nearest neighbor clustering algorithm, a cost and sigma associated with a support vector machine, and the like. Executor node 280 can execute the first machine learning experiment by at least applying the machine learning model having first experiment parameter set 470a to a training dataset and a validation dataset generated by preparer node 240.

[0073] In some example embodiments, orchestrator node 230 may continue to trigger the execution of additional machine learning trials if there are sufficient remaining computing resources (including time budget) to support the execution of additional machine learning trials. For example, Figure 4 Orchestrator node 230 is shown sending a second set of experiment parameters 470b to executor node 280 that triggers execution of a second machine learning experiment at executor node 280, and sending a third set of experiment parameters 470c to trigger execution of a third machine learning experiment at executor node 280.

[0074] Executor node 280 can return a first test result 475a of executing the first machine learning test, a second test result 475b of executing the second machine learning test, and a third test result 475c of executing the third machine learning test to orchestrator node 230. First test result 475a, second test result 475b, and third test result 475c can correspond to the performance of one or more machine learning models having first test parameter set 470a, second test parameter set 470b, and third test parameter set 470c. Furthermore, it should be understood that executor node 280 can send first test result 475a, second test result 475b, and third test result 475c to orchestrator node 230 at least by storing first test result 475a, second test result 475b, and third test result 475c in shared persistence 115 (e.g., experiment persistence 300).

[0075] For example, executor node 280 may evaluate first trial result 475a, second trial result 475b, and / or third trial result 475c relative to a target metric specified by user 125 as part of the initial configuration of the machine learning model to identify a machine learning model having a set of parameters and / or hyperparameters suitable for performing the task. Executor node 280 may select, for example, first trial parameter set 470a and a machine learning model associated with first trial parameter set 470a based at least on first trial result 475a being superior to second trial result 475b and third trial result 475c relative to the target metric specified by user 125. The target metric may be the accuracy of the machine learning model, in which case first trial result 475a may be superior by exhibiting a higher target metric than second trial result 475b and third trial result 475c. Alternatively and / or additionally, the target metric may be logarithmic loss, in which case first trial result 475a may be superior by exhibiting a lower target metric than second trial result 475b and third trial result 475c.

[0076] exist Figure 4In the example shown, orchestrator node 230 may send one or more messages to client 120 indicating the status of a machine learning experiment. For example, orchestrator node 230 may send third message 460c to client 120 including first experiment result 475a of a first machine learning experiment executed by executor node 280. Alternatively and / or additionally, orchestrator node 230 may send fourth message 460d to client 120 including third experiment result 475c of a third machine learning experiment executed by executor node 280 and an indication that the machine learning experiment is complete. Orchestrator node 230 may communicate with client 120 via user interface node 270, which may be configured to generate and / or update user interface 150 to display at least a portion of the content of third message 460c and / or fourth message 460d at client 120.

[0077] refer to Figure 5 , the pipeline engine 110 can deploy the data processing pipeline 250 to an infrastructure 500 having one or more cloud computing platforms (e.g., MICROSOFT AZURE, AMAZON AWS, GOOGLE CLOUD, etc.), including, for example, a first cloud computing platform 510a, a second cloud computing platform 510b, etc. The first cloud computing platform 510a and the second cloud computing platform 510b can have different timing definitions. For example, factors affecting the runtime of the data processing pipeline 250 (including storage capacity, technical limitations, response time, etc.) can vary between the first cloud computing platform 510a and the second cloud computing platform 510b. Even on the same cloud computing platform, these factors may evolve over time due to changes in individual user provisioning (including, for example, storage type, data center, etc.). Optimal allocation of the time budget available to the user 125 at the client 120 for executing the data processing pipeline 250 may require an accurate and consistent estimate of the runtime associated with the data processing pipeline 250. However, differences in timing definitions between the first cloud computing platform 510a and the second cloud computing platform 510b may prevent the runtime of the data processing pipeline 250 from being determined with sufficient accuracy and consistency.

[0078] Reference again Figure 3AThe orchestrator node 230 of the data processing pipeline 250a may include a runtime estimator 320, as well as an optimizer 330 and a budget counter 340. In some example embodiments, the runtime estimator 320 may be configured to determine a runtime estimate for the data processing pipeline 250a executing one or more machine learning experiments, each of which includes a different type of machine learning model and / or a different set of experiment parameters. For example, the runtime estimator 320 may define checkpoints for collecting timing information to avoid inconsistencies introduced by different timing definitions across different cloud computing platforms (including, for example, the first cloud computing platform 510a, the second cloud computing platform 510b, etc.). The runtime estimator 320 may generate a runtime estimate for each individual machine learning experiment and for a process including a machine learning experiment. The runtime estimator 320 may further identify, based at least on the runtime estimates for the individual machine learning experiments, one or more machine learning experiments that can be executed without exceeding the available time budget. Before the orchestrator node 230 triggers execution of the one or more machine learning experiments at the executor node 280, the optimizer 330 may optimize the execution of the one or more machine learning experiments. Orchestrator node 230 may continue to trigger the execution of successive machine learning trials until budget counter 340 signals exhaustion of the available time budget.

[0079] In some example embodiments, timing differences between different cloud computing platforms (such as first cloud computing platform 510a and second cloud computing platform 510b) can be reconciled by defining one or more checkpoints, each of which corresponds to an event associated with the execution of a machine learning experiment. The runtime estimator 320 can determine the runtime of the machine learning experiment by at least collecting timing information associated with the one or more checkpoints and determining the amount of time that elapsed between two or more consecutive checkpoints. In this way, instead of relying on timing statistics provided by the first cloud computing platform 510a and / or the second cloud computing platform 510b, a runtime estimate for the data processing pipeline 250 can be determined based on a first time associated with a first checkpoint (corresponding to the time when the input dataset was received at the preparer node 240) and a second time associated with a second checkpoint (corresponding to the time when the corresponding training dataset and / or validation dataset generated by the preparer node 240 was sent to the executor node 280 for execution of the machine learning experiment corresponding to the data processing pipeline 250).

[0080] As described above, runtime estimator 320 may collect timing information associated with various checkpoints in order to determine the runtime associated with data processing pipeline 250, including, for example, the runtime of a machine learning experiment performed by executing data processing pipeline 250. To further illustrate, Figure 6A Depicted is a schematic diagram illustrating various components of runtime associated with a data processing pipeline 250, according to some example embodiments. Figure 6A As shown, the runtime associated with the data processing pipeline 250 may include non-computational time and computational time. The non-computational time may include setup time associated with provisioning a compute instance and starting execution of a machine learning experiment, which may correspond to the amount of time required to deploy and / or start the executor node 280. For example, the setup time may correspond to the amount of time that elapses between a first time when a set of experiment parameters is sent to the executor node 280 and a second time when the executor node 280 executes the corresponding machine learning experiment. The non-computational time may include input time for transferring a training dataset and / or a validation dataset from a central storage system (e.g., shared persistence 115) to a local compute instance including the experiment persistence 300. The computational time component of the runtime associated with the data processing pipeline 250 may include training and evaluation time, which may correspond to the amount of time required to perform computations associated with the machine learning experiment, including training the corresponding machine learning model and evaluating its performance. Compute time may also include output time, which may correspond to the amount of time required to write the results of the machine learning model, including the trained machine learning model, the output of the machine learning model, and corresponding log files to a central storage system such as shared persistence 115.

[0081] In order for the runtime estimator 320 to collect timing information associated with each checkpoint, the messages exchanged between the various nodes of the data processing pipeline 250 may be extended to include additional fields to accommodate the timing information. For example, Figure 6B Depicted is a timing diagram illustrating an example of an extended message exchange according to some example embodiments. Figure 6B As shown, orchestrator node 230 (e.g., orchestration engine 235 communicatively coupled to runtime estimator 320) may send first message 600a triggering execution of a machine learning experiment to executor node 280 before receiving second message 600b conveying the results of the machine learning model from executor node 280. First message 600a and / or second message 600b may be expanded to include timing information associated with the setup, input, training and evaluation, and output of the machine learning experiment to enable determination of the runtime associated with the machine learning experiment.

[0082] In some example embodiments, the runtime associated with the data processing pipeline 250 can be determined based on timing information associated with test trials performed at a cloud computing platform (e.g., the first cloud computing platform 510a or the second cloud computing platform 510b) using benchmark datasets covering various use cases and data characteristics (e.g., datasets with multiple rows or multiple features, difficult classification tasks, or multiple text columns to be preprocessed, etc.). To ensure reproducible results, the test trials can be performed using a predefined configuration and a fixed seed for the random number generator used during the optimization process. The test trials can be performed upon initial deployment to the cloud computing platform, at predetermined intervals, and / or in response to subsequent changes to the cloud computing platform (e.g., upgrades to computing resources, modifications to storage systems, etc.).

[0083] Alternatively and / or additionally, the runtime associated with the data processing pipeline 250 can be determined based on timing information collected from user trials. When there is insufficient timing information associated with the user trials, the timing information associated with the user trials can be supplemented by timing information associated with the test trials. However, to determine the runtime associated with the data processing pipeline 250, the runtime estimator 320 can be configured to prioritize timing information associated with the user trials and / or more recent timing information. For example, when determining the runtime of the data processing pipeline 250, the runtime estimator 320 can assign a lower weight to timing information associated with the test trials and / or less recent timing information.

[0084] In some example embodiments, the runtime estimator 320 may include one or more runtime estimation models trained based at least on timing information associated with test trials and / or user trials to determine the runtime of a process comprising one or more separate machine learning trials executed by executing the data processing pipeline 250. The one or more runtime estimation models may include a computational time model trained to determine a computational time component of the runtime associated with the data processing pipeline 250. Additionally, the one or more runtime estimation models may include a non-computational time model trained to determine a non-computational time component of the runtime associated with the data processing pipeline 250. Each of the one or more runtime estimation models may be, for example, a machine learning model implemented using the data processing pipeline 250 (or a different data processing pipeline). For example, a linear regression model, a neural network, a hybrid model, an rank-preserving regression model, etc.

[0085] Because the setup time can be constant and the input time can scale linearly with the size of the training dataset and / or validation dataset, a linear regression model can be used to implement a non-computational time model. In contrast, a more complex machine learning model that provides better flexibility and / or robustness can be used to implement the computational time model. For example, a hybrid model (e.g., a Gaussian mixture model, etc.) or a neural network that can provide not only scaling values ​​but also a probability distribution of runtimes can be used to implement the computational time model. Alternatively and / or additionally, an rank-preserving regression model that can implement specific constraints to ensure, for example, non-negative runtime estimates, monotonicity of runtime estimates with respect to dataset size, etc. can be used to implement the computational time model.

[0086] In some example embodiments, the runtime estimator 320 may be configured to generate a runtime estimate for a process comprising one or more machine learning trials to generate a machine learning model comprising a set of model parameters and hyperparameters for performing a task. The estimate may provide the runtime of the process to the client 120 (e.g., via the user interface 150), for example, so that the user 125 may allocate a sufficient time budget based on the runtime estimate of the process to generate an optimal machine learning model for the task associated with the input data set. The runtime estimate associated with each machine learning trial further enables identification of one or more machine learning trials that can be executed without exceeding the available time budget.

[0087] For example, runtime estimator 320 can determine a runtime estimate for a process of generating a machine learning model to perform a task. If user 125 at client 120 fails to allocate sufficient time budget for the process, user 125 can be alerted via user interface 150. Orchestrator node 230 can also refrain from executing data processing pipeline 250 to identify a machine learning model if the runtime required to identify the machine learning model exceeds the available time budget. Alternatively and / or additionally, if the runtime of a first machine learning experiment does not exceed the available time budget, but the runtime of a second machine learning model does exceed the available time budget, orchestrator node 230 can determine to execute the first machine learning experiment instead of the second machine learning experiment.

[0088] In some example embodiments, runtime estimator 320 may generate one or more runtime estimates based on at least an input dataset associated with a task (including, for example, object recognition, natural language processing, information retrieval, speech recognition, classification, regression, etc.). For example, upon receiving the input dataset, orchestrator node 230 may invoke preparer node 240 to perform a consistency check on the input dataset and determine corresponding task metadata, including, for example, task type, column type, target column, etc. To maximize efficiency, preparer node 240 may omit at least some of the pre-processing of the input dataset that is not required to generate the runtime estimates. Orchestrator node 230 may generate a hyperparameter space for the task based on at least the task metadata determined by preparer node 240. The task metadata data and the hyperparameter space for the task may be forwarded to runtime estimator 320 to determine runtime estimates for individual machine learning experiments and for processes including one or more machine learning models.

[0089] To determine a reliable runtime estimate for an input dataset and the corresponding task, a computation time model can be marginalized across various possible trial parameter sets. That is, runtime estimates for the input dataset and the corresponding task can be generated to account for possible variations in the trial parameter sets. A numerical approximation can be used to represent the marginalization of the computation time model. For example, the computation time model can be used to determine the computation time of a sample of trial parameter sets that follows a uniform or empirical distribution of trial parameter sets from one or more previous machine learning experiments. This sample can provide a distribution of computation times for calculating the average computation time across the various trial parameter sets. Alternatively and / or additionally, other information (such as percentiles) can be extracted from the distribution of computation times to calculate a more complex budget estimate. For example, the computation time can be determined by calculating the numerical convolution of the distribution of computation times and / or analytical convolution and extracting the mean (or different values) from the resulting distribution to produce a more accurate estimate in the case of a non-Gaussian distribution. Unlike the computational component of the runtime, the non-computational component of the runtime can be independent of the trial parameter set and can be calculated once based on the size of the input dataset using the non-computational time model.

[0090] In some example embodiments, runtime estimator 320 may determine a runtime estimate for one or more individual machine learning trials. Alternatively and / or additionally, runtime estimator 320 may determine a runtime estimate for a process that includes one or more machine learning trials to generate a machine learning model for performing a task, the machine learning model including a set of model parameters and hyperparameters for performing the task associated with an input data set. As described above, orchestrator node 230 may refrain from executing data processing pipeline 250 if the runtime estimate for the identified machine learning model exceeds the available time budget. Additionally, user interface 150 may be updated to display one or more time budget suggestions, including, for example, a suggested time budget for completing a process, a warning when user 125 fails to allocate sufficient time budget, and the like.

[0091] An estimate of the runtime for executing one or more machine learning trials to generate a machine learning model may correspond to the sum of the average compute time and the estimated non-compute time multiplied by a multiplier corresponding to the expected number of machine learning trials to generate the machine learning model, divided by a parallelization factor corresponding to the number of machine learning trials to be executed simultaneously (e.g., on multiple executor nodes 280). For example, if generating a machine learning model requires an average compute time of 25 seconds, a non-compute time of 5 seconds, and 50 machine learning trials, then an estimate of the runtime without parallelization may be 25 minutes. If five machine learning trials can be executed in parallel, then a wall-clock based estimate of the runtime may be 5 minutes.

[0092] In some example embodiments, the expected number of machine learning trials required to generate a machine learning model (including a set of model parameters and hyperparameters for performing a task associated with the input dataset) can be determined based on benchmark datasets covering various use cases and data characteristics (e.g., datasets with multiple rows or multiple features, difficult classification tasks, or multiple text columns to be preprocessed, etc.). For example, for each benchmark dataset, the i-th machine learning trial can produce optimal performance such that the machine learning model is identified by the i-th machine learning trial. A multiplier can be extracted from a distribution of various values ​​of i associated with the benchmark dataset. For example, the multiplier can be the 75th percentile (or another percentile) of the distribution, the median of the distribution, the mean of the distribution, etc. The value of the multiplier can be selected to ensure that the runtime estimate includes sufficient time for generating the machine learning model. It should be understood that using an index of machine learning trials rather than cumulative runtime can eliminate skew associated with the size of the input dataset, the complexity of the corresponding task, etc.

[0093] Executing the data processing pipeline 250 may include executing one or more machine learning experiments, each of the one or more machine learning experiments including a different type of machine learning model and / or a different set of experiment parameters. A machine learning model (including a set of model parameters and hyperparameters) for performing the task may be identified based on the machine learning experiment with the best performance. However, without an accurate or consistent run time estimate for each machine learning experiment, the user 125 may allocate an insufficient time budget and / or the available time budget may not be used effectively, such that the process or individual machine learning experiments may exceed the available time budget or an excessive amount of the allocated time budget remains unused.

[0094] In some example embodiments, a runtime estimate for a process and / or a runtime estimate for each machine learning trial included in the process may be provided (e.g., as a recommendation displayed via user interface 150 at client 120) so that user 125 can allocate a sufficient time budget. Furthermore, orchestrator node 230 may thereby identify, based at least on the runtimes of the respective machine learning trials, one or more machine learning trials that can be executed without exceeding the available time budget. In doing so, orchestrator node 230 may prevent execution of one or more machine learning trials whose runtime estimates exceed the available time budget. For example, orchestrator node 230 may identify a machine learning trial whose runtime does not exceed the available time budget as the next machine learning trial to be executed by executor node 280. However, it should be understood that the runtime estimates for machine learning trials may be associated with a certain degree of uncertainty, including a certain probability of exceeding the available time budget.

[0095] Figure 7A A flow chart illustrating a process 700 for generating a data processing pipeline configured to generate a machine learning model is depicted in accordance with some example embodiments. Figure 1 、 Figure 2A-2C 、 Figure 3A-3C 、 Figure 4-Figure 5 、 Figure 6A-Figure 6B and Figure 7A-7B , process 700 can be performed by the pipeline engine 110 to generate, for example, a data processing pipeline 250 configured to generate a machine learning model. The machine learning model generated by the data processing pipeline can be a machine learning model having a set of parameters and / or hyperparameters for performing cognitive tasks such as, for example, object recognition, natural language processing, information retrieval, speech recognition, classification, and / or regression.

[0096] At 702, the pipeline engine 110 may generate a user interface configured to receive one or more inputs for constructing a data processing pipeline to generate a machine learning model. For example, the pipeline engine 110 may generate a user interface 150 that may be configured to display a selection of operator nodes (including, for example, the orchestrator node 230, the preparer node 240, and the executor node 280) at the client 120. The selection of the operator nodes displayed at the client 120 as part of the user interface 150 may also include selecting one or more auxiliary operator nodes, including, for example, the start node 260, the user interface node 270, and the like. As part of the data processing pipeline, the start node 260 may be configured to receive input for configuring a process that includes one or more machine learning experiments, while the user interface node 270 may be configured to output the progress and / or results of the one or more machine learning experiments. Alternatively, instead of displaying a selection of operator nodes, the user interface 150 may display one or more dialog boxes prompting the user 125 to select one or more operator nodes to include in the data processing pipeline.

[0097] At 704, the pipeline engine 110 may respond to one or more inputs received from the client 120 by adding at least an orchestrator node, a preparer node, and an executor node to a graph representing the data processing pipeline, such that the data processing pipeline is configured to generate a machine learning model trained to perform a task. For example, the pipeline engine 110 may generate a graph representing the data processing pipeline 250 configured to generate a machine learning model. Figure 2B In the example shown, the data processing pipeline 250 can be constructed to include an orchestrator node 230, a preparer node 240, and an executor node 280. Figure 2B As shown, the data processing pipeline 250 can be constructed to include a start node 260 and a user interface node 270. As described above, the data processing pipeline 250 can be executed to generate a machine learning model for performing a task associated with an input data set. As part of the data processing pipeline 250, the start node 260 can be configured to receive input for configuring a process for generating the machine learning model, while the progress and results of the process can be output by the user interface node 270.

[0098] Orchestrator node 230, preparer node 240, start node 260, user interface node 270, and executor node 280 may be interconnected via one or more directed edges indicating a data flow between orchestrator node 230, preparer node 240, start node 260, user interface node 270, and executor node 280. For example, orchestrator node 230 and preparer node 240 may be interconnected via a first directed edge indicating that an output of orchestrator node 230 may be provided as an input to preparer node 240, and a second directed edge indicating that an output of preparer node 240 may be provided as an input to orchestrator node 230. Alternatively and / or additionally, orchestrator node 230 and executor node 280 may be interconnected via a third directed edge indicating that an output of orchestrator node 230 may be provided as an input to executor node 280, and a fourth directed edge indicating that an output of executor node 280 may be provided as an input to orchestrator node 230.

[0099] At 706, pipeline engine 110 may generate a corresponding data processing pipeline including an orchestrator node, a preparer node, and an executor node based on at least the graph. For example, in some example embodiments, pipeline engine 110 may generate data processing pipeline 250 including orchestrator node 230, preparer node 240, start node 260, user interface node 270, and executor node 280 based on at least the corresponding graph.

[0100] At 708, pipeline engine 110 may generate a machine learning model trained to perform the task by at least executing the data processing pipeline. For example, pipeline engine 110 may generate a machine learning model trained to perform the task by at least executing data processing pipeline 250 including orchestrator node 230, preparer node 240, executor node 280, start node 260, and user interface node 270. Executing data processing pipeline 250 may include executing one or more data processing operations associated with each of orchestrator node 230, preparer node 240, executor node 280, start node 260, and user interface node 270.

[0101] In some example embodiments, orchestrator node 230 may be executed to coordinate at least the operations of preparer node 240 and executor node 280. For example, orchestrator node 230 may respond to receiving an initial configuration for implementing a machine learning model from start node 260 by at least triggering generation of a training dataset and a validation dataset at orchestrator node 240. Upon receiving an indication from preparer node 240 that preparer node 240 has generated the training dataset and the validation dataset, orchestrator node 230 may trigger execution of one or more machine learning experiments at executor node 280, each of the one or more machine learning experiments applying a different type of machine learning model and / or different experiment set parameters to the training dataset and / or validation dataset generated by preparer node 240. Furthermore, orchestrator node 230 may be executed to determine a machine learning model including a set of model parameters and hyperparameters for performing a specified task based at least on the results of the machine learning experiments executed by executor node 280.

[0102] Figure 7B A flow chart illustrating an example of a process 750 for allocating a time budget for executing a machine learning data processing pipeline is depicted, according to some example embodiments. Figure 1 、 Figure 2A-2C 、 Figure 3A-3C 、 Figure 4-Figure 5 、 Figure 6A-6B and Figure 7A-7B Process 750 may be performed by orchestrator node 230 including, for example, runtime estimator 320 , as part of operation 708 of process 700 .

[0103] At 752, orchestrator node 230 may determine a first runtime for a process that generates a machine learning model for performing a task associated with an input dataset. In some example embodiments, runtime estimator 320 may determine a runtime estimate for a process that performs one or more machine learning trials to identify a machine learning model for performing the task. The runtime estimate for the process may correspond to the sum of the average compute time and the estimated non-compute time multiplied by a multiplier corresponding to an expected number of machine learning trials to identify the machine learning model, and divided by a parallelization factor corresponding to the number of machine learning trials to be executed in parallel (e.g., on multiple executor nodes 280). The number of machine learning trials required to identify a machine learning model may be determined based on benchmark datasets covering various use cases and data characteristics (e.g., datasets with multiple rows or multiple features, difficult classification tasks, or multiple text columns to be preprocessed, etc.).

[0104] In some example embodiments, the runtime estimate of a process may include non-computational time (e.g., setup time and input time) and computational time (e.g., training and evaluation time and output time). The runtime estimator 320 may apply a non-computational time model (e.g., a linear regression model) to determine the non-computational time of a machine learning experiment, and apply a computational time model (e.g., a complex machine learning model) to determine the computational time of a machine learning experiment. In addition, the average computational time and the estimated non-computational time may be determined based on timing information collected from test experiments (e.g., performed using a benchmark dataset) and / or actual user experiments. Test experiments may be executed upon initial deployment to the cloud computing platform, at predetermined time intervals, and / or in response to subsequent changes to the cloud computing platform (e.g., upgrades to computing resources, modifications to storage systems, etc.). When the timing information associated with a user experiment is insufficient, the timing information associated with the user experiment may be supplemented by the timing information associated with the test experiment. However, as described above, as more user experiments are executed to provide sufficient timing information, the timing information associated with the user experiment may be superseded and / or replace the timing information associated with the test experiment.

[0105] At 754, orchestrator node 230 may perform a process of identifying a machine learning model that performs a task associated with the input dataset based at least on the first runtime not exceeding the available time budget. For example, orchestrator node 230 may refuse to execute a process whose runtime estimate exceeds the available time budget. Alternatively and / or additionally, orchestrator node 230 may provide the runtime estimate of the process via user interface 150 at client 120 to enable user 125 to allocate a sufficient time budget for the process.

[0106] At 756, orchestrator node 230 may determine a second run time for the first machine learning trial and a third run time for the second machine learning trial included in the process. In some example embodiments, run time estimator 320 may determine a run time estimate for each machine learning trial in the process. The run time estimate for each machine learning trial may be determined based at least on the corresponding set of trial parameters. As described above, the process of generating a machine learning model may begin with a random set of trial parameters, but subsequent sets of trial parameters may be generated by combining results from earlier machine learning trials. For example, information-based optimization techniques (e.g., Bayesian optimization, etc.) may be applied to identify regions of the hyperparameter space that are most likely to include model parameters and hyperparameter sets. In this way, with each successive update, the sampling of the hyperparameter space may transition from uniform sampling to information-weighted sampling using a Bayesian (or other) approach.

[0107] At 758, in response to determining that the second runtime of the first machine learning trial does not exceed the available time budget but the third runtime of the second machine learning trial does exceed the available time budget, orchestrator node 230 may execute the first machine learning trial instead of the second machine learning trial. In some example embodiments, runtime estimator 320 may be configured to maximize the use of the available time budget, including by selecting a machine learning trial whose runtime estimate does not exceed the available time budget for execution by executor node 280. For example, given n number of trial parameter sets (e.g., sampled from a uniform distribution and / or generated by an information-based optimizer), orchestrator node 230 may eliminate one or more trial parameter sets whose runtime exceeds the available time budget. Thus, if the computation time associated with the first machine learning trial does not exceed the remaining time budget adjusted for non-computation time associated with the process, orchestrator node 230 may determine to execute the first machine learning trial. Conversely, if the computation time associated with the second machine learning trial exceeds the available time budget, orchestrator node 230 may avoid executing the second machine learning trial.

[0108] Figure 8 A block diagram illustrating a computing system 800 consistent with implementations of the current subject matter is depicted. Figure 1 、 Figure 2A-2C 、 Figure 3A-3C 、 Figure 4-Figure 5 、 Figure 6A-6B and Figure 7A-7B and Figure 8 , computing system 800 may be used to implement pipeline engine 110 and / or any components therein.

[0109] like Figure 8 As shown, computing system 800 may include a processor 810, a memory 820, a storage device 830, and an input / output device 840. Processor 810, memory 820, storage device 830, and input / output device 840 may be interconnected via a system bus 850. Processor 810 is capable of processing instructions for execution within computing system 800. Such executed instructions may implement, for example, one or more components of pipeline engine 110. In some example embodiments, processor 810 may be a single-threaded processor. Alternatively, processor 810 may be a multi-threaded processor. Processor 810 is capable of processing instructions stored in memory 820 and / or storage device 830 to display graphical information of a user interface provided via input / output device 840.

[0110] The memory 820 is a computer-readable medium, such as a volatile or non-volatile medium, that stores information within the computing system 800. For example, the memory 820 may store a data structure representing a configuration object database. The storage device 830 can provide persistent storage for the computing system 800. The storage device 830 may be a solid-state drive, a floppy disk device, a hard disk device, an optical disk device, a magnetic tape device, or other suitable persistent storage device. The input / output device 840 provides input / output operations for the computing system 800. In some example embodiments, the input / output device 840 includes a keyboard and / or a pointing device. In various embodiments, the input / output device 840 includes a display unit for displaying a graphical user interface.

[0111] According to some example embodiments, the input / output device 840 may provide input / output operations for a network device. For example, the input / output device 840 may include an Ethernet port or other network port to communicate with one or more wired and / or wireless networks (e.g., a local area network (LAN), a wide area network (WAN), the Internet).

[0112] In some example embodiments, the computing system 800 can be used to execute various interactive computer software applications that can be used to organize, analyze, and / or store data in various formats. Alternatively, the computing system 800 can be used to execute any type of software application. These applications can be used to perform various functions, such as planning functions (e.g., generating, managing, editing electronic spreadsheet documents, word processing documents, and / or any other objects), computing functions, communication functions, and the like. The application can include various plug-in functions or can be an independent computing product and / or function. When activated within the application, the function can be used to generate a user interface provided via the input / output device 840. The user interface can be generated by the computing system 800 and presented to the user (e.g., on a computer screen monitor, etc.).

[0113] One or more aspects or features of the subject matter described herein can be implemented in digital electronic circuits, integrated circuits, specially designed ASICs, field programmable gate arrays (FPGAs) computer hardware, firmware, software, and / or combinations thereof. These different aspects or features can include implementations in one or more computer programs executable and / or interpretable on a programmable system, the programmable system including at least one programmable processor, which can be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device. A programmable system or computing system can include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The relationship between the client and the server is generated by the computer program running on each computer, and has a client-server relationship with each other.

[0114] These computer programs may also be referred to as programs, software, software applications, applications, components or codes, including machine instructions for programmable processors, and may be implemented in high-level procedural languages ​​and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the term "machine-readable medium" refers to any computer program product, device and / or equipment (such as a disk, optical disk, memory and programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives a machine instruction as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor. A machine-readable medium (such as, for example, a non-transitory solid-state memory or a magnetic hard drive or any equivalent storage medium) may store such machine instructions non-transitorily. A machine-readable medium (such as, for example, a processor cache or other random access memory associated with one or more physical processor cores) may alternatively or additionally store such machine instructions in a transient manner.

[0115] To provide for interaction with a user, one or more aspects or features of the subject matter described herein may be implemented on a computer having a display device for displaying information to the user, such as, for example, a cathode ray tube (CRT) or a liquid crystal display (LCD) or a light emitting diode monitor (LED), and a keyboard and pointing device, such as, for example, a mouse or trackball, through which the user may provide input to the computer. Other types of devices may also be used to provide for interaction with the user. For example, the feedback provided to the user may be any form of sensory feedback, such as, for example, visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. Other possible input devices include touch screens or other touch-sensitive devices, such as single-point or multi-point resistive or capacitive trackpads, voice recognition hardware and software, optical scanners, optical pointers, digital image capture devices and associated interpretation software, and the like.

[0116] In the above description and claims, phrases such as "at least one of" or "one or more of" may appear after a combined list of elements or features. The term "and / or" may also appear in a list of two or more elements or features. Unless implicitly or explicitly contradicted by the context in which it is used, such phrases are intended to mean any one of the elements or features listed individually, or any combination of any one of the listed elements or features with any other listed elements or features. For example, the phrases "at least one of A and B," "one or more of A and B," and "A and / or B" are each intended to mean "only A, only B, or A and B together." A similar interpretation applies to lists of three or more items. For example, the phrases "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, and / or C" are each intended to mean "only A, only B, only C, A and B together, A and C together, B and C together, or A, B, and C together." In the above description and claims, the use of the term "based on" is intended to mean "based at least in part on," so that unlisted features or elements are also permissible.

[0117] Depending on the desired configuration, the subject matter described herein may be embodied in systems, devices, methods and / or articles. The implementations set forth in the foregoing description do not represent all implementations consistent with the subject matter described herein. Instead, they are merely some examples consistent with aspects related to the described subject matter. Although some variations have been described in detail above, other modifications or additions are possible. In particular, in addition to the features and / or variations set forth herein, further features and / or variations may also be provided. For example, the above implementations may be directed to various combinations and subcombinations of the disclosed features and / or combinations and subcombinations of several further features disclosed above. In addition, the logical flows depicted in the accompanying drawings and / or described herein do not necessarily require the specific order shown or to be performed in order to achieve the desired results. Other implementations may be within the scope of the appended claims.

Claims

1. A system comprising: at least one data processor; as well as at least one memory storing instructions that, when executed by the at least one data processor, result in operations comprising: receiving one or more inputs from a client device for constructing a data processing pipeline, the data processing pipeline configured to implement a process of generating a machine learning model for performing a task associated with an input data set, the process comprising a plurality of machine learning trials, the input data set comprising a plurality of rows and columns; wherein the data processing pipeline comprises a preparer node, wherein the preparer node is configured to generate a training dataset for training a machine learning model to perform a task based on at least the input dataset; each of the plurality of machine learning experiments applies a corresponding machine learning model and a corresponding set of experiment parameters to a training dataset, wherein the corresponding set of experiment parameters is identified by applying Bayesian optimization; wherein each trial parameter set includes one or more hyperparameters, wherein the values ​​taken by the hyperparameters during each machine learning trial are adjusted based on the number of columns in the input dataset and / or the number of unique values ​​on the columns in the input dataset; wherein Bayesian optimization starts with a random set of hyperparameters and combines the corresponding results to identify a region in the hyperparameter space that is most likely to include model parameters and hyperparameters associated with the machine learning model identified as being best suited for the task; determining a first runtime for a process of generating the machine learning model, wherein the first runtime is determined based at least on an expected number of machine learning trials required to generate the machine learning model and / or a number of multiple machine learning trials executed in parallel; and executing the process to generate the machine learning model based at least on the first runtime not exceeding an available time budget; The data processing pipeline includes an executor node configured to execute the multiple machine learning experiments, wherein the executor node is configured to evaluate the test results of the corresponding machine learning experiments and select a corresponding one of the test parameter sets based on whether the corresponding test result is better than other test results relative to a target metric, wherein the target metric is the accuracy of the corresponding machine learning model or the logarithmic loss of the corresponding machine learning model.

2. The system according to claim 1, wherein: The first runtime includes non-computational time associated with provisioning a compute instance and commencing execution of the plurality of machine learning experiments, and wherein the non-computational time is determined at least by applying a linear regression model configured to determine the non-computational time based at least on a size of the training dataset and / or a validation dataset.

3. The system according to claim 1, wherein: The first runtime includes computational time associated with training a machine learning model, evaluating performance of the machine learning model, and outputting the trained machine learning model, wherein the computational time is computed by applying a computational time model configured to provide a probability distribution of runtimes over different sets of trial parameters.

4. The system according to claim 3, wherein: The computational time model includes a machine learning model, a hybrid model, a neural network, an rank-preserving regression model, an instance-based model, a regularized model, a decision tree, a random forest, a Bayesian model, a clustering model, an association model, a dimensionality reduction model and / or an integrated model generated by executing the data processing pipeline.

5. The system according to claim 3, wherein: The computation time model is trained based on at least timing information collected from one or more test trials and / or user trials.

6. The system according to claim 5, wherein: Training of the computational time model prioritizes more recent timing information and / or timing information from user trials.

7. The system according to claim 5, wherein: The timing information is collected at one or more checkpoints, and wherein the timing information includes an amount of time that elapses between two or more consecutive checkpoints.

8. The system according to claim 1, wherein: The expected amount of machine learning experiments is determined as an amount of machine learning experiments performed to generate a machine learning model against one or more benchmark datasets, and wherein the one or more benchmark datasets cover different use cases and input datasets with different characteristics.

9. The system of claim 1 , further comprising: updating a user interface at the client device to display the first runtime; as well as An indication is received from the client device to allocate a time budget corresponding to the first runtime.

10. The system of claim 1, further comprising: determining a second run time of a first machine learning experiment included in the plurality of machine learning experiments, the second run time being determined based on at least a first experiment parameter set associated with the first machine learning experiment; determining a third run time for a second machine learning trial included in the plurality of machine learning trials, the third run time being determined based on at least a second trial parameter set associated with the second machine learning trial; and Based at least on the second runtime not exceeding the available time budget and the third runtime exceeding the available time budget, the first machine learning trial is executed but the second machine learning trial is not executed.

11. The system according to claim 10, wherein: Execution of the first machine learning experiment includes applying a first machine learning model having a first set of experiment parameters to the training dataset and the validation dataset.

12. The system of claim 11, further comprising: A third machine learning experiment is performed by at least applying the first machine learning model with the second set of trial parameters, the second machine learning model with the first set of trial parameters, or the second machine learning model with the third set of trial parameters to the training dataset and the validation dataset.

13. The system according to claim 1, wherein: The data processing pipeline includes an orchestrator node, a preparer node, and an executor node, wherein the preparer node is configured to generate a validation dataset for evaluating performance of a machine learning model trained to perform a task based on at least the input dataset, and wherein the orchestrator node is configured to identify a machine learning model for performing the task associated with the input dataset based on at least results of the plurality of machine learning trials.

14. The system according to claim 13, wherein: The machine learning model for performing a task is generated by at least executing the data processing pipeline based on a corresponding graphical representation of the data processing pipeline, the data processing pipeline including the orchestrator node, the preparer node, and the executor node.

15. The system of claim 1, wherein: The machine learning models include neural networks, regression models, instance-based models, regularization models, decision trees, random forests, Bayesian models, clustering models, association models, dimensionality reduction models and / or ensemble models.

16. A computer-implemented method comprising: receiving one or more inputs from a client device for constructing a data processing pipeline, the data processing pipeline configured to implement a process of generating a machine learning model for performing a task associated with an input data set, the process comprising a plurality of machine learning trials, the input data set comprising a plurality of rows and columns; wherein the data processing pipeline comprises a preparer node, wherein the preparer node is configured to generate a training dataset for training a machine learning model to perform a task based on at least the input dataset; each of the plurality of machine learning experiments applies a corresponding machine learning model and a corresponding set of experiment parameters to a training dataset, wherein the corresponding set of experiment parameters is identified by applying Bayesian optimization; wherein each trial parameter set includes one or more hyperparameters, wherein the values ​​taken by the hyperparameters during each machine learning trial are adjusted based on the number of columns in the input dataset and / or the number of unique values ​​on the columns in the input dataset; wherein Bayesian optimization starts with a random set of hyperparameters and combines the corresponding results to identify a region in the hyperparameter space that is most likely to include model parameters and hyperparameters associated with the machine learning model identified as being best suited for the task; determining a first runtime for a process of generating the machine learning model, wherein the first runtime is determined based at least on an expected number of machine learning trials required to generate the machine learning model and / or a number of multiple machine learning trials executed in parallel; and executing the process to generate the machine learning model based at least on the first runtime not exceeding an available time budget; Wherein, performing the process includes evaluating the test results of the corresponding machine learning test, and selecting a corresponding one of the test parameter sets based on whether the corresponding test result is better than other test results relative to a target metric, wherein the target metric is the accuracy of the corresponding machine learning model or the logarithmic loss of the corresponding machine learning model.

17. A non-transitory computer-readable medium storing instructions that, when executed by at least one data processor, result in operations comprising: receiving one or more inputs from a client device for constructing a data processing pipeline, the data processing pipeline configured to implement a process of generating a machine learning model for performing a task associated with an input data set, the process comprising a plurality of machine learning trials, the input data set comprising a plurality of rows and columns; wherein the data processing pipeline comprises a preparer node, wherein the preparer node is configured to generate a training dataset for training a machine learning model to perform a task based on at least the input dataset; each of the plurality of machine learning experiments applies a corresponding machine learning model and a corresponding set of experiment parameters to a training dataset, wherein the corresponding set of experiment parameters is identified by applying Bayesian optimization; wherein each trial parameter set includes one or more hyperparameters, wherein the values ​​taken by the hyperparameters during each machine learning trial are adjusted based on the number of columns in the input dataset and / or the number of unique values ​​on the columns in the input dataset; wherein Bayesian optimization starts with a random set of hyperparameters and combines the corresponding results to identify a region in the hyperparameter space that is most likely to include model parameters and hyperparameters associated with the machine learning model identified as being best suited for the task; determining a first runtime for a process of generating the machine learning model, wherein the first runtime is determined based at least on an expected number of machine learning trials required to generate the machine learning model and / or a number of multiple machine learning trials executed in parallel; and executing the process to generate the machine learning model based at least on the first runtime not exceeding an available time budget; Wherein, performing the process includes evaluating the test results of the corresponding machine learning test, and selecting a corresponding one of the test parameter sets based on whether the corresponding test result is better than other test results relative to a target metric, wherein the target metric is the accuracy of the corresponding machine learning model or the logarithmic loss of the corresponding machine learning model.

Images