Hyperparameter space optimization of machine learning data processing pipelines

Abstract

A data processing pipeline can be generated to include a coordinator node, a preparer node, and an executor node. The preparer node can generate a training dataset. The executor node can perform a machine learning trial by applying a machine learning model and / or a different set of trial parameters to the training dataset. The coordinator node can identify a machine learning model for performing a task based on results of the machine learning trial. Data associated with execution of the data processing pipeline can be collected for storage in a tracking database. A report including denormalized and enriched data from the tracking database can be generated. A hyperparameter space of the machine learning model can be analyzed based on the report. A root cause of at least one failure associated with execution of the data processing pipeline can be identified based on the analysis.

Description

Technical Field

[0001] This article describes topics that generally involve machine learning, and more specifically, the hyperparameter space of the data processing pipeline used to implement machine learning models. Background Technology

[0002] Machine learning models can be trained to perform various cognitive tasks, including object recognition, natural language processing, information retrieval, speech recognition, classification, regression, and so on. For example, an Enterprise Resource Planning (ERP) system may include an issue tracking system configured to generate tickets in response to errors reported via one or more phone calls, emails, SMS messages, social media posts, web chat, etc. The issue tracking system can generate tickets to include a textual description of the error associated with the ticket. Therefore, to determine the appropriate response for resolving the error associated with the ticket, the ERP system may include a machine learning model trained to perform text classification. For example, a machine learning model can be trained to prioritize tickets corresponding to the severity of the error, at least based on the textual description of the error. Summary of the Invention

[0003] Systems, methods, and artifacts, including computer program artifacts, are provided for optimizing the hyperparameter space of a machine learning data processing pipeline. In one aspect, a system comprising at least one data processor and at least one memory is provided. The at least one memory may store instructions that, when executed by the at least one data processor, cause operations. These operations may include: collecting data associated with the execution of the data processing pipeline for storage in a tracking database, the data processing pipeline being executed to generate a machine learning model having a set of hyperparameters for performing a task associated with an input dataset, the execution of the data processing pipeline including performing multiple machine learning trials, each of the multiple machine learning trials applying different types of machine learning models and / or different sets of trial parameters to a training dataset, and the machine learning model having the set of hyperparameters for performing the task being identified at least based on the results of the multiple machine learning trials; generating a report based on at least a portion of the data associated with the execution of the data processing pipeline; analyzing the hyperparameter space of the machine learning model based on at least a portion of the report; and identifying the root cause of at least one failure associated with the execution of the data processing pipeline, at least based on the analysis of the hyperparameter space.

[0004] In some variations, one or more of the features disclosed herein, including the following characteristics, may optionally be included in any feasible combination. A logical table can be generated by at least denormalizing the data associated with the execution of the data processing pipeline. Reports can be generated based on the logical table. Each row of the logical table may correspond to one of multiple machine learning experiments. Each column of the logical table may correspond to a value describing the multiple machine learning experiments, the corresponding experiments, and / or the results of the multiple machine learning experiments.

[0005] In some variations, multiple machine learning trials can be ranked based at least on a target metric. A column corresponding to the ranking of each machine learning trial included in the ranked trials can be added to a logical table.

[0006] In some variations, the relative deviation from the target metric associated with the validation and / or test datasets can be determined for each of multiple machine learning trials. A column corresponding to the relative deviation can be added to a logical table.

[0007] In some variations, reports can be generated by applying association rule algorithms to produce one or more association rules that link one or more hyperparameters of a machine learning model to the results of multiple machine learning experiments.

[0008] In some variations, association rules supported by a sub-threshold proportion of data associated with the execution of the data processing pipeline can be excluded from one or more association rules applied to generate a report.

[0009] In some variations, reports can be generated by applying interpretability techniques to calculate the impact of hyperparameters of the machine learning model on the target metric.

[0010] In some variations, at least one fault may include a combination of one or more hyperparameter values ​​with unexpected behavior, a machine learning model with below-threshold performance, and / or poor scaling behavior.

[0011] In some variations, one or more corrective actions corresponding to the root cause of at least one fault may be performed. These corrective actions may include removing hyperparameters, quantizing hyperparameters with continuous values, and / or limiting and / or rescaling the range of hyperparameters.

[0012] In some variations, the data associated with the execution of the data processing pipeline may include one or more task metadata, target performance metrics, and hyperparameter values.

[0013] In some variations, the data processing pipeline may include a coordinator node, a preparer node, and multiple executor nodes. The preparer node can be configured to generate a training dataset based at least on the input dataset. The multiple executor nodes can be configured to perform multiple machine learning experiments by applying different types of machine learning models and / or different sets of experiment parameters to the training dataset. The coordinator node can be configured to identify a set of machine learning models with hyperparameters for performing the task, based at least on the results of the multiple machine learning experiments.

[0014] In some variations, machine learning models may include neural networks, regression models, instance-based models, regularized models, decision trees, random forests, Bayesian models, clustering models, association models, dimensionality reduction models, and / or ensemble models.

[0015] On the other hand, a method is provided for optimizing the hyperparameter space of a machine learning data processing pipeline. The method may include: collecting data associated with the execution of the data processing pipeline to store in a tracking database, the data processing pipeline being executed to generate a machine learning model with a set of hyperparameters for performing a task associated with an input dataset, the execution of the data processing pipeline including executing multiple machine learning trials, each of the multiple machine learning trials applying different types of machine learning models and / or different sets of trial parameters to a training dataset, and the machine learning model with the set of hyperparameters for performing the task being identified at least based on the results of the multiple machine learning trials; generating a report based on at least a portion of the data associated with the execution of the data processing pipeline; analyzing the hyperparameter space of the machine learning model based on at least a portion of the report; and identifying at least one root cause of a failure associated with the execution of the data processing pipeline, at least based on the analysis of the hyperparameter space.

[0016] In some variations, one or more features disclosed herein, including the following characteristics, may optionally be included in any feasible combination. The method may also include: generating a logical table by at least denormalizing data associated with the execution of the data processing pipeline; the report is generated based on the logical table, each row of which corresponds to one of a plurality of machine learning experiments, and each column of which corresponds to a value describing the plurality of machine learning experiments, the corresponding experiments, and / or the results of the plurality of machine learning experiments.

[0017] In some variations, the method may further include: ranking multiple machine learning trials based at least on a target metric; and adding a column corresponding to the ranking of each machine learning trial included in the ranked multiple machine learning trials to a logical table.

[0018] In some variations, the method may further include: determining, for each of a plurality of machine learning trials, a relative deviation from a target metric associated with a validation dataset and / or a test dataset; and adding a column corresponding to the relative deviation to a logical table.

[0019] In some variations, reports can be generated by applying association rule algorithms to produce one or more association rules that link one or more hyperparameters of a machine learning model to the results of multiple machine learning experiments. Association rules supported by data below a threshold proportion associated with the execution of the data processing pipeline can be excluded from the one or more association rules applied to generate the report.

[0020] In some variations, reports can be generated by applying interpretability techniques to calculate the impact of hyperparameters of the machine learning model on the target metric.

[0021] In some variations, the method may further include: performing one or more corrective actions corresponding to the root cause of at least one fault, the one or more corrective actions including removing hyperparameters, quantizing hyperparameters with continuous values, and / or limiting and / or rescaling the range of hyperparameters.

[0022] In another aspect, a computer program product including a non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium may include program code that causes operations when executed by at least one data processor. These operations may include: collecting data associated with the execution of a data processing pipeline for storage in a tracking database, the data processing pipeline being executed to generate a machine learning model with a set of hyperparameters for performing a task associated with an input dataset, the execution of the data processing pipeline including performing multiple machine learning trials, each of the multiple machine learning trials applying different types of machine learning models and / or different sets of trial parameters to a training dataset, and the machine learning model with the set of hyperparameters for performing the task being identified at least based on the results of the multiple machine learning trials; generating a report based on at least a portion of the data associated with the execution of the data processing pipeline; analyzing the hyperparameter space of the machine learning model based on at least a portion of the report; and identifying the root cause of at least one failure associated with the execution of the data processing pipeline, at least based on the analysis of the hyperparameter space.

[0023] Implementations of the present subject matter may include methods consistent with the descriptions provided herein, as well as articles comprising machine-readable media operable to cause one or more machines (e.g., computers, etc.) to perform operations implementing one or more described features. Similarly, computer systems comprising one or more processors and one or more memories coupled to the one or more processors are also described. Memory that may include non-transitory computer-readable or machine-readable storage media may include, encode, store, etc., one or more programs that cause one or more processors to perform one or more operations described herein. Computer-implemented methods consistent with one or more implementations of the present subject matter may be implemented by one or more data processors residing in a single computing system or multiple computing systems. Such multiple computing systems may be interconnected and may exchange data and / or commands or other instructions via one or more connections (including, for example, connections to networks (e.g., the Internet, wireless wide area networks, local area networks, wide area networks, wired networks, etc.), direct connections between one or more of the multiple computing systems, etc.).

[0024] Details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features and advantages of the subject matter described herein will become apparent from the description and drawings and from the claims. Although certain features of the currently disclosed subject matter have been described for illustrative purposes with respect to machine learning data processing pipelines, it should be readily understood that such features are not intended to be limiting. The claims appended to this disclosure are intended to define the scope of the protected subject matter. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate certain aspects of the subject matter disclosed herein and, together with the description, help to explain some principles associated with the disclosed embodiments. In the drawings,

[0026] Figure 1A A system diagram illustrating a data processing pipeline generator system according to some example embodiments is depicted;

[0027] Figure 2A A schematic diagram illustrating an example of a data processing pipeline with modular pipeline elements according to some example embodiments is depicted;

[0028] Figure 2B Examples depicting graphs representing a data processing pipeline according to some example embodiments;

[0029] Figure 2C Examples of operator nodes configured to generate machine learning models are depicted according to some example embodiments;

[0030] Figure 3AA block diagram is depicted illustrating an example communication flow between operator nodes that are configured to generate machine learning models according to some example embodiments of a data processing pipeline.

[0031] Figure 3B A flowchart is depicted illustrating examples of data processing operations performed by executor nodes that form a data processing pipeline configured to generate machine learning models, according to some example embodiments.

[0032] Figure 3C An example of a user interface according to some example embodiments is depicted;

[0033] Figure 3D Another example of a user interface according to some example embodiments is depicted;

[0034] Figure 3E A schematic diagram illustrating an example of hyperparameter space optimization according to some example embodiments is depicted;

[0035] Figure 4A A sequence of diagrams illustrating examples of a process for generating a machine learning model trained to perform a task, according to some example embodiments;

[0036] Figure 4B A sequence diagram illustrating another example of a process for generating a machine learning model trained to perform a task, according to some example embodiments;

[0037] Figure 5A Examples of data for generating reports, according to some example embodiments, are described;

[0038] Figure 5B An example of hyperparameter space reporting according to some example embodiments is depicted;

[0039] Figure 5C A graph depicts an example of hyperparameters having several values ​​that result in the performance of multiple models, according to some example embodiments;

[0040] Figure 5D A graph depicting examples illustrating the relationship between execution time, classification encoding, and feature selection according to some example embodiments;

[0041] Figure 6A A graph depicting the average effect of hyperparameters associated with a random forest classifier according to some example embodiments is presented;

[0042] Figure 6B A graph depicting examples illustrating the relationship between hyperparameters and area under the curve measure of a classification model according to some example embodiments;

[0043] Figure 6CA graph depicting an example illustrating the relationship between the learning rate and the target metric of an example classification model according to some example embodiments;

[0044] Figure 7A A flowchart illustrating a process for generating a data processing pipeline configured to generate a machine learning model, according to some example embodiments, is depicted.

[0045] Figure 7B A flowchart illustrating an example of a process for hyperparameter space optimization according to some example embodiments is depicted; and

[0046] Figure 8 A block diagram illustrating a computing system according to some example embodiments is depicted.

[0047] When applicable, similar reference numerals indicate similar structures, features, or elements. Detailed Implementation

[0048] A data processing pipeline can include a series of operations for processing data, including collecting and / or manipulating data, such as exceptionally large and / or complex datasets known as "big data." A data processing pipeline can be graphically represented as multiple operator nodes interconnected by one or more directed edges in a directed graph. Each operator node can correspond to a data processing operation performed on the data traversed by that operator node. Simultaneously, the directed edges connecting two operator nodes can indicate the data flow between the data processing operations corresponding to each operator node. Therefore, a graph representing a data processing pipeline (such as a directed graph) can be constructed by interconnecting a series of operator nodes with at least one or more directed edges.

[0049] A graph representing a data processing pipeline can be constructed to include one or more operator nodes configured to generate machine learning models trained to perform a task. For example, the graph can be constructed to include a coordinator node, one or more preparer nodes, and one or more executor nodes. The coordinator node can be configured to coordinate the operations of the preparer nodes and one or more executor nodes. For example, each preparer node can be configured to generate training and validation datasets based at least on an input dataset associated with the task. Simultaneously, each executor node can be configured to apply different types of machine learning models and / or different sets of parameters to the training and validation datasets generated by the preparer nodes. The coordinator node can be configured to determine, at least based on the performance of different types of machine learning models, a machine learning model consisting of a set of model parameters and hyperparameters used to perform the task associated with the input dataset. Therefore, a machine learning model trained to perform a task can be generated by executing at least a data processing pipeline including a coordinator node, one or more preparer nodes, and one or more executor nodes.

[0050] In some example embodiments, one or more issues associated with the execution of a data processing pipeline may require the pipeline's configuration to undergo one or more optimizations. For example, a data processing pipeline may be optimized to avoid pauses during which it fails to generate results for machine learning experiments for an excessively long period. Alternatively and / or additionally, a data processing pipeline may be optimized to avoid excessive resource consumption, including, for example, lengthy experiment run times, peak memory consumption, large runtime artifacts, etc. In some cases, optimization may be necessary to improve unexpectedly poor results associated with a particular input dataset. Although rare, data processing pipelines may sometimes require optimization to avoid failures that prevent the pipeline from fully executing.

[0051] The configuration of a data processing pipeline can include a set of hyperparameters spanning a hyperparameter space. Each hyperparameter, which can be associated with different values, controls a certain aspect of the data processing pipeline. An example of a hyperparameter could be the depth of the trees included in a random forest model. It should be understood that the type of algorithm used by the operator nodes included in the data processing pipeline can itself be a hyperparameter. For example, whether to use Z-scaling or min / max scaling to normalize the input dataset could be one of the hyperparameters associated with the data processing pipeline. Therefore, optimizing a data processing pipeline can include determining the overall configuration of the data processing pipeline as well as the configuration of each operator node included in the data processing pipeline. This can be achieved by identifying the set of hyperparameters that produce optimal results relative to a defined target metric. However, the hyperparameter space may contain many hyperparameters, many of which are conditional hyperparameters that are only valid when they exist in certain combinations. Meanwhile, a relatively small number of hyperparameters may actually contribute to problems encountered during the execution of the data processing pipeline. The task of identifying the optimal set of hyperparameters can be challenging, especially when optimization requires identifying hyperparameters that cause rare and / or unreproducible problems.

[0052] In some example implementations, hyperparameter space optimization can be achieved by collecting data associated with the execution of the data processing pipeline (including, for example, runtime information, task metadata, experiment configuration, etc.). Structured reports based on this data can be generated to enable analysis of the hyperparameter space, thereby identifying the root causes of failures associated with the execution of the data processing pipeline, such as pauses, excessive resource consumption, poor results, and malfunctions. Reports can be generated based on a set schedule and / or in response to triggering events including user input requesting report generation. Corrective actions to address these failures can be determined based on at least a portion of the report.

[0053] Figure 1A A system diagram illustrating a data processing pipeline generator system 100 according to some example embodiments is depicted. (Reference) Figure 1A The data processing pipeline generator system 100 may include a pipeline controller 110. For example... Figure 1A As shown, the pipeline controller 110 and the client device 120 can be communicatively connected via network 130. The client device 120 can be a processor-based device, including, for example, a smartphone, tablet computer, wearable device, virtual assistant, Internet of Things (IoT) device, etc. Meanwhile, network 130 can be any wired and / or wireless network, including, for example, a Public Land Mobile Network (PLMN), Wide Area Network (WAN), Local Area Network (LAN), Virtual Local Area Network (VLAN), Internet, etc.

[0054] Client device 120 can interact with pipeline controller 110 to generate one or more data processing pipelines. For example... Figure 1A As shown, the pipeline controller 110 may be associated with a user interface 150, which is configured to receive one or more inputs at a client device 120 from a user 125 for editing a graph representing the data processing pipeline, and to output the progress and / or results of executing the data processing pipeline to the user 125 at the client device 120. In some example embodiments, one or more inputs may edit the graph representing the data processing pipeline to include one or more operator nodes configured to generate, at least based on the input dataset, a machine learning model trained to perform a task associated with the input dataset. For example, the graph representing the data processing pipeline may be edited to include a coordinator node, one or more preparer nodes, and one or more executor nodes.

[0055] In some example embodiments, a coordinator node can be configured to coordinate the operations of one or more executor nodes. Simultaneously, each executor node can be configured to apply different sets of different types of machine learning models and / or parameters to training and validation datasets generated by one or more preparer nodes based on the input dataset. Furthermore, the coordinator node can be configured to identify the machine learning model used to perform the task associated with the input dataset, at least based on its performance associated with different sets of different types of machine learning models and / or parameters. Therefore, a machine learning model trained to perform a task can be generated by executing at least a data processing pipeline including a coordinator node, one or more preparer nodes, and one or more executor nodes. For example, a data processing pipeline can be executed to generate a machine learning model trained to perform cognitive tasks such as object recognition, natural language processing, information retrieval, speech recognition, classification, and / or regression. The machine learning model can be any type of machine learning model, including, for example, neural networks, regression models, instance-based models, regularized models, decision trees, random forests, Bayesian models, clustering models, association models, dimensionality reduction models, ensemble models, etc.

[0056] To further illustrate, Figure 2A A schematic diagram illustrating an example of a data processing pipeline with modular pipeline elements according to some example embodiments is depicted. Figure 2A As shown, a data processing pipeline can include different combinations of elements for data preparation, feature engineering, feature selection, model training, ensemble, etc. Each element of the data processing pipeline can be associated with one or more hyperparameters. A machine learning model for performing a task associated with an input dataset can be identified by at least evaluating the performance of the data processing pipeline across different combinations of pipeline elements and hyperparameters. For example, an executor node can be configured to execute one or more machine learning trials, each corresponding to a different combination of pipeline elements and hyperparameters. Furthermore, a coordinator node can identify a machine learning model for performing a task associated with an input dataset based at least on the performance of one or more machine learning trials. When used herein, the “optimal” machine learning model for performing the task can refer to a type of machine learning model and the corresponding combination of parameters and hyperparameters that produces the best performance across one or more machine learning trials.

[0057] Figure 2B Examples depicting a graphical representation of a data processing pipeline 200 according to some example embodiments are provided. (Reference) Figure 2B The data processing pipeline 200 may include multiple operator nodes, such as a first operator node 210a, a second operator node 210b, a third operator node 210c, a fourth operator node 210d, and a fifth operator node 210e. Each of the first operator node 210a, the second operator node 210b, the third operator node 210c, the fourth operator node 210d, and / or the fifth operator node 210e may correspond to a data processing operation performed on the data that has passed through the operator node.

[0058] also, Figure 2B The first operator node 210a, the second operator node 210b, the third operator node 210c, the fourth operator node 210d, and / or the fifth operator node 210e are shown as interconnected by one or more directed edges. The directed edges can indicate the data flow between data processing operations corresponding to the operator nodes interconnected by the directed edges. For example, the first edge 220a can interconnect the first operator node 210a and the fourth operator node 210d to at least indicate that the output of the data processing operation corresponding to the first operator node 210a is provided as input to the data processing operation corresponding to the fourth operator node 210d. Alternatively and / or additionally, the second edge 220b interconnecting the second operator node 210b and the fourth operator node 210d can indicate that the output of the data processing operation corresponding to the fourth operator node 210d can be provided as input to the data processing operation corresponding to the second operator node 210b.

[0059] In some example embodiments, the data processing pipeline can be constructed to include one or more specific operator nodes to implement machine learning models trained to perform cognitive tasks such as object recognition, natural language processing, information retrieval, speech recognition, classification, and / or regression. Figure 2C Examples of operator nodes configured to generate machine learning models in a data processing pipeline 250, according to some example embodiments, are depicted. Figure 2C As shown, the data processing pipeline 250 can be constructed to include a coordinator node 230 and one or more preparer nodes, for example, preparer node 240. Furthermore, the data processing pipeline 250 can be constructed to include one or more executor nodes, including, for example, executor node 280. Additionally and / or alternatively, the data processing pipeline 250 can be constructed to include one or more auxiliary operator nodes, including, for example, a start node 260, a user interface node 270, and a graph terminator node 290. The start node 260 can receive initial configuration to generate, for example, a machine learning model specified by user 125 at client device 120. Meanwhile, the user interface node 270 can be configured to generate and / or update the user interface 150 to display the progress of the execution of the data processing pipeline 200 at client device 120. The graph terminator node 290 can be invoked to terminate the execution of the data processing pipeline 250.

[0060] In some example embodiments, preparer node 240 may be configured to validate and preprocess, for example, an input dataset received from client device 120. Furthermore, preparer node 240 may be configured to generate a training dataset and a validation dataset based at least on the input dataset. For example, the input dataset may include text associated with one or more errors reported to an issue tracking system. Preparer node 240 may validate the input dataset and terminate further processing of the input dataset in response to identifying one or more errors present in the input dataset. After validating the input dataset, preparer node 240 may preprocess the input dataset, including by removing invalid data rows and / or columns from the input dataset and encoding any text included in the input dataset. Preparer node 240 may divide the validated and preprocessed input dataset into a training dataset for training a machine learning model to perform text classification and a validation dataset for evaluating the performance of the trained machine learning model in performing text classification.

[0061] In some example embodiments, executor node 280 may be configured to perform one or more machine learning trials based on a training dataset and / or a validation dataset generated by preparer node 240. Each machine learning trial may include applying a machine learning model with a specific set of trial parameters to the training dataset and / or the validation dataset. The set of trial parameters may include one or more parameters of the machine learning model, for example, initial weights applied to the machine learning model before training. Furthermore, the set of trial parameters may include one or more hyperparameters of the machine learning model, including, for example, the learning rate (e.g., step size) of a neural network, the value of the constant k in a k-nearest neighbor clustering algorithm, the cost associated with a support vector machine, and sigma, etc. It should be understood that executor node 280 may be configured to perform a sequence of machine learning trials, each trial including different types of machine learning models and / or different sets of trial parameters. For example, executor node 280 may perform a first machine learning trial by applying at least a first machine learning model with a first set of trial parameters to the training dataset and / or the validation dataset. Executor node 280 may also perform a second machine learning trial by applying at least a first machine learning model with a second set of trial parameters or a second machine learning model with a third set of trial parameters to the training dataset and / or the validation dataset.

[0062] In some example embodiments, coordinator node 230 may be configured to coordinate the operations of preparer node 240 and executor node 280. Coordinator node 230 may respond to receiving an initial configuration for implementing a machine learning model from start node 260 by triggering the generation of training and validation datasets at least at preparer node 240. This initial configuration may include, for example, task type (e.g., classification, regression, etc.), target columns (e.g., columns in the training dataset corresponding to benchmark truth labels), target metric (e.g., accuracy), column names and / or types in the training and validation datasets, computational resource budget (e.g., maximum execution time, etc.), paths to output directories, paths to the training dataset, paths to the validation dataset, etc.

[0063] Upon receiving an indication from preparer node 240 that preparer node 240 has generated training and validation datasets, coordinator node 230 can determine a machine learning model, including a set of model parameters and hyperparameters, for performing the task associated with the input dataset (e.g., classifying text associated with an issue tracking system). For example, coordinator node 230 can determine the machine learning model, including a set of model parameters and hyperparameters, for performing the task associated with the input dataset by triggering the execution of one or more machine learning trials at executor node 280, each machine learning trial including different types of machine learning models and / or different sets of trial parameters. Coordinator node 230 can trigger an amount of machine learning trials at executor node 280 consistent with the computational resource budget specified in the initial configuration for implementing the machine learning model. For example, coordinator node 230 can trigger additional machine learning trials based at least on the amount of remaining computational resources sufficient to support the execution of additional machine learning trials.

[0064] In some example embodiments, the set of experimental parameters for machine learning experiments can be randomly selected from a hyperparameter space that includes parameters governing the configuration of the data processing pipeline 250 and the configuration of each node within the data processing pipeline 250. A machine learning model comprising a set of model parameters and hyperparameters for performing a task associated with the input dataset can be identified by applying information-based optimization techniques (e.g., a Bayesian hyperparameter optimizer, etc.). This can begin with a random set of experimental parameters and then incorporate the corresponding results to identify, within the available time budget, the region in the hyperparameter space most likely to include the set of model parameters and hyperparameters associated with the identified machine learning model as best suited for the task associated with the input dataset. For each successive update, sampling of the hyperparameter space can be transformed from uniform sampling (e.g., from a uniform distribution of experimental parameters to produce random sampling, where each experimental parameter has an equal probability of being sampled) to information-weighted sampling using Bayesian (or other) methods.

[0065] The execution of a particular type of machine learning model may depend on hyperparameters, such as, for example, the number of trees created when training a random forest model, the solver algorithm applied to train a multilayer perceptron model, etc. The corresponding hyperparameter space can include different types of hyperparameters, including, for example, categorical non-numeric values, quantizable integer values, quantizable continuous values, etc. Examples of categorical non-numeric values ​​include the solver algorithm for a multilayer perceptron model, a Boolean flag controlling the intercept fitting in a linear regression model, and switches that select a particular algorithm at each step of the data processing pipeline that implements a machine learning model. Quantizable integer values ​​can be restricted to certain multiples of the base values, such as the maximum depth of a random forest model (e.g., the range from the first value x to the second value y) or the number of trees in a random forest model described by a finite set of quantizations (e.g., {20, 30, 40, ..., 240, 250}). Similarly, quantizable continuous values ​​can also be restricted to certain multiples of the base values, such as the L1 penalty in a resilient network regression model (e.g., the range from the first value x to the second value y in multiples of the third value z) and the learning rate in an XGBoost model (e.g., the range in multiples of 10). -3 Multiples from 10 -3 Up to 1).

[0066] While sampling over the hyperparameter space is typically uniform, with values ​​having the same probability of occurrence, logarithmic scaling can be applied to some numerical hyperparameters, such as the learning rate. The effect of logarithmic scaling is to apply uniform sampling across the order of magnitude of the corresponding hyperparameter space. Therefore, 10 -4 The learning rate may be related to the 10^6 logarithmic scaling sampling. -1 The learning rate is the same, while using general uniform sampling, 10 -1 The learning rate is likely to be much smaller. In some cases, the hyperparameter space can be further expanded by rules that exclude certain combinations of parameter values. For example, unlike other types of machine learning models, multilayer perceptron models may always require numerical normalization to obtain good results. Therefore, it is possible to prevent the hyperparameter space from including both denormalization of numerical values ​​and combinations of parameters of the multilayer perceptron model. Figure 2A In this context, doing so may be equivalent to excluding Figure 2A Some combinations of the pipe elements shown.

[0067] Executor node 280 can execute a first machine learning experiment including a first machine learning model with a first set of experimental parameters and a second machine learning experiment including a first machine learning model with a second set of experimental parameters or a second machine learning model with a third set of experimental parameters. Therefore, coordinator node 230 can select a machine learning model, including a set of model parameters and hyperparameters, for performing the task associated with the input dataset, based at least on the corresponding performance of the first machine learning model with the first set of experimental parameters, the first machine learning model with the second set of experimental parameters, and / or the second machine learning model with the third set of experimental parameters. When selecting a machine learning model, including a set of model parameters and hyperparameters, for performing the task associated with the input dataset, coordinator node 230 can terminate the execution of data processing pipeline 250, including by sending a message to graph terminator node 290 to terminate the execution of data processing pipeline 250.

[0068] As described above, executor node 280 can execute the first machine learning experiment and the second machine learning experiment sequentially. However, it should be understood that the data processing pipeline 250 can be configured to include multiple executor nodes, and coordinator node 230 can coordinate the multiple executor nodes to execute multiple machine learning experiments in parallel. Furthermore, the data processing pipeline 250 can be configured to include multiple preparer nodes, and coordinator node 230 can coordinate the multiple preparer nodes to generate input datasets and validation datasets in parallel.

[0069] In some example embodiments, coordinator node 230, preparer node 240, and executor node 280 can communicate via one or more messages. However, these messages may not include intermediate data, such as training and validation datasets generated by preparer node 240. Instead, coordinator node 230, preparer node 240, and executor node 280 can exchange intermediate data via shared persistence 115 accessible to coordinator node 230, preparer node 240, and executor node 280. For example, coordinator node 230 may store at least a portion of the initial configuration of the input dataset associated with the specified task in shared persistence 115. Similarly, preparer node 240 may store training and validation datasets generated based on the input dataset in shared persistence 115. Instead of directly sending the input dataset, training dataset, and validation dataset, coordinator node 230, preparer node 240, and executor node 280 can exchange the input dataset, training dataset, and / or validation data by at least sending messages indicating the availability of the input dataset, training dataset, and / or validation dataset in shared persistence 115. Therefore, preparer node 240 can access shared persistence 115 to obtain input datasets in order to generate training and validation datasets, while executor node 280 can access shared persistence 115 to obtain training and validation datasets for use during the execution of one or more machine learning experiments.

[0070] Figure 3A A block diagram depicts an example communication flow between operator nodes of a data processing pipeline 250 configured to generate machine learning models according to some example embodiments. (Reference) Figure 3A Coordinator node 230, preparer node 240, and executor node 280 can exchange intermediate data by at least accessing experimental persistence 300, which can form Figure 1A This is part of the shared persistence 115 shown. (As shown) Figure 3A As shown, user 125 at client device 120 can interact with user interface 150 to specify an initial configuration for a machine learning model to perform tasks such as object recognition, natural language processing, information retrieval, speech recognition, classification, and / or regression via application programming interface 310. In response to receiving the initial configuration of the machine learning model, coordinator node 230 can trigger the generation of a training dataset at preparer node 240 to train the machine learning model to perform the specified task and to generate a validation dataset to evaluate the performance of the trained machine learning model on the specified task. Preparer node 240 can store the training and validation datasets in experimental persistence 300. Furthermore, preparer node 240 can send a first message to coordinator node 230 notifying it of the availability of the training and validation datasets.

[0071] In response to a first message from preparer node 240, coordinator node 230 may send a second message to executor node 280 to trigger the execution of one or more machine learning experiments. For example, executor node 280 may respond to the second message by executing at least a first machine learning experiment comprising a first machine learning model with a first set of experimental parameters and / or a second machine learning experiment comprising a first machine learning model with a second set of experimental parameters or a second machine learning model with a third set of experimental parameters. executor node 280 may also store in experiment persistence 300 the results of machine learning experiments, for example, corresponding to the respective performance of the first machine learning model with the first set of experimental parameters, the first machine learning model with the second set of experimental parameters, and / or the second machine learning model with the third set of experimental parameters. To identify a machine learning model comprising a set of model parameters and hyperparameters used to perform a specified task, coordinator node 230 may at least access experiment persistence 300 to evaluate the results of the machine learning experiments, for example, relative to a target metric specified by user 125 as part of the initial configuration of the machine learning model.

[0072] A data processing pipeline 250, including a coordinator node 230, a preparer node 240, and an executor node 280, can be executed to perform feature extraction, feature preprocessing, and training of a machine learning model. For example, feature extraction can be performed to generate numerical features based on one or more columns of data from an input dataset, including encoding categorical features and / or extracting values ​​from data fields in each column. Feature preprocessing may include normalizing the values ​​occupying one or more columns in the input dataset. Therefore, a machine learning model can be trained at least by applying it to the numerical columns generated through feature extraction and / or feature preprocessing.

[0073] In some example embodiments, the data processing pipeline 250 can be configured to dynamically adapt based on metrics and / or configurations of the input dataset. Furthermore, the data processing pipeline 250 can dynamically adapt based on one or more previous processing operations within the data processing pipeline 250. Therefore, some operations can be omitted from the data processing pipeline 250 to reduce operational costs, minimize training time, and improve the accuracy of the resulting machine learning model. For example, whether executor node 280 performs feature selection can depend on the amount of available features. That is, if more than a threshold amount of features are available, executor node 280 can perform feature selection. Executor node 280 can also avoid subjecting embedded columns to any additional processing to avoid distorting the embedding space. If the input dataset does not include any columns with text data, executor node 280 can omit any text encoding, thereby reducing the hyperparameter space. Furthermore, executor node 280 can exclude one or more columns from the input dataset that are determined to have information values ​​below a threshold, for example, columns with target cross-entropy below a threshold.

[0074] In some example embodiments, the quantization of hyperparameters can be adjusted based on metrics of the input dataset, including, for example, the number of columns and / or the number of unique values ​​among columns containing certain data types. When used herein, “quantization” of hyperparameters can refer to discrete values ​​that the hyperparameters might take during each machine learning experiment. For example, if a hyperparameter column sampling rate of 10% and 12% produces substantially the same results, the hyperparameter column sampling rate could vary in 20% increments for each machine learning experiment.

[0075] Figure 3B A flowchart illustrating examples of data processing operations performed by an executor node 280 forming a data processing pipeline 250 configured to generate a machine learning model, according to some example embodiments, is depicted. In some example embodiments, the executor node 280 may respond to a second message from the coordinator node 230 at least by accessing shared persistence 115 to obtain a training dataset and / or validation dataset generated by the preparer node 240. The executor node 280 may perform a sequence of data processing operations on the training dataset and / or validation dataset, wherein each data processing operation applies a different transformation to the training dataset and / or validation dataset. Figure 3B As shown, executor node 280 can perform data processing operations, such as column selection, feature selection, text encoding, classification encoding, interpolation, normalization, and classification.

[0076] In some example embodiments, the executor node 280 performing a single machine learning experiment can generate a corresponding candidate machine learning model with a specific set of parameters and / or hyperparameters. The executor node 280 can store the candidate machine learning model in shared persistence 115 (e.g., experimental persistence 300). Furthermore, the executor node 280 can send the results of the machine learning experiment to the coordinator node, which can correspond to the performance of the candidate machine learning model on a validation dataset. For example, the executor node 280 can store the results of the machine learning experiment in shared persistence 115 (e.g., experimental persistence 300), allowing the coordinator node 230 to access shared persistence 115 (e.g., experimental persistence 300) to obtain the results of the machine learning experiment. As noted, the coordinator node 230 can access shared persistence 115 (e.g., experimental persistence 300) to evaluate the results of one or more machine learning experiments and identify a machine learning model, including the set of model parameters and hyperparameters, for performing the task specified by user 125 at client device 120.

[0077] Figure 3C -D depicts an example of a user interface 150 according to some example embodiments. For example... Figure 3C As shown in -D, the user interface 150 can be updated to display the progress and results of one or more machine learning experiments at the client device 120. For example, the user interface 150 can be updated to display model accuracy, calibration curves, confusion matrix, importance of each feature (e.g., the relevance of each column in the training dataset of the machine learning model), etc., at the client device 120. Figure 3C In the example of the user interface 150 shown, progress and results associated with various types of machine learning models can be sorted to identify one or more machine learning models with the best results. Figure 3D An example of a user interface 150 is depicted that displays the progress and results of a single type of machine learning model (e.g., an XGBoost classifier).

[0078] In some example embodiments, optimization of the hyperparameter space can be achieved through a trace database 315 and a reporting engine 325. The trace database 315 may be a relational database communicatively connected to the pipeline controller 110. During the experimental phase, the trace database 110 can be centrally used and populated with data about the hyperparameters. To enhance security and privacy, the trace database may be strictly independent of experimental persistence 300, which may be a client-specific database containing experimental results accessible to a single client. Conversely, the trace database 315 may be a central database from which the pipeline controller 110 collects anonymized experimental data.

[0079] Reporting engine 325 may be part of pipeline controller 110. Reporting engine 325 may be configured to analyze experimental results and generate complex hyperparameter space reports. To enhance privacy and security, access to reporting engine 325 may be restricted to certain users, such as developers. As noted, during the execution of data processing pipeline 250, various components of pipeline engine 110, such as coordinator node 230 and executor node 280, may exchange messages to convey status information. For example, coordinator node 230 may send one or more messages to executor node 280 containing task metadata, experiment configuration, and trial configuration (e.g., hyperparameter values). Executor node 280 may send one or more messages to coordinator node 230 containing the results of each machine learning experiment (including, for example, timing information, performance metrics such as accuracy, etc.).

[0080] Coordinator node 230 can be configured to collect information capable of hyperparameter space analysis and optimization for storage in tracking database 315. This information may include task metadata, experiment configurations, trial configurations, and performance metrics. Coordinator node 230 can avoid collecting confidential information. Instead, coordinator node 230 can collect personally identifiable information from each user's dataset, such as the number of rows, the number of categorical features, etc. Reporting engine 325 can therefore analyze the collected data based on standard product terms and conditions. Furthermore, reporting engine 325 can perform analysis across different systems (e.g., different production systems) and on a set of predefined benchmark datasets running the pipeline controller 110. This allows reporting engine 325 to collect insights from multiple scenarios simultaneously.

[0081] In some example embodiments, the data collected by coordinator node 230 for tracking database 310 may need to be formatted in some way to generate reports. This can be done at tracking database 310, for example, by denormalizing the data. For example, data collected by coordinator node 230, including task metadata, experiment configurations, and target performance metrics, can be added to a single logical table. In the resulting database view of the data, each row may correspond to a single machine learning experiment, and the columns within each row may include values ​​describing the machine learning experiment, the corresponding experiment, and the machine learning experiment result. Table 1 below depicts an example of denormalized data, including task metadata, experiment configurations (e.g., with hyperparameter values), and experiment results. It should be understood that the table can contain many more columns, especially to accommodate the large amount of task metadata and experiment configurations associated with the actual machine learning pipeline.

[0082] Table 1

[0083]

[0084] In some example embodiments, the database capabilities of tracking database 315 can be used to enrich denormalized data, for example, by adding labels and other derived quantities. Examples of enriching target information include adding rankings to metrics within experiments with different datasets to make the metrics comparable (e.g., sorting machine learning experiments associated with the experiment by the target metric and adding a new column indicating the ranking of each experiment). Enrichment may also include determining the relative deviation of the target metric (e.g., area under the curve) on the validation dataset and the same target metric on the test dataset to determine a quantity indicating an overfitting trend. That is, the target metric can be determined for both the validation and target datasets, and the deviation between these two values ​​can be used to detect overfitting.

[0085] In some cases, enrichment can include labeling one or more machine learning trials included in the denormalized data. For example, a label (e.g., the binary value of a column in the denormalized data) can be added to each trial that performs worse than the result of a majority-voting model, which is an information-theoretic worst-case reasonable model. For each trial, it should be understood that the majority-voting model can be the first machine learning trial performed during training. For classification tasks, the majority-voting model can always predict the majority class with the probability seen in the training dataset. For regression tasks, depending on the chosen target metric, the majority-voting model can predict the mean or median of the distribution of target values ​​in the training dataset.

[0086] In some example embodiments, a flag can be added to each machine learning trial with the worst training runtime (e.g., training runtime in the longest 10th percentile (or another configurable percentile)) to identify the machine learning model with the longest training time. Alternatively and / or additionally, a flag can be added to each machine learning trial with the worst inference time (e.g., inference runtime in the longest 10th percentile (or another configurable percentile)) to identify the machine learning model that takes the longest time to perform inference on the validation dataset. One or more flags can also be added to each machine learning trial with the worst objective metric (e.g., objective metric in the lowest 10th percentile (or another configurable percentile)) to identify the model with the worst accuracy. For example, a flag can be added for each of the three task types, including binary classification, multi-class classification, and regression. In some cases, a flag can be added to each machine learning trial that ends with an error (such as an anomalous termination of the model training on the training dataset or performing inference on the validation dataset).

[0087] In some example embodiments, in addition to (or instead of) the aforementioned enrichment, the denormalized data in the tracking database 315 may be enriched and / or filtered based on, for example, one or more user inputs received from the client device 120. For example, the denormalized data may be enriched with flags specified by the user input. Alternatively and / or additionally, the denormalized data may be filtered to limit the analysis of the hyperparameter space to one or more specific versions of the pipeline controller 110, machine learning experiments with certain characteristics (e.g., performing only regression tasks or machine learning experiments with at least two numerical columns), or machine learning experiments from a specific computing environment (e.g., certain production systems or benchmark results from a build system).

[0088] In some example embodiments, the reporting engine 325 may generate a report based at least on denormalized data from the tracking database 315. The report may further be generated based on a target variable specified by one or more user inputs from the client device 120. In some cases, the report may be generated in response to a request from the client device 120. However, it should be understood that report generation may also be automatic, at fixed time intervals, or triggered by certain events (e.g., new releases). Furthermore, reports may be generated based on association rules or interpretability capabilities with intermediate models.

[0089] Association rules can be used to analyze the impact of classification hyperparameters and classification task metadata on binary objective information, such as labels added as part of enriching denormalized data. Doing so allows analysis of the effects associated with the selection of certain algorithms in the machine learning pipeline (e.g., model selection, imputation methods for missing values, or task type), and can identify pipeline elements associated with poor results or above-average error rates.

[0090] To generate reports based on association rules, Report Engine 325 can remove non-classifiable features from denormalized data. Examples of non-classifiable features include numerical features such as the number of rows or hyperparameters of the maximum depth of a random forest. Report Engine 325 can apply prior algorithms (or similar algorithms such as ECLAT, FP-growth, etc.) to generate a set of rules that associate categorical features with the target information. Table 2 depicts examples of the resulting rules.

[0091] Table 2

[0092]

[0093] As shown in Table 2, each rule can include one or more conditions (left side) that lead to one or more conditions (right side). For example, a flag with a value of "1" can identify machine learning experiments with poor results. Therefore, the last rule in the example shown in Table 2 indicates that when using a multilayer perceptron model as a prediction algorithm, poor results may occur if the machine learning pipeline is executed without standardizing the numerical values.

[0094] In some example embodiments, the association rule algorithm can be configured to generate an appropriate number of the most useful association rules. For example, the association rule algorithm can impose a threshold on the minimum support required for each rule. That is, each association rule must be associated with a percentage of denormalized data (e.g., a certain percentage of rows) of a threshold proportion. This threshold can be set below the proportion of positive signs. Thus, if 4% of all machine learning trials produce positive signs, the minimum support threshold can be set to no more than 2%. Alternatively and / or additionally, the association rule algorithm can impose a threshold on the amount of conditions associated with each rule. This threshold can be set to 1, but can be increased to 2 if there are no rules with fewer than two conditions.

[0095] In some cases, the report generator 325 can filter association rules that fail to extract the relationship between hyperparameters and the presence of labels associated with the target metric. In doing so, the report generator 325 can exclude association rules that do not explain why a label appears (or does not appear). Therefore, a condition affirming a label can be a condition on the right-hand side of an association rule. This also means that the remaining association rules will have categorical feature conditions on the left-hand side. Table 3 depicts examples of such association rules.

[0096] Table 3

[0097]

[0098] When filtering association rules, the report generator 325 can rank the association rules based on the likelihood of each rule in interpreting the results of the machine learning pipeline. The ranking of association rules can be based on a ranking rule that first considers the complexity of each rule, quantified by the number of conditions on its left side, since a simpler rule with fewer conditions on its left side may be superior to a more complex rule with more conditions. When two rules have the same complexity, the ranking considers the importance of each rule (in descending order), quantified by the frequency of "boosting" or the association rule being true. Finally, when two rules have the same complexity and importance, they can be ranked (in descending order) based on the support associated with each association rule. Support can indicate the frequency of conditions appearing on the left side of the association rule. Since the right side of an association rule is fixed, support can be correlated with confidence. To further illustrate, Table 4 below depicts examples of association rules that have been ranked based on complexity, importance, and support.

[0099] Table 4

[0100]

[0101] Instead of association rules, the reporting engine 325 can also generate reports based on the interpretability of intermediate models. For detailed evaluations of excessive resource consumption or poor results, the reporting engine 325 can use the entire set of denormalized data (e.g., task metadata, complete trial configuration, and target metrics) as input. Target metrics can include one or more numeric values ​​specifying the expected accuracy and / or training time of the machine learning model. Alternatively, target metrics can include one or more Boolean flags indicating errors, such as the machine learning model providing poor results or excessive training time.

[0102] Reporting engine 325 can select different subsets of the data and train a random forest model as an intermediate model on each subset to predict the target metric for each machine learning trial within that subset. For example, reporting engine 325 can construct a random forest model that takes all hyperparameter values ​​as input and predicts whether a given machine learning trial exhibits excessive runtime. This random forest model is not used to predict machine learning trials with new hyperparameter values. It should be understood that this intermediate model is configured to streamline the logical relationship between the prepared data and the target metric for subsequent extraction and utilization.

[0103] To obtain concise information, the reporting engine 325 can apply interpretability or explainability methods configured to generate black-box models. For example, the reporting engine 325 can apply Shapley Additive Explanations (SHAP) to extract information and generate corresponding reports. While interpretability techniques typically explain the predictions of machine learning models and rate the importance of various input features, in some example embodiments, interpretability techniques such as Shapley Additive Explanations can be reused to extract logical connections included in trained random forest models, discarding individual explanations from intermediate models. It should be understood that the Shapley Additive Explanation library can apply game theory methods to explain the output of machine learning models, including linking optimal credit allocation with local explanations using classic Shapley values. Therefore, Shapley Additive Explanations is a solution concept derived from cooperative game theory. Furthermore, Shapley Additive Explanations can provide high-speed, accurate algorithms for tree ensemble methods, such as the random forest-based intermediate models generated by the reporting engine 325. Since machine learning models are typically trained on some training data, libraries such as Shapley additive interpretations (or alternative interpretations such as Locally Interpretable Model-Independent Interpretations (LIME)) can be applied to explain how the trained machine learning model arrives at specific results in order to make predictions on previously unseen data. Traditionally, interpretability methods have not been used to extract the logical connections of machine learning models trained solely to extract such connections.

[0104] The Shapley additive interpretation library can compute the impact of individual features (such as hyperparameters) on intermediate models predicting target information (e.g., the accuracy or training runtime of a machine learning model). For all machine learning trials in a dataset, the library can further provide visualizations of this impact as a function of feature values, such as possible hyperparameter values. Since features can interact with each other, the Shapley additive interpretation library supports visualization modes where features are color-coded to reveal interactions between different features. Furthermore, the impact of multiple features (e.g., hyperparameters) can be aggregated by averaging the absolute impact of each feature on each entry in the input dataset. This information is evaluated to determine the impact of each hyperparameter on the defined target information.

[0105] To support different types of tasks (e.g., binary classification, multi-class classification, regression, etc.) with dedicated algorithms and target metrics (e.g., accuracy for classification tasks and mean absolute error for regression tasks), the reporting engine 325 can run analyses for each type of task separately. Therefore, the reports output by the reporting engine 325 can include separate sections for each type of task. In cases where a type of task is not associated with any machine learning experiment, for example, due to filters indicated by one or more user inputs, the corresponding section of the report can remain blank.

[0106] Figure 5A Table 500 depicts an example of the data used to generate the report. (Example...) Figure 5A As shown, each part of the report can be further segmented using different portions of the input data to interpret the relationship with the target metric. For example, the report structure can conform to the type of task, thus using row-based data segmentation (see numbers 1 and 2 in Table 500). For sub-sections of the report, the data can be divided into subsets of columns using features from the task metadata (see number 3 in Table 500), features representing hyperparameters (see number 4 in Table 500), or features associated with the task metadata and features associated with hyperparameters (see number 5 in Table 500) to predict the target metric. In some cases, the report can also be generated using data corresponding to features representing switches controlling the pipeline layout (e.g., hyperparameters for selecting algorithm modules). Reports are generated in this way at least because the characteristics of the dataset in the task metadata and the trial configuration can exhibit a high degree of correlation, which often leads to confusing results. For example, if categorical variables are present, categorical coding features may be effective, in which case, even if the actual cause of the error is the presence of categorical coding modules, Shapley Additive Interpretation (SHAP) might identify the number of categorical variables in the task metadata as the root cause of the error. Null features (e.g., any hyperparameters associated with classification codes in the absence of any classification input variables) are automatically excluded from subsequent analysis.

[0107] The report generator 325 can create a structured report for each subset following the process described above. The report is generated and provided electronically as a structured document (e.g., an HTML document, a PDF file, etc.). The report may include a set of charts and a corresponding set of numerical tables that provide numerical values ​​for the visualizations in the charts (if available). The charts and tables can describe the importance and impact of all features (including hyperparameters and task metadata entries) relative to the defined target metrics.

[0108] Figure 5B An example of report 550 is described according to some example embodiments. Figure 5BThe example of Report 550 shown includes a bar chart summarizing the importance of each input feature (e.g., hyperparameters or task metadata features) in the studied subset relative to the target metric (e.g., accuracy or training time). Report 550 also includes a scatter plot summarizing the importance and distribution of each input feature in the studied subset relative to the target metric. Furthermore, Report 550 includes a scatter plot visualizing the effects of the primary variable (x-axis) and secondary variables (colors). This scatter plot is described in more detail in... Figure 5C As shown in the image. Figure 5C As shown, the scatter plot depicts examples of hyperparameters with several values ​​that produce various model performances. It's worth noting that the secondary axes, encoded with different colors, indicate a good interaction between the maximum number of features and the number of trees to be used when creating a random forest model. Therefore, depending on the value of the maximum number of features, the secondary parameters may improve or degrade the model's performance.

[0109] Figure 5D Another example of a scatter plot is shown, which provides a visualization of the relationship between the target metric execution time and the classification encoding (main x-axis) and feature selector (color). Figure 5D The scatter plot shown in the figure illustrates that both hash encoder modules and one-hot encoder modules can lead to long training execution times, and are therefore good candidates for further performance analysis. Figure 5D The “ shown <n a>The value can correspond to an experiment that does not have a corresponding pipeline module. That is, the dataset may lack classification features or feature selection operations.

[0110] In some example implementations, the reported content can be used to identify and perform various corrective actions. Examples of corrective actions include identifying hyperparameter values ​​with unexpected behavior, removing certain hyperparameters, quantizing continuous hyperparameter values, limiting and rescaling parameter ranges, identifying unfavorable algorithm combinations, detecting poor scaling behavior, and identifying pipeline modules with technical problems.

[0111] In some example implementations, the content of the reports can be applied to identify hyperparameter values ​​exhibiting unexpected behavior. For instance, during testing, in rare cases, the model created using the XGBoost algorithm performed worse than the majority-voting algorithm, which should be the worst information-theoretic reasonable model. The root cause of the performance was identified using filters on the XGBoost model and by generating reports with a flag indicating performance below a threshold. The XGBoost algorithm was determined to have parameters that control the boosting method when building new trees. However, one available method, the DART booster, alters the behavior of the model's inference steps. Without considering this behavior, the results were unexpectedly poor, although only in extremely rare and infrequent cases (e.g., once in eighteen trials).

[0112] In some example implementations, the content of the report can be applied to remove certain hyperparameters. The average impact of hyperparameters associated with a particular module can be used to identify some hyperparameters to be removed. Figure 6A An example of a random forest algorithm is depicted, where the minimum number of samples used for splitting when creating trees in a random forest has the least impact compared to other hyperparameters (such as the minimum number of samples per leaf and the maximum depth). This hyperparameter (reported as having the least impact on the target metric (e.g., the area under the curve in this example)) can be a candidate to be removed.

[0113] In some example implementations, the content of the report can be applied to quantify continuous hyperparameter values. Figure 6B An example of a hyperparameter with two characteristics is illustrated: the bagging temperature parameter in a CatBoost classification model. First, the bagging temperature hyperparameter has a very small impact on model performance over a wide range of values. Second, at very small values, the bagging temperature hyperparameter tends to be associated with poor results. Therefore, the bagging temperature hyperparameter can be quantized, allowing it to take certain values ​​(e.g., 0.1, 0.3, 0.5, 0.7, and 0.9) instead of others. Quantization of the bagging temperature hyperparameter can lead to faster hyperparameter optimization because the parameter space is smaller than the continuous range of values.

[0114] In some example implementations, the content of the report can be applied to limit and rescale the parameter range. Figure 6C Examples of hyperparameter constraints and rescaling applicable to parameter ranges are depicted. The learning rate used when creating a CatBoost classification model exhibits three characteristics. First, at small values, the classification model provides both good and bad results. Second, within a wide range of values ​​(e.g., 0.2–0.8), the hyperparameters have no effect on the classification model's results. Third, at high values, the classification model performs poorly. Therefore, the CatBoost classification model can be optimized by constraining the values ​​of the hyperparameters to a certain range (e.g., 0–0.5). Alternatively and / or additionally, the CatBoost classification model can be optimized by switching from uniform sampling to uniform logarithmic sampling, making the possibilities equal across all orders of magnitude.

[0115] In some example implementations, the report's content can be applied to identify unfavorable algorithm combinations. A report on classification hyperparameters, with model performance as the target metric, can identify which switches lead to poor model performance. Therefore, the report's content could identify that the multilayer perceptron algorithm produces poor results if numeric values ​​are left unscaled. This observation suggests that certain algorithm combinations should be prohibited in the hyperparameter space to avoid poor results. For example, the combination of "no normalization" of numeric input features and the multilayer perceptron model could be disallowed.

[0116] In some example implementations, the reported content can be used to detect poor scaling behavior. Examining the impact of model parameters and metadata on a set of benchmark datasets may reveal poor scaling behavior of the XGBoost classification algorithm when applied to multi-class classification problems. While it is expected that the algorithm will take more time when there are more label values, the XGBoost algorithm exhibits behavior worse than linear scaling, rather than the expected linear scaling behavior. Therefore, these empirical results suggest that the XGBoost algorithm should be excluded from classification problems where the number of label values ​​exceeds a threshold (e.g., 5 or more).

[0117] In some example implementations, the report content can be applied to identify pipeline modules with technical problems. For experiments marked as "invalid" in the report due to feature importance, the measures describing the importance of each column of the model do not sum to the expected value of 1. Automatic reporting of classification hyperparameters indicates that a problem exists when the classification encoding switch is set to integer encoder, in which case a certain type of pipeline module is active to encode categorical variables. This information allows analysis of a particular pipeline module to identify problems in the module's processing feature information.

[0118] Figure 4A Sequence diagrams illustrating process 400 for generating a machine learning model trained to perform a task, according to some example embodiments, are depicted. Referring to Figures 1, 2A-C, 3A-C, and 4A, process 400 may be executed by pipeline controller 110 as part of execution data processing pipeline 250 to generate a machine learning model with a set of parameters and / or hyperparameters for performing cognitive tasks (e.g., object recognition, natural language processing, information retrieval, speech recognition, classification, and / or regression).

[0119] Coordinator node 230 (e.g., coordination engine 235) can receive initial configuration 455 from client device 120 for implementing a machine learning model to perform cognitive tasks (e.g., object recognition, natural language processing, information retrieval, speech recognition, classification, and / or regression). In response to receiving the initial configuration 455 from client device 120, coordinator node 230 can trigger at preparer node 240 the generation of a training dataset for training the machine learning model to perform the task and the generation of a validation dataset for evaluating the performance of the trained machine learning model to perform the task. Figure 4A As shown, coordinator node 230 can trigger the generation of training and validation datasets at least by sending a first message 460a to preparer node 240. Preparer node 240 can respond to the first message 460a by generating the training and validation datasets and performing one or more preparation tasks, such as embedding and / or encoding various types of data (e.g., text data, numerical data, spatial data, categorical data, etc.). When preparer node 240 completes the generation of the training and validation datasets and the preparation tasks, preparer node 240 can send a second message 460b to coordinator node 230, notifying coordinator node 230 of the availability of the training and validation datasets and the results of the preparation tasks. Preparer node 240 can store the training and validation datasets in shared persistence 115 (e.g., experimental persistence 300), where the training and validation datasets are accessible by coordinator node 230 and executor node 280. Therefore, preparer node 240 can avoid sending the training and validation datasets directly to coordinator node 230 in the second message 460b.

[0120] Coordinator node 230 can respond to the second message 460b at least by triggering the execution of the first machine learning experiment at executor node 280. For example... Figure 4A As shown, coordinator node 230 can trigger the execution of a first machine learning experiment by sending at least a first set 470a of experimental parameters to executor node 280. The first set 470a of experimental parameters may include one or more parameters of the machine learning model, such as the initial weights applied to the machine learning model before training. Furthermore, the first set 470a of experimental parameters may include one or more hyperparameters of the machine learning model, including, for example, the learning rate (e.g., step size) of the neural network, the value of the constant k in the k-nearest neighbor clustering algorithm, the cost associated with the support vector machine, and sigma, etc. Executor node 280 can execute the first machine learning experiment by applying the machine learning model with at least the first set 470a of experimental parameters to the training dataset and validation dataset generated by preparer node 240.

[0121] In some example embodiments, if there are sufficient remaining computing resources (e.g., execution time, etc.) to support the execution of additional machine learning experiments, the coordinator node 230 may continue to trigger the execution of additional machine learning experiments. For example, Figure 4A Coordinator node 230 is shown as sending a second set 470b of test parameters to executor node 280 to trigger the execution of a second machine learning experiment at executor node 280, and sending a third set 470c of test parameters to trigger the execution of a third machine learning experiment at executor node 280.

[0122] The executor node 280 can return to the coordinator node 230 a first test result 475a of executing a first machine learning experiment, a second test result 475b of executing a second machine learning experiment, and a third test result 475c of executing a third machine learning experiment. The first test result 475a, the second test result 475b, and the third test result 475c can correspond to the performance of one or more machine learning models having a first set 470a of test parameters, a second set 470b of test parameters, and a third set 470c of test parameters. Furthermore, it should be understood that the executor node 280 can send the first test result 475a, the second test result 475b, and the third test result 475c to the coordinator node 230 at least by storing them in shared persistence 115 (e.g., experimental persistence 300).

[0123] According to some example embodiments, executor node 280 may evaluate a first trial result 475a, a second trial result 475b, and / or a third trial result 475c relative to a target metric specified by user 125 as part of the initial configuration of a machine learning model, in order to identify a machine learning model with a set of parameters and / or hyperparameters for performing the task. Executor node 280 may, for example, select a first set of trial parameters 470a and a machine learning model associated with the first set of trial parameters 470a, for example, based at least on the fact that the first trial result 475a is better than the second trial result 475b and the third trial result 475c relative to the target metric specified by user 125. The target metric may be the accuracy of the machine learning model, in which case the first trial result 475a may be better by exhibiting a higher target metric than the second trial result 475b and the third trial result 475c. Alternatively and / or additionally, the target metric may be log loss, in which case the first trial result 475a may be better by exhibiting a lower target metric than the second trial result 475b and the third trial result 475c.

[0124] exist Figure 4A In the example shown, coordinator node 230 may send one or more messages to client device 120 indicating the status of a machine learning experiment. For example, coordinator node 230 may send a third message 460c to client device 120 including a first experiment result 475a of a first machine learning experiment performed by executor node 280. Alternatively and / or additionally, coordinator node 230 may send a fourth message 460d to client device 120 including a third experiment result 475c of a third machine learning experiment performed by executor node 280 and an indication of the completion of the machine learning experiment. Coordinator node 230 may communicate with client device 120 via user interface node 270, which may be configured to generate and / or update user interface 150 to display at least a portion of the contents of the third message 460c and / or the fourth message 460d at client device 120.

[0125] Refer again Figure 3A The coordinator node 230 of the data processing pipeline 250a may include an optimizer 330, a budget counter 340, and a coordinator engine 235. In some example embodiments, the optimizer 330 may optimize the execution of one or more machine learning experiments before the coordinator node 230 triggers the execution of one or more machine learning experiments at the executor node 280. The optimizer 330 may optimize the execution of one or more machine learning experiments to reduce overhead, adapt to large training datasets, and eliminate inconsistencies in the results of executing the data processing pipeline 250. The coordinator node 230 may continue to trigger the execution of successive machine learning experiments until the budget counter 340 signals that the available time budget has been exhausted.

[0126] To further illustrate, Figure 4B A sequence diagram illustrating another example of a process 450 for generating a machine learning model trained to perform a task, according to some example embodiments, is depicted. Figure 4B As shown, coordinator node 230 (e.g., coordination engine 235) can trigger the generation of training and validation datasets at least by sending a first message 460a to preparer node 240. Preparer node 240 can respond to the first message 460a by generating the training and validation datasets and performing one or more preparation tasks, such as embedding and / or encoding various types of data (e.g., text data, numerical data, spatial data, categorical data, etc.). When preparer node 240 completes the generation of the training and validation datasets and the preparation tasks, preparer node 240 can send a second message 460b to coordinator node 230 to notify coordinator node 230 of the availability of the training and validation datasets and the results of the preparation tasks. As noted, the training and validation datasets can be stored in shared persistence 115 (e.g., experimental persistence 300), where they can be accessed by coordinator node 230 and executor node 280.

[0127] Upon receiving the second message 460b from the preparer node 240, the coordinator node 230 (e.g., the coordination engine 235) may send one or more messages to the optimizer 330 requesting an optimization strategy for executing one or more machine learning trials (e.g., a first machine learning trial with a first set 470a of trial parameters, a second machine learning trial with a second set 470b of trial parameters, a third machine learning trial with a third set 470c of trial parameters, etc.). For example, as... Figure 4B As shown, the coordination engine 235 can send a fifth message 460e to the optimizer 330, and the optimizer 330 can respond by sending an optimization strategy 465 for executing machine learning experiments to the coordination engine 235. The coordination engine 235 can trigger the execution of machine learning experiments at the executor node 280, which can be executed according to the corresponding set of experiment parameters and the optimization strategy.

[0128] Figure 7A A flowchart illustrating a process 700 for generating a data processing pipeline configured to generate a machine learning model, according to some example embodiments, is depicted. In some example embodiments, process 700 may be executed by a pipeline controller 110 to generate, for example, a data processing pipeline 250 configured to generate a machine learning model. The machine learning model generated by the data processing pipeline may be a machine learning model with a set of parameters and / or hyperparameters for performing cognitive tasks (e.g., object recognition, natural language processing, information retrieval, speech recognition, classification, and / or regression).

[0129] At 702, pipeline controller 110 can generate a user interface configured to receive one or more inputs for constructing a data processing pipeline for generating machine learning models. For example, pipeline controller 110 can generate a user interface 150 configured to display selections of operator nodes (including, for example, coordinator node 230, preparer node 240, and executor node 280) at client device 120. The selection of operator nodes displayed at client device 120 as part of user interface 150 may also include one or more auxiliary operator nodes, including, for example, start node 260, user interface node 270, etc. As part of the data processing pipeline, start node 260 can be configured to receive inputs configuring a process including one or more machine learning experiments, while user interface node 270 can be configured to output the progress and / or results of one or more machine learning experiments. Alternatively, instead of displaying selections of operator nodes, user interface 150 may display one or more dialog boxes prompting user 125 to select one or more operator nodes to include in the data processing pipeline.

[0130] At 704, the pipeline controller 110 can configure the data processing pipeline to generate a machine learning model trained to perform a task in response to one or more inputs received from the client device 120, at least by adding a coordinator node, a preparer node, and an executor node to the graph representing the data processing pipeline. For example, the pipeline controller 110 can generate a graph representing a data processing pipeline 250 configured to generate a machine learning model. Figure 2B In the example shown, the data processing pipeline 250 can be constructed to include a coordinator node 230, a preparer node 240, and an executor node 280. Furthermore, as... Figure 2B As shown, the data processing pipeline 250 can be constructed to include a start node 260 and a user interface node 270. As indicated, the data processing pipeline 250 can be executed to generate a machine learning model for performing tasks associated with an input dataset. As part of the data processing pipeline 250, the start node 260 can be configured to receive inputs from a configuration process to generate the machine learning model, while the progress and results of the process can be output by the user interface node 270.

[0131] Coordinator node 230, preparer node 240, start node 260, user interface node 270, and executor node 280 can be interconnected via one or more directed edges indicating the data flow between them. For example, coordinator node 230 and preparer node 240 can be interconnected via a first directed edge indicating that the output of coordinator node 230 can serve as an input to preparer node 240, and a second directed edge indicating that the output of preparer node 240 can serve as an input to coordinator node 230. Alternatively and / or additionally, coordinator node 230 and executor node 280 can be interconnected via a third directed edge indicating that the output of coordinator node 230 can serve as an input to executor node 280, and a fourth directed edge indicating that the output of executor node 280 can serve as an input to coordinator node 230.

[0132] At 706, the pipeline controller 110 can generate a corresponding data processing pipeline including a coordinator node, a preparer node, and an executor node, at least based on a graph. For example, in some example embodiments, the pipeline controller 110 can generate a data processing pipeline 250 including a coordinator node 230, a preparer node 240, a start node 260, a user interface node 270, and an executor node 280, at least based on a corresponding graph.

[0133] At 708, the pipeline controller 110 can generate a machine learning model trained to perform a task, at least by executing a data processing pipeline. For example, the pipeline controller 110 can generate a machine learning model trained to perform a task, at least by executing a data processing pipeline 250 that includes a coordinator node 230, a preparer node 240, an executor node 280, a start node 260, and a user interface node 270. Executing the data processing pipeline 250 may include executing one or more data processing operations associated with each of the coordinator node 230, the preparer node 240, the executor node 280, the start node 260, and the user interface node 270.

[0134] In some example embodiments, coordinator node 230 may be executed to coordinate the operations of at least preparer node 240 and executor node 280. For example, coordinator node 230 may respond to receiving an initial configuration for implementing a machine learning model from start node 260, at least by triggering the generation of training and validation datasets at preparer node 240. Upon receiving an indication from preparer node 240 that preparer node 240 has generated training and validation datasets, coordinator node 230 may trigger the execution of one or more machine learning trials at executor node 280, each machine learning trial applying a different type of machine learning model and / or a different set of trial parameters to the training and / or validation datasets generated by preparer node 240. Furthermore, coordinator node 230 may be executed to determine, at least based on the results of the machine learning trials executed by executor node 280, a machine learning model, including a set of model parameters and hyperparameters, for performing a specified task.

[0135] Figure 7B A flowchart illustrating an example of a process 750 for hyperparameter space optimization according to some example embodiments is depicted. In some example embodiments, process 750 may be executed by a pipeline controller 110 including, for example, a coordinator node 230, a tracking database 315, and a reporting engine 325.

[0136] In 752, the pipeline controller 110 can store data associated with the execution of the data processing pipeline that implements the machine learning model in the tracking database 315. In some example embodiments, the coordinator node 230 can be configured to collect information on hyperparameter space analysis and optimization to store in the tracking database 315. This information may include task metadata, experiment configurations, trial configurations, and performance metrics. As noted, the coordinator node 230 can avoid collecting confidential information. Instead, the coordinator node 230 can collect personally identifiable information from each user's dataset, such as the number of rows, the number of categorical features, etc.

[0137] At 754, the pipeline controller 110 can query the tracking database 315 to generate a report based on at least a portion of the data stored in the tracking database 315. In some example embodiments, the report generator 325 can generate a structured report based on at least a portion of the data stored in the tracking database 315. This report can be generated and provided electronically as a structured document (e.g., an HTML document, a PDF file, etc.). Furthermore, the report can be generated based on a set schedule and / or in response to a triggering event including user input requesting the generation of a report. The report may include a set of charts and a corresponding set of numerical tables that provide numerical values ​​for the effects (if available) of the visualizations in the charts. The charts and tables can describe the importance and impact of all features (including hyperparameters and task metadata entries) relative to defined target metrics.

[0138] In some example embodiments, the reporting engine 325 can generate reports by applying association rule algorithms to analyze the impact of classification hyperparameters and classification task metadata information on binary target information (such as flags added as part of enriching denormalized data). Alternatively and / or additionally, the reporting engine 325 can also generate reports based on the interpretability of intermediate models. For example, the reporting engine 325 can select different subsets of data from the tracking database 315 and train a random forest model as an intermediate model on each subset to predict the target metric for each machine learning trial within the subset. Using interpretability methods, interpretability techniques such as Shapley additive interpretation can be reused to extract the logical connections included in the trained random forest model, while discarding individual interpretations of the intermediate model.

[0139] In 756, the pipeline controller 110 can analyze the hyperparameter space of the machine learning model based on at least a portion of the report to identify the root cause of at least one failure associated with the execution of the data processing pipeline implementing the machine learning model. In some example embodiments, the contents of the report can be used to identify the root causes of one or more failures observed during the execution of the data processing pipeline (e.g., pauses, excessive resource consumption, poor results, failures, etc.) and perform various corrective actions. Examples of corrective actions include identifying hyperparameter values ​​with unexpected behavior, removing certain hyperparameters, quantizing continuous hyperparameter values, limiting and rescaling parameter ranges, identifying unfavorable algorithm combinations, detecting poor scaling behavior, and identifying pipeline modules with technical problems.

[0140] In view of the above-described embodiments of the subject matter, this application discloses the following list of examples, wherein a feature of an example, considered alone or in combination with more than one feature of the example, and optionally in combination with one or more features of one or more other examples, is also a further example falling within the scope of this disclosure:

[0141] Example 1: A system comprising: at least one data processor; and at least one memory storing instructions that, when executed by the at least one data processor, result in operations including: collecting data associated with the execution of a data processing pipeline for storage in a tracking database, the data processing pipeline being executed to generate a machine learning model having a set of hyperparameters for performing a task associated with an input dataset, the execution of the data processing pipeline including executing multiple machine learning trials, each of the multiple machine learning trials applying different types of machine learning models and / or different sets of trial parameters to a training dataset, and the machine learning model having a set of hyperparameters for performing the task being identified at least based on the results of the multiple machine learning trials; generating a report based on at least a portion of the data associated with the execution of the data processing pipeline; analyzing the hyperparameter space of the machine learning model based on at least a portion of the report; and identifying at least one root cause of a failure associated with the execution of the data processing pipeline, at least based on the analysis of the hyperparameter space.

[0142] Example 2: The system of Example 1 further includes: generating a logical table by at least denormalizing the data associated with the execution of the data processing pipeline, the report being generated based on the logical table, each row of the logical table corresponding to one of a plurality of machine learning experiments, and each column of the logical table corresponding to a value describing the plurality of machine learning experiments, the corresponding experiments, and / or the results of the plurality of machine learning experiments.

[0143] Example 3: The system of Example 2 further includes: ranking multiple machine learning trials based at least on a target metric; and adding a column corresponding to the ranking of each machine learning trial included in the ranked multiple machine learning trials to a logical table.

[0144] Example 4: A system of any one of Examples 2 to 3, further comprising: ranking a plurality of machine learning trials based at least on a target metric; and adding a column corresponding to the ranking of each machine learning trial included in the ranked plurality of machine learning trials to a logical table.

[0145] Example 5: A system of any of Examples 1 to 4, wherein a report is generated by applying an association rule algorithm to generate one or more association rules that link one or more hyperparameters of a machine learning model to the results of multiple machine learning experiments.

[0146] Example 6: The system of Example 5, wherein association rules supported by a proportion of data below a threshold associated with the execution of the data processing pipeline are excluded from one or more association rules applied to generate a report.

[0147] Example 7: A system of any of Examples 1 to 6, wherein a report is generated by applying interpretability techniques to compute the impact of hyperparameters of a machine learning model on a target metric.

[0148] Example 8: A system of any one of Examples 1 to 7, wherein at least one failure includes a combination of one or more hyperparameter values ​​with unexpected behavior, a machine learning model with below-threshold performance, and / or poor scaling behavior.

[0149] Example 9: The system according to any one of Examples 1 to 8 further includes: performing one or more corrective actions corresponding to the root cause of at least one fault, the one or more corrective actions including removing hyperparameters, quantizing hyperparameters having continuous values ​​and / or limiting and / or rescaling the range of hyperparameters.

[0150] Example 10: A system of any of Examples 1 to 9, wherein the data associated with the execution of the data processing pipeline includes one or more task metadata, target performance metrics, and hyperparameter values.

[0151] Example 11: A system according to any one of Examples 1 to 10, wherein the data processing pipeline includes a coordinator node, a preparer node, and a plurality of executor nodes, wherein the preparer node is configured to generate a training dataset based at least on an input dataset, wherein the plurality of executor nodes are configured to perform a plurality of machine learning trials by applying at least different types of machine learning models and / or sets of different trial parameters to the training dataset, and wherein the coordinator node is configured to identify a set of machine learning models with hyperparameters for performing a task based at least on the results of the plurality of machine learning trials.

[0152] Example 12: A system of any one of Examples 1 to 11, wherein the machine learning model includes neural networks, regression models, instance-based models, regularized models, decision trees, random forests, Bayesian models, clustering models, association models, dimensionality reduction models, and / or ensemble models.

[0153] Example 13: A method comprising: collecting data associated with the execution of a data processing pipeline to store in a tracking database, the data processing pipeline being executed to generate a machine learning model with a set of hyperparameters for performing a task associated with an input dataset, the execution of the data processing pipeline including performing multiple machine learning trials, each of the multiple machine learning trials applying a different type of machine learning model and / or a different set of trial parameters to a training dataset, and the machine learning model with the set of hyperparameters for performing the task being identified at least based on the results of the multiple machine learning trials; generating a report based on at least a portion of the data associated with the execution of the data processing pipeline; analyzing the hyperparameter space of the machine learning model based on at least a portion of the report; and identifying at least one root cause of a failure associated with the execution of the data processing pipeline, based at least on the analysis of the hyperparameter space.

[0154] Example 14: Method Example 13, further includes: generating a logical table by at least denormalizing the data associated with the execution of the data processing pipeline, the report being generated based on the logical table, each row of the logical table corresponding to one of multiple machine learning experiments, and each column of the logical table corresponding to a value describing the multiple machine learning experiments, the corresponding experiment, and / or the results of the multiple machine learning experiments.

[0155] Example 15: The method of Example 14 further includes: ranking multiple machine learning trials based at least on a target metric; and adding a column corresponding to the ranking of each machine learning trial included in the ranked multiple machine learning trials to a logical table.

[0156] Example 16: The method of any one of Examples 14-15 further includes: determining, for each of the multiple machine learning trials, a relative deviation from the target metric associated with the validation dataset and / or test dataset; and adding a column corresponding to the relative deviation to a logical table.

[0157] Example 17: A method of any of Examples 13-16, wherein a report is generated by applying an association rule algorithm to generate one or more association rules that link one or more hyperparameters of a machine learning model to the results of multiple machine learning experiments, and wherein association rules supported by a sub-threshold proportion of data associated with the execution of the data processing pipeline are excluded from the one or more association rules applied to generate the report.

[0158] Example 18: A method from any of Examples 13 to 17, wherein a report is generated by applying interpretability techniques to compute the impact of hyperparameters of a machine learning model on a target metric.

[0159] Example 19: The method of any one of Examples 13 to 18 further includes: performing one or more corrective actions corresponding to the root cause of at least one fault, the one or more corrective actions including removing hyperparameters, quantizing hyperparameters with continuous values ​​and / or limiting and / or rescaling the range of hyperparameters.

[0160] Example 20: A non-transitory computer-readable medium storing instructions that, when executed by at least one data processor, result in operations including: collecting data associated with the execution of a data processing pipeline for storage in a tracking database, the data processing pipeline being executed to generate a machine learning model with a set of hyperparameters for performing a task associated with an input dataset, the execution of the data processing pipeline including executing multiple machine learning trials, each of the multiple machine learning trials applying a different type of machine learning model and / or a different set of trial parameters to a training dataset, and the machine learning model with the set of hyperparameters for performing the task being identified at least based on the results of the multiple machine learning trials; generating a report based on at least a portion of the data associated with the execution of the data processing pipeline; analyzing the hyperparameter space of the machine learning model based on at least a portion of the report; and identifying the root cause of at least one failure associated with the execution of the data processing pipeline, at least based on the analysis of the hyperparameter space.

[0161] Figure 8 A block diagram illustrating a computing system 800 consistent with an implementation of the present subject is depicted. Referring to Figures 1-8, the computing system 800 can be used to implement the pipeline controller 110 and / or any of its components.

[0162] like Figure 8 As shown, the computing system 800 may include a processor 810, a memory 820, a storage device 830, and an input / output device 840. The processor 810, memory 820, storage device 830, and input / output device 840 may be interconnected via a system bus 850. The processor 810 is capable of processing instructions for execution within the computing system 800. Such executed instructions may implement one or more components, such as the pipeline controller 110. In some example embodiments, the processor 810 may be a single-threaded processor. Alternatively, the processor 810 may be a multi-threaded processor. The processor 810 is capable of processing instructions stored in the memory 820 and / or the storage device 830 to display graphical information for a user interface provided via the input / output device 840.

[0163] Memory 820 is a computer-readable medium, such as volatile or non-volatile, that stores information within computing system 800. For example, memory 820 may store data structures representing a database of configuration objects. Storage device 830 provides persistent storage for computing system 800. Storage device 830 may be a solid-state drive, floppy disk device, hard disk device, optical disk device, tape device, or other suitable persistent storage device. Input / output device 840 provides input / output operations for computing system 800. In some example embodiments, input / output device 840 includes a keyboard and / or pointing device. In various embodiments, input / output device 840 includes a display unit for displaying a graphical user interface.

[0164] According to some example embodiments, input / output device 840 can provide input / output operations for network devices. For example, input / output device 840 may include an Ethernet port or other networking port to communicate with one or more wired and / or wireless networks (e.g., local area network (LAN), wide area network (WAN), Internet).

[0165] In some example embodiments, the computing system 800 can be used to execute various interactive computer software applications that can be used to organize, analyze, and / or store data in various formats. Alternatively, the computing system 800 can be used to execute any type of software application. These applications can be used to perform various functions, such as planning functions (e.g., generating, managing, and editing spreadsheet documents, word processing documents, and / or any other objects), computing functions, communication functions, etc. Applications may include various additional functions or may be standalone computing products and / or functions. Once activated within an application, the function can be used to generate a user interface provided through the input / output device 840. The user interface may be generated by the computing system 800 and presented to the user (e.g., on a computer screen monitor, etc.).

[0166] One or more aspects or features of the subject matter described herein can be implemented in digital electronic circuits, integrated circuits, specially designed ASICs, field-programmable gate arrays (FPGAs), computer hardware, firmware, software, and / or combinations thereof. These aspects or features can be implemented in one or more computer programs executable and / or interpretable on a programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to transfer data and instructions to the storage system, at least one input device, and at least one output device. The programmable system or computing system can include client devices and servers. Client devices and servers are typically geographically separated and typically interact via a communication network. The relationship between client devices and servers is generated by computer programs running on respective computers and having client-server relationships with each other.

[0167] These computer programs, also referred to as programs, software, software applications, applications, components, or code, include machine instructions for programmable processors and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. When used herein, the term "machine-readable medium" means any computer program product, apparatus, and / or device for providing machine instructions and / or data to a programmable processor, for example, such as disks, optical disks, memories, and programmable logic devices (PLDs), including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" means any signal used to provide machine instructions and / or data to a programmable processor. Machine-readable media can store such machine instructions non-transitory, for example, non-transitory machine instructions stored in a non-transitory manner, such as non-transitory solid-state memory or magnetic hard disk drives or any equivalent storage medium. Machine-readable media can alternatively or additionally store such machine instructions transiently, for example, machine instructions stored transiently in a processor cache or other random access memory associated with one or more physical processor cores.

[0168] To provide interaction with the user, one or more aspects or features of the subject matter described herein can be implemented on a computer having a display device, such as a cathode ray tube (CRT), liquid crystal display (LCD), or light-emitting diode (LED) monitor, for displaying information to the user, and a keyboard and pointing device, such as a mouse or trackball, through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user. For example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including sound, speech, or tactile input. Other possible input devices include touchscreens or other touch-sensitive devices, such as single-point or multi-point resistive or capacitive touchpads, voice recognition hardware and software, optical scanners, optical pointers, digital image capture devices, and associated interpretation software, etc.

[0169] In the foregoing description and claims, phrases such as "at least one of..." or "one or more of..." may appear after a list of connected elements or features. The term "and / or" may also appear in a list of two or more elements or features. Unless the context in which it is used implies or explicitly contradicts this, such phrases are intended to mean any element or feature listed individually or any recited element or feature combined with any other recited element or feature. For example, the phrases "at least one of A and B;", "one or more of A and B;", and "A and / or B" are each intended to mean "A alone, B alone, or A and B together". A similar interpretation applies to lists containing three or more items. For example, the phrases "at least one of A, B, and C;", "one or more of A, B, and C;", and "A, B, and / or C" each mean "A alone, B alone, C alone, A and B together, A and C together, B and C together, or A and B and C together". The use of the term "based on" in the foregoing and claims is intended to mean "at least partially based on", thus allowing for the inclusion of unrecited features or elements.

[0170] The subject matter described herein can be implemented in systems, apparatuses, methods, and / or articles according to desired configurations. The embodiments set forth in the foregoing description do not represent all embodiments consistent with the subject matter described herein. Rather, they are merely examples of aspects consistent with the described subject matter. While some variations have been described in detail above, other modifications or additions are possible. In particular, further features and / or variations may be provided in addition to those set forth herein. For example, the above embodiments may involve various combinations and sub-combinations of the disclosed features and / or combinations and sub-combinations of several other features disclosed above. Furthermore, the logical flows depicted in the drawings and / or described herein do not necessarily require the specific order or sequence shown to achieve the desired results. Other embodiments may be within the scope of the appended claims.< / n>

Claims

1. A system comprising: At least one data processor; and At least one memory storing instructions that, when executed by the at least one data processor, cause an operation including the following: Data associated with the execution of a data processing pipeline is collected and stored in a tracking database. The data processing pipeline is executed to generate a set of machine learning models with hyperparameters for performing tasks associated with an input dataset. The execution of the data processing pipeline includes executing multiple machine learning trials, each of which applies different types of machine learning models and / or different sets of trial parameters to a training dataset. The machine learning models with hyperparameters for performing tasks are identified at least based on the results of the multiple machine learning trials, wherein the machine learning models are configured to perform various cognitive tasks. A report is generated based on at least a portion of the data associated with the execution of the data processing pipeline; The hyperparameter space of the machine learning model is analyzed based on at least a portion of the report; Based at least on the analysis of the hyperparameter space, identify the root cause of at least one failure associated with the execution of the data processing pipeline; and Perform one or more corrective actions corresponding to the root cause of at least one fault, said one or more corrective actions including removing hyperparameters, quantizing hyperparameters with continuous values, and / or limiting and / or rescaling the range of hyperparameters. The data processing pipeline includes a coordinator node, a preparer node, and multiple executor nodes. The preparer node is configured to generate the training dataset based on the input dataset. The plurality of executor nodes are configured to execute the plurality of machine learning experiments by applying at least different types of machine learning models and / or different sets of experimental parameters to the training dataset. The coordinator node is configured to coordinate the operations of the preparer node and multiple executor nodes in the following manner: In response to receiving the initial configuration for implementing the machine learning model, the generator node triggers the generation of the training and validation datasets; and Upon receiving an indication from the preparer node that the preparer node has generated the training and validation datasets, the executor node triggers an execution of a number of machine learning experiments consistent with the computational resource budget described in the initial configuration. The coordinator node triggers additional machine learning experiments based on the amount of remaining computational resources sufficient to support the execution of these additional machine learning experiments. The coordinator node is configured to identify a set of machine learning models with hyperparameters for performing the task, based at least on the results of the plurality of machine learning trials. At least one of these faults involves excessive resource consumption, including peak memory consumption of the actuator node. Each machine learning test generates a candidate machine learning model, which has a specific set of hyperparameters sampled from the hyperparameter space.

2. The system according to claim 1, further comprising: A logical table is generated by at least denormalizing the data associated with the execution of the data processing pipeline. The report is generated based on the logical table, where each row corresponds to one of the plurality of machine learning trials, and each column corresponds to a value describing the plurality of machine learning trials, the corresponding experiment, and / or the result of the plurality of machine learning trials.

3. The system according to claim 2, further comprising: The multiple machine learning experiments are ranked based at least on the target metric; and Add a column corresponding to the ranking of each machine learning experiment included in the sorted multiple machine learning experiments to the logical table.

4. The system according to claim 2, further comprising: For each of the plurality of machine learning trials, determine the relative deviation from the target metric associated with the validation dataset and / or test dataset; and Add the column corresponding to the relative deviation to the logical table.

5. The system according to claim 1, wherein, The report is generated by applying an association rule algorithm to generate one or more association rules that link one or more hyperparameters of the machine learning model to the results of the multiple machine learning experiments.

6. The system according to claim 5, wherein, Association rules supported by a sub-threshold proportion of data associated with the execution of the data processing pipeline are excluded from the one or more association rules applied to generate the report.

7. The system according to claim 1, wherein, The report is generated by applying interpretability techniques to calculate the impact of the hyperparameters of the machine learning model on the target metric.

8. The system according to claim 1, wherein, The at least one fault also includes one or more hyperparameter values ​​with unexpected behavior, a combination of machine learning models with below-threshold performance, and / or poor scaling behavior.

9. The system according to claim 1, wherein, The data associated with the execution of the data processing pipeline includes one or more task metadata, target performance metrics, and hyperparameter values.

10. The system according to claim 1, wherein, The machine learning models include neural networks, regression models, instance-based models, regularized models, decision trees, random forests, Bayesian models, clustering models, association models, dimensionality reduction models, and / or ensemble models.

11. A computer-implemented method, comprising: Data associated with the execution of a data processing pipeline is collected and stored in a tracking database. The data processing pipeline is executed to generate a set of machine learning models with hyperparameters for performing tasks associated with an input dataset. The execution of the data processing pipeline includes executing multiple machine learning trials, each of which applies different types of machine learning models and / or different sets of trial parameters to a training dataset. The machine learning models with hyperparameters for performing tasks are identified at least based on the results of the multiple machine learning trials, wherein the machine learning models are configured to perform various cognitive tasks. A report is generated based on at least a portion of the data associated with the execution of the data processing pipeline; The hyperparameter space of the machine learning model is analyzed based on at least a portion of the report; Based at least on the analysis of the hyperparameter space, identify the root cause of at least one failure associated with the execution of the data processing pipeline; and Perform one or more corrective actions corresponding to the root cause of at least one fault, said one or more corrective actions including removing hyperparameters, quantizing hyperparameters with continuous values, and / or limiting and / or rescaling the range of hyperparameters. The data processing pipeline includes a coordinator node, a preparer node, and multiple executor nodes. The preparer node is configured to generate the training dataset based on the input dataset. The plurality of executor nodes are configured to execute the plurality of machine learning experiments by applying at least different types of machine learning models and / or different sets of experimental parameters to the training dataset. The operations of coordinating the preparer node and multiple executor nodes include: In response to receiving the initial configuration for implementing the machine learning model, the generator node triggers the generation of the training and validation datasets; and Upon receiving an indication from the preparer node that the preparer node has generated the training and validation datasets, the executor node triggers an execution of a number of machine learning experiments consistent with the computational resource budget described in the initial configuration. The coordinator node triggers additional machine learning experiments based on the amount of remaining computational resources sufficient to support the execution of these additional machine learning experiments. The coordinator node is configured to identify a set of machine learning models with hyperparameters for performing the task, based at least on the results of the plurality of machine learning trials. At least one of these faults involves excessive resource consumption, including peak memory consumption of the actuator node. Each machine learning test generates a candidate machine learning model, which has a specific set of hyperparameters sampled from the hyperparameter space.

12. The method of claim 11, further comprising: A logical table is generated by at least denormalizing the data associated with the execution of the data processing pipeline. The report is generated based on the logical table, where each row corresponds to one of the plurality of machine learning trials, and each column corresponds to a value describing the plurality of machine learning trials, the corresponding experiment, and / or the result of the plurality of machine learning trials.

13. The method of claim 12, further comprising: The multiple machine learning experiments are ranked based at least on the target metric; and Add a column corresponding to the ranking of each machine learning experiment included in the sorted multiple machine learning experiments to the logical table.

14. The method of claim 12, further comprising: For each of the plurality of machine learning trials, determine the relative deviation from the target metric associated with the validation dataset and / or test dataset; and Add the column corresponding to the relative deviation to the logical table.

15. The method according to claim 11, wherein, The report is generated by applying an association rule algorithm to generate one or more association rules that link one or more hyperparameters of the machine learning model to the results of the plurality of machine learning experiments, wherein association rules supported by a sub-threshold proportion of data associated with the execution of the data processing pipeline are excluded from the one or more association rules applied to generate the report.

16. The method according to claim 11, wherein, The report is generated by applying interpretability techniques to calculate the impact of the hyperparameters of the machine learning model on the target metric.

17. A non-transitory computer-readable medium storing instructions that, when executed by at least one data processor, cause an operation comprising: Data associated with the execution of a data processing pipeline is collected and stored in a tracking database. The pipeline is executed to generate a set of machine learning models with hyperparameters for performing tasks associated with an input dataset. The execution of the pipeline includes executing multiple machine learning trials, each of which applies different types of machine learning models and / or different sets of trial parameters to a training dataset. The machine learning model with the set of hyperparameters for performing the task is identified at least based on the results of the multiple machine learning trials. Machine learning models are configured to perform a variety of cognitive tasks; A report is generated based on at least a portion of the data associated with the execution of the data processing pipeline; The hyperparameter space of the machine learning model is analyzed based on at least a portion of the report; and Based at least on the analysis of the hyperparameter space, identify the root cause of at least one failure associated with the execution of the data processing pipeline, and Perform one or more corrective actions corresponding to the root cause of at least one fault, said one or more corrective actions including removing hyperparameters, quantizing hyperparameters with continuous values, and / or limiting and / or rescaling the range of hyperparameters. The data processing pipeline includes a coordinator node, a preparer node, and multiple executor nodes. The preparer node is configured to generate the training dataset based on the input dataset. The plurality of executor nodes are configured to execute the plurality of machine learning experiments by applying at least different types of machine learning models and / or different sets of experimental parameters to the training dataset. The operations of coordinating the preparer node and multiple executor nodes include: In response to receiving the initial configuration for implementing the machine learning model, the generator node triggers the generation of the training and validation datasets; and Upon receiving an indication from the preparer node that the preparer node has generated the training and validation datasets, the executor node triggers an execution of a number of machine learning experiments consistent with the computational resource budget described in the initial configuration. The coordinator node triggers additional machine learning experiments based on the amount of remaining computational resources sufficient to support the execution of these additional machine learning experiments. The coordinator node is configured to identify a set of machine learning models with hyperparameters for performing the task, based at least on the results of the plurality of machine learning trials. At least one of these faults involves excessive resource consumption, including peak memory consumption of the actuator node. Each machine learning test generates a candidate machine learning model, which has a specific set of hyperparameters sampled from the hyperparameter space.

Images