Automated machine learning pipeline exploration and deployment

Abstract

Techniques for automated machine learning (ML) pipeline exploration and deployment are described. An automated ML pipeline generation system allows a user to easily build an optimized ML pipeline by providing a dataset, identifying target columns in the dataset, and providing an exploration budget. Multiple candidate ML pipelines can be identified and evaluated through an exploration process, and the best ML pipeline can be provided to the requesting user or can be deployed for production inference. The user can configure, monitor, and adjust the exploration at multiple points throughout the process.

Description

BACKGROUND

[0001] The field of machine learning has been widely accepted as an important driver of the future of technology. Organizations around the world are now seeking to use machine learning techniques to solve a wide variety of problems, such as optimizing aspects of their products, internal processes, customer experience, etc. While the high-level view of machine learning sounds simple - e.g., provide training data to a computer to allow the computer to automatically learn from the training data to generate a model that can make predictions on other data - implementing machine learning in practice can be very difficult.

[0002] For example, many organizations are racing to implement ML solutions to their business problems, such as classification models for fraud detection and customer churn prediction or regression models for risk assessment. However, building these solutions often requires hiring experts in machine learning, who are in short supply and difficult to retain. Due to lack of deep training in mathematics and statistics, most software engineers struggle to make good judgments about why one ML algorithm would be better than another, much less find the right algorithm parameters. In addition to this, users often need to clean or pre-process data in order to build good models, which most users do not have the knowledge or ability to do. Instead, users often use datasets as-is, try one or two algorithms that are easy to use, and accept whatever level of quality they achieve - or give up altogether. As a result, organizations struggle to achieve the desired model quality. Moreover, even for data scientists with full ML knowledge, they can need to spend a significant amount of their time experimenting with different ML solutions in order to find the best solution for a particular problem.

[0003] These and other difficulties are due in part to the underlying algorithmic and mathematical complexity of machine learning algorithms, which are often developed by academic researchers or individuals at the forefront of the field. In addition, generating, updating, and deploying useful models is also difficult, which can be very time and resource consuming and fraught with complexity. Moreover, machine learning models tend to be very specific to a particular use case and operating environment, and thus any changes to the underlying environment or use case can require a completely new model to be generated. Furthermore, building and deploying machine learning techniques is very different from traditional software engineering, and requires practices and architectures that are different from what traditional software engineering development teams are familiar with. BRIEF DESCRIPTION OF DRAWINGS

[0004] Various embodiments according to the present disclosure will be described with reference to the drawings, in which:

[0005] Figure 1 is a diagram illustrating an environment for automated machine learning pipeline exploration and deployment, according to some embodiments.

[0006] Figure 2 FIG. 1 is a diagram illustrating one example user interface for creating an automated machine learning pipeline exploration, in accordance with some embodiments.

[0007] Figure 3 FIG. 2 is a diagram illustrating one example user interface for configuring exploration budgets and preferred processing resources for automated machine learning pipeline exploration, in accordance with some embodiments.

[0008] Figure 4 FIG. 3 is a diagram illustrating one example user interface for configuring advanced options for automated machine learning pipeline exploration, in accordance with some embodiments.

[0009] Figure 5 FIG. 4 is a diagram illustrating one example set of processing jobs for automated machine learning pipeline exploration and deployment, in accordance with some embodiments.

[0010] Figure 6 FIG. 5 is a diagram illustrating one example user interface for monitoring trials of automated machine learning pipeline exploration, in accordance with some embodiments.

[0011] Figure 7 FIG. 6 is a diagram illustrating one example user interface for viewing results of trials of automated machine learning pipeline exploration, in accordance with some embodiments.

[0012] Figure 8 FIG. 7 is a diagram illustrating one example interactive code exploration user interface for viewing and / or modifying automated machine learning pipeline exploration, in accordance with some embodiments.

[0013] Figure 9 FIG. 8 is a flow diagram illustrating operations of a method for automated machine learning pipeline exploration and deployment, in accordance with some embodiments.

[0014] Figure 10 FIG. 9 is a block diagram of an illustrative operating environment in which machine learning models are trained and hosted, in accordance with some embodiments.

[0015] Figure 11 FIG. 10 illustrates an example provider network environment, in accordance with some embodiments.

[0016] Figure 12 FIG. 11 is a block diagram of an example provider network that provides storage services and hardware virtualization services to customers, in accordance with some embodiments.

[0017] Figure 13 FIG. 12 is a block diagram of an example computer system that can be used in some embodiments. DETAILED DESCRIPTION

[0018] The present disclosure relates to methods, devices, systems, and non-transitory computer-readable storage media for automated machine learning pipeline exploration and deployment. According to some embodiments, an automated machine learning (ML) pipeline generation system (or "AMPGS") allows a user to easily build optimized ML pipelines by simply providing a dataset, identifying target columns in the dataset, and providing an exploration budget. An ML pipeline is a series of processing steps that collectively operate on input data to generate an ML inference result, which can include, for example, feature and target pre-processing operations (or, a "feature pre-processing pipeline" of the overall ML pipeline, which can include the use of one or more ML models), followed by an inference ML model. The AMPGS system can generate a single trained ML pipeline that can be deployed to perform real-time or batch inference, and in some embodiments, allows the user to view details of the resulting ML model, details of the training jobs that led to the generation of the ML model, parameters and operations of the ML pipeline, and optionally some or all of these details of other ML pipelines that have been explored but ultimately determined to be less performant. In some embodiments, the AMPGS system also produces feature processing code and interactive code exploration "notebooks" that train the user on the components of one or more ML pipelines, providing an explanation of the steps taken by the system. In some embodiments, the AMPGS thus provides a "white box" approach by showing the user the incremental steps or jobs that were performed to arrive at the final ML model produced for inference purposes, which can also enable the user to modify and repeat the process to iteratively fine-tune the ML pipeline according to their particular needs.

[0019] In some embodiments, a user is thus able to quickly build ML models - e.g., classification and regression models - without any substantial ML knowledge. For example, in some embodiments, a user can provide a tabular dataset and identify target columns in the dataset for prediction, and the AMPGS system then automatically explores ML pipeline solutions with different combinations of data pre-processors, algorithms, and / or algorithm parameter settings to find a "best" model. In some embodiments, the user can then deploy this best model (in terms of the ML pipeline) directly to a production environment (e.g., with a single click) or iterate on one or more recommended solutions to further improve model quality.

[0020] Figure 1 FIG. 1 is a diagram illustrating an environment for automated machine learning pipeline exploration and deployment, according to some embodiments. In Figure 1Among other things, an automated machine learning pipeline generation system 102 (or "AMPGS") implemented as part of the machine learning service 110 develops, evaluates, and / or deploys ML pipelines on behalf of users 109. The AMPGS 102 (and the ML service 110) can be implemented using one or more computing devices in one or more networks and / or geographic locations as software, hardware, or a combination of both. In some embodiments, the AMPGS 102 is implemented within the multi-tenant provider network 100 and operates as part of the ML service 110 to provide the ML-related operations described herein as a service to users 109.

[0021] The provider network 100 (or "cloud" provider network) provides users 109 with the ability to utilize one or more various types of computing-related resources, such as: compute resources (e.g., executing virtual machine (VM) instances and / or containers, executing batch jobs, executing code rather than pre-configured servers), data / storage resources (e.g., object storage, block-level storage, data archival storage, databases and database tables, etc.), network-related resources (e.g., configuring virtual networks (including groups of compute resources), content delivery networks (CDNs), domain name services (DNS)), application resources (e.g., databases, application build / deployment services), access policies or roles, identity policies or roles, machine images, routers, and other data processing resources, etc. These and other computing resources can be provided as services, such as: hardware virtualization services that can execute compute instances, storage services 114 that can store data objects, etc. Users 109 (or "customers") of the provider network 100 can utilize one or more user accounts associated with a customer account, although these items can be used somewhat interchangeably depending on the context of use. Users can interact with the provider network 100 via one or more interfaces over one or more intermediate networks 106 (e.g., the Internet), such as by using application programming interface (API) calls, via consoles implemented as websites or applications, etc. The one or more interfaces can be part of, or function as a front-end to, a control plane of the provider network 100, which includes "backend" services that support and enable the services that can be provided more directly to customers.

[0022] For example, a cloud provider network (or simply "cloud") can refer to a large pool of accessible virtualized computing resources (such as compute, storage, and networking resources, applications, and services). The cloud can provide convenient, on-demand network access to a shared, configurable computing resource pool that can be programmatically provisioned and released in response to customer commands. These resources can be dynamically provisioned and reconfigured to adjust for variable load traffic. Cloud computing can thus be viewed as applications delivered as a service over a publicly accessible network 106 (e.g., the Internet, a cellular communication network) as well as the hardware and software in cloud provider data centers that provide those services.

[0023] Generally, the traffic and operations of a provider network can be broadly subdivided into two categories: control plane operations carried over a logical control plane and data plane operations carried over a logical data plane. The data plane represents the movement of user data through the distributed computing system, while the control plane represents the movement of control signals through the distributed computing system. The control plane generally includes one or more control plane components that are distributed across and implemented by one or more control servers. Control plane traffic generally includes management operations, such as system configuration and management (e.g., resource placement, hardware capacity management, diagnostic monitoring, system state information). The data plane includes customer resources (e.g., compute instances, containers, block storage volumes, databases, file storage) implemented on the provider network. Data plane traffic generally includes non-management operations, such as transferring customer data to and from customer resources. The control plane components are generally implemented on a separate set of servers from the data plane servers, and control plane traffic and data plane traffic can be sent over separate / different networks.

[0024] To provide compute resource services, the provider network 100 generally relies on virtualization technology. For example, virtualization technology can be used to provide users with the ability to control or utilize compute instances (e.g., VMs using a guest operating system (O / S) that operates using a hypervisor, which can or can not further run on top of an underlying host O / S; containers that can or can not operate in a VM; instances that can execute on “bare metal” hardware without an underlying hypervisor), where one or more compute instances can be implemented using a single electronic device. Thus, users can directly utilize compute instances hosted by the provider network (e.g., provided by a hardware virtualization service) to perform various computing tasks. Additionally or alternatively, users can indirectly utilize compute instances by submitting code to be executed by the provider network (e.g., via an on-demand code execution service), which in turn utilizes compute instances to execute the code, without the user generally having any control or knowledge of the underlying compute instance(s) involved.

[0025] For example, in various embodiments, "serverless" functions can include code that is provided by a user or other entity (such as the provider network itself) that can be executed on-demand. Serverless functions can be maintained by an on-demand code execution service within the provider network 100 and can be associated with a particular user or account or generally accessible by multiple users / accounts. Serverless functions can be associated with a uniform resource locator (URL), uniform resource identifier (URI), or other indicia that can be used to invoke the serverless function. Serverless functions can be executed by a compute instance (such as a virtual machine, container, etc.) when triggered or invoked. In some embodiments, serverless functions can be invoked through an application programming interface (API) call or a specially formatted hypertext transfer protocol (HTTP) request message. Thus, a user can define a serverless function that can be executed on-demand without the user having to maintain specialized infrastructure to execute the serverless function. Rather, the serverless function can be executed on-demand using resources maintained by the provider network 100. In some embodiments, these resources can be kept in a "ready" state (e.g., with a pre-initialized runtime environment configured to execute serverless functions) allowing for near real-time execution of serverless functions.

[0026] As described herein, in some embodiments, the AMPGS 102 makes the ML model building process easier and faster. Rather than requiring the user 109 to decide which ML algorithm to use, the AMPGS 102 can automatically select multiple different ML algorithms from its locally supported list of high-performance algorithms and evaluate some or all of them. The AMPGS 102 can also automatically evaluate different hyperparameter settings for those algorithms in an effort to improve the resulting quality of the discovered ML models. In some embodiments, the user 109 also does not need to be busy with data cleaning or pre-processing, as the AMPGS 102 can automatically apply different types of data pre-processors to the data before passing the data through the ML algorithms to train the ML models. In some embodiments, the AMPGS 102 also makes the details and artifacts of the ML pipelines that it has evaluated fully accessible to the user, such as the corresponding source code, allowing advanced users such as data scientists to quickly run baselines and iterate on the results to further improve the model quality.

[0027] To begin the process, in some embodiments, a user 109 can provide or otherwise identify a dataset 108 for use in ML pipeline exploration. For example, as shown at optional circle (1), the user 109 can utilize an application 103 (e.g., a web application implementing a console 105 of the provider network 100, a spreadsheet plug-in, a standalone application, another web application of another entity that utilizes the AMPGS 102 as a backend, a database, or a hybrid SQL environment, etc.) executed by a computing device 104 to cause the computing device 104 to upload the dataset 108 to a storage location (e.g., provided by a storage service 114, such as an object storage service of the provider network 100).

[0028] For example, the dataset 108 can be a columnar dataset that includes rows (or entries) of data values, where the data values can be arranged according to one or more columns (or attributes) and can have the same data type or different data types. In some cases, the dataset 108 can include headers or other metadata that describes the names of the columns or data types, although in some cases such metadata can not be present. For example, the dataset 108 can have been provided by the user as a plain text file (e.g., a comma-separated values (CSV) or tab-separated values (TSV) file), an exported database table or structure, an application-specific file such as a spreadsheet, etc. However, in some embodiments, the dataset 108 can also be a database table (hosted by a database service or another system of the provider network 100), data from a web page or web service, etc.

[0029] The user 109 can utilize the computing device 104 at circle (2) to initiate ML pipeline exploration through the AMPGS 102. For example, the user 109 can cause the computing device 104 to issue one or more requests (e.g., API calls) to the ML service 110, which can occur by the user 109 configuring an ML exploration job using one or more user interfaces such as those depicted in FIG. 1. Figures 2-4 Such user interfaces can be provided by the ML service 110 (or by another service of the provider network), and the user 109 can provide user input via these interfaces that ultimately is provided to the AMPGS 102 to configure, initiate, and / or modify the ML exploration job.

[0030] For example, Figure 2is a diagram illustrating one example user interface 200 for creating an automated machine learning pipeline exploration, in accordance with some embodiments. This user interface and other user interfaces are shown to include various example data fields and user interface input elements (e.g., text input boxes, drop-down boxes, etc.); it should be understood that these are examples and representative of only one implementation - many other data fields and elements can be implemented in various embodiments by those of skill in the art.

[0031] As shown, the user interface 200 allows the user 109 to provide a name 205 and description 210 for the exploration. The user interface 200 also allows the user to provide user interface elements 215 to identify a dataset to be used for model training purposes, such as by selecting a file or storage location (e.g., from a set of storage objects within a storage service associated with the user’s account), providing a resource identifier (e.g., a uniform resource locator (URL)), directly uploading a dataset, etc.

[0032] The user interface 200 also includes a user interface element 220 in which the user can identify a column from the dataset that should be reasoned over by the model (i.e., the value that the model should ultimately output as its prediction for a given input data). The user interface element 220 can provide a list of columns to select from (e.g., after the AMPGS 102 has obtained the dataset identified via element 215 and has identified the columns contained in the dataset), allow the user to provide an identifier for the column (e.g., a column name or number), etc.

[0033] In some embodiments, the user can also configure exploration budget information and compute resource utilization settings for the ML pipeline exploration. Such configuration options enable the user to flexibly control the speed, thoroughness, and / or expense of the exploration based on their specific needs. Figure 3 is a diagram illustrating one example user interface 300 for configuring an exploration budget and preferred processing resources for an automated machine learning pipeline exploration, in accordance with some embodiments. In this user interface 300, the user can control how the AMPGS 102 performs the ML pipeline exploration, for example, by using user interface element 305 to set a maximum number of training jobs (e.g., a number of ML models that should be trained in whole or in part) that should be run during the exploration. The user can also indicate via the other two input elements 310 how long any particular training can last, in this example, the input elements 310 allow the user to indicate a number (here, “4”) and a corresponding unit of time (here, “hrs” means hours, although other options can be selected, such as seconds, minutes, days, weeks, etc.), at which point the training jobs can be stopped. In some embodiments, the user can also use input element 317 to provide a maximum cost (or budget) associated with the ML pipeline exploration process, allowing the system to stop or configure its exploration to prevent exceeding this user-specified cost.

[0034] In this example, the user can also use the user interface 300 input elements 315 / 320 to indicate what type of compute instances (e.g., virtual machine types, where multiple different options with different resource characteristics can be made available, such as different amounts of memory, processing power, network availability, storage, etc.) will be used for the ML model training job (via element 315), and the number of those instances that will be used at any point in time (via element 320), allowing the user to control the speed and expense at which exploration occurs. However, in some embodiments, the user can select “automatic” (or similar, meaning “automatic”), and thus the system can select its own desired type and / or amount of instances, e.g., based on previous exploration runs and / or based on a budget provided by the user.

[0035] In some embodiments, a user with sufficient ML knowledge can be enabled to customize additional aspects of the ML pipeline exploration process. Figure 4 FIG. 4 is a diagram showing one example user interface 400 for configuring advanced options for automated machine learning pipeline exploration, according to some embodiments.

[0036] In this user interface 400, the user can specify via input element 405 what type of machine learning problem is to be encountered, enabling the user to control what type of ML algorithm will be used for training. For example, there are many different types of machine learning problems known to those skilled in the art, such as binary classification, multiclass classification, linear regression, etc. In some embodiments, the user can be able to select an “automatic detection” option, as the type of machine learning problem can be inferred based on the type and / or values of the “target column” specified by the user.

[0037] The user can also specify a target metric for the exploration. For example, the user interface 400 includes a user interface element 410 where the user can select (e.g., from a list of metrics) a target metric type that will be used to evaluate which ML pipeline is best for the user. In some embodiments, the target metric can be “automatic” (automatic), allowing the AMPGS 102 to use its own selected metric or metrics to determine the best-performing ML pipeline. Various types of metrics can be used and are known to those skilled in the art, including but not limited to mean squared error (MSE), classification accuracy, log loss, area under the curve (AUC), mean absolute error (MAE), root mean squared error (RMSE), root mean squared log error (RMSLE), R-squared, F1 score, etc.

[0038] In some embodiments, the user can also configure, via user interface element 415, which types of algorithms are available for the AMPGS 102 to test. The selection of particular algorithms that can be used can enable advanced users to exclude certain algorithms or explore a limited number of particular algorithms of interest. The particular algorithms can be populated in the interface 400 based on the selected problem type 405, as particular algorithms are well known to be used for particular ML problem types. Various types of algorithms can be included, such as (but not limited to) XGBoost, linear learners, k-nearest neighbors (KNN), factorization machines, multi-layer perceptron (MLP), neural networks, etc., as known to those of skill in the art.

[0039] The example user interface 400 also includes element 420 that allows the user to indicate (e.g., yes or no) whether the AMPGS 102 should explore creating ensemble models that are based on combining multiple other models. In some cases, ensemble models can provide improved accuracy, such as via applying an ensemble method that combines several base models to produce one best predictive model, such as via using a bagging method (e.g., building multiple models by different subsamples of a dataset), a random forest type method, a boosting method (e.g., building multiple models where some models can learn to correct errors of other models), a stacking method (e.g., a supervised model or algorithm that combines predictions from multiple other models), or other methods known to those of skill in the art.

[0040] The example user interface 400 also includes element 425 that allows the user to indicate whether the exploration should just be a “dry run,” which means that the AMPGS 102 will identify multiple pipelines to explore and provide the user with data describing those models, but not actually perform any ML model training. In such cases, the AMPGS 102 can provide details via other user interfaces (e.g., of a console or interactive code application) that identify the details of the multiple candidate pipelines that the user can inspect. The user can then indicate (e.g., via a transport request message) to the AMPGS 102 to perform a full exploration or just perform a particular portion of the exploration — e.g., implement one (or more) of the candidate pipelines, or even run through the pipeline one step at a time (e.g., just perform data pre-processing and return the results, perform training of a model, etc.).

[0041] The user interface 400 can also allow the user to configure other aspects of the exploration, such as configuring particular permissions settings (e.g., providing an identifier of a user account or “role” that is used by the AMPGS 102 when performing operations within a provider network) or security settings (e.g., implementing network isolation for those instances that perform the exploration work so that they cannot make any outbound network calls, placing one or more instances within a particular dedicated network, encrypting or decrypting data according to a user-provided key or algorithm).

[0042] Thereafter, the user can be provided with a user interface that allows the user to start an ML pipeline exploration, resulting in the issuance of a command to the AMPGS 102 via a request message sent (as Figure 1 These configuration settings can additionally or alternatively be provided without the use of such a graphical user interface, and instead can be issued by another type of application program (executing in the provider network or outside the provider network) that can or can not have obtained parallel user instructions regarding these settings.

[0043] Thus, an ML pipeline exploration job can be initiated via these example user interfaces 200 / 300 / 400 or via other means (e.g., configuration files or database entries). Returning to Figure 1 Upon receiving a message indicating that a job is desired to be launched, at circle (3), the AMPGS 102 can initialize the use of the pipeline recommender system 112 to start the ML pipeline exploration.

[0044] In some embodiments, the pipeline recommender system 112 is responsible for determining the set of ML pipelines to explore and to be optimized by the pipeline optimizer system 116. In some embodiments, the pipeline recommender system 112 can analyze the user-provided dataset and infer one or more of the probabilistic patterns of the dataset, target leakage, feature importance, the type of ML problem (e.g., classification, regression, etc.) based on the user-identified target column, etc. The pipeline recommender system 112 can use the user-provided dataset and the knowledge learned from the collected metadata to recommend a promising and diverse set of feature processing pipelines to be applied to the customer dataset along with the code of the feature processing models. The pipeline recommender system 112 can also recommend the hardware (compute instance types and counts), the identifiers of which hyperparameters to tune (and optionally their effective search spaces), and the degree of parallelization that the pipeline optimizer system 116 should use when exploring the ML pipelines. Thereafter, the pipeline optimizer system 116 can use the pipelines recommended by the pipeline recommender system 112 to start an optimization job, which typically involves running multiple training jobs to identify the highest performance pipeline. As the optimization job progresses, the pipeline optimizer system 116 can discard low performance models and can tune the hyperparameters of the highest performance models.

[0045] In some embodiments, a user can utilize both the pipeline recommender system 112 (to generate candidate ML pipelines for exploration) and the pipeline optimizer system 116 (to explore and evaluate candidate ML pipelines). However, in some embodiments, a user can use portions of the AMPGS 102 independently - e.g., only use the pipeline recommender system 112 to identify candidate ML pipelines (and thereafter use this data elsewhere), or only use the pipeline optimization system 116 to evaluate candidate ML pipelines (e.g., using candidate ML pipelines that can have been developed with or without use of the pipeline recommender system 112). Thus, in some embodiments, these components are designed such that they are not tightly coupled, and thus can optionally be used independently of one another.

[0046] For example, the operations of the AMPGS 102 (and the pipeline recommender system 112 and / or the pipeline optimizer system 116) can be implemented as shown with respect to Figure 5 Figure 5 is a diagram illustrating an example set of processing jobs for automated machine learning pipeline exploration and deployment, in accordance with some embodiments.

[0047] At block 505, the data builder component of the AMPGS 102 can obtain a dataset and perform preliminary operations on the dataset (e.g., by sending a request to a storage service based on a provided resource identifier). For example, in some embodiments, the data builder splits the dataset into three parts - one for training, one for validation, and one for testing. In some embodiments, each of the three parts is equal in size (e.g., the same number of rows or a very similar number of rows (e.g., 334 rows, 333 rows, and 333 rows given a dataset of 1,000 rows)), although in other embodiments, the parts have dissimilar sizes (e.g., the training dataset can be much larger than the validation or testing datasets). The preliminary operations can also include generating metadata describing the dataset (e.g., the total number of rows, the number of columns, the data types of the columns, value distributions, and other statistics based on the values of the columns), which can be used as part of subsequent processing, cleaning the dataset, etc.

[0048] ​Next, the pipeline recommender system 112 can analyze the dataset (e.g., via use of one or more ML models and target columns of the dataset, etc.) to recommend pipelines (combinations of one or more pre-processing tasks and machine learning algorithms) for exploration. For example, the pipeline recommender system 112 can utilize one or more ML models that have been trained to identify particular pipeline types that have worked well on particular datasets based on characteristics of those datasets (e.g., number and / or types of columns, types of columns to be inferred, etc.). Additionally or alternatively, the pipeline recommender system 112 can use a database (or other data structure) of prior pipelines (and characteristics of the datasets involved) and compare corresponding dataset / inference characteristics of the current dataset to identify the most similar historical pipelines that can be recommended.

[0049] Additionally or alternatively, the pipeline recommender system 112 can generate multiple different pre-processing pipelines (and corresponding training datasets) according to configured “strategies” - e.g., a “baseline” strategy that will one-hot encode (1HE) all categorical variables, median impute nulls with an identifier; a “quadratic” strategy that will hash encode any detected categorical variables, bucket numerical features, and add Cartesian product features of predictive feature combinations: instance-type x instance-count, num_layers x instance-type, dataset_size x num_layers; a “log-bucketing” strategy that will hash encode any categorical variables, bucket numerical variables, and log transform all numerics. For example, the feature processing pipeline can be generated using two machine learning jobs that can be executed using the batch inference system 142 (which can train and / or host machine learning models in batch), as shown at optional circle (3B): a training job to learn the transformation; and then a batch processing job that applies the transformation to the dataset to generate a transformed dataset for exploration of different ML models.

[0050] The pipeline recommender system 112 can identify a fixed number of pipelines for exploration or a number provided by a user during exploration configuration. As one example, the pipeline recommender system 112 can recommend up to ten pipelines for exploration, such as (1) “apply one-hot encoding and principal component analysis (as a feature pre-processor / transformation), followed by hyperparameter tuning using ‘XGBOOST’ algorithm,” (2) “apply one-hot encoding and principal component analysis (as a feature pre-processor / transformation), followed by hyperparameter tuning using ‘linear learner’ algorithm,” (3) “apply principal component analysis (as a feature pre-processor / transformation), followed by hyperparameter tuning using ‘XGBOOST’ algorithm,” and so on.

[0051] Returning temporarily to Figure 1In some embodiments, after the pipeline recommender system 112 identifies different ML pipelines to evaluate, details about these pipelines can be provided to the user 109 at optional circle (4), where the user 109 can provide confirmation to continue completing the exploration, stop the exploration, run only a portion of the exploration (e.g., continue with only some of the recommended pipelines, run only some portions of one pipeline), etc.

[0052] Assuming the full exploration is to continue, referring to Figure 5 The feature pre-processing analyzers 510A (e.g., each implemented by a separate one or more computing instances) can run for each different pre-processing step / transform identified within the recommended ML pipeline plan. Each feature pre-processing analyzer 510A-510M can perform initialization of the pre-processing tasks so that the tasks can later be implemented by the feature pre-processors 515A-515N, and this analysis can be performed at least partially in parallel.

[0053] For example, a feature pre-processing analyzer 510A for one-hot encoding can need to run an ML job to figure out how many columns it should produce. It can be the case, therefore, that one column of a dataset can have 10,000 unique values, with many of these values referenced only once or twice. Thus, the feature pre-processing analyzer 510A can determine to generate columns for only the top X (e.g., 200) values, and generate one column for any other values as a “catch-all.” Thus, the feature pre-processing analyzer 510A will perform this analysis, which can include identifying which all different categories are referenced within the column, what is the count of each of these values, etc., to ultimately determine which columns should be generated. Similar types of pre-processing analysis can be performed for other types of pipeline pre-processing tasks, e.g., for principal component analysis, learning what the results (e.g., which features should be used) are, etc.

[0054] Thereafter, the actual pre-processing can be performed by a set of feature pre-processors 515A-515N, where each pre-processor can implement a particular pre-processing step or can implement multiple pre-processing steps (e.g., for one pipeline). The feature pre-processors 515A-515N can thus optionally use data from the feature pre-processing analyzers at least partially in parallel to apply the pre-processing operations / transforms to the dataset (e.g., one row at a time) to produce a transformed output dataset (or portions thereof).

[0055] Notably, in some embodiments during the pre-processing analysis and / or actual pre-processing, the AMPGS 102 can provide updates to the user 109 (e.g., via the application 103, such as the console 105 and / or interactive code application 107 such as a Jupyter Notebook), who can view intermediate results, stop the tasks (or the entire ML pipeline exploration), modify the tasks, etc.

[0056] Next, the pipeline optimizer system 116 (also refer to Figure 1 circle (5)) can utilize the pre-processed dataset and the pipeline recommendations (including the identifiers of the particular ML algorithms used to generate the ML models) and can cause to train a plurality of ML models accordingly, e.g., by invoking the model training system 120 described herein (optionally in parallel), utilizing an ML framework, etc., according to any user-specified configuration parameters (e.g., how many experiments to run in total, how many experiments can be run at a time, what type of computing instances to use, etc.). The results of each training include the model artifacts as well as the values of the target metrics to be optimized. The pipeline optimizer system 116 can use this information, e.g., via using additional one or more ML models or databases, to identify additional ML pipeline variants to test (e.g., with different hyperparameter values) that can lead to the generation of good models. Such hyperparameter tuning systems and techniques are known to those skilled in the art and can be used to work to find increasingly better pipelines.

[0057] Notably, in some embodiments during these training experiments, the AMPGS 102 can provide updates to the user 109 via the application 103 (e.g., via the console 105 and / or an interactive code application 107 such as a Jupyter Notebook), who can view intermediate results, stop particular training (or the entire ML pipeline exploration), etc.

[0058] The model integrator 525 can also be used to create one or more ensemble models based on the use of some of the explored models (if desired by the user or configured by the system). For example, as described above, one or more ensemble techniques known to those skilled in the art (e.g., using some aggregation or selection of the outputs) can be used to use multiple models in an ensemble. Based on all the results, the output generator 530 can produce an output in the form of one or more comprehensive results in the form of updated result data for the user to view (e.g., at optional circle (6)), packages of pipeline exploration artifacts 122 (e.g., including code 123 for implementing the pre-processing pipeline and / or one or more inference models 124 for the ML models) to be stored at a storage location (refer to optional circle (7A)) for later retrieval by the user or another application, results stored to a database, etc. Figure 1

[0059] Returning to Figure 1 ​In some scenarios, the user 109 can review the results of the exploration and select a deployed pipeline for use. In such cases, the user’s computing device 104 can issue a command to deploy a particular pipeline, and in response, at optional circle (7B), the AMPGS 102 can send one or more commands (e.g., API calls) to a model hosting system 140 described further herein to “host” the pipeline - e.g., launch or reserve one or more compute instances, run the pipeline code 126, configure an endpoint associated with the pipeline (from which inference requests can be received from external clients and provided to the pipeline), etc. As one example, the command can include a “create model” API call that combines code for the model (e.g., inference code implemented within a container) with model artifacts (e.g., data describing weights associated with various aspects of the model) to produce a model that can be hosted (e.g., by the model hosting system 140 or batch inference system 142), where the create model request can include one or more of: a name for the model, an identifier for a container with inference code, one or more identifiers for other containers in the inference pipeline, a set of permissions or roles for the model when executing, a set of user-defined tags to be associated with the model, etc. The one or more commands can also include “create endpoint” (or “create transformation job” in the case of batch inference) API calls for all pipeline components needed to deploy the model (e.g., one or more containers for pre-processing, one or more containers for actual inference, associated routing, etc.).

[0060] Thus, as described above, at optional circle (7C), the AMPGS 102 can send one or more commands (e.g., API calls) to a batch inference system 142 to “host” the pipeline for performing batch inference. The batch inference system 142 can be implemented similarly to the model hosting system 140 described herein, but can be optimized for performing inference in batches (e.g., via input data stored at a storage service location), storing results back to a storage location, etc., rather than performing real-time synchronous inference. Such one or more deployments can also be performed automatically at the end of the pipeline exploration process, where the AMPGS 102 can automatically deploy a “best” performance pipeline. Thereafter, a client application 130A-130B at optional circle (8) internal to the provider network (such as an application executed by an application running through an internal service 132 such as a hardware virtualization service or code execution service) or external to the provider network (e.g., through the computing device 104) can issue inference requests to an endpoint associated with the deployed pipeline code 126, enabling the pipeline to generate inferences and return the results to the application.

[0061] As shown throughout, the ongoing status of the exploration and / or intermediate (or final) results can be provided to the user in a "white box" manner, allowing the user to fully see what is being suggested to do, what is actually being done, and how the results are coming out. For example, Figure 6 is a diagram illustrating one example user interface 600 for viewing the results of an experiment for monitoring automated machine learning pipeline exploration, in accordance with some embodiments. In this user interface 600, a first panel 605 displays the stage of the exploration - e.g., a pre-processing stage, a model building stage, a post-processing stage, etc. In this example, the model building stage is in progress.

[0062] A second panel 610 displays an overview of the "experiments" (or model training) for the exploration - here, three models are actively being trained (referred to as experiment 55, experiment 56, and experiment 57), and some of these columns in the table can enable selection to view additional information about each experiment, ranging from plots of intermediate results of the experiment, details of any warnings / issues, details of the particular experiment (e.g., identifier of the pre-processing, model algorithm type, hyperparameters being tested), etc. A third panel 615 displays various plots describing the overall exploration results to date - here, model loss over time as the models are being trained, model accuracy over time as the models are being trained, etc.

[0063] For additional details, Figure 7 is a diagram illustrating one example user interface 700 for viewing the results of an experiment for monitoring automated machine learning pipeline exploration, in accordance with some embodiments. For example, when a user selects a particular experiment in the second panel 610 of Figure 6 This user interface 700 can be displayed when a user selects a particular experiment in the second panel 610 of and provides additional details about the particular experiment. In this example, a first panel 705 presents user interface input elements (here, buttons) that allow the user to further explore the pipeline and / or deploy the pipeline while displaying the name of the experiment. A second panel 710 allows the user to view / download each of the sets of data resulting from the dataset splits, while a third panel 715 allows the user to view / download pre-processing related information (e.g., pre-processing code - such as pre-processing ML models - and / or resulting datasets), and a fourth panel 720 allows the user to view / download other pipeline artifacts (e.g., model code / weights, hyperparameter values).

[0064] Figure 8 is a diagram illustrating one example interactive code exploration user interface 800 for viewing and / or modifying automated machine learning pipeline exploration, in accordance with some embodiments. As Figure 8As shown, the user can be presented with an interactive code exploration (e.g., via a web browser that is a web application) that allows the user to explore the code, run the code, modify and run the code, etc. In some embodiments, the user can be presented with code for performing ML pipeline exploration to allow the user to see which particular pipelines to recommend testing, which pre-processing operations to use, etc. Moreover, the user can “run” this code in its original form or in a modified form (e.g., by the user editing the code) (e.g., via selecting a user interface element (such as a button) that causes the application to send a request to the provider network to actually execute the code). In this example, the interactive code exploration user interface 800 includes a first code portion 805 that includes some defining code, e.g., to assign an exploration name, to create a storage location path, to import code from a library, etc.

[0065] The interactive code exploration user interface 800 also includes a second code portion 810 that defines a number of different ways to pre-process a training dataset. In this example, the dataset is configured with code to pre-process the dataset in a variety of different ways to compare which technique or techniques work best, which can include training a feature processing model, and then running a “batch” transformation job to actually apply the transformation.

[0066] As shown, a first pipeline “FP_BASELINE” is defined with a first “FP0” step that is defined (as a training job to learn a transformation) with values for a source directory, instance type and count, and other unspecified values such as a version of an ML framework to use, a set of labels to apply, a feature processing strategy to use (e.g., a baseline strategy that performs one-hot encoding for all categorical variables and performs median imputation for null values with an identifier; a secondary strategy that performs hash encoding for categorical variables, performs bucketing for numerical values, adds a Cartesian product feature for a combination of prediction features; or other strategies). A supplemental “BT0” step for the first pipeline is also defined (as a batch processing job to apply the transformation to the dataset), which can define such values as an instance type, an instance count, an output path, a set of labels, etc. Together, these two steps can form one feature processing pipeline, and multiple pairs of these steps (i.e., many feature processing pipelines) can be defined to create a number of different feature processing pipelines.

[0067] The interactive code exploration user interface 800 can also include a portion 815 to display results of a pipeline that is running, and finally a code portion 820 to define different combinations of machine learning models and pipelines, each combination including values for a name, an ML algorithm to use, a set of hyperparameters, an identifier of a storage location to store a particular set of input values generated by one of the feature processing pipelines, etc.

[0068] Figure 9 is a flowchart showing operations 900 of a method for automated machine learning pipeline exploration and deployment, according to some embodiments. Some or all of the operations 900 (or other processes described herein, or variations, and / or combinations thereof) are performed under the control of one or more computer systems configured with executable instructions (e.g., computer programs, one or more computer programs, or one or more applications) and are implemented, for example, as code (e.g., executable instructions, one or more computer programs, or one or more applications) executed together on one or more processing units. The code is stored on a computer-readable storage medium, for example, in the form of one or more computer programs. The computer-readable storage medium is non-transitory. In some embodiments, one or more (or all) of the operations 900 are performed by the AMPGS 102 of the other figures.

[0069] The operations 900 include, at block 905, receiving a request message initiated by a computing device of a user to identify machine learning (ML) pipelines based at least in part on a dataset, the request message identifying the dataset and an exploration budget. In some embodiments, the exploration budget indicates a maximum number of ML models to be trained as part of training, and wherein the number of the plurality of ML models is the maximum number. In some embodiments, the exploration budget can also be a maximum amount of time associated with aspects of training or a maximum cost associated with training.

[0070] In some embodiments, the request message includes a resource identifier of a storage location of the dataset, and the operations 900 further include obtaining, by a machine learning service of the multi-tenant provider network, the dataset from a storage service via use of the resource identifier.

[0071] The operations 900 further include, at block 910, generating a plurality of ML pipeline plans based at least in part on the dataset.

[0072] In some embodiments, the operations 900 further include, at block 915, transmitting, to the computing device, a first one or more messages to cause the computing device to present information associated with the plurality of ML pipeline plans to the user, and at block 920, receiving a second request message initiated by the computing device to begin exploring at least two ML pipeline plans of the plurality of ML pipeline plans.

[0073] In some embodiments, the information associated with the plurality of ML pipeline plans includes, for each ML pipeline plan of the plurality of ML pipeline plans, one or more pre-processing stages and an identifier of an ML model algorithm type. In some embodiments, the information is presented to the user by the computing device via the interactive code application, and the operations 900 further include receiving a third request message initiated by the computing device of the user, the third request message indicating a request to execute one pipeline of one ML pipeline plan of the plurality of ML pipeline plans. In some embodiments, the operations 900 further include receiving a fourth request message initiated by the computing device of the user, the fourth request message indicating a request to execute a modified pipeline of one ML pipeline plan of the plurality of ML pipeline plans, wherein the modified pipeline is specified by the user based on one ML pipeline plan.

[0074] The operations 900 further include, at block 925, training a plurality of ML models corresponding to the plurality of ML pipeline plans based on at least a portion of the dataset and an exploration budget.

[0075] In some embodiments, the operations 900 further include partitioning the dataset into a plurality of sets, the plurality of sets including a training set, wherein the training of the plurality of ML models utilizes at least the training set.

[0076] In some embodiments, the training of the plurality of ML models is performed at least partially in parallel, as at least two ML models of the plurality of ML models are actively trained at least partially at the same point in time.

[0077] The operations 900 further include, at block 930, transmitting data to the computing device of the user indicating results of the training.

[0078] In some embodiments, the operations 900 further include receiving a fifth request message initiated by the computing device of the user, the fifth request message indicating a request to deploy an ML pipeline corresponding to one ML pipeline plan of the plurality of ML pipeline plans; transmitting a sixth request message to cause a model hosting system of a provider network to deploy the ML pipeline behind an endpoint; and transmitting an identifier of the endpoint to the computing device or a storage location.

[0079] In some embodiments, the transmitting of the data to the computing device of the user causes the computing device of the user to present results to the user, the results including identifiers of a plurality of trials corresponding to the plurality of ML models, wherein for each trial of the plurality of trials, the results include a value of a target metric generated by the training of the corresponding ML model. In some embodiments, the request further identifies the target metric.

[0080] In some embodiments, the operations 900 further include generating another ML pipeline including an ensemble ML model based on at least two ML models of the plurality of ML models.

[0081] Figure 10 is a block diagram of an illustrative operating environment in which machine learning models are trained and hosted in accordance with some embodiments. The operating environment includes an end user device 1002 (e.g., a PC or mobile device such as computing device 104), a model training system 120, a model hosting system 140, a training data storage 1060, a training metric data storage 1065, a container data storage 1070, a trained model data storage 1075, and a model prediction data storage 1080. The machine learning service 110 described herein can include one or more of these entities, such as the model hosting system 140, the model training system 120, etc.

[0082] In some embodiments, a user interacts with the model training system 120 via a user device 1002 to provide data that causes the model training system 120 to train one or more machine learning models. Generally, a machine learning model can be viewed as one or more equations that are “trained” using a set of data. In some embodiments, the model training system 120 provides ML functionality as a web service, and thus passing messages between a user device 1002 and the model training system 120 (or provider network 100) and / or between components of the model training system 120 (or provider network 100) can utilize HTTP messaging passing data in a machine-readable file format such as XML or JSON.

[0083] A user device 1002 can interact with the model training system 120 via a front end 1029 of the model training system 120. For example, the user device 1002 can provide a training request to the front end 1029 that includes a container image (or multiple container images, or identifiers of one or more locations that store container images), an indicator of input data (e.g., an address or location of the input data), one or more hyperparameter values (e.g., values that indicate how an algorithm will operate, how many algorithms to run in parallel, how many clusters to split data into, etc.), and / or information that describes a computer machine on which to train a machine learning model (e.g., a graphics processing unit (GPU) instance type, a central processing unit (CPU) instance type, an amount of memory to allocate, a type of virtual machine instance to use for training, etc.).

[0084] In some embodiments, the container image can include one or more layers, where each layer represents executable instructions. Some or all of the executable instructions together represent an algorithm that defines a machine learning model. The executable instructions (e.g., the algorithm) can be written in any programming language (e.g., Python, Ruby, C++, Java, etc.). In some embodiments, the algorithm is pre-generated and obtained by the user via the user device 1002 from an algorithm repository (e.g., a network-accessible marketplace, a data store provided by a machine learning training service, etc.). In some embodiments, the algorithm is fully user-generated or partially user-generated (e.g., a user provides code modifications or configures existing algorithm code).

[0085] In some embodiments, the user device 1002 can provide an algorithm written in any programming language in the training request, rather than providing a container image (or an identifier thereof) in the training request. The model training system 120 packages the algorithm into a container (optionally, with other code, such as a “base” ML algorithm supplemented with user-provided code) that is ultimately loaded into a virtual machine instance 1022 for training a machine learning model, as described in more detail below. For example, a user via the user device 1002 can develop an algorithm / code using an application (e.g., an interactive web-based programming environment) and cause the algorithm / code to be provided to the model training system 120, possibly as part of (or referenced in) a training request, where this algorithm / code can be containerized, e.g., on its own or with an existing container having a machine learning framework.

[0086] In some embodiments, the user device 1002 provides an indicator of a container image (e.g., an indication of an address or location where the container image is stored) in the training request, rather than providing the container image in the training request. For example, the container image can be stored in the container data store 1070, and this container image can have been previously created / uploaded by the user. The model training system 120 can retrieve the container image from the indicated location and create a container using the retrieved container image. The container is then loaded into a virtual machine instance 1022 for training a machine learning model, as described in more detail below.

[0087] In some implementations, model training system 120 may use information provided by user device 1002 to train a machine learning model in one or more pre-built virtual machine instances 1022. Specifically, model training system 120 may include a single physical computing device or multiple physical computing devices interconnected using one or more computing networks (not shown), wherein one or more physical computing devices host one or more virtual machine instances 1022. Model training system 120 may handle the acquisition and configuration of computing power (e.g., containers, instances, etc., described in more detail below) based on information describing the computing machines on which the machine learning model provided by user device 1002 is to be trained. Model training system 120 may then use the computing power to train the machine learning model, as described in more detail below. Model training system 120 may automatically scale based on the capacity of training requests received from user device 1002 via front end 1029, thereby alleviating the burden on the user of worrying about over-utilization (e.g., acquiring too few computing resources and suffering performance problems) or under-utilization (e.g., acquiring more computing resources than necessary to train the machine learning model and thus overpaying).

[0088] In some implementations, virtual machine instance 1022 is used to perform tasks. For example, such tasks may include training a machine learning model. Figure 10 As shown, each virtual machine instance 1022 includes an operating system (OS) 1024, a language runtime 1026, and one or more ML training containers 1030. Typically, an ML training container 1030 is a logical unit created within the virtual machine instance using resources available on the instance and can be used to isolate task execution from other processes (e.g., task execution) occurring within the instance. In some implementations, the ML training container 1030 is formed by one or more container images and a top container layer. Each container image may also include one or more image layers, where each image layer represents executable instructions. As described above, some or all of the executable instructions together represent an algorithm that defines a machine learning model. Changes made to the ML training container 1030 (e.g., creation of new files, modification of existing files, file deletion, etc.) are stored in the top container layer. If the ML training container 1030 is deleted, the top container layer is also deleted. However, one or more container images that form part of the deleted ML training container 1030 may remain unchanged. The ML training container 1030 can be implemented as, for example, a Linux container (LXC), a Docker container, etc.

[0089] In some embodiments, the ML training container 1030 can include, individually, the runtime 1034, the code 1037, and dependencies 1032 required by the code 1037. The runtime 1034 can be defined by one or more executable instructions that form at least a portion of a container image used to form the ML training container 1030 (e.g., one or more executable instructions in a container image that define an operating system and / or runtime for running in a container formed from the container image). The code 1037 includes one or more executable instructions that form at least a portion of a container image used to form the ML training container 1030. For example, the code 1037 includes executable instructions in a container image that represent algorithms that define a machine learning model, which can reference (or utilize) code or libraries from the dependencies 1032. The runtime 1034 is configured to execute the code 1037 in response to an instruction to start machine learning model training. Execution of the code 1037 results in the generation of model data, as described in more detail below.

[0090] In some embodiments, the code 1037 includes executable instructions that represent algorithms that define different machine learning models. For example, the code 1037 includes a first set of executable instructions that represent a first algorithm that defines a first machine learning model, and a second set of executable instructions that represent a second algorithm that defines a second machine learning model. In some embodiments, the virtual machine instance 1022 executes the code 1037 and trains all of the machine learning models. In some embodiments, the virtual machine instance 1022 executes the code 1037, thereby selecting one of the machine learning models to train. For example, the virtual machine instance 1022 can identify a type of training data indicated by the training request, and select the machine learning model corresponding to the identified type of training data to train (e.g., execute executable instructions that represent an algorithm that defines the selected machine learning model).

[0091] In some embodiments, the runtime 1034 is the same as the runtime 1026 utilized by the virtual machine instance 1022. In some embodiments, the runtime 1034 is different than the runtime 1026 utilized by the virtual machine instance 1022.

[0092] In some embodiments, the model training system 120 uses one or more container images included in the training request (or container images retrieved from the container data store 1070 in response to the received training request) to create and initialize the ML training container 1030 in the virtual machine instance 1022. For example, the model training system 120 creates the ML training container 1030 that includes the one or more container images and / or a top container layer.

[0093] Before starting the training process, in some embodiments, the model training system 120 retrieves the training data from the location indicated in the training request. For example, the location indicated in the training request can be a location in the training data storage device 1060. Accordingly, the model training system 120 retrieves the training data from the indicated location in the training data storage device 1060. In some embodiments, the model training system 120 does not retrieve the training data before starting the training process. Rather, the model training system 120 streams the training data from the indicated location during the training process. For example, the model training system 120 can initially retrieve a portion of the training data and provide the retrieved portion to the virtual machine instance 1022 that is training the machine learning model. Once the virtual machine instance 1022 has applied and used the retrieved portion or once the virtual machine instance 1022 is about to use all of the retrieved portion (e.g., the buffer storing the retrieved portion is almost empty), the model training system 120 can retrieve a second portion of the training data and provide the second retrieved portion to the virtual machine instance 1022, and so on.

[0094] In some embodiments, to perform the machine learning model training, the virtual machine instance 1022 executes code 1037 stored in the ML training container 1030. For example, the code 1037 includes some or all of the executable instructions that form the container image of the ML training container 1030 in which it is initialized. Accordingly, the virtual machine instance 1022 executes some or all of the executable instructions (that form the container image of the ML training container 1030 in which it is initialized) to train the machine learning model. The virtual machine instance 1022 executes some or all of the executable instructions in accordance with the hyperparameter values included in the training request. As an illustrative example, the virtual machine instance 1022 trains the machine learning model by identifying values for certain parameters (e.g., coefficients, weights, centroids, etc.). The identified values depend on the hyperparameters that define how the training is performed. Accordingly, the virtual machine instance 1022 can execute the executable instructions to initiate a machine learning model training process, where the training process is run using the hyperparameter values included in the training request. The execution of the executable instructions can include the virtual machine instance 1022 applying, as input parameters to some or all of the executed instructions, the training data retrieved by the model training system 120.

[0095] In some embodiments, executing the executable instructions causes the virtual machine instance 1022 (e.g., the ML training container 1030) to generate model data. For example, the ML training container 1030 generates model data and stores the model data in a file system of the ML training container 1030. The model data includes characteristics of the machine learning model being trained, such as a number of layers in the machine learning model, hyperparameters of the machine learning model, coefficients of the machine learning model, weights of the machine learning model, and the like. In particular, the generated model data includes values for defining the characteristics of the machine learning model being trained. In some embodiments, executing the executable instructions causes the ML training container 1030 to be modified such that the model data is written to a top container layer of the ML training container 1030, and / or one or more container images that form part of the ML training container 1030 are modified to include the model data.

[0096] The virtual machine instance 1022 (or the model training system 120 itself) extracts the generated model data from the ML training container 1030 and stores the generated model data in the training model data storage 1075 in an entry associated with the virtual machine instance 1022 and / or the machine learning model being trained. In some embodiments, the virtual machine instance 1022 generates a single file that includes the model data and stores the single file in the training model data storage 1075. In some embodiments, the virtual machine instance 1022 generates multiple files during the process of training the machine learning model, where each file includes model data. In some embodiments, each model data file includes the same or different model data information (e.g., one file identifies the structure of the algorithm, another file includes a list of coefficients, etc.). The virtual machine instance 1022 can package the multiple files into a single file once training is complete and store the single file in the training model data storage 1075. Alternatively, the virtual machine instance 1022 stores the multiple files in the training model data storage 1075. The virtual machine instance 1022 stores the one or more files in the training model data storage 1075 while the training process is ongoing and / or after the training process is complete.

[0097] In some embodiments, the virtual machine instance 1022 periodically stores the one or more model data files in the training model data storage device 1075 while the training process is ongoing. Thus, the one or more model data files can be stored in the training model data storage device 1075 at different times during the training process. Each set of model data files corresponding to a particular time or each set of model data files existing in the training model data storage device 1075 as of a particular time can be checkpoints representing different versions of the partially trained machine learning model during different stages of the training process. Thus, before the training is completed, a user can submit a deployment and / or execution request via the user device 1002 to deploy and / or execute one version of the partially trained machine learning model (e.g., the trained machine learning model as of a certain stage of the training process) in a manner described below. The one version of the partially trained machine learning model can be based on some or all of the model data files stored in the training model data storage device 1075.

[0098] In some embodiments, the virtual machine instance 1022 executes the code 1037 stored in the plurality of ML training containers 1030. For example, the algorithm included in the container image can be in a format that allows for parallelization of the training process. Thus, the model training system 120 can create multiple copies of the container image provided in the training request and cause the virtual machine instance 1022 to load each container image copy in a separate ML training container 1030. The virtual machine instance 1022 can then execute the code 1037 stored in the ML training containers 1030 in parallel. The virtual machine instance 1022 can also provide configuration information to each ML training container 1030 (e.g., information indicating that the ‘N’ ML training containers 1030 collectively train the machine learning model and that the particular ML training container 1030 receiving the configuration information is the Xth ML training container 1030 of the N), which can be included in the resulting model data. In some embodiments, by parallelizing the training process, the model training system 120 can significantly reduce the training time.

[0099] In some embodiments, multiple virtual machine instances 1022 execute code 1037 stored in multiple ML training containers 1030. For example, the resources for training a particular machine learning model can exceed the limits of a single virtual machine instance 1022. However, the algorithms included in the container image can be in a format that allows for parallelization of the training process. Thus, the model training system 120 can create multiple copies of the container image provided in the training request, cause multiple virtual machine instances 1022 to initialize, and cause each virtual machine instance 1022 to load a copy of the container image in one or more separate ML training containers 1030. The virtual machine instances 1022 can then each execute the code 1037 stored in the ML training containers 1030 in parallel. The model training system 120 can also provide configuration information to each ML training container 1030 via the virtual machine instances 1022 (e.g., information indicating that N ML training containers 1030 are collectively training a machine learning model and that the particular ML training container 1030 receiving the configuration information is the Xth ML training container 1030 of the N, information indicating that M virtual machine instances 1022 are collectively training a machine learning model and that the particular ML training container 1030 receiving the configuration information is initialized in the Yth virtual machine instance 1022 of the M, etc.), which can be included in the resulting model data. As described above, in some embodiments, by parallelizing the training process, the model training system 120 can significantly reduce the training time.

[0100] In some embodiments, the model training system 120 includes multiple physical computing devices, and two or more of the physical computing devices host one or more virtual machine instances 1022 executing the code 1037. Thus, in addition to on different virtual machine instances 1022 and / or ML training containers 1030, parallelization can occur on different physical computing devices.

[0101] In some embodiments, the model training system 120 includes an ML model evaluator 1028. The ML model evaluator 1028 can monitor the virtual machine instance 1022 as a machine learning model is being trained, thereby obtaining generated model data and processing the obtained model data to generate model metrics. For example, the model metrics can include quality metrics such as an error rate of the machine learning model being trained, a statistical distribution of the machine learning model being trained, a latency of the machine learning model being trained, a confidence level of the machine learning model being trained (e.g., a confidence level of how accurate the machine learning model being trained is known to be, etc. The ML model evaluator 1028 can obtain model data and evaluation data for the machine learning model being trained from the training data storage device 1060. The evaluation data is separate from the data used to train the machine learning model and includes both input data and expected output (e.g., known results), and thus the ML model evaluator 1028 can define the machine learning model using the model data and execute the machine learning model by providing the input data as input to the machine learning model. The ML model evaluator 1028 can then compare the output of the machine learning model to the expected output and determine one or more quality metrics of the machine learning model being trained based on the comparison (e.g., the error rate can be a difference or distance between the machine learning model output and the expected output).

[0102] In some embodiments, the ML model evaluator 1028 generates model metrics periodically during the training process and stores the model metrics in the training metrics data store 1065. While the machine learning model is being trained, a user can access and retrieve the model metrics from the training metrics data store 1065 via the user device 1002. The user can then use the model metrics to determine whether to adjust the training process and / or stop the training process. For example, the model metrics can indicate that the machine learning model is underperforming (e.g., has an error rate above a threshold, has a statistical distribution that is not an expected or desired distribution (e.g., not a binomial distribution, a Poisson distribution, a geometric distribution, a normal distribution, a Gaussian distribution, etc.), has an execution latency above a threshold, has a confidence level below a threshold) and / or is progressively deteriorating (e.g., a quality metric continues to worsen over time). In response, in some embodiments, the user can transmit a request to the model training system 120 via the user device 1002 to modify the machine learning model being trained (e.g., transmit a modification request). The request can include a new or modified container image, a new or modified algorithm, new or modified hyperparameters, and / or new or modified information describing the computer machines on which to train the machine learning model. The model training system 120 can modify the machine learning model accordingly. For example, the model training system 120 can cause the virtual machine instance 1022 to optionally delete the existing ML training container 1030, create and initialize a new ML training container 1030 using some or all of the information included in the request, and execute the code 1037 stored in the new ML training container 1030 to restart the machine learning model training process. As another example, the model training system 120 can cause the virtual machine instance 1022 to modify the execution of the code stored in the existing ML training container 1030 according to the data provided in the modification request. In some embodiments, the user can transmit a request to the model training system 120 via the user device 1002 to stop the machine learning model training process. The model training system 120 can then instruct the virtual machine instance 1022 to delete the ML training container 1030 and / or delete any model data stored in the training model data store 1075.

[0103] As described below, in some embodiments, model data stored in training model data storage device 1075 is used by model hosting system 140 to deploy machine learning models. Alternatively or additionally, user device 1002 or another computing device (not shown) may retrieve model data from training model data storage device 1075 to implement learning algorithms in an external device. As an illustrative example, a robotic device may include sensors for capturing input data. User device 1002 may retrieve model data from training model data storage device 1075 and store the model data in the robotic device. The model data defines a machine learning model. Therefore, the robotic device may provide the captured input data as input to the machine learning model, thereby producing an output. The robotic device may then perform actions (e.g., move forward, raise an arm, generate sound, etc.) based on the resulting output.

[0104] Although virtual machine instance 1022 is in Figure 10 While shown as a single group of virtual machine instances 1022, some embodiments of this application separate virtual machine instances 1022 that are actively assigned to perform tasks from those that are not. For example, virtual machine instances 1022 that are actively assigned to perform tasks are grouped into an "active pool," while those that are not actively assigned to perform tasks are placed in a "warming pool." In some embodiments, virtual machine instances 1022 in the warming pool may be pre-initialized in response to training requests with the operating system, language runtime, and / or other software required to enable rapid execution of tasks (e.g., rapid initialization of machine learning model training in one or more ML training containers 1030).

[0105] In some embodiments, the model training system 120 includes a processing unit, a network interface, a computer-readable media driver, and an input / output device interface, all of which can communicate with each other via a communication bus. The network interface provides connectivity to one or more networks or computing systems. Thus, the processing unit can receive information and instructions from other computing systems or services (e.g., user device 1002, model hosting system 140, etc.). The processing unit can also communicate to and from the memory of the virtual machine instance 1022 and also provide output information for an optional display via the input / output device interface. The input / output device interface can also accept input from optional input devices. The memory may contain computer program instructions (which are grouped into modules in some embodiments), which the processing unit executes to implement one or more aspects of this disclosure.

[0106] In some implementations, the model hosting system 140 includes a single physical computing device or multiple physical computing devices interconnected using one or more computing networks (not shown), wherein one or more physical computing devices host one or more virtual machine instances 1042. The model hosting system 140 can handle the acquisition and configuration of computing power (e.g., containers, instances, etc.) based on the demand for executing a trained machine learning model. The model hosting system 140 can then use the computing power to execute the machine learning model, as described in more detail below. The model hosting system 140 can automatically scale based on the capacity of execution requests received from the user device 1002 via the front end 1049 of the model hosting system 140, thereby alleviating the burden on the user of worrying about overutilization (e.g., acquiring too few computing resources and suffering performance problems) or underutilization (e.g., acquiring more computing resources than necessary to run the machine learning model and thus overpaying).

[0107] In some implementations, virtual machine instance 1042 is used to perform tasks. For example, these tasks may include executing machine learning models. Figure 10 As shown, each virtual machine instance 1042 includes an operating system (OS) 1044, a language runtime 1046, and one or more ML scoring containers 1050. The ML scoring container 1050 is similar to the ML training container 1030 in that it is a logical unit created within the virtual machine instance using resources available on the instance, and can be used to isolate task execution from other processes (e.g., task execution) occurring within the instance. In some implementations, the ML scoring container 1050 is formed by one or more container images and a top container layer. Each container image also includes one or more image layers, where each image layer represents executable instructions. As described above, some or all of the executable instructions together represent an algorithm that defines a machine learning model. Changes made to the ML scoring container 1050 (e.g., creation of a new file, modification of an existing file, file deletion, etc.) are stored in the top container layer. If the ML scoring container 1050 is deleted, the top container layer is also deleted. However, one or more container images forming a part of the deleted ML scoring container 1050 may remain unchanged. The ML scoring container 1050 may be implemented, for example, as a Linux container.

[0108] In some embodiments, the ML scoring containers 1050 each include a runtime 1054, code 1056, and dependencies 1052 (e.g., supporting software such as libraries) required by the code 1056. The runtime 1054 can be defined by one or more executable instructions that form at least a portion of a container image used to form the ML scoring container 1050 (e.g., one or more executable instructions in a container image that define an operating system and / or runtime for running in a container formed from the container image). The code 1056 includes one or more executable instructions that form at least a portion of a container image used to form the ML scoring container 1050. For example, the code 1056 includes executable instructions in a container image that represent an algorithm that defines a machine learning model, which can reference the dependencies 1052. The code 1056 can also include model data that represents characteristics of the defined machine learning model, as described in more detail below. The runtime 1054 is configured to execute the code 1056 in response to an instruction to start execution of the machine learning model. Execution of the code 1056 results in the generation of an output (e.g., a prediction or “inference” result), as described in more detail below.

[0109] In some embodiments, the runtime 1054 is the same as the runtime 1046 utilized by the virtual machine instance 1042. In some embodiments, the runtime 1054 is different than the runtime 1046 utilized by the virtual machine instance 1042.

[0110] In some embodiments, the model hosting system 140 uses one or more container images included in the deployment request (or container images retrieved from the container data store 1070 in response to a received deployment request) to create and initialize the ML scoring container 1050 in the virtual machine instance 1042. For example, the model hosting system 140 creates the ML scoring container 1050 that includes the one or more container images and / or a top container layer.

[0111] As described above, in some embodiments, the user device 1002 can submit a deployment request and / or an execution request to the model hosting system 140 via the front end 1049. The deployment request causes the model hosting system 140 to deploy a trained machine learning model into the virtual machine instance 1042. For example, the deployment request can include an identification of an endpoint (e.g., an endpoint name, such as an HTTP endpoint name) and an identification of one or more trained machine learning models (e.g., locations of one or more model data files stored in the training model data store 1075). Optionally, the deployment request also includes an identification of one or more container images stored in the container data store 1070.

[0112] Upon receiving the deployment request, the model hosting system 140 causes one or more ML scoring containers 1050 in one or more hosted virtual machine instances 1042 to be initialized. In embodiments in which the deployment request includes an identification of one or more container images, the model hosting system 140 forms the one or more ML scoring containers 1050 from the one or more identified container images. For example, the container images identified in the deployment request can be the same container images used to form the ML training containers 1030 used to train the machine learning model corresponding to the deployment request. Accordingly, the code 1056 of the one or more ML scoring containers 1050 includes one or more executable instructions in the one or more container images that represent an algorithm defining the machine learning model. In embodiments in which the deployment request does not include an identification of container images, the model hosting system 140 forms the one or more ML scoring containers 1050 from one or more container images stored in the container data store 1070 that are suitable for executing the one or more identified trained machine learning models. For example, the suitable container images can be container images that include executable instructions that represent an algorithm defining the one or more identified trained machine learning models.

[0113] In some embodiments, the model hosting system 140 also forms the one or more ML scoring containers 1050 by retrieving model data corresponding to the one or more identified trained machine learning models. For example, the deployment request can identify a location of one or more model data files stored in the training model data store 1075. In embodiments in which a single model data file is identified in the deployment request, the model hosting system 140 retrieves the identified model data file from the training model data store 1075 and inserts the model data file into a single ML scoring container 1050, which forms part of the code 1056. In some embodiments, the model data file is archived or compressed (e.g., formed from a package of separate files). Accordingly, the model hosting system 140 dearchives or decompresses the model data file to obtain a plurality of separate files and inserts the separate files into the ML scoring container 1050. In some embodiments, the model hosting system 140 stores the model data file in the same location as the model data file was stored in the ML training container 1030 that generated the model data file. For example, the model data file was originally stored in the top container layer of the ML training container 1030 at a certain offset, and the model hosting system 140 then stores the model data file in the top container layer of the ML scoring container 1050 at the same offset.

[0114] In implementations in which the deployment request identifies multiple model data files, the model hosting system 140 retrieves the identified model data files from the trained model data storage 1075. The model hosting system 140 can insert the model data files into the same ML scoring container 1050, into different ML scoring containers 1050 initialized in the same virtual machine instance 1042, or into different ML scoring containers 1050 initialized in different virtual machine instances 1042. As an illustrative example, the deployment request can identify multiple model data files corresponding to different trained machine learning models because the trained machine learning models are related (e.g., the output of one trained machine learning model is used as input to another trained machine learning model). Thus, the user can desire to deploy multiple machine learning models to ultimately receive a single output that depends on the outputs of the multiple machine learning models.

[0115] In some implementations, the model hosting system 140 associates one or more initialized ML scoring containers 1050 with an endpoint identified in the deployment request. For example, each of the one or more initialized ML scoring containers 1050 can be associated with a network address. The model hosting system 140 can map the one or more network addresses to the identified endpoint, and the model hosting system 140 or another system (e.g., a routing system, not shown) can store the mapping. Thus, the user device 1002 can use the endpoint to reference one or more trained machine learning models stored in the one or more ML scoring containers 1050. This allows the network addresses of the ML scoring containers 1050 to change without causing the user operating the user device 1002 to change the way the user references the trained machine learning models.

[0116] Once the one or more ML scoring containers 1050 are initialized, the one or more ML scoring containers 1050 are ready to execute one or more trained machine learning models. In some implementations, the user device 1002 transmits an execution request to the model hosting system 140 via the front end 1049, where the execution request identifies an endpoint and includes input (e.g., a set of input data) to a machine learning model. The model hosting system 140 or another system (e.g., a routing system, not shown) can obtain the execution request, identify one or more ML scoring containers 1050 corresponding to the identified endpoint, and route the input to the one or more identified ML scoring containers 1050.

[0117] In some embodiments, the virtual machine instance 1042 executes the code 1056 stored in the identified ML scoring container 1050 in response to the model hosting system 140 receiving the execution request. In particular, execution of the code 1056 causes executable instructions in the code 1056 corresponding to the algorithm to read the model data files stored in the ML scoring container 1050, use the inputs included in the execution request as input parameters, and generate a corresponding output. As an illustrative example, the algorithm can include coefficients, weights, layers, cluster centroids, etc. The executable instructions in the code 1056 corresponding to the algorithm can read the model data files to determine values for the coefficients, weights, layers, cluster centroids, etc. The executable instructions can include input parameters, and the inputs included in the execution request can be supplied by the virtual machine instance 1042 as the input parameters. With the machine learning model characteristics and input parameters provided, execution of the executable instructions by the virtual machine instance 1042 can be completed, resulting in an output.

[0118] In some embodiments, the virtual machine instance 1042 stores the output in the model prediction data store 1080. Alternatively or additionally, the virtual machine instance 1042 transmits the output to the user device 1002 via the front end 1049, submitting the execution result.

[0119] In some embodiments, the execution request corresponds to a set of related trained machine learning models. Accordingly, the ML scoring container 1050 can transmit the output to a second ML scoring container 1050 initialized in the same virtual machine instance 1042 or in a different virtual machine instance 1042. The virtual machine instance 1042 that caused the second ML scoring container 1050 to be initialized can then execute a second code 1056 stored in the second ML scoring container 1050, providing the received output as input parameters to executable instructions in the second code 1056. The second ML scoring container 1050 also includes model data files stored therein that are read by the executable instructions in the second code 1056 to determine values for characteristics that define the machine learning model. Execution of the second code 1056 results in a second output. The virtual machine instance 1042 that caused the second ML scoring container 1050 to be initialized can then transmit the second output to the model prediction data store 1080 and / or the user device 1002 via the front end 1049 (e.g., if no more trained machine learning models are needed to generate the output) or to a third ML scoring container 1050 initialized in the same or a different virtual machine instance 1042 (e.g., if output from one or more additional trained machine learning models is needed), and the process mentioned above can be repeated with respect to the third ML scoring container 1050.

[0120] While the virtual machine instance 1042 is executing the code 1056 stored in the ML scoring container 1050, the model hosting system 140 can receive a second execution request from the user device 1002. The second execution request can include a second set of input parameters, and the model hosting system 140 can identify a second ML scoring container 1050 that includes a second code 1056 corresponding to a second algorithm that is to be executed using the second set of input parameters. The model hosting system 140 can cause a second virtual machine instance 1042 to initialize the second ML scoring container 1050, and the second virtual machine instance 1042 can execute the second code 1056 stored in the second ML scoring container 1050, providing the second set of input parameters as input parameters to executable instructions in the second code 1056. The second ML scoring container 1050 also includes model data files stored therein that are read by the executable instructions in the second code 1056 to determine values for characteristics that define the machine learning model. Execution of the second code 1056 results in a second output. The second virtual machine instance 1042 can then transmit the second output to the model prediction data store 1080 and / or the user device 1002 via the front end 1049 (e.g., if no more trained machine learning models are needed to generate the output) or to a third ML scoring container 1050 initialized in the same or a different virtual machine instance 1042 (e.g., if output from one or more additional trained machine learning models is needed), and the process mentioned above can be repeated with respect to the third ML scoring container 1050. Figure 10The middle is shown as a single grouping of virtual machine instances 1042, but some embodiments of the present application separate virtual machine instances 1042 that are actively allocated to perform tasks from those that are not actively allocated to perform tasks. For example, those virtual machine instances 1042 that are actively allocated to perform tasks are grouped into an “active pool,” while those virtual machine instances 1042 that are not actively allocated to perform tasks are placed into a “warm pool.” In some embodiments, those virtual machine instances 1042 in the warm pool can be pre-initialized with operating systems, language runtimes, and / or other software needed for rapid task-enabled execution (e.g., rapid initialization of one or more ML scoring containers 1050, rapid execution of code 1056 in one or more ML scoring containers, etc.) in response to deployment and / or execution requests.

[0121] In some embodiments, the model hosting system 140 includes a processing unit, a network interface, computer-readable media drives, and an input / output device interface, all of which can communicate with one another via a communication bus. The network interface can provide connectivity to one or more networks or computing systems. Thus, the processing unit can receive information and instructions from other computing systems or services (e.g., the user device 1002, the model training system 120, etc.). The processing unit can also communicate to and from the memory of the virtual machine instances 1042 and also provide output information for an optional display via the input / output device interface. The input / output device interface can also accept input from optional input devices. The memory can contain computer program instructions (which are grouped into modules in some embodiments) that the processing unit executes in order to implement one or more aspects of the present disclosure.

[0122] In some embodiments, the operating environment supports multiple different types of machine learning models, such as multi-armed bandit models, reinforcement learning models, ensemble machine learning models, deep learning models, etc.

[0123] Figure 10 The model training system 120 and the model hosting system 140 depicted in the middle are not intended to be limiting. For example, the model training system 120 and / or the model hosting system 140 can also operate within a computing environment having fewer or more numbers of devices than shown. Thus, Figure 10 For example, the model training system 120 and / or the model hosting system 140 can operate within a computing environment having fewer or more numbers of devices than shown. Thus, Figure 10The descriptions of the model training system 120 and / or the model hosting system 140 in this document can be considered illustrative, rather than limiting of the present disclosure. For example, the model training system 120 and / or the model hosting system 140, or various components thereof, can implement various web service components, hosting or “cloud” computing environments, and / or peer-to-peer network configurations to implement at least a portion of the processes described herein. In some embodiments, the model training system 120 and / or the model hosting system 140 are implemented directly in hardware or software executed by a hardware device, and can for example include one or more physical or virtual servers implemented on physical computer hardware configured to execute computer-executable instructions for performing the various features described herein. The one or more servers can be geographically dispersed or geographically co-located, for example in one or more points of presence (POPs) or regional data centers.

[0124] The front end 1029 processes all training requests received from the user devices 1002 and provisions the virtual machine instances 1022. In some embodiments, the front end 1029 serves as a front door to all other services provided by the model training system 120. The front end 1029 processes the requests and ensures that the requests are properly authorized. For example, the front end 1029 can determine whether a user associated with a training request is authorized to initiate a training process.

[0125] Similarly, the front end 1049 processes all deployment and execution requests received from the user devices 1002 and provisions the virtual machine instances 1042. In some embodiments, the front end 1049 serves as a front door to all other services provided by the model hosting system 140. The front end 1049 processes the requests and ensures that the requests are properly authorized. For example, the front end 1049 can determine whether a user associated with a deployment request or an execution request is authorized to access the indicated model data and / or execute the indicated machine learning model.

[0126] The training data storage 1060 stores training data and / or evaluation data. The training data can be data used to train a machine learning model and the evaluation data can be data used to evaluate the performance of a machine learning model. In some embodiments, the training data and the evaluation data have common data. In some embodiments, the training data and the evaluation data have no common data. In some embodiments, the training data includes input data and an expected output. While the training data storage 1060 is depicted as being located outside of the model training system 120 and the model hosting system 140, this is not intended to be limiting. For example, in some embodiments not shown, the training data storage 1060 is located inside at least one of the model training system 120 or the model hosting system 140.

[0127] In some embodiments, the training metrics data storage 1065 stores model metrics. While the training metrics data storage 1065 is depicted as being external to the model training system 120 and the model hosting system 140, this is not intended to be limiting. For example, in some embodiments not shown, the training metrics data storage 1065 is internal to at least one of the model training system 120 or the model hosting system 140.

[0128] The container data storage 1070 stores container images, such as container images used to form the ML training container 1030 and / or the ML scoring container 1050, which can be retrieved by the various virtual machine instances 1022 and / or 1042. While the container data storage 1070 is depicted as being external to the model training system 120 and the model hosting system 140, this is not intended to be limiting. For example, in some embodiments not shown, the container data storage 1070 is internal to at least one of the model training system 120 or the model hosting system 140.

[0129] The training model data storage 1075 stores model data files. In some embodiments, some of the model data files consist of a single file, while other model data files are packages of multiple separate files. While the training model data storage 1075 is depicted as being external to the model training system 120 and the model hosting system 140, this is not intended to be limiting. For example, in some embodiments not shown, the training model data storage 1075 is internal to at least one of the model training system 120 or the model hosting system 140.

[0130] In some embodiments, the model prediction data storage 1080 stores output (e.g., execution results) generated by the ML scoring container 1050. While the model prediction data storage 1080 is depicted as being external to the model training system 120 and the model hosting system 140, this is not intended to be limiting. For example, in some embodiments not shown, the model prediction data storage 1080 is internal to at least one of the model training system 120 or the model hosting system 140.

[0131] Although the model training system 120, the model hosting system 140, the training data storage device 1060, the training metric data storage device 1065, the container data storage device 1070, the training model data storage device 1075, and the model prediction data storage device 1080 are illustrated as separate components, this is not intended to be limiting. In some embodiments, any one or all of these components can be combined to perform the functions described herein. For example, any one or all of these components can be implemented by a single computing device, or by multiple different computing devices such as computer servers that are logically or physically grouped together to function as a server system. Any one or all of these components can communicate via a shared internal network, and the collective system (e.g., also referred to herein as a machine learning service) can communicate with one or more of the user devices 1002 via one or more networks 106.

[0132] Various example user devices 1002 are illustrated in Figure 10 , including desktop computers, laptop computers, and mobile phones, each provided by way of illustration and not limitation. Generally, the user devices 1002 can be any computing device such as a desktop computer, laptop computer, or tablet computer, personal computer, wearable computer, server, personal digital assistant (PDA), hybrid PDA / mobile handset, mobile handset, e-book reader, set-top box, voice command device, camera, digital media player, etc. In some embodiments, the model training system 120 and / or the model hosting system 140 provide one or more user interfaces, command line interfaces (CLIs), application programming interfaces (APIs), and / or other programming interfaces to the user devices 1002 for submitting training requests, deployment requests, and / or execution requests. In some embodiments, the user devices 1002 can execute a standalone application that interacts with the model training system 120 and / or the model hosting system 140 to submit training requests, deployment requests, and / or execution requests.

[0133] In some embodiments, network 106 includes any wired network, wireless network, or combination thereof. For example, network 106 can be a personal area network, a local area network, a wide area network, an over-the-air broadcast network (e.g., for radio or television), a cable network, a satellite network, a cellular telephone network, or combination thereof. As another example, network 106 can be a publicly accessible network of linked networks, such as the Internet, which can be operated by a variety of different parties. In some embodiments, network 106 can be a private or semi-private network, such as a corporate or university intranet. Network 106 can include one or more wireless networks, such as a Global System for Mobile Communications (GSM) network, a Code Division Multiple Access (CDMA) network, a Long Term Evolution (LTE) network, or any other type of wireless network. Network 106 can use protocols and components that communicate via the Internet or any of the other aforementioned types of networks. For example, protocols used by network 106 can include HTTP, HTTP Secure (HTTPS), Message Queue Telemetry Transport (MQTT), Constrained Application Protocol (CoAP), and the like. Protocols and components that communicate via the Internet or any of the other aforementioned types of communication networks are well known to those skilled in the art and are therefore not described in greater detail herein.

[0134] Figure 11 An example provider network (or "service provider system") environment is shown in accordance with some embodiments. Provider network 1100 can provide resource virtualization to customers via one or more virtualization services 1110 that allow customers to purchase, lease, or otherwise obtain instances 1112 of virtualized resources (including, but not limited to, computing resources and storage resources) implemented on devices within one or more provider networks in one or more data centers. Local Internet Protocol (IP) addresses 1116 can be associated with resource instances 1112; local IP addresses are internal network addresses for resource instances 1112 on provider network 1100. In some embodiments, provider network 1100 can also provide public IP addresses 1114 and / or ranges of public IP addresses (e.g., Internet Protocol version 4 (IPv4) or Internet Protocol version 6 (IPv6) addresses) that customers can obtain from provider 1100.

[0135] Conventionally, the provider network 1100 can allow customers of the service provider (e.g., customers operating one or more customer networks 1150A-1150C including one or more customer devices 1152) to dynamically associate at least some public IP addresses 1114 assigned or allocated to the customer with particular resource instances 1112 assigned to the customer via the virtualization service 1110. The provider network 1100 can also allow the customer to remap a public IP address 1114 previously mapped to one virtualized computing resource instance 1112 allocated to the customer to another virtualized computing resource instance 1112 also allocated to the customer. For example, a customer of a service provider (such as an operator of one or more customer networks 1150A-1150C) can use virtualized computing resource instances 1112 and public IP addresses 1114 provided by the service provider to implement a customer-specific application and present the customer's application on an intermediate network 1140 such as the Internet. Other network entities 1120 on the intermediate network 1140 can then generate traffic to a destination public IP address 1114 published by one or more customer networks 1150A-1150C; the traffic is routed to the service provider data center and, at the data center, via a network substrate to a local IP address 1116 of a virtualized computing resource instance 1112 that is currently mapped to the destination public IP address 1114. Similarly, response traffic from the virtualized computing resource instance 1112 can be routed back via the network substrate to the source entity 1120 on the intermediate network 1140.

[0136] As used herein, a local IP address refers to an internal or "private" network address of a resource instance in a provider network, for example. A local IP address can be within an address block reserved by Internet Engineering Task Force (IETF) Request for Comments (RFC) 1918 and / or have an address format specified by IETF RFC 4193 and can change within a provider network. Network traffic originating from outside the provider network is not directly routed to a local IP address; rather, the traffic uses a public IP address that is mapped to the local IP address of the resource instance. A provider network can include network devices or equipment that provide network address translation (NAT) or similar functionality to perform the mapping from a public IP address to a local IP address and vice versa.

[0137] A public IP address is an Internet routable network address assigned to a resource instance by a service provider or customer. Traffic routed to a public IP address is translated, e.g., via 1:1 NAT, and forwarded to a corresponding local IP address of the resource instance.

[0138] Some public IP addresses can be assigned by the provider network infrastructure to particular resource instances; these public IP addresses can be referred to as standard public IP addresses, or simply standard IP addresses. In some embodiments, the mapping of standard IP addresses to local IP addresses of resource instances is a default launch configuration for all resource instance types.

[0139] At least some public IP addresses can be allocated to, or obtained by, a customer of the provider network 1100; the customer can then assign the public IP addresses it allocates to particular resource instances allocated to the customer. These public IP addresses can be referred to as customer public IP addresses, or simply customer IP addresses. Instead of being assigned to resource instances by the provider network 1100 as in the case of standard IP addresses, customer IP addresses can be assigned to resource instances by the customer, e.g., via an API provided by the service provider. Unlike standard IP addresses, customer IP addresses are allocated to a customer account, and can be remapped to other resource instances as needed or desired by the respective customer. A customer IP address is associated with a customer account (rather than a particular resource instance), and the customer controls the IP address until the customer chooses to release it. Unlike conventional static IP addresses, customer IP addresses allow a customer to mask resource instance or availability zone failures by remapping the customer's public IP addresses to any resource instance associated with the customer account. For example, a customer IP address enables a customer to resolve a customer resource instance or software problem by remapping the customer IP address to a replacement resource instance.

[0140] Figure 12 is a block diagram of an exemplary provider network that provides storage services and hardware virtualization services to customers in accordance with some embodiments. The hardware virtualization services 1220 provide a plurality of computing resources 1224 (e.g., VMs) to customers. For example, the computing resources 1224 can be rented or leased to customers of the provider network 1200 (e.g., customers that implement customer networks 1250). Each computing resource 1224 can be provisioned with one or more local IP addresses. The provider network 1200 can be configured to route packets from the local IP addresses of the computing resources 1224 to public Internet destinations, and from public Internet sources to the local IP addresses of the computing resources 1224.

[0141] Provider network 1200 can provide customer networks 1250, coupled to intermediate network 1240, e.g., via local network 1256, with the ability to implement virtual computing systems 1292 via hardware virtualization services 1220 coupled to intermediate network 1240 and provider network 1200. In some embodiments, hardware virtualization services 1220 can provide one or more APIs 1202 (e.g., network service interfaces) via which customer networks 1250 can access functionality provided by hardware virtualization services 1220, e.g., via consoles 1294 (e.g., web-based applications, standalone applications, mobile applications, etc.). In some embodiments, at provider network 1200, each virtual computing system 1292 at customer network 1250 can correspond to computing resources 1224 that are rented, leased, or otherwise provided to customer network 1250.

[0142] A customer can access functionality of storage services 1210 from virtual computing systems 1292 and / or instances of another customer device 1290 (e.g., via consoles 1294), e.g., via one or more APIs 1202, to access data from and store data to storage resources 1218A-1218N of virtual data storage devices 1216 (e.g., folders or "buckets," virtualized volumes, databases, etc.) provided by provider network 1200. In some embodiments, a virtualized data storage gateway (not shown) can be provided at customer network 1250, which can locally cache at least some data (e.g., frequently accessed or critical data), and can communicate with storage services 1210 via one or more communication channels to upload new or modified data from the local cache, so that a primary storage device (virtualized data storage device 1216) of the data is maintained. In some embodiments, a user via virtual computing systems 1292 and / or on another customer device 1290 can install and access virtual data storage device 1216 volumes via storage services 1210 acting as a storage virtualization service, and these volumes can appear local (virtualized) storage devices 1298 to the user.

[0143] While not shown in Figure 12 One or more virtualization services can also be accessed from resource instances within provider network 1200 via one or more APIs 1202. For example, a customer, a device service provider, or other entity can access a virtualization service from within a respective virtual network on provider network 1200 via API 1202 to request allocation of one or more resource instances within the virtual network or within another virtual network.

[0144] Illustrative System

[0145] In some embodiments, a system implementing part or all of the techniques described herein can include a general purpose computer system that includes or operates in connection with one or more computer accessible media, such as Figure 13 The computer system 1300 is shown comprising one or more processors 1310 coupled to a system memory 1320 via an input / output (I / O) interface 1330. Although Figure 13 The computer system 1300 is shown as a single computing device, but in various embodiments, the computer system 1300 can comprise one computing device or any number of computing devices configured to work together as the single computer system 1300.

[0146] In various embodiments, the computer system 1300 can be a uniprocessor system including one processor 1310, or a multiprocessor system including several processors 1310 (e.g., two, four, eight, or another suitable number). The processor(s) 1310 can be any suitable processor capable of executing instructions. For example, in various embodiments, the processor(s) 1310 can be general- purpose or embedded processors implementing any of a variety of instruction set architectures (ISAs), such as the x86, ARM, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISA. In multiprocessor systems, each of the processor(s) 1310 can (but not necessarily) implement the same ISA.

[0147] The system memory 1320 can store instructions and data accessible by the processor(s) 1310. In various embodiments, the system memory 1320 can be implemented using any suitable memory technology, such as random access memory (RAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), nonvolatile / Flash-type memory, or any other type of memory. In the illustrated embodiment, program instructions and data used by the

[0148] In one embodiment, I / O interface 1330 can be configured to coordinate I / O traffic between processor 1310, system memory 1320, and any peripheral devices in the device, including network interface 1340 or other peripheral interfaces. In some embodiments, I / O interface 1330 can perform any necessary protocol, timing or other data transformations to convert data signals from one component (e.g., system memory 1320) into a format suitable for use by another component (e.g., processor 1310). In some embodiments, I / O interface 1330 can include support for devices attached through various types of peripheral buses, such as a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard, for example. In some embodiments, the functionality of I / O interface 1330 can be split into two or more separate components, such as a north bridge and a south bridge, for example. Also, in some embodiments some or all of the functionality of the I / O interface 1330, such as an interface to system memory 1320, can be incorporated directly into the processor 1310.

[0149] Network interface 1340 can be configured to allow data to be exchanged between computer system 1300 and other devices 1360 attached to a network 1350 (e.g., other computer systems or devices as discussed above, with regard to the peripheral Figure 1 device). In various embodiments, network interface 1340 can support communication via any suitable wired or wireless general data networks, such as types of Ethernet network, for example. Additionally, network interface 1340 can support communication via telecommunications / telco networks, such as analog voice networks or digital fiber communications networks, via storage area networks (SAN) such as Fibre Channel SANs, or via any other suitable type of network and / or protocol.

[0150] In some embodiments, computer system 1300 includes one or more offload cards 1370 (including one or more processors 1375, and possibly one or more network interfaces 1340) that are connected using I / O interface 1330 (e.g., a bus that implements a version of the Peripheral Component Interconnect Express (PCI-E) standard or another interconnect such as QuickPath Interconnect (QPI) or UltraPath Interconnect (UPI)). For example, in some embodiments, computer system 1300 can act as a host electronic device that hosts a compute instance (e.g., operating as part of a hardware virtualization service), and one or more offload cards 1370 execute a virtualization manager that can manage compute instances executing on the host electronic device. As an example, in some embodiments, one or more offload cards 1370 can perform compute instance management operations such as pausing and / or unpausing compute instances, starting and / or terminating compute instances, performing memory transfer / copy operations, etc. In some embodiments, these management operations can be performed by one or more offload cards 1370 in cooperation with a hypervisor (e.g., at the request of the hypervisor) executed by other processors 1310A-1310N of computer system 1300. However, in some embodiments, the virtualization manager implemented by one or more offload cards 1370 can entertain requests from other entities (e.g., from the compute instances themselves), and can not cooperate with (or serve) any separate hypervisor.

[0151] In some embodiments, system memory 1320 can be one embodiment of a computer-accessible medium configured to store program instructions and data as described above. However, in other embodiments, program instructions and / or data can be received, sent or stored upon different types of computer-accessible media. Generally speaking, a computer-accessible medium can include non-transitory storage media or memory media such as magnetic or optical media, e.g., disk or DVD / CD coupled

[0152] The various implementations discussed or presented herein can be implemented in a variety of diverse operating environments, which in some cases can include one or more user computers, computing devices, or processing devices that can be used to operate any of a number of applications. User or client devices can include any of a number of general purpose personal computers, such as desktop or laptop computers running a standard operating system, as well as cellular, wireless, and handheld devices running mobile software and capable of supporting a number of networking and messaging protocols. Such a system also can include a number of workstations running any of a variety of commercially-available operating systems and other known applications for purposes such as development and database management. These devices also can include other electronic devices, such as virtual terminals, thin clients, gaming systems, and / or other devices capable of communicating via a network.

[0153] Most implementations utilize at least one network to support communications using any of a number of widely available protocols, such as Transmission Control Protocol / Internet Protocol (TCP / IP), File Transfer Protocol (FTP), Universal Plug and Play (UPnP), Network File System (NFS), Common Internet File System (CIFS), Extensible Messaging and Presence Protocol (XMPP), AppleTalk, etc. The one or more networks can include, for example, a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), the Internet, an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network, and any combination thereof.

[0154] In implementations that use a web server, the web server can run any of a variety of server or mid-tier applications, including HTTP servers, File Transfer Protocol (FTP) servers, Common Gateway Interface (CGI) servers, data servers, Java servers, business application servers, etc. The server(s) also can be capable of executing programs or scripts in response requests from user devices, e.g., by executing one or more web applications that can be implemented any programming language, such as C, C# or C++) or any scripting language such as Perl, Python, PHP, or TCL, and combinations thereof. The server(s) can also include database servers, including without limitation those available from Oracle(R), Microsoft(R), Sybase(R), IBM(R), etc. Database servers can be relational or non-relational (e.g., "NoSQL"), distributed or non-distributed, etc.

[0155] The environments disclosed herein can include various data stores, and other storage media as discussed above. These can reside in a variety of locations, such as on a storage medium local to (and / or resident to) one or more of the computers or remote from any or all of the computers across the network. In a particular set of embodiments, the information can reside in a storage-area network (SAN) familiar to those skilled in the art. Similarly, any necessary files for performing the functions attributed to the computers, servers, or other network devices can be stored locally and / or remotely, as appropriate. Where a system includes computers in the form of a computerized device, such a device can include a bus, and at least one hardware element coupled with the bus, including, for example, at least one central processing unit (CPU), at least one input device, and / or at least one output device. Such a system can further include one or more storage devices, such as disks, tape, optical storage, and the like, as well as a computer readable storage medium such as random access memory (RAM) or read-only memory (ROM), and a removable media device, a memory stick, flash drive, etc.

[0156] Such devices also can include a computer-readable storage media reader, a communications device (e.g., a modem, a network card (wireless or wired), an infrared communication device, etc.), and working memory as discussed above. The computer-readable storage media reader can be connected with a computer-readable storage medium, or multiple computer-readable storage media, to read from or write to the computer-readable storage medium or media. The computer-readable storage media can be a non-transitory computer readable storage medium, and can be a remote, local, fixed, and / or removable storage device as discussed above. The system and various devices also can include a computer program product, or software program, tangibly embodied in a computer-readable storage medium, and comprising machine instructions, or a plurality of machine instructions, for example, software, that, when executed, e.g., by a processor, cause a machine to perform operations in accordance with the described embodiments. The machine instructions can include, for example, instructions and data for performing the operations described in detail above. The user computer system can further include a communications device such as a modem, a network card (wireless or wired), an infrared communication device, etc. that enables communication between the computer system and a network (e.g., the Internet, home intranet, etc.), such that software applications executed within the computer system can communicate with and / or receive data from other devices connected to the network.

[0157] Storage media and computer readable media for containing code, or portions of code, can include any appropriate media known or used in the art, including storage media and communication media, such as but not limited to volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage and / or transmission of information such as computer readable instructions, data structures, program modules or other data, including RAM, ROM, Electrically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disk (DVD), or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by a system device. Based on the disclosure and teachings provided herein, a person of ordinary skill in the related art will appreciate other ways and / or methods to implement the various embodiments.

[0158] In the foregoing description, various embodiments have been described. For purposes of explanation, specific configurations and details are set forth in order to provide a thorough understanding of the embodiments. However, it will also be apparent to one skilled in the art that the embodiments can be practiced without the specific details. Furthermore, well-known features can have been omitted or simplified in order not to obscure the embodiment being described.

[0159] Bracketed text and blocks with dashed borders (e.g., long dashes, short dashes, dot-dash, and dots) are used herein to illustrate optional operations that add additional features to some embodiments. However, this notation should not be taken to mean that these are the only options or optional operations and / or that blocks with solid borders are not optional.

[0160] In various embodiments, reference numerals having a suffix letter (e.g., 1218A-1218N) can be used to indicate that there can be one or more instances of the referenced entity, and when there are multiple instances, each instance need not be identical, but can instead share some general characteristics or act in concert. Further, unless there is an explicit indication to the contrary, the use of a particular suffix is not intended to imply a particular quantity of the entity. Thus, in various embodiments, two entities using the same or different suffix letters can or can not have the same number of instances.

[0161] References to “one implementation,” “an implementation,” “example implementation,” etc., indicate that the implementation described can include a particular feature, structure, or characteristic, but every implementation can not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same implementation. Further, when a particular feature, structure, or characteristic is described in connection with an implementation, it is submitted that it is within the knowledge of those skilled in the art to effect such feature, structure, or characteristic in connection with other implementations whether or not explicitly described.

[0162] Further, in the various implementations described above, unless specifically stated otherwise, the conjunctive language such as “at least one of A, B, or C” is intended to be understood under its full breadth, such that A, B, or C, or any combination thereof, is contemplated (e.g., A, B, and / or C). As such, the conjunctive language is not generally intended and should not be understood to imply that the given implementation requires at least one of A, at least one of B, and at least one of C each to be present.

[0163] At least some implementations of the technology disclosed can be described in terms of the following clauses:

[0164] 1. A computer-implemented method comprising:

[0165] receiving, at an endpoint of a provider network, a request message initiated by a computing device of a user to identify a machine learning (ML) pipeline based at least in part on a data set, the request message identifying the data set, an exploration budget, and a target metric;

[0166] generating a plurality of ML pipeline plans based at least in part on the data set, wherein each ML pipeline plan identifies at least one pre-processing stage and one ML model algorithm type;

[0167] sending, to the computing device of the user, a message identifying the plurality of ML pipeline plans;

[0168] receiving a message initiated by the computing device, the message indicating a request to perform ML pipeline exploration based on one or more of the plurality of ML pipeline plans;

[0169] initiating the ML pipeline exploration, the ML pipeline exploration comprising:

[0170] performing a plurality of pre-processing stages identified within the plurality of ML pipeline plans at least in part in parallel to produce a plurality of processed data sets, wherein each of the pre-processing stages utilizes at least some values of the data set or values derived based on the data set; and

[0171] performing a plurality of ML model training jobs at least partially in parallel, each execution training an ML model using one of the ML model algorithm types utilizing at least one of the plurality of processed datasets; and

[0172] transmitting, to the computing device of the user, data indicating results of the ML pipeline exploration, the results indicating values of the target metric for each of the plurality of ML model training jobs.

[0173] 2. The computer-implemented method of clause 1, wherein the exploration budget indicates at least one of:

[0174] a maximum number of ML model training jobs to perform;

[0175] a maximum amount of time associated with aspects of the ML pipeline exploration; or

[0176] a maximum cost associated with the ML pipeline exploration.

[0177] 3. The computer-implemented method of any of clauses 1-2, further comprising:

[0178] receiving a second request message initiated by the computing device of the user, the second request message indicating a request to deploy an ML pipeline corresponding to one of the plurality of ML model training jobs;

[0179] transmitting a message to cause a model hosting system of the provider network to deploy the ML pipeline behind an endpoint; and

[0180] transmitting, to the computing device or a storage location, an identifier of the endpoint.

[0181] 4. A computer-implemented method comprising:

[0182] receiving a first request message initiated by a computing device of a user to identify a machine learning (ML) pipeline based at least in part on a dataset, the first request message identifying the dataset and an exploration budget;

[0183] generating a plurality of ML pipeline plans based at least in part on the dataset;

[0184] transmitting, to the computing device, a first one or more messages to cause the computing device to present information associated with the plurality of ML pipeline plans to the user;

[0185] receiving a second request message initiated by the computing device to begin exploring at least two of the plurality of ML pipeline plans;

[0186] training a plurality of ML models corresponding to the at least two ML pipeline plans based on at least a portion of the dataset and the exploration budget; and

[0187] transmitting a second one or more messages to the computing device of the user indicating results of the training.

[0188] 5. The computer-implemented method of clause 4, wherein the exploration budget indicates at least one of:

[0189] a maximum number of ML models to be trained as part of the training;

[0190] a maximum amount of time associated with aspects of the training; or

[0191] a maximum cost associated with the training.

[0192] 6. The computer-implemented method of any of clauses 4-5, wherein the request message includes a resource identifier of a storage location of the dataset, and wherein the method further comprises:

[0193] obtaining, by a machine learning service of a multi-tenant provider network, the dataset from a storage service via use of the resource identifier.

[0194] 7. The computer-implemented method of clause 6, further comprising:

[0195] segmenting the dataset into a plurality of sets, the plurality of sets including a training set,

[0196] wherein the training of the plurality of ML models utilizes at least the training set.

[0197] 8. The computer-implemented method of any of clauses 4-7, wherein the training of the plurality of ML models is performed at least partially in parallel in that at least two of the plurality of ML models are actively trained at least partially at the same point in time.

[0198] 9. The computer-implemented method of any of clauses 4-8, wherein for each of the plurality of ML pipeline plans, the information identifies one or more pre-processing stages and ML model algorithm types.

[0199] 10. The computer-implemented method of clause 9, wherein the information is presented to the user by the computing device via an interactive code application, and wherein the method further comprises:

[0200] receiving a third request message initiated by the computing device of the user, the third request message indicating a request to execute one stage of one ML pipeline plan of the plurality of ML pipeline plans.

[0201] 11. The computer-implemented method of clause 10, further comprising:

[0202] receiving a fourth request message initiated by the computing device of the user, the fourth request message indicating a request to execute a modified pipeline of one ML pipeline plan of the plurality of ML pipeline plans, wherein the modified pipeline is generated by the user based on the one ML pipeline plan.

[0203] 12. The computer-implemented method of any of clauses 4-10, further comprising:

[0204] receiving a fifth request message initiated by the computing device of the user, the fifth request message indicating a request to deploy an ML pipeline corresponding to one ML pipeline plan of the plurality of ML pipeline plans;

[0205] transmitting a sixth request message to cause a model hosting system of a provider network to deploy the ML pipeline behind an endpoint; and

[0206] transmitting an identifier of the endpoint to the computing device or a storage location.

[0207] 13. The computer-implemented method of any of clauses 4-12, wherein the transmitting data to the computing device of the user causes the computing device of the user to present the results to the user, the results including identifiers of a plurality of trials corresponding to the plurality of ML models, wherein for each trial of the plurality of trials, the results include a value of a target metric generated by the training of the corresponding ML model.

[0208] 14. The computer-implemented method of clause 13, wherein the request further identifies the target metric.

[0209] 15. The computer-implemented method of any of clauses 4-14, further comprising:

[0210] generating another ML pipeline including an ensemble ML model based on at least two ML models of the plurality of ML models.

[0211] 16. A system comprising:

[0212] a storage service implemented by a first one or more electronic devices of a provider network, the storage service to receive a data set transmitted on behalf of a user and store the data set to a storage location; and

[0213] a machine learning (ML) service implemented by a second one or more electronic devices of the provider network, the ML service including instructions that, upon execution, cause the ML service to:

[0214] receive a request message initiated by a computing device of the user to identify ML pipelines based at least in part on a dataset, the request message identifying the dataset and an exploration budget;

[0215] generate a plurality of ML pipeline plans based at least in part on the dataset;

[0216] transmit information associated with the plurality of ML pipeline plans to the computing device;

[0217] train a plurality of ML models corresponding to the plurality of ML pipeline plans based on at least a portion of the dataset and the exploration budget; and

[0218] transmit data indicative of results of the training to the computing device of the user.

[0219] 17. The system of clause 16, wherein the ML service causes the plurality of ML models to be trained at least in part in parallel in that at least two of the plurality of ML models are actively training at least in part at the same point in time.

[0220] 18. The system of any of clauses 16-17, wherein the information associated with the plurality of ML pipeline plans includes, for each ML pipeline plan of the plurality of ML pipeline plans, an identifier of one or more pre-processing stages and an ML model algorithm type.

[0221] 19. The system of any of clauses 16-18, wherein the information is presented to the user by the computing device via an interactive code application, and wherein the instructions, upon execution, further cause the ML service to:

[0222] receive a second request message initiated by the computing device of the user, the second request message indicating a request to execute one stage or pipeline of one of the plurality of ML pipeline plans; and

[0223] cause execution of the one stage or pipeline.

[0224] 20. The system of clause 19, wherein the instructions, upon execution, further cause the ML service to:

[0225] receiving a third request message initiated by the computing device of the user, the third request message indicating a request to perform a modified stage of one of the plurality of ML pipeline plans, wherein the modified stage is specified by the user based on the one stage; and

[0226] causing performance of the modified stage.

[0227] Therefore, the specification and drawings should be considered as illustrative only and not in a limiting sense. It will be apparent, however, that various modifications and changes can be made to the application without departing from its broader aspect and scope as set forth in the claims that follow. The specification and drawings are accordingly to be regarded as merely illustrative.

Claims

1. A computer-implemented method comprising: receiving a first request message initiated by a computing device of a user to identify machine learning (ML) pipelines based at least in part on a dataset, the first request message identifying the dataset and an exploration budget; generating a plurality of ML pipeline plans based at least in part on the dataset; transmitting, to the computing device, a first one or more messages to cause the computing device to present information to the user associated with the plurality of ML pipeline plans, wherein for each ML pipeline plan of the plurality of ML pipeline plans, the information identifies one or more pre-processing stages for pre-processing at least some values of the dataset; receiving a second request message initiated by the computing device to begin exploring at least two ML pipeline plans of the plurality of ML pipeline plans; training a plurality of ML models corresponding to the at least two ML pipeline plans based on at least a portion of a pre-processed dataset and the exploration budget; and transmitting, to the computing device of the user, a second one or more messages indicating results of the training.

2. The computer-implemented method of claim 1, wherein the exploration budget indicates at least one of: a maximum number of ML models to train as part of the training; a maximum amount of time associated with aspects of the training; or a maximum cost associated with the training.

3. The computer-implemented method of any one of claims 1-2, wherein the request message includes a resource identifier of a storage location of the dataset, and wherein the method further comprises: obtaining, by a machine learning service of a multi-tenant provider network, the dataset from a storage service via use of the resource identifier.

4. The computer-implemented method of claim 3, further comprising: segmenting the dataset into a plurality of sets, the plurality of sets including a training set, wherein the training of the plurality of ML models utilizes at least the training set.

5. The computer-implemented method of any one of claims 1-2, wherein the training of the plurality of ML models is performed at least partially in parallel in that at least two ML models of the plurality of ML models are actively trained at least partially at a same point in time.

6. The computer-implemented method of any one of claims 1-2, wherein for each ML pipeline plan of the plurality of ML pipeline plans, the information further identifies an ML model algorithm type.

7. The computer-implemented method of claim 6, wherein the information is presented to the user by the computing device via an interactive code application, and wherein the method further comprises: receiving a third request message initiated by the computing device of the user, the third request message indicating a request to execute one stage of one ML pipeline plan of the plurality of ML pipeline plans.

8. The computer-implemented method of claim 7, further comprising: receiving a fourth request message initiated by the computing device of the user, the fourth request message indicating a request to execute a modified pipeline of one of the plurality of ML pipeline plans, wherein the modified pipeline is generated by the user based on the one of the ML pipeline plans.

9. The computer-implemented method of any one of claims 1-2, further comprising: receiving a fifth request message initiated by the computing device of the user, the fifth request message indicating a request to deploy an ML pipeline corresponding to one of the plurality of ML pipeline plans; transmitting a sixth request message to cause a model hosting system of a provider network to deploy the ML pipeline behind an endpoint; and transmitting, to the computing device or a storage location, an identifier of the endpoint.

10. The computer-implemented method of any one of claims 1-2, wherein the transmitting data to the computing device of the user causes the computing device of the user to present the results to the user, the results including identifiers of a plurality of trials corresponding to the plurality of ML models, wherein for each trial of the plurality of trials, the results include a value of a target metric generated by the training of the corresponding ML model.

11. The computer-implemented method of claim 10, wherein the request further identifies the target metric.

12. The computer-implemented method of any one of claims 1-2, further comprising: generating another ML pipeline including an ensemble ML model based on at least two of the plurality of ML models.

13. A computer system, comprising: a storage service implemented by a first one or more electronic devices of a provider network, the storage service to receive a data set transmitted on behalf of a user and store the data set to a storage location; and a machine learning (ML) service implemented by a second one or more electronic devices of the provider network, the ML service including instructions that, when executed, cause the ML service to: receive a request message initiated by a computing device of the user to identify ML pipelines based at least in part on a data set, the request message identifying the data set and an exploration budget; generate a plurality of ML pipeline plans based at least in part on the data set; transmit, to the computing device, information associated with the plurality of ML pipeline plans, wherein the information associated with the plurality of ML pipeline plans includes, for each ML pipeline plan of the plurality of ML pipeline plans, an identifier of one or more preprocessing stages to preprocess at least some values of the data set; train a plurality of ML models corresponding to the plurality of ML pipeline plans based on at least a portion of the preprocessed data set and the exploration budget; and transmit, to the computing device of the user, data indicating results of the training. ​ 14. The computer system of claim 13, wherein the ML service causes the plurality of ML models to be trained at least partially in parallel because at least two ML models of the plurality of ML models are actively trained at least partially at the same point in time.

15. The computer system of any one of claims 13-14, wherein the information associated with the plurality of ML pipeline plans further comprises, for each ML pipeline plan of the plurality of ML pipeline plans, an identifier of an ML model algorithm type.

Images