Automated Machine Learning Pipeline Deployment

Abstract

Techniques for self-service machine learning are provided. A request to deploy a machine learning model is received, the request specifying whether the machine learning model should be deployed for batch inference or real-time inference. In response to determining that a deployment pipeline for the machine learning model is unavailable, a deployment pipeline is instantiated for the machine learning model, the deployment pipeline including: reading the machine learning model definition from a registry containing trained machine learning model definitions; validating the machine learning model definition using one or more test exemplars; and instantiating an inference pipeline including the machine learning model. Input data is processed using the inference pipeline.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 400,289, filed August 23, 2022, and U.S. Provisional Patent Application No. 63 / 400,306, filed August 23, 2022, the entire contents of each of which are incorporated herein by reference.

[0002] FIELD Embodiments of the present disclosure relate to machine learning. More particularly, embodiments of the present disclosure relate to automated self-service machine learning pipelines. [Background technology]

[0003] Increasingly, artificial intelligence (AI) and machine learning (ML) are being used in a wide variety of deployments and solutions to perform a variety of tasks. For example, ML models are trained and used to perform speech recognition, image classification, outcome prediction for various events or occurrences, and similar processes. In traditional systems, the actual process of designing, training, and deploying a model architecture is laborious, tedious, time-consuming, and complex. For example, a data scientist must manually define the model architecture, manually perform various operations and processes, instantiate the training process, manually train (or oversee the training), manually evaluate the resulting model, manually perform various operations and processes, instantiate the model for deployment, and finally, deploy the model. Each step in these processes adds significant complexity, requires attention from highly trained data scientists, adds delays or lags to the operations, and potentially introduces human error or inaccuracy.

[0004] Thus, AI and ML systems are significantly limited in their use and deployment because the actual process of training and deploying them is laborious and difficult. Improved systems and techniques for providing automated model training and deployment are needed. Summary of the Invention [Means for solving the problem]

[0005] According to one embodiment presented in the present disclosure, a method is provided that includes receiving a request to deploy a machine learning model, the request specifying whether the machine learning model should be deployed for batch inference or real-time inference, and, in response to determining that a deployment pipeline for the machine learning model is unavailable, instantiating a deployment pipeline for the machine learning model that includes retrieving the machine learning model definition from a registry that includes trained machine learning model definitions, validating the machine learning model definition using one or more test exemplars, instantiating an inference pipeline that includes the machine learning model, and processing input data using the inference pipeline.

[0006] According to one embodiment presented in the present disclosure, a system is provided that includes a memory comprising computer-executable instructions and one or more processors configured to execute the computer-executable instructions and cause a processing system to perform operations including receiving a request to deploy a machine learning model, the request specifying whether the machine learning model should be deployed for batch inference or real-time inference, instantiating a deployment pipeline for the machine learning model that includes retrieving the machine learning model definition from a registry that includes trained machine learning model definitions, validating the machine learning model definition using one or more test exemplars, and instantiating an inference pipeline that includes the machine learning model, and processing input data using the inference pipeline.

[0007] According to one embodiment presented in the present disclosure, a non-transitory computer-readable medium is provided comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform operations including receiving a request to deploy a machine learning model, the request specifying whether the machine learning model should be deployed for batch inference or real-time inference; in response to determining that a deployment pipeline for the machine learning model is unavailable, reading the machine learning model definition from a registry that contains trained machine learning model definitions; validating the machine learning model definition using one or more test exemplars; instantiating an inference pipeline that includes the machine learning model; and processing input data using the inference pipeline.

[0008] According to one embodiment presented in the present disclosure, a method is provided that includes receiving a request to perform continuous learning for a machine learning model, the request specifying retraining logic comprising one or more trigger criteria, automatically instantiating an inference pipeline including the machine learning model, automatically instantiating the retraining logic including the one or more trigger criteria, processing input data using the inference pipeline, and, in response to determining that the one or more trigger criteria are satisfied, automatically using the retraining logic to read new training data from a specified repository and train the machine learning model using the new training data, thereby generating a refined machine learning model using the retraining logic.

[0009] According to one embodiment presented in the present disclosure, a system is provided that includes a memory comprising computer-executable instructions and one or more processors configured to execute the computer-executable instructions and cause a processing system to perform operations including receiving a request to perform continuous learning for a machine learning model, the request specifying retraining logic comprising one or more trigger criteria, automatically instantiating an inference pipeline including the machine learning model, automatically instantiating the retraining logic including the one or more trigger criteria, processing input data using the inference pipeline, and, in response to determining that the one or more trigger criteria are satisfied, automatically using the retraining logic to read new training data from a specified repository and train the machine learning model using the new training data, thereby generating a refined machine learning model using the retraining logic.

[0010] According to one embodiment presented in the present disclosure, a non-transitory computer-readable medium comprising computer-executable instructions is provided that, when executed by one or more processors of a processing system, causes the processing system to perform operations including receiving a request to perform continuous learning for a machine learning model, the request specifying retraining logic comprising one or more trigger criteria; automatically instantiating an inference pipeline including the machine learning model; automatically instantiating the retraining logic including the one or more trigger criteria; processing input data using the inference pipeline; and, in response to determining that the one or more trigger criteria are satisfied, automatically using the retraining logic to read new training data from a specified repository; and training the machine learning model using the new training data, thereby generating a refined machine learning model using the retraining logic.

[0011] The following description and the related drawings set forth in detail certain illustrative features of the one or more embodiments. [Brief explanation of the drawings]

[0012] The accompanying drawings depict certain aspects of one or more embodiments and therefore should not be considered limiting of the scope of the present disclosure.

[0013] [Figure 1] Figure 1 depicts an exemplary environment for an improved artificial intelligence / machine learning pipeline.

[0014] [Figure 2] Figure 2 depicts an example architecture for an automated self-service machine learning pipeline.

[0015] [Figure 3] FIG. 3 depicts an example workflow for self-service machine learning model deployment.

[0016] [Figure 4] FIG. 4 depicts an example workflow for automated continuous learning pipeline deployment.

[0017] [Figure 5] FIG. 5 is a flow diagram depicting an exemplary method for self-service machine learning deployment.

[0018] [Figure 6] FIG. 6 is a flow diagram depicting an exemplary method for real-time inference using automatically deployed models.

[0019] [Figure 7] FIG. 7 is a flow diagram depicting an exemplary method for batch inference using automatically deployed models.

[0020] [Figure 8] FIG. 8 is a flow diagram depicting an exemplary method for automated continuous learning development.

[0021] [Figure 9] FIG. 9 is a flow diagram depicting an exemplary method for automatically training a machine learning model using a deployed pipeline.

[0022] [Figure 10] FIG. 10 is a flow diagram depicting an exemplary method for automatically deploying machine learning models.

[0023] [Figure 11] FIG. 11 is a flow diagram depicting an exemplary method for automatically performing continuous training of a machine learning model.

[0024] [Figure 12] FIG. 12 depicts an exemplary computing device configured to implement various aspects of the disclosure.

[0025] Additional aspects of the present disclosure can be found in the attached appendix.

[0026] To facilitate understanding, the same reference numerals have been used, where possible, to designate the same elements common to the figures. It is contemplated that elements and features of one embodiment may be beneficially incorporated in other embodiments without further recitation. DETAILED DESCRIPTION OF THE INVENTION

[0027] Detailed Description Aspects of the present disclosure provide apparatuses, methods, processing systems, and computer-readable media for automated machine learning operations. For example, in some embodiments, techniques and architectures are provided for enabling automated (e.g., self-service) deployment of machine learning models based on simple definitions rather than requiring complex configurations and deep technical understanding. In some embodiments, techniques and architectures are provided for enabling automated (e.g., self-service) training and continuous learning of machine learning models based on similarly simple definitions (as opposed to the complex configurations and technical understanding required in conventional systems).

[0028] In conventional systems, a user (e.g., a data scientist or engineer) is required to manually build the required infrastructure and train and use machine learning models. For example, a user may configure containers or computing instances, launch microservices, and similar processes. Furthermore, in many conventional systems, only one user or entity (e.g., someone logged into a production account) can perform the various operations required to instantiate or deploy a trained model.

[0029] In aspects of the present disclosure, a user can instead simply provide an automated system with a model definition and / or configuration file (e.g., indicating whether the model should be deployed as a real-time or batch inference endpoint). The system can then automatically instantiate any required infrastructure, perform any associated operations or evaluations (e.g., validate the model), and deploy and / or train the model according to the configuration. This substantially reduces the time, effort, and expertise required to work with and deploy machine learning models, enabling ML to be used for a broader and more diverse range of solutions that would otherwise be too niche to justify the effort. Furthermore, aspects of the present disclosure facilitate rapid continuous learning and automated updates, ensuring continued success and improved model accuracy. In addition, aspects of the present disclosure can reduce human error in the process, thereby resulting in a more reliable and accurate computing system. Additionally, aspects of the present disclosure can automatically, intelligently, and dynamically reuse infrastructure when relevant, thereby reducing the computational burden of the training and / or deployment process (compared to traditional solutions where users perform the process manually and rarely or never reuse previous infrastructure).

[0030] As used herein, a "pipeline" generally refers to a set of components, operations, and / or processes used to perform a task. For example, a deployment pipeline may refer to a set of components, operations, and / or processes for deploying a machine learning model for inference. An inference pipeline may refer to a set of components, operations, and / or processes for performing inference using a machine learning model. A training pipeline may refer to a set of components, operations, and / or processes for training or refining a machine learning model based on training data. Aspects of the present disclosure provide automated deployment and use of such pipelines to perform self-service machine learning (e.g., inference and / or training).

[0031] In some embodiments, automated machine learning model deployment (referred to in some aspects as self-service machine learning) is provided. In one embodiment, a deployment request or submission can be received from a user to instantiate a model for inference. The request may specify, for example, the model architecture or definition, whether the model should be deployed as a batch or real-time inference system, how to access input data and / or where to provide output, and the like. In one embodiment, if a deployment pipeline exists for the architecture, the system can reuse this existing pipeline to deploy the model. If such a pipeline does not exist, the system can instantiate one.

[0032] In at least one embodiment, as discussed above, deploying a deployment pipeline (also referred to as "instantiating," "spawning," or "creating" a pipeline) can include instantiating a set of components or processes and performing the sequence of operations required to deploy a model. The deployment pipeline can then be used to actually deploy the model (e.g., to instantiate an inference pipeline for the model). In some embodiments, the deployment pipeline reads the model definition and configuration (from a request or from a registry, as discussed in more detail below), optionally validates the model (e.g., to ensure it behaves deterministically), and finally, actually instantiates a new endpoint or inference pipeline that is used to serve the model to users.

[0033] In one embodiment, when input is ready for processing (e.g., when a user provides input data for real-time inference and / or when batch data is ready for processing), the system processes the input using an instantiated inference pipeline. As discussed above, deploying an inference pipeline can include instantiating a set of components or processes to perform the sequence of operations required to process the input data using a model. For example, the inference pipeline may optionally perform preprocessing on the input data, pass the data through a model, generate output, and return the output as appropriate. In this way, the system provides fast and automated deployment of trained models for inference.

[0034] In some embodiments, automated continuous learning of machine learning models (referred to in some aspects as self-service training and / or continuous learning) is provided. In one such embodiment, a request can be received from a user to instantiate a continuous learning pipeline. For example, the request may include a training script / container (e.g., defining how training should be performed), a continuous training configuration file (e.g., retraining schedule or criteria), and a model deployment configuration file (e.g., a configuration file used to define how the model is deployed for inference, such as whether real-time or batch inference should be used).

[0035] In some embodiments, the training container can be read or provided in a central location, and a training schedule can be instantiated (e.g., by subscribing to input table updates or using a timer or other trigger criteria). In some embodiments, the training pipeline can be deployed and used immediately upon receiving a submission / request. This training pipeline generates / trains a machine learning model based on the provided architecture. For example, in one embodiment, the training pipeline can read new training data (e.g., from a defined storage location or database as indicated in the request), use the data to refine the model, and store the refined model in a model registry. In some aspects, the model is stored along with a model deployment configuration file (which may be provided in the request) and an associated indicator or flag indicating that it is ready for deployment.

[0036] In some embodiments, storing the model in the registry with this flag can automatically initiate the deployment process, as discussed above. The deployed model can then be used for inference, as discussed above.

[0037] In embodiments, model inference may have an independent schedule from the continuous training pipeline. Similarly, new (refined) models can also be deployed as different versions (enabling model versioning), so that it is possible to have several different model versions in production (e.g., until older models are retired).

[0038] In one embodiment, once the trigger criteria for retraining are met, the retraining logic and / or pipeline and associated configuration files (from the request) can be used to perform the retraining by accessing the training containers and configurations from a central location (and the file locations referenced therein), reading new data, etc., as discussed above. This process may then be repeated indefinitely, continually providing newly refined models.

[0039] An example environment for artificial intelligence / machine learning pipelines FIG. 1 depicts an exemplary environment 100 for an improved artificial intelligence / machine learning pipeline.

[0040] In the illustrated environment 100, the machine learning system 115 is communicatively linked to the data repository 105 and one or more applications 125. In embodiments, the data repository 105, the machine learning system 115, and the applications 125 may be coupled using any suitable technology. The connections may include wireless connections, wired connections, or a combination of wired and wireless connectivity. In at least one aspect, the data repository 105, the machine learning system 115, and the applications 125 are communicatively coupled via the Internet.

[0041] Although a single data repository 105 is depicted for conceptual clarity, in embodiments there may be any number of such repositories. Additionally, although depicted as a separate component for conceptual clarity, in some embodiments data repository 105 may be implemented or stored within other components, such as within machine learning system 115 and / or application 125.

[0042] In the illustrated example, data repository 105 stores data 110. Data 110 may generally correspond to a wide variety of data, such as training data for machine learning models, input data during runtime (e.g., for batch inference), output data (e.g., generated inferences), and the like. As shown, machine learning system 115 uses data 110 in conjunction with one or more machine learning models. For example, as discussed in more detail below, machine learning system 115 may read or access data 110 and train or refine machine learning models using an automated training and / or continuous learning pipeline. Similarly, as discussed in more detail below, machine learning system 115 may read or access data 110 as input to an automated inference pipeline.

[0043] As shown, users 120 can interact with machine learning system 115 to perform various machine learning-related tasks. For example, users 120 may be data scientists, engineers, or other users who desire to train and / or deploy machine learning models. In some embodiments, users can also provide requests or submissions to machine learning system 115 to trigger automated instantiation and / or deployment of machine learning models and training pipelines, as discussed in more detail below.

[0044] In some aspects, user 120 may indicate a model definition (either included in the request or as a pointer to the model, which may be stored in a registry, such as in data repository 105) along with a configuration that specifies how the model should be deployed. For example, the configuration may indicate that the model should be launched in batch mode and specific storage locations (e.g., particular tables or other storage structures in data repository 105) from which input data can be accessed and / or specific storage locations (e.g., particular tables or other storage structures in data repository 105) from which output data should be stored. In response, machine learning system 115 may automatically deploy the model, as appropriate.

[0045] Similarly, in some aspects, user 120 may indicate model definitions and training configurations, enabling machine learning system 115 to automatically instantiate the training process. For example, the configurations may specify where training data will be stored (e.g., a particular table or other storage structure within data repository 105), the contents of the training criteria (e.g., whether retraining should be performed whenever new data becomes available at that location, when a certain amount of data or exemplars is available, when a defined period of time has passed, and the like), whether machine learning system 115 should automatically deploy newly refined models, whether newly refined models should replace previous models (e.g., whether previous inference pipelines should be closed when new ones are created), and the like.

[0046] In the illustrated embodiment, a set of applications 125 may interface with machine learning system 115 for various purposes. For example, applications 125 may use trained machine learning models to generate predictions or suggestions for users 130. In an embodiment, applications 125 may use the models locally (e.g., machine learning system 115 may deploy them to applications 125) or may access models hosted by machine learning system 115 (e.g., using an application programming interface (API)). In an embodiment, applications 125 themselves may be hosted in any suitable location, including on a user device (e.g., on a personal device of user 130), in a cloud-based deployment (accessible via the user device), and the like.

[0047] As shown, application 125 can optionally transmit data to data repository 105. For example, for batch inference, user 130 may use application 125 to provide or store input data in an appropriate location within data repository 105 (application 125 may know the appropriate location based on the configuration used to instantiate the model, as discussed above). Machine learning system 115 can then automatically retrieve the data, process it, and generate output data, as discussed above. In some embodiments, application 125 may similarly use data repository 105 to provide input data for real-time inference. In other aspects, application 125 may directly provide input data to machine learning system 115 for real-time inference.

[0048] In some embodiments, the machine learning system 115 can provide the data directly to the requesting user 130. For example, the machine learning system 115 may provide the generated output to the application 125 that provided the input data. In some embodiments, the machine learning system 115 stores the output data in an appropriate location in the data repository 105, allowing the application 125 to retrieve or access it.

[0049] In at least one embodiment, some or all of the applications 125 can be used to provide or enable continuous learning. In one such embodiment, the application 125 may store the labeled exemplar in the data repository 105 once the label is known. For example, after using input data to generate an inference (e.g., a predicted future value for a variable based on current data), the application 125 may subsequently determine an actual value for the variable. This actual value can then be used as a label for previous data used to generate the inference, and the labeled exemplar can be stored in the data repository 105 (e.g., in a location used for continuous training of the model). This can allow the machine learning system 115 to automatically retrieve and use it to refine the model, as discussed above.

[0050] An example architecture for an automated self-service machine learning pipeline Figure 2 depicts an example architecture 200 for an automated self-service machine learning pipeline. The architecture shows one exemplary implementation of a machine learning system, such as machine learning system 115 of Figure 1. While the depicted example includes various individual components for conceptual clarity, the operations of each component may be performed by any number of components, collectively or independently.

[0051] In the illustrated example, development component 205 is used to define machine learning models (e.g., by user 120 of FIG. 1 ). In one embodiment, projects 210A-B within development component 205 may each correspond to an ongoing machine learning project. For example, project 210A may correspond to a data scientist developing a machine learning model for classifying images based on the content they depict, while project 210B may correspond to a data scientist developing a machine learning model for identifying spoken keywords in audio data. In general, development component 205 may be implemented using any suitable technology and may reside in any suitable location. For example, development component 205 may correspond to one or more separate computing devices used by a user to develop models, may correspond to a machine learning system application or interface, and the like.

[0052] In some embodiments, a user may use the development component 205 to define a model's architecture, model configuration, and the like. For example, using the development component 205, a user may create a project 210 for training a specific model architecture (e.g., a neural network). Using the development component 205, a user may specify information such as the model's hyperparameters (e.g., number of layers, learning rate, and the like), as well as information related to the features to be used, pre-processing they wish to apply to input data, and the like. In some embodiments, the development component 205 may also be used by a user to perform operations such as data exploration (e.g., investigating potential data sources for a model), feature engineering, and the like.

[0053] In the illustrated example, once the model architecture is ready to begin training and / or the model is ready for deployment, the deployment component 205 can provide relevant data to the deployment component 215. For example, a user may provide a submission to the deployment component 215 that includes the model architecture or definition, configuration files, and the like.

[0054] In the illustrated example, deployment component 215 includes model registry 220 and feature registry 225. While depicted as separate components for conceptual clarity, in some aspects model registry 220 and feature registry 225 may be combined into a single registry or data store. In one embodiment, model registry 220 is used to store model definitions and / or configuration files defined using development component 205. For example, a user may provide a model definition (e.g., indicating architecture, hyperparameters, and the like) for a given project 210 as a submission to deployment component 215, which stores it in model registry 220. In some embodiments, deployment component 215 can also store provided configurations along with the model definition in model registry 220 (e.g., specifying whether the model should be instantiated as a real-time inference model or a batch inference model).

[0055] In some embodiments, a flag, tag, tag, or other indication can also be stored along with the model in model registry 220. As discussed above, this flag can be used to indicate whether the model is ready for training and / or deployment. For example, a user may set a flag or otherwise cause model registry 220 to update when relevant, such as when the architecture is ready to begin training, when the model is trained and ready for deployment, and the like.

[0056] In an embodiment, feature registry 225 may include information related to features and / or pre-processing that may be applied to a model. For example, feature registry 225 may include definitions for data transducers or other components that may be used to clean, normalize, or otherwise pre-process input data.

[0057] As shown, deployment component 215 is coupled with service provision component 230. Service provision component 230 can generally access definitions and configurations in model registry 220 and instantiate pipelines 235, 240, and / or 245. For example, based on a user submission (or based on a flag associated with a model in model registry 220), machine learning system 115 may automatically retrieve the model definition and configuration and use it to instantiate the corresponding pipeline.

[0058] As one example, if a given model's configuration (or the configuration included in a user request or submission) indicates that the model should be instantiated for real-time inference, the service delivery component 230 may spawn a real-time inference pipeline 235. As another example, based on the submission, request, and / or tag, the service delivery component 230 can additionally or alternatively instantiate a batch inference pipeline 240 and / or a continuous training pipeline 245.

[0059] In the illustrated example, real-time inference pipeline 235 includes a copy or instance of model 250A and API 255, which can be used to enable or provide access to model 250A (e.g., to application 270A). For example, application 270A may use API 255 to provide input data to real-time inference pipeline 235, which then processes it with model 250A and generates an output inference. This output can then be returned back to application 270A via API 255.

[0060] In the depicted example, batch inference pipeline 240 includes feature storage 260A, a copy or instance of model 250B, and predictor 265A. For example, application 270B or other entity may provide input data to be processed in batches, which can be stored in features 260A. When appropriate trigger criteria are met (e.g., defined in the configuration), batch inference pipeline 240 reads the data, processes it with model 250B, and stores the output data in prediction 265A.

[0061] As shown, the continuous training pipeline 245 includes feature storage 260B, a copy or instance of model 250C, and prediction storage 265B. For example, application 270C may provide input data to be processed in real time or in batches, which can optionally be stored in feature 260B. The continuous training pipeline 245 can then process this data using model 250C to generate predictions 265B, which are returned to the requesting application 270C. In the illustrated example, application 270C may optionally store labeled exemplars (e.g., newly labeled data) in feature 260B or other repository to enable continuous training. In some aspects, when appropriate trigger criteria are met (e.g., defined in the configuration), the continuous training pipeline 240 reads the new labeled training data and uses it to refine or update model 250C. In some aspects, as discussed above, the refined model can then be stored in the model registry 220, which may trigger the automatic creation of another inference pipeline for the refined model.

[0062] Example Workflow for Self-Service Model Deployment 3 depicts an example workflow 300 for self-service machine learning model deployment. For example, workflow 300 may be used to instantiate real-time and / or batch inference pipelines. In some embodiments, workflow 300 is performed by a machine learning system, such as machine learning system 115 of FIG. 1.

[0063] In the illustrated example, model 250 is provided to model registry 220. For example, as discussed above, a user (e.g., a data scientist) may provide a request or submission that includes model 250 and request that it be instantiated for inference. In some embodiments, as discussed above, model 250 corresponds to a model definition, which specifies information about the model or associated data, such as its design and / or architecture, hyperparameters, and the like.

[0064] Although not included in the illustrated example, in some aspects, model 250 also includes (or is associated with) one or more configuration files that indicate how the model should be instantiated. For example, the configuration may indicate whether model 250 is ready for deployment, whether it should be deployed for batch or real-time inference, the specific input data to be used, pre-processing that should be applied to the input data, and the like.

[0065] In the illustrated workflow 300, the model evaluator 305 can monitor the model registry 220 to enable automated deployment of machine learning models. For example, the model evaluator 305 may identify new models to be stored in the model registry 220 and periodically scan the registry and do the same. In some aspects, the model evaluator 305 can identify any models with a deployment flag indicating that they are ready for deployment. For example, as discussed above, a user (or another system) may add a model 250 to the model registry 220 with a deployment indicator or flag, or may set a deployment indicator or flag for a model 250 already stored in the registry 220.

[0066] In some aspects, the model evaluator 305 may additionally or alternatively evaluate other criteria prior to deployment, such as whether the model group to which the model 250 belongs exists and has the appropriate indicators (e.g., whether the group to which the model belongs also has the "Deploy" flag set to true), whether the model is approved / registered (in addition to having the "Deploy" indicator), whether the model 250 has the appropriate links to configuration files, and similar processes.

[0067] In the depicted example, if the model evaluator 305 determines that the relevant criteria are satisfied such that the model 250 is ready for deployment, the deployment pipeline component 310 is triggered to begin the deployment process. In some aspects, the deployment pipeline component 310 may also perform some evaluation to determine whether a deployment pipeline already exists for the model 250. In some embodiments, for a given model 250, the system can deploy multiple instances of the model using a single model deployment pipeline. For example, the same pipeline may be used to deploy a model as a real-time inference endpoint and a batch inference endpoint.

[0068] In at least one embodiment, the deployment pipeline can similarly be reused across multiple versions of the same model. For example, in one such embodiment, different versions of model 250 (e.g., with different weights, such as after a retraining or refinement operation) may be deployed by the same deployment pipeline if the architecture remains the same (e.g., the new version of the model uses the same input data, the same preprocessing, and the like).

[0069] In the illustrated example, therefore, the deployment pipeline component 310 can first determine whether a deployment pipeline already exists for the shown model definition. If so, the deployment pipeline component 310 can refrain from instantiating a new deployment pipeline and instead deploy the model using the existing deployment pipeline. If such a pipeline does not exist, in the illustrated example, the deployment pipeline component 310 can instantiate one (as indicated by arrow 312).

[0070] In some embodiments, in addition to or instead of checking whether a deployment pipeline already exists, the deployment pipeline component 310 can evaluate various other criteria before proceeding. For example, the deployment pipeline component 310 may check whether required tags exist in the model's configuration file (e.g., whether tags indicating "batch inference" or "real-time inference" exist, whether the deployment tag is set to true, and the like).

[0071] As discussed above, instantiating a deployment pipeline may generally involve instantiating, creating, deploying, or starting a set of components or other processes (e.g., software modules) to perform the sequence of operations required to deploy model 250. For example, deployment pipeline component 310 may create deployment pipeline 315, which includes validation component 320 and / or deployment component 325. Although two separate components are depicted in deployment pipeline 315 for conceptual clarity, in some aspects the respective operations may be combined or distributed across any number of components.

[0072] Additionally, other components or operations not depicted in the illustrated example may be included. For example, in at least one embodiment, the system may monitor the model registry 220 using one or more state change rules and update the deployment pipeline 315 accordingly. For example, if the state or status of a model and / or model group changes from "approved" to "pending" or "rejected," and / or if the model deployment flag changes from "true" to "false," the system may automatically undeploy the model (e.g., by deleting the production account from it, deleting the deployment pipeline, and similar processes). This can then be redeployed using workflow 300 if the state changes, as discussed above.

[0073] In some embodiments, instantiating the deployment pipeline 315 is performed based, at least in part, on the configuration associated with the model 250. That is, different operations or processes may be used to deploy the model depending on whether pre-processing should be performed on the input data, the pre-processing used, whether the deployment mode is real-time or batch inference, and the like.

[0074] The deployment pipeline 315 is generally used to deploy an inference pipeline that generates inferences or predictions using the depicted model 250. The validation component 320 may generally be used to validate and / or perform integration tests for the model 250. For example, the validation component 320 may be used to verify that the model 250 operates deterministically. Some models may perform non-deterministically (e.g., with some degree of randomness) in their predictions, which may be undesirable for the system. In some aspects, the validation component 320 may therefore process input data (e.g., sample data included in a registry with the model 250) multiple times to verify that output predictions are identical. That is, the validation component 320 may process the test exemplar more than once, compare the generated outputs, and determine whether they match. If applicable, the validation component 320 may verify that the model behaves deterministically and can proceed with deployment. In one embodiment, if the mode is not deterministic, the validation component 320 can further prevent processing (eg, prevent the model from being deployed).

[0075] As another validation example, the validation component 320 may verify that incorrect or otherwise inappropriate input data results in an appropriate error or other output. That is, the validation component 320 may process this data using a model using a test exemplar that does not satisfy one or more criteria specified in the model's configuration (e.g., in a registry). For example, the criteria may specify the appropriate length of the input data (e.g., the number of dimensions in a vector), specific features to be used as inputs, and the like. In one embodiment, the text data may fail one or more of these criteria. Rather than generating a faulty output (e.g., an unreliable prediction), in one embodiment, the validation component 320 can verify that the model returns an error or otherwise does not produce an output guess.

[0076] As another validation example, the validation component 320 may verify that the model operates correctly based on test data indicated in the configuration. For example, the validation component 320 may process valid inputs (e.g., supplied or indicated by a user), generate output guesses, and verify that the output guesses are valid (e.g., that the outputs themselves are valid guesses and / or that the outputs match the appropriate or correct outputs for the test data as indicated in the configuration data).

[0077] In at least one embodiment, validation component 320 can determine the tests to perform based, at least in part, on the configuration associated with model 250. For example, the configuration may specify the tests to perform, or validation component 320 may determine the relevant tests based on the specific architecture or design of the model (e.g., based on the input data it uses, how that input data is formatted, and the like).

[0078] In one embodiment, if the validation component 320 determines that any aspect of validation and integration has failed, the deployment pipeline 315 can be stopped. That is, the deployment pipeline 315 may prevent any further processing and may not instantiate or deploy the model inference pipeline. In some embodiments, the validation component 320 and / or the deployment pipeline 315 can additionally or alternatively generate and provide an alert or other notification (e.g., to a user associated with the model 250, such as the data scientist or other user who designed it, or the user who provided the request / submission and deployed it). In one embodiment, this notification may indicate which validation tests failed, next steps to be taken (e.g., how to remediate them), and the like.

[0079] In the illustrated example, if validation component 320 confirms that the relevant tests are successful and the model is validated, deployment component 325 may be triggered, as indicated by arrow 327, to instantiate and / or deploy inference pipeline 330.

[0080] In some embodiments, as discussed above, deploying the inference pipeline 330 may generally include instantiating, creating, deploying, or starting a set of components or other processes (e.g., software modules) to perform inference using the model 250. For example, the deployment component 325 may determine (e.g., based on configurations included with the model and / or submission or request) whether the model 250 is being deployed for batch or real-time inference and proceed accordingly (e.g., instantiating appropriate systems or components for each).

[0081] In the illustrated example, deployment component 325 creates an inference pipeline 330, which includes a model instance 335 that corresponds to model 250. That is, model instance 335 may be a copy of model 250. As discussed above, deployment pipeline 315 may create multiple inference pipelines 330 for inference, each with a corresponding model instance 335. In some embodiments, instantiating an inference pipeline 330 may include starting or triggering an endpoint (e.g., a virtual machine or container) to host the model instance 335.

[0082] Although not included in the illustrated example, in some embodiments, the inference pipeline 330 can optionally include other components, such as a feature pipeline. That is, the deployment component 325 may read or determine (e.g., based on the configuration files of the model 250 in the model registry 220) the transformations or other pre-processing that should be applied to the input data and use this information to create a feature pipeline (e.g., a sequence of components or processes) to implement the indicated operations within the inference pipeline 330. In at least one embodiment, the configuration defines the feature pipeline itself, or otherwise points to or indicates specific transformations or other operations to be applied to the input data.

[0083] In some embodiments, as discussed above, the inference pipeline 330 may additionally or alternatively include other components, such as an API (e.g., API 255 of FIG. 2 ) that enables connectivity between the model instance 335 and applications using the inference pipeline 330, data stores (or pointers to data stores) in which input and / or output data is stored, and the like.

[0084] The inference pipeline 330 (or a pointer to it) can then be returned or provided to the entity that requested the deployment or provided the submission. For example, a pointer or link to the inference pipeline 330 may be returned to allow a user or other entity to begin using the inference pipeline 330.

[0085] In this manner, aspects of the present disclosure may enable automated deployment of trained machine learning models in a self-service manner, reducing or eliminating the need for manual configuration and instantiation of required components and systems required by traditional approaches, allowing models to be deployed more quickly, more accurately, and more reliably than traditional approaches.

[0086] Example Workflow for Deploying a Continuous Learning Pipeline 4 depicts an example workflow 400 for automated continuous learning pipeline deployment. For example, workflow 400 may be used to instantiate a training pipeline. In some embodiments, workflow 400 is performed by a machine learning system, such as machine learning system 115 of FIG. 1.

[0087] In the illustrated example, model 250A can be provided to model registry 220, as discussed above. For example, a user may submit model 250A, along with a corresponding configuration, to model registry 220 and request that the model be trained and / or deployed for continuous learning. In some embodiments, model 250A may be an untrained model (e.g., a model definition that specifies an architecture and hyperparameters, but without trained weights or other learnable parameters, or with random values ​​for these parameters). In other embodiments, model 250A may be a trained model.

[0088] In one embodiment, when model 250A is a trained model, model evaluator 305 may identify one or more flags indicating that it is ready for deployment, as discussed above. This may trigger the deployment process, generally discussed above with reference to FIG. 2, in which model evaluator 305 evaluates various criteria before triggering deployment pipeline component 310, which in turn evaluates one or more criteria before using an existing deployment pipeline 315 or instantiating a new one (indicated by arrow 427), which in turn performs various evaluations and actions to create inference pipeline 330 for model 250A (indicated by arrow 429).

[0089] The inference pipeline 330 can then be used for inference, as discussed above. In at least one aspect, before, during, or after this process, a training component 405 can additionally perform various operations and instantiate the training pipeline 410 (as indicated by arrow 407). In some embodiments, the training component 405 may also be used if the model 250A has not yet been trained. That is, the training component 405 may be used to provide initial training of the model.

[0090] As shown, training component 405 may monitor model registry 220, similar to model evaluator 305. In one embodiment, training component 405 can determine whether model 250A in model registry 220 is ready for training. For example, training component 405 may determine whether a training and / or refinement flag or indicator (e.g., in its configuration file) is associated with the model. When training component 405 detects such an indicator, training component 405 can automatically instantiate training pipeline 410 (as indicated by arrow 407).

[0091] As discussed above, instantiating the training pipeline 410 may generally correspond to instantiating, creating, deploying, or otherwise initiating a set of components or other processes (e.g., software modules) to perform the sequence of operations required to train the model 250A. For example, the training component 405 may create the training pipeline 410, which includes the update component 415 and / or the evaluation component 420. While two separate components of the training pipeline 410 are depicted for conceptual clarity, in embodiments, the respective operations may be combined or distributed across any number of components. Similarly, other operations and components than those included within the depicted workflow 400 may also be used.

[0092] In the illustrated example, update component 415 may generally be used to retrieve training data for model 250A (e.g., from data 425) and refine the model (e.g., update one or more learnable parameters) based on the training data. While depicted as a single repository for conceptual clarity, in some embodiments, data 425 may be distributed across any number of systems and repositories. For example, update component 415 may retrieve or receive input examples from one data store, look up target outputs / indicators in another data store, and perform similar processing.

[0093] In some embodiments, data 425 is indicated in the configuration of the model and / or in a request or submission requesting that the model be trained. That is, the submission and / or configuration may indicate a specific storage location within data 425 (e.g., a database table or other repository) where training data for model 250A can be found.

[0094] In some embodiments, the particular operations used by the training component 415 may vary depending on the particular model architecture. That is, the training component 405 may instantiate different components or processes for the update component 415 depending on the particular architecture (e.g., depending on whether the model 250A is a neural network, a random forest model, or the like). In this way, the system can provide training automatically and dynamically without requiring the user to understand or manually instantiate such components.

[0095] As an example, if model 250A is an artificial neural network, update component 415 may pass input training samples through the model, generate output guesses, and compare these guesses against ground truth indicators included with the input data (e.g., classification or numeric values). The difference between the generated output and the actual desired output may be used to define a loss, which may be used to update the model parameters (e.g., to update the weights of one or more layers of the mode using backpropagation and gradient descent).

[0096] In some aspects, the update component 415 can perform this training or refinement process based on the submission and / or configuration of the model 250A. For example, the update component 415 may determine training hyperparameters (e.g., learning rate) based on the configuration, determine whether to use batches of training data (e.g., batch gradient descent) or individual training samples (e.g., stochastic gradient descent), and the like.

[0097] In the illustrated example, once training is complete, the trained model is passed to evaluation component 420. In an embodiment, training may be deemed “complete” based on various criteria, some or all of which may be specified in the configuration and / or submission of model 250A. For example, termination criteria may include refining the model using a defined number of exemplars, refining the model using training data until a defined period of time has passed, refining the model until a minimum desired model accuracy is reached, refining the model until all of the exemplars available in data 425 have been used, and similar processes.

[0098] In one embodiment, the evaluation component 420 may optionally perform various evaluations on the updated model. For example, the evaluation component 420 may process test data (e.g., a subset of training exemplars represented for the model in data 425) and determine model accuracy, inference time (e.g., the time it takes to process one test sample using the trained model), and the like. In some aspects, the evaluation component 420 may determine aspects of the model itself, such as its size (e.g., the number of parameters and / or required storage space). Generally, the evaluation component 420 may collect a wide variety of performance metrics for the model. These metrics may be stored (in data 425) along with the updated model in the model registry 220 (e.g., in a configuration file), along with the training data, stored to a user (e.g., transmitted or displayed to a user or other entity that initiated the training process), and similarly processed.

[0099] In the illustrated workflow 400, the training pipeline 410 outputs the updated model 250B and stores it back in the model registry 220. In some embodiments, the training pipeline 410 can automatically set a deployment flag or indicator for the model 250B such that the model evaluator 305 automatically initiates the deployment process for it, as discussed above.

[0100] Although not included in the illustrated embodiment, in some aspects, once a model is deployed in the inference pipeline 330, the training component 405 can monitor one or more trigger criteria to determine when retraining is required. For example, the training component 405 can use a time-based trigger (e.g., to enable periodic retraining, such as weekly). In some aspects, the training component 405 uses an event-based trigger, such as user input or the addition of new training data in the presented data 425, or monitors whether the deployed model (in the inference pipeline 330) produces correct predictions.

[0101] For example, users of inference pipeline 330 may use the developed model to generate output inferences or predictions based on their input data. In some aspects, participating entities may optionally subsequently determine actual output indicators for the data (e.g., the model provides a prediction for the future, and the actual value can subsequently be determined). Such entities may then optionally create and store new training samples (e.g., in the depicted portion of data 425), each new training sample including the input data and the corresponding ground truth output value or indicator.

[0102] In one embodiment, when the training component 405 determines that one or more trigger criteria are met, the instantiated training pipeline 410 can be used to further refine the model and generate another new model 250. As described above, this new model is again stored in the registry, which may automatically initiate another deployment process (which may reuse the previously created deployment pipeline 315) and instantiate a new inference pipeline 330 containing the new model. In some embodiments, as discussed above, the previous inference pipeline 330 (with the old model version) may remain deployed. In other embodiments, the system may automatically terminate the previous pipeline in favor of the new one.

[0103] In this way, workflow 400 may continue to iterate indefinitely, or to refine and evolve the model over time until defined criteria are met. This can provide seamless continuous learning, allowing the model to be repeatedly updated for improved accuracy and performance without requiring any further input or effort from the user or entity that provided the initial submission or request. This is a significant improvement over conventional systems.

[0104] An exemplary method for self-service machine learning deployment 5 is a flow diagram depicting an example method 500 for self-service machine learning deployment. In some embodiments, method 500 provides additional details regarding workflow 300 of FIG. 3. In some embodiments, method 500 is performed by a machine learning system, such as machine learning system 115 of FIG. 1.

[0105] In block 505, the machine learning system receives a request to deploy the machine learning model. In some aspects, this request is referred to as a submission of the machine learning model for deployment, as discussed above. For example, as discussed above, the request may specify a model definition, configuration information indicating how the model should be deployed, and the like. In some aspects, receiving the request includes identifying or receiving the model definition in a registry (e.g., model registry 220 of FIG. 2 ), where the model is associated with a flag or indicator that indicates or requires deployment. That is, rather than receiving an explicit user request, the machine learning system may identify the model (in the registry) using a deployment tag, where the model and tag may have been generated and / or added to the registry by a user, automatically by another system (e.g., from a training pipeline), and the like.

[0106] At block 510, the machine learning system determines whether a deployment pipeline exists for the model definition. That is, as discussed above, the machine learning system may instantiate a new deployment pipeline for the new model, but may reuse a previously created pipeline for a model that has already been deployed (e.g., if the same model has already been deployed, or if a different version of the model, such as one with the same model architecture but with different values ​​for learnable parameters, has been deployed). Although not included in the illustrated example, in some embodiments, the machine learning system may also perform other evaluations or checks to ensure that the configuration file is complete and ready for deployment.

[0107] At block 510, if the machine learning system determines that a deployment pipeline for the indicated model definition already exists, method 500 continues to block 520. If the machine learning system determines that such a pipeline does not exist, method 500 proceeds to block 515. At block 515, the machine learning system instantiates or creates a deployment pipeline for the indicated model. For example, as discussed above, the machine learning system may create, start, instantiate, or otherwise generate a set of components or processes (e.g., software modules), such as one or more virtual machines, to deploy the model. In some aspects, as discussed above, the machine learning system may create a deployment pipeline based, at least in part, on details of the indicated model definition. For example, different validation operations may be included in the pipeline, or different components may be used to test the model, depending on the particular architecture. Method 500 then continues to block 520.

[0108] In block 520, the machine learning system retrieves the model definition and configuration indicated in the request using a deployment pipeline (which may be newly generated or reused from a previous deployment). For example, the machine learning system may retrieve the model definition (e.g., architecture and hyperparameters, input features, and the like) and configuration information (e.g., pre-processing operations, data storage locations, deployment type, and the like) for the model indicated in the request from a model registry. In some aspects, this includes copying or moving the model definition and configuration from the model repository into a central memory or repository and / or into a repository or memory of a deployment pipeline.

[0109] At block 525, the machine learning system optionally validates the model using the deployment pipeline. For example, as discussed above with reference to validation component 320 of FIG. 3, the machine learning system may perform one or more tests (e.g., using test data included in the request or indicated in the model configuration) to verify that the model operates deterministically, that the model correctly generates errors with respect to the driving data, that the model generates correct and / or properly formed outputs with respect to correctly formed data, and the like. Although not included in the illustrated example, in some aspects, if model validation fails, the machine learning system may stop the deployment process and generate an alert, error, or notification indicating the problem.

[0110] After validation, method 500 continues at block 530, where the machine learning system instantiates an inference pipeline for the model definition. For example, as discussed above, the machine learning system may instantiate, generate, create, or otherwise initiate one or more components or modules (e.g., virtual machines) to perform inference using the depicted model. In some embodiments, as discussed above, instantiating the inference pipeline may include reading or accessing the feature pipeline definition (to be used to pre-process data for the model) and using this definition to instantiate or create a set of operations used to pre-process data prior to inference.

[0111] As discussed above, the inference process may include steps such as receiving or accessing input, formatting or pre-processing it, passing it through a model, generating output inferences, and / or returning or storing the generated outputs.

[0112] Advantageously, using method 500, a machine learning system can automatically perform the required validation and testing and deploy machine learning models using dynamically generated pipelines and systems, thereby enabling faster model deployment and prototyping and more diverse and varied uses of machine learning models in more versatile deployments and implementations.

[0113] Exemplary Methods for Automated Real-Time Inference 6 is a flow diagram depicting an example method 600 for real-time inference using automatically deployed models. In some embodiments, method 600 is performed using an instantiated inference pipeline (e.g., created in block 530 of FIG. 5). In some embodiments, method 600 is performed by a machine learning system, such as machine learning system 115 of FIG. 1.

[0114] In block 605, the machine learning system receives or accesses input data from a requesting entity. For example, using an API (e.g., API 255 of FIG. 2), the requesting entity (which may be an automated application, a user-controlled application, and the like) can provide data to be used as input to the model to generate an output prediction. In general, the formatting and content of the input may vary substantially depending on the particular model and implementation. For example, in an image classification embodiment, the input may include one or more images. In a weather prediction embodiment, the input may include time-series data related to weather.

[0115] At block 610, the machine learning system identifies a corresponding inference pipeline for the input data. In some aspects, the input is provided to the corresponding inference pipeline directly by the requesting entity (e.g., using a corresponding API). In other embodiments, the input request may indicate a model to be used, and the machine learning system can identify an appropriate pipeline (e.g., identify an inference pipeline that uses a recently trained or refined version of the indicated model).

[0116] At block 615, the machine learning system may optionally pre-process the input data using an inference pipeline. For example, as discussed above, the inference pipeline may include a feature pipeline or component that uses one or more transformations, operations, or other processes to prepare the input data for processing using the machine learning model. Generally, these pre-processing steps may vary depending on the particular implementation and configuration of the model. For example, the model designer may specify that normalization should be used, that the input should be converted to a vector encoding, and the like.

[0117] At block 620, the machine learning system generates an output guess using an inference pipeline by processing the input data (or prepared / preprocessed input data) using the deployed model. As discussed above, the actual operations for processing data using the model may vary depending on the particular model architecture. Similarly, the format and content of the output guess may vary depending on the particular implementation or model. For example, the output guess may include a classification of the input data, a numeric value for the data (e.g., generated using a regression model), and the like. In some aspects, the output may further include a confidence score or other value generated by the model. This confidence score may indicate, for example, the probability or likelihood that the output guess is accurate (e.g., the probability that the input data belongs to the generated category).

[0118] At block 625, the machine learning system then returns the generated output to the requesting entity (e.g., via an API). In this manner, method 600 enables an automatically generated inference pipeline to automatically receive and process input data and return generated output. This significantly reduces complexity in the machine learning process, reduces errors, and generally improves the operation of the machine learning system (and the operation of requesting entities that rely on such predictions).

[0119] Example Methods for Automated Batch Inference 7 is a flow diagram depicting an example method 700 for batch inference using automatically deployed models. In some embodiments, method 700 is performed using an instantiated inference pipeline (e.g., created in block 530 of FIG. 5). In some embodiments, method 700 is performed by a machine learning system, such as machine learning system 115 of FIG. 1.

[0120] At block 705, the machine learning system determines whether one or more inference criteria are satisfied. In some aspects, the inference criteria are specified in a configuration or request used to instantiate the inference pipeline. For example, the criteria may specify that the machine learning system should process batches of data periodically (e.g., process any stored data in a timely manner), in response to an event or occurrence (e.g., when the number of input samples meets or exceeds a minimum number of samples), and the like. If the machine learning system determines that the inference criteria are not satisfied, method 700 repeats block 705.

[0121] If the machine learning system determines that the inference criteria are met, method 700 continues at block 710. At block 710, the machine learning system receives or accesses input data (from one or more requesting entities) for the batch inference process. For example, as discussed above, a requesting entity (which may be an automated application, a user-controlled application, and the like) can provide data to be used as input to a model to a repository or storage location (e.g., a database table). Once the inference criteria are met, the machine learning system can retrieve or access these stored samples for processing (e.g., retrieve them from a specified storage repository or location).

[0122] At block 715, the machine learning system identifies a corresponding inference pipeline for the input data. As discussed above, in some aspects, the input is provided directly to the corresponding inference pipeline by the requesting entity (e.g., using a corresponding API). In other embodiments, the input request may indicate a model to be used, and the machine learning system can identify an appropriate pipeline (e.g., identify an inference pipeline that uses a recently trained or refined version of the indicated model).

[0123] At block 720, the machine learning system can optionally pre-process the input data using an inference pipeline. For example, as discussed above, the inference pipeline may include a feature pipeline or component that uses one or more transformations, operations, or other processes to prepare the input data for processing using the machine learning model. Generally, these pre-processing steps may vary depending on the particular implementation and configuration of the model. For example, a model designer may specify that normalization should be used, that the input should be converted to vector encoding, and the like. In some aspects, the machine learning system can process the input data sequentially (e.g., process one sample at a time). In at least one aspect, the machine learning system processes some or all of the input samples in parallel (e.g., using one or more feature pipelines).

[0124] At block 725, the machine learning system generates output inferences using an inference pipeline by processing the input data samples (or prepared / preprocessed input data) using the deployed model. As discussed above, the actual operations for processing data using a model may vary depending on the particular model architecture. Similarly, the format and content of the output inferences may vary depending on the particular implementation or model. For example, an output inference for a given data sample may include a classification of the input sample, a numerical value for the sample (e.g., generated using a regression model), and the like. In some aspects, the output may further include a corresponding confidence score or other value generated by the model for each output inference / input data sample. This confidence score may indicate, for example, the probability or likelihood that a given output inference is accurate (e.g., the probability that the corresponding input data belongs to the generated category).

[0125] At block 730, the machine learning system then stores the generated output data in a designated location or repository (e.g., the same database table or a different database table from which the input data was accessed). Method 700 then returns to block 705 to begin the process again.

[0126] In this manner, method 700 enables an automatically generated inference pipeline to automatically receive and process input data in batches to generate output inferences, which significantly reduces complexity in the machine learning process, reduces errors, and generally improves the operation of the machine learning system (and the operation of requesting entities that rely on such predictions).

[0127] An exemplary method for automated continuous learning 8 is a flow diagram depicting an example method 800 for automated continuous learning development. In some embodiments, method 800 provides additional details regarding workflow 400 of FIG. 4. In some embodiments, method 800 is performed by a machine learning system, such as machine learning system 115 of FIG. 1.

[0128] In block 805, the machine learning system receives a request to deploy a continuous learning pipeline for the model definition. In some aspects, this request is referred to as a submission of a machine learning model for training or refinement, as discussed above. For example, as discussed above, the request may specify a model definition, configuration information indicating how the model should be deployed, training configurations such as where training data is stored and retraining criteria, and the like. In some aspects, receiving the request includes identifying or receiving a model definition in a registry (e.g., model registry 220 of FIG. 4 ), where the model is associated with a flag or indicator that indicates or requires deployment with continuous learning. That is, rather than receiving an explicit user request, the machine learning system may identify the model using a training / continuous learning tag (in the registry), where the model and tag may have been generated and / or added to the registry by a user, automatically by another system (e.g., from a training pipeline), and the like.

[0129] In block 810, the machine learning system creates a training schedule based on the request. For example, as discussed above, the machine learning system may create one or more event listeners (e.g., to monitor whether new training data is being added to the storage repository), one or more timers (e.g., to determine whether an indicated period of time has elapsed), and the like. Generally, the training schedule may be used to control when and how a model is trained or updated. In at least one embodiment, the training schedule is implemented by the training component 405 of FIG. 4.

[0130] At block 815, the machine learning system may instantiate and / or launch a training pipeline (e.g., training pipeline 410) to train or update the model, as discussed above. In some embodiments, rather than immediately launching the training pipeline and training the model, the machine learning system may first deploy the current version of the model for inference, as discussed above. In embodiments, the training pipeline is generally used to generate new versions of models. For example, as discussed above, the training pipeline may receive training data, read it, or otherwise access it (e.g., from a specified repository or location indicated in the request and / or configuration file), and use the data to update model parameters. In some embodiments, the machine learning system may then store the newly updated model in a model registry with a flag indicating that it is ready for deployment for inference. One exemplary method for launching a training pipeline is discussed in further detail below with reference to FIG. 9.

[0131] At block 820, the machine learning system identifies or detects the presence of the newly trained model in the model registry. For example, as discussed above, the machine learning system (e.g., model evaluator 305 of FIG. 3) may detect or identify the presence or addition of the newly trained model in the registry (e.g., based on a deployment flag). In response, at block 825, the machine learning system deploys the newly trained model for inference. In some aspects, this deployment process may be implemented using method 500 of FIG. 5.

[0132] In block 830, the machine learning system determines whether one or more training criteria (also referred to as update criteria, retraining criteria, refinement criteria, and the like) are satisfied. For example, the machine learning system may use a training schedule (e.g., an event listener and / or a timer) to determine whether the model should be retrained or updated as part of a continuous learning deployment. As discussed above, the training criteria may include a variety of considerations, such as periodic retraining, retraining based on the occurrence of an event, and the like.

[0133] At block 830, if the machine learning system determines that the training criteria are not met, method 800 repeats block 830. If the training criteria are met, method 800 returns to block 815 and starts the training pipeline again using (new) training data. In this way, the machine learning system can iteratively update the model using new data, thereby ensuring continuous updates and maximizing model accuracy and reliability.

[0134] Advantageously, using method 800, a machine learning system can train, refine, monitor, and deploy machine learning models using dynamically generated pipelines and systems to automatically perform the required training, validation and testing, and deployment. By doing so, the machine learning system enables faster model training and deployment and more versatile deployment and use of a greater variety and variety of machine learning models in implementations.

[0135] Exemplary Method for Model Training Using a Training Pipeline 9 is a flow diagram depicting an example method 900 for automatically training a machine learning model using a deployed pipeline. In some embodiments, method 900 provides additional details relative to block 815 of FIG. 8. In some embodiments, method 900 is performed by a machine learning system, such as machine learning system 115 of FIG. 1.

[0136] At block 905, the machine learning system accesses training data for the model. For example, as discussed above, the model configuration may define one or more memory locations or repositories (e.g., database tables or other data structures) in which the training data is stored. In some aspects, as discussed above, the training data is stored in a single data repository (e.g., with input data and corresponding output indicators in a single storage device). In other aspects, the data may be distributed (e.g., with input data stored in one or more different locations and corresponding output indicators in one or more other locations). In some embodiments, accessing the training includes reading or accessing each training exemplar independently (e.g., using each to refine the model separately). In other aspects, the machine learning system may access multiple exemplars (e.g., to perform batch training).

[0137] At block 910, the machine learning system refines the machine learning model based on the training data. As discussed above, this refinement process generally involves updating one or more parameters of the model (such as weights in a neural network) to better fit the training data. During this refinement process, the model learns to make more accurate and reliable predictions for the input data during runtime.

[0138] At block 915, the machine learning system determines whether at least one training exemplar remains in the indicated repository. If so, method 900 returns to block 905. If not, method 900 continues at block 920, where the machine learning system can optionally evaluate a newly trained or refined model.

[0139] For example, as discussed above, a machine learning system may retrieve or access test data (e.g., from a designated repository), process it using a model, generate output predictions, and compare the generated outputs to corresponding signatures or ground truth for the test samples. In this manner, the machine learning system can determine performance metrics such as model accuracy and reliability.

[0140] In block 925, the machine learning system stores the newly trained model in a model registry along with a deployment flag or indicator indicating that it is primed and ready for deployment. In some aspects, as discussed above, this allows the machine learning system (e.g., via model evaluator 305 of FIG. 3) to automatically detect the model and begin the deployment process. In some aspects, as discussed above, the performance metrics (determined in block 920) may also be stored with the model, allowing a user to inspect the model's performance at any given time (e.g., for a given version) and changes over time (e.g., across versions).

[0141] Exemplary Methods for Automated Model Development 10 is a flow diagram depicting an example method 1000 for automatically deploying a machine learning model. In some embodiments, method 1000 provides additional details regarding workflow 300 of FIG. 3 and / or method 500 of FIG. 5. In some embodiments, method 1000 is performed by a machine learning system, such as machine learning system 115 of FIG. 1.

[0142] In block 1005, a request to deploy a machine learning model (e.g., model 250 of FIG. 3) is received, the request specifying whether the machine learning model should be deployed for batch inference or real-time inference.

[0143] In block 1010, the machine learning model definition is read from a registry containing trained machine learning model definitions (e.g., model registry 220 of FIG. 2).

[0144] At block 1015, the machine learning model definition is validated (eg, by validation component 320 of FIG. 3) using one or more test exemplars.

[0145] At block 1020, an inference pipeline (e.g., inference pipeline 330 of FIG. 3) containing the machine learning model is instantiated.

[0146] In some aspects, the operations of blocks 1010, 1015, and 1020 may be collectively referred to as instantiating a deployment pipeline for the machine learning model. In some aspects, blocks 1010, 1015, and 1020 may be performed in response to determining that a deployment pipeline for the machine learning model is unavailable.

[0147] In block 1025, the input data is processed using the speculation pipeline.

[0148] Exemplary Methods for Automated Model Training 11 is a flow diagram depicting an example method 1100 for automatically performing continuous training of a machine learning model. In some embodiments, method 1100 provides additional details regarding workflow 400 of FIG. 4 and / or method 800 of FIG. 8. In some embodiments, method 1100 is performed by a machine learning system, such as machine learning system 115 of FIG. 1.

[0149] In block 1105, a request to perform continuous learning for a machine learning model (e.g., model 250A of FIG. 4) is received, the request specifying retraining logic comprising one or more trigger criteria.

[0150] In block 1110, an inference pipeline (e.g., inference pipeline 330 of FIG. 4) containing the machine learning model is automatically instantiated.

[0151] In block 1115, retraining logic, including one or more trigger criteria, is automatically instantiated (eg, by training component 405 of FIG. 4).

[0152] In block 1120, the input data is processed using the speculation pipeline.

[0153] In block 1125, retraining logic is used to retrieve new training data (eg, data 425 in FIG. 4) from a designated repository.

[0154] In block 1130, retraining logic is used to generate a refined machine learning model (e.g., model 250B of FIG. 4) by training the machine learning model using the new training data.

[0155] In some aspects, the operations of blocks 1125 and 1130 may be performed automatically in response to determining that one or more trigger criteria are satisfied.

[0156] Exemplary Computing Device for Automated Model Deployment and / or Training 12 depicts an exemplary computing device configured to implement various aspects of the present disclosure. While depicted as a physical device, in embodiments, computing device 1200 may be implemented using a virtual device and / or across several devices (e.g., in a cloud environment). In one embodiment, computing device 1200 corresponds to one or more systems in a healthcare platform, such as a machine learning system (e.g., machine learning system 115 of FIG. 1).

[0157] As shown, computing device 1200 includes a CPU 1205, memory 1210, storage 1215, a network interface 1225, and one or more I / O interfaces 1220. In the illustrated embodiment, CPU 1205 reads and executes programming instructions stored in memory 1210 and stores and reads application data resident in storage 1215. CPU 1205 generally represents a single CPU and / or GPU, multiple CPUs and / or GPUs, a single CPU and / or GPU with multiple processing cores, and the like. Memory 1210 is included to generally represent random access memory. Storage 1215 may be any combination of disk drives, flash-based storage devices, and the like, and may include fixed and / or removable storage devices such as fixed disk drives, removable memory cards, caches, optical storage devices, network-attached storage (NAS), or storage area networks (SANs).

[0158] In some embodiments, I / O devices 1235 (keyboard, monitor, etc.) are connected via I / O interface 1220. Additionally, via network interface 1225, computing device 1200 can be communicatively coupled to one or more other devices and components (e.g., via a network, which may include the Internet, a local network, and the like). As shown, CPU 1205, memory 1210, storage 1215, network interface 1225, and I / O interface 1220 are communicatively coupled by one or more buses 1230.

[0159] In the illustrated embodiment, memory 1210 includes a model runner component 1250 and a training component 1255, which may implement one or more embodiments discussed above. While depicted as separate components for conceptual clarity, in embodiments, the operations of the depicted components (and others not shown) may be combined or distributed across any number of components. Furthermore, while depicted as software resident in memory 1210, in embodiments, the operations of the depicted components (and others not shown) may be implemented using hardware, software, or a combination of hardware and software.

[0160] In one embodiment, the model runner component 1250 may be used to automatically deploy machine learning models, as discussed above. For example, the model runner component 1250 (which may correspond to the model evaluator 305 and / or the deployment pipeline component 310 in FIG. 3 , respectively) may monitor a model registry, identify models that are ready for deployment, and / or receive requests or submissions to deploy models. In response, the model runner component 1250 may automatically deploy the model by creating a deployment pipeline (if one does not exist), by using the deployment pipeline to validate and deploy the model in an inference pipeline, and the like.

[0161] In one embodiment, training component 1255 may be used to automatically train or refine machine learning models, as discussed above. For example, training component 1255 (which may correspond to training component 405 in FIG. 4) may receive training requests or submissions (or identify models in a registry that are ready for training) and automatically instantiate and use a training pipeline to train the models, deploy them, and / or retrain them when appropriate.

[0162] In the illustrated example, storage device 1215 includes training data 1270, one or more machine learning models 1275, and one or more corresponding configurations 1280. In one embodiment, training data 1270 (which may correspond to data 425 of FIG. 4 ) may include any data used to train, refine, or test a machine learning model, as discussed above. Models 1275 may correspond to model definitions stored in a model registry (e.g., model registry 220 of FIGS. 2, 3, and / or 4 ) as discussed above. Configurations 1280 generally correspond to configurations or information associated with a model, such as how each model 1275 should be deployed, whether each model is ready for deployment, how training should be performed, and the like, as discussed above. While depicted as residing within storage device 1215 for conceptual clarity, training data 1270, models 1275, and configurations 1280 may be stored in any suitable location, including memory 1210 or one or more remote systems distinct from computing device 1200.

[0163] Additional Considerations The foregoing description is provided to enable any person skilled in the art to practice various embodiments described herein. The examples discussed herein do not limit the scope, applicability, or embodiments set forth in the claims. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments. For example, changes may be made to the function and arrangement of elements discussed without departing from the scope of the disclosure. Various examples may omit, substitute, or add various procedures or components as appropriate. For example, described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. Also, features described with respect to some examples may be combined in other examples. For example, an apparatus may be implemented, or a method may be practiced, using any number of aspects described herein. Additionally, the scope of the disclosure is intended to cover such apparatuses or methods practiced using other structure, functionality, or structure and functionality, in addition to, or other than, the various aspects of the disclosure described herein. It should be understood that any aspect of the disclosure disclosed herein may be embodied by one or more elements of a claim.

[0164] As used herein, the word "exemplary" means "serving as an example, instance, or illustration." Any aspect described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects.

[0165] As used herein, phrases referring to "at least one of" a list of items refer to any combination of the items, including single members. By way of example, "at least one of a, b, or c" is intended to cover a, b, c, ab, ac, bc, and abc, as well as any combination with multiple identical elements (e.g., aa, aaa, aab, aac, abb, acc, bb, bbb, bbc, cc, and ccc, or any other permutation of a, b, and c).

[0166] As used herein, the term "determining" encompasses a wide variety of actions. For example, "determining" may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up a table, database, or another data structure), resolving, and the like. "Determining" may also include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory), and the like. "Determining" may also include resolving, selecting, choosing, establishing, and the like.

[0167] The methods disclosed herein include one or more steps or actions for achieving the method. The method steps and / or actions may be interchanged without departing from the scope of the claims. In other words, unless a specific order of steps or actions is specified, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims. Furthermore, various operations of the methods described above may be performed by any suitable means capable of performing the corresponding functions. The means may include various hardware and / or software components and / or modules, including, but not limited to, circuits, application-specific integrated circuits (ASICs), or processors. Generally, where operations are illustrated in figures, those operations may have corresponding counterpart means-plus-function components with similar numbering.

[0168] Embodiments of the present invention may be provided to end users through a cloud computing infrastructure. Cloud computing generally refers to the provisioning of scalable computing resources as a service over a network. More formally, cloud computing may be defined as computing power that provides an abstraction between computing resources and their underlying technical architecture (e.g., servers, storage, networks), enabling opportunistic, on-demand network access to a shared pool of configurable computing resources that can be rapidly provisioned and released with minimal management effort or service provider interaction. Thus, cloud computing allows users to access virtual computing resources (e.g., storage, data, applications, and even fully virtualized computing systems) in the "cloud," regardless of the underlying physical systems used to provide the computing resources (or the location of those systems).

[0169] Typically, cloud computing resources are provided to users on a pay-per-use basis, and users are charged only for the computing resources they actually use (e.g., the amount of storage space consumed by the user or the number of virtualized systems instantiated by the user). Users can access any of the resources residing in the cloud at any time and from any location across the Internet. In the context of the present invention, users may access applications or systems (e.g., machine learning system 115 of FIG. 1 ) or related data available in the cloud. For example, a machine learning system may run on a computing system in the cloud and automatically train, deploy, and / or monitor machine learning models based on user requests or submissions. In such cases, the machine learning system may maintain a model registry and / or processing pipeline in the cloud. Doing so allows users to access this information from any computing system attached to a network connected to the cloud (e.g., the Internet).

[0170] The following claims are not limited to the embodiments set forth herein but are intended to be accorded the full scope consistent with the language set forth in the claims. Within the claims, reference to an element in the singular does not mean "one and only one," unless specifically so stated, but rather "one or more." Unless specifically stated otherwise, the term "some" refers to one or more. No claim element is to be construed under the provisions of 35 U.S.C. § 112(f) unless the element is expressly recited using the phrase "means for," or, in the case of a method claim, the element is recited using the phrase "step for." All structural and functional equivalents of the elements of various aspects described throughout this disclosure that are known or later become known to those skilled in the art are expressly incorporated herein by reference and intended to be encompassed by the claims. Furthermore, nothing disclosed herein is intended to be made available to the public, regardless of whether such disclosure is expressly recited in the claims.

[0171] Illustrative notes Example implementations are described in the numbered appendices below.

[0172] Appendix 1: A method comprising: receiving a request to deploy a machine learning model, the request specifying whether the machine learning model should be deployed for batch inference or real-time inference; and in response to determining that a deployment pipeline for the machine learning model is unavailable, instantiating a deployment pipeline for the machine learning model, where instantiating the deployment pipeline comprises reading the machine learning model definition from a registry containing trained machine learning model definitions, validating the machine learning model definition using one or more test exemplars, and instantiating an inference pipeline including the machine learning model; and processing input data using the inference pipeline.

[0173] Appendix 2: The method of Appendix 1, wherein reading the machine learning model from the registry further includes reading a feature pipeline definition for the machine learning model from the registry, the feature pipeline definition indicating how to pre-process input data for the machine learning model, and instantiating the inference pipeline includes generating a feature pipeline based on the feature pipeline definition.

[0174] Appendix 3: The method of any one of Appendixes 1-2, wherein the request specifies deploying a machine learning model for real-time inference, and the method further includes receiving input data from a requesting entity; generating prepared data by processing the input data using a feature pipeline; generating an output inference by processing the prepared data using a machine learning model; and providing the output inference to the requesting entity.

[0175] Appendix 4: The method of any one of Appendixes 1-3, wherein the request specifies deploying a machine learning model for batch inference, and the request further specifies a memory location for the batch inference.

[0176] Appendix 5: The method of any one of Appendixes 1-4, further comprising receiving input data from a requesting entity; storing the input data in a specified memory location; and, in response to determining that one or more inference criteria are satisfied, reading the input data from the specified memory location; processing the input data using a feature pipeline to generate prepared data; processing the prepared data using a machine learning model to generate an output inference; and storing the output inference in a specified memory location.

[0177] Appendix 6: The method of any one of Appendixes 1-5, further including receiving a second request to deploy the machine learning model, and in response to determining that a deployment pipeline for the machine learning model is available, not instantiating a new deployment pipeline for the machine learning model based on the second request, and instantiating a new inference pipeline including the second instance of the machine learning model using the deployment pipeline.

[0178] Appendix 7: The method of any one of Appendixes 1-6, wherein validating the machine learning model definition includes generating first output data by processing a first test exemplar using the machine learning model, generating second output data by processing the first test exemplar using the machine learning model, and verifying that the first output data matches the second output data.

[0179] Appendix 8: The method of any one of Appendixes 1-7, wherein validating the machine learning model definition includes processing a first test exemplar using the machine learning model, where the first test exemplar does not satisfy one or more model criteria specified in the registry, and verifying that the inference pipeline returns an error for the first test exemplar.

[0180] Appendix 9: The method of any one of Appendixes 1-8, further comprising receiving a plurality of machine learning model definitions, receiving a plurality of configuration files for the plurality of machine learning model definitions, and storing the plurality of machine learning model definitions and the plurality of configuration files in a registry.

[0181] Appendix 10: A method including receiving a request to perform continuous learning for a machine learning model, the request specifying retraining logic comprising one or more trigger criteria; automatically instantiating an inference pipeline including the machine learning model; automatically instantiating the retraining logic including the one or more trigger criteria; processing input data using the inference pipeline; and in response to determining that the one or more trigger criteria are satisfied, automatically using the retraining logic to read new training data from a specified repository; and training the machine learning model using the new training data, thereby generating a refined machine learning model using the retraining logic.

[0182] Appendix 11: The method of Appendix 10, further comprising storing the refined machine learning model in a registry containing trained machine learning models, and storing an indication that the refined machine learning model is ready for deployment.

[0183] Appendix 12: The method of any one of Appendixes 10-11, further comprising automatically instantiating a new inference pipeline including the refined machine learning model, and processing new input data using the new inference pipeline including the refined machine learning model.

[0184] Addendum 13: The method of any one of Addendums 10-12, wherein automatically instantiating a new inference pipeline including the refined machine learning model includes reading the refined machine learning model from a registry.

[0185] Appendix 14: The method of any one of Appendixes 10-13, further comprising generating performance metrics by evaluating the refined machine learning model using test data, and storing the performance metrics in a registry.

[0186] Appendix 15: The method of any one of Appendixes 10-14, wherein the designated repository is indicated in the request.

[0187] Appendix 16: The method of any one of Appendixes 10-15, further comprising receiving a request to deploy a continuous training pipeline for the machine learning model, the request specifying one or more trigger criteria.

[0188] Addendum 17: The method of any one of Addendums 10-16, wherein input data is received from a requesting entity, and the method further includes processing the input data to generate an output guess; and transmitting the output guess to the requesting entity, wherein the requesting entity stores the input data and corresponding ground truth in a repository designated as new training data.

[0189] Clause 18: The method of any one of Clauses 10-17, wherein the request further specifies deploying the machine learning model for one of batch inference or real-time inference.

[0190] Supplementary Note 19: The method of any one of Supplementary Notes 10-18, wherein automatically instantiating an inference pipeline for the machine learning model further includes reading a feature pipeline definition for the machine learning model, the feature pipeline definition indicating instructions for pre-processing input data for the machine learning model, and generating the feature pipeline based on the feature pipeline definition.

[0191] Appendix 20: A system comprising: a memory comprising computer-executable instructions; and one or more processors configured to execute the computer-executable instructions to cause a processing system to perform a method described in any one of Appendixes 1-19.

[0192] Appendix 21: A system, comprising means for carrying out the method described in any one of appendices 1-19.

[0193] Appendix 22: A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to perform a method described in any one of Appendixes 1-19.

[0194] Appendix 23: A computer program product embodied on a computer-readable storage medium, the computer program product comprising code for performing the method of any one of Appendixes 1-19.

Claims

1. 1. A method comprising: receiving a request to deploy a machine learning model, the request specifying whether the machine learning model should be deployed for batch inference or real-time inference; instantiating a deployment pipeline for the machine learning model in response to determining that a deployment pipeline for the machine learning model is unavailable, wherein instantiating the deployment pipeline includes: Retrieving the machine learning model definition from a registry containing trained machine learning model definitions; validating the machine learning model definition using one or more test exemplars; instantiating an inference pipeline including the machine learning model; and processing input data using said inference pipeline; A method comprising:

2. Retrieving the machine learning model from the registry further includes retrieving a feature pipeline definition for the machine learning model from the registry, the feature pipeline definition indicating how to pre-process input data for the machine learning model; instantiating the inference pipeline includes generating a feature pipeline based on the feature pipeline definition. The method of claim 1.

3. The request specifies deploying the machine learning model for real-time inference, and the method further comprises: receiving input data from a requesting entity; processing the input data using the feature pipeline to generate prepared data; generating an output inference by processing the prepared data using the machine learning model; and providing the output guess to the requesting entity; and The method of claim 2 , comprising:

4. the request specifies deploying the machine learning model for batch inference; The request further specifies a storage location for the batch inference. The method of claim 2.

5. The method further comprises: receiving input data from a requesting entity; storing said input data in said defined memory location; In response to determining that one or more presumptive criteria are satisfied, reading the input data from the defined memory location; processing the input data using the feature pipeline to generate prepared data; generating an output inference by processing the prepared data using the machine learning model; and storing said output guess in said defined memory location; The method of claim 4, comprising:

6. receiving a second request to deploy the machine learning model; In response to determining that a deployment pipeline for the machine learning model is available, avoid instantiating a new deployment pipeline for the machine learning model based on the second request; and instantiating a new inference pipeline that includes a second instance of the machine learning model using the deployment pipeline; and The method of claim 1 further comprising:

7. Validating the machine learning model definition includes: generating first output data by processing a first test exemplar using the machine learning model; generating second output data by processing the first test exemplar using the machine learning model; and verifying that the first output data matches the second output data; The method of claim 1 , comprising:

8. Validating the machine learning model definition includes: using the machine learning model to process a first test exemplar, the first test exemplar failing to satisfy one or more model criteria defined in the registry; and verifying that the speculation pipeline returns an error for the first test exemplar; The method of claim 1 , comprising:

9. receiving a plurality of machine learning model definitions; receiving a plurality of configuration files for the plurality of machine learning model definitions; storing the plurality of machine learning model definitions and a plurality of configuration files in the registry; The method of claim 1 further comprising:

10. A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to: receiving a request to deploy a machine learning model, the request specifying whether the machine learning model should be deployed for batch inference or real-time inference; instantiating a deployment pipeline for the machine learning model in response to determining that a deployment pipeline for the machine learning model is unavailable, wherein instantiating the deployment pipeline includes: Retrieving the machine learning model definition from a registry containing trained machine learning model definitions; validating the machine learning model definition using one or more test exemplars; instantiating an inference pipeline including the machine learning model; and processing input data using said inference pipeline; 10. A non-transitory computer-readable medium for performing operations including:

11. Retrieving the machine learning model from the registry further includes retrieving a feature pipeline definition for the machine learning model from the registry, the feature pipeline definition indicating how to pre-process input data for the machine learning model; instantiating the inference pipeline includes generating a feature pipeline based on the feature pipeline definition. The non-transitory computer-readable medium of claim 10.

12. The operation further comprises: receiving a second request to deploy the machine learning model; In response to determining that a deployment pipeline for the machine learning model is available, avoid instantiating a new deployment pipeline for the machine learning model based on the second request; and instantiating a new inference pipeline that includes a second instance of the machine learning model using the deployment pipeline; and 11. The non-transitory computer-readable medium of claim 10, comprising:

13. Validating the machine learning model definition includes: generating first output data by processing a first test exemplar using the machine learning model; generating second output data by processing the first test exemplar using the machine learning model; and verifying that the first output data matches the second output data; 11. The non-transitory computer-readable medium of claim 10, comprising:

14. Validating the machine learning model definition includes: using the machine learning model to process a first test exemplar, the first test exemplar failing to satisfy one or more model criteria defined in the registry; and verifying that the speculation pipeline returns an error for the first test exemplar; 11. The non-transitory computer-readable medium of claim 10, comprising:

15. The operation further comprises: receiving a plurality of machine learning model definitions; receiving a plurality of configuration files for the plurality of machine learning model definitions; storing the plurality of machine learning model definitions and a plurality of configuration files in the registry; 11. The non-transitory computer-readable medium of claim 10, comprising:

16. 1. A system comprising: a memory comprising computer-executable instructions; one or more processors that execute the computer-executable instructions to provide the system with: receiving a request to deploy a machine learning model, the request specifying whether the machine learning model should be deployed for batch inference or real-time inference; instantiating a deployment pipeline for the machine learning model in response to determining that a deployment pipeline for the machine learning model is unavailable, wherein instantiating the deployment pipeline includes: Retrieving the machine learning model definition from a registry containing trained machine learning model definitions; validating the machine learning model definition using one or more test exemplars; instantiating an inference pipeline including the machine learning model; and processing input data using said inference pipeline; one or more processors configured to perform operations including: A system comprising:

17. Retrieving the machine learning model from the registry further includes retrieving a feature pipeline definition for the machine learning model from the registry, the feature pipeline definition indicating how to pre-process input data for the machine learning model; instantiating the inference pipeline includes generating a feature pipeline based on the feature pipeline definition.

17. The system of claim 16.

18. The operation further comprises: receiving a second request to deploy the machine learning model; In response to determining that a deployment pipeline for the machine learning model is available, avoid instantiating a new deployment pipeline for the machine learning model based on the second request; and instantiating a new inference pipeline that includes a second instance of the machine learning model using the deployment pipeline; and 17. The system of claim 16, comprising:

19. Validating the machine learning model definition includes: generating first output data by processing a first test exemplar using the machine learning model; generating second output data by processing the first test exemplar using the machine learning model; and verifying that the first output data matches the second output data; 17. The system of claim 16, comprising:

20. The operation further comprises: receiving a plurality of machine learning model definitions; receiving a plurality of configuration files for the plurality of machine learning model definitions; storing the plurality of machine learning model definitions and a plurality of configuration files in the registry; 17. The system of claim 16, comprising:

21. 1. A method comprising: receiving a request to perform continuous learning for a machine learning model, the request specifying retraining logic comprising one or more trigger criteria; automatically instantiating an inference pipeline that includes the machine learning model; and automatically instantiating the retraining logic including the one or more trigger criteria; processing input data using the inference pipeline; automatically in response to determining that the one or more trigger criteria are satisfied; using said retraining logic to retrieve new training data from a designated repository; generating a refined machine learning model using the retraining logic by training the machine learning model using the new training data; and A method comprising:

22. storing the refined machine learning model in a registry containing trained machine learning models; storing an indication that the refined machine learning model is ready for deployment; and 22. The method of claim 21 further comprising:

23. automatically instantiating a new inference pipeline that includes the refined machine learning model; and processing new input data using the new inference pipeline including the refined machine learning model; and 23. The method of claim 22, further comprising:

24. 24. The method of claim 23, wherein automatically instantiating the new inference pipeline including the refined machine learning model comprises reading the refined machine learning model from the registry.

25. generating a performance metric by evaluating the refined machine learning model using test data; and storing said performance metrics in said registry; 23. The method of claim 22, further comprising:

26. The method of claim 21 , wherein the specified repository is indicated in the request.

27. The input data is received from a requesting entity, and the method further comprises: generating output predictions by processing the input data; transmitting the output guess to the requesting entity, which stores the input data and corresponding ground truth as new training data in the designated repository; 22. The method of claim 21, comprising:

28. 22. The method of claim 21, wherein the request further specifies deploying the machine learning model for one of batch inference or real-time inference.

29. Automatically instantiating the inference pipeline for the machine learning model further comprises: reading a feature pipeline definition for the machine learning model, the feature pipeline definition indicating instructions for pre-processing input data for the machine learning model; and generating a feature pipeline based on the feature pipeline definition; 22. The method of claim 21, comprising:

30. A non-transitory computer-readable medium comprising computer-executable instructions that, when executed by one or more processors of a processing system, cause the processing system to: receiving a request to perform continuous learning for a machine learning model, the request specifying retraining logic comprising one or more trigger criteria; automatically instantiating an inference pipeline that includes the machine learning model; and automatically instantiating the retraining logic including the one or more trigger criteria; processing input data using the inference pipeline; automatically in response to determining that the one or more trigger criteria are satisfied; using said retraining logic to retrieve new training data from a designated repository; generating a refined machine learning model using the retraining logic by training the machine learning model using the new training data; and 10. A non-transitory computer-readable medium for performing operations including:

31. The operation further comprises: storing the refined machine learning model in a registry containing trained machine learning models; storing an indication that the refined machine learning model is ready for deployment; and 31. The non-transitory computer readable medium of claim 30, comprising:

32. automatically instantiating a new inference pipeline that includes the refined machine learning model; and processing new input data using the new inference pipeline including the refined machine learning model; and 32. The non-transitory computer-readable medium of claim 31, further comprising:

33. The operation further comprises: generating a performance metric by evaluating the refined machine learning model using test data; and storing said performance metrics in said registry; 32. The non-transitory computer readable medium of claim 31, comprising:

34. The input data is received from a requesting entity, and the operation further comprises: generating output predictions by processing the input data; transmitting the output guess to the requesting entity, which stores the input data and corresponding ground truth as new training data in the designated repository; 31. The non-transitory computer readable medium of claim 30, comprising:

35. Automatically instantiating the inference pipeline for the machine learning model further comprises: reading a feature pipeline definition for the machine learning model, the feature pipeline definition indicating instructions for pre-processing input data for the machine learning model; and generating a feature pipeline based on the feature pipeline definition; 31. The non-transitory computer readable medium of claim 30, comprising:

36. 1. A system comprising: a memory comprising computer-executable instructions; one or more processors that execute the computer-executable instructions to provide the system with: receiving a request to perform continuous learning for a machine learning model, the request specifying retraining logic comprising one or more trigger criteria; automatically instantiating an inference pipeline that includes the machine learning model; and automatically instantiating the retraining logic including the one or more trigger criteria; processing input data using the inference pipeline; automatically in response to determining that the one or more trigger criteria are satisfied; using said retraining logic to retrieve new training data from a designated repository; generating a refined machine learning model using the retraining logic by training the machine learning model using the new training data; and one or more processors configured to perform operations including: A system comprising:

37. The operation further comprises: storing the refined machine learning model in a registry containing trained machine learning models; storing an indication that the refined machine learning model is ready for deployment; and 37. The system of claim 36, comprising:

38. automatically instantiating a new inference pipeline that includes the refined machine learning model; and processing new input data using the new inference pipeline including the refined machine learning model; and 38. The system of claim 37, further comprising:

39. The operation further comprises: generating a performance metric by evaluating the refined machine learning model using test data; and storing said performance metrics in said registry; 38. The system of claim 37, comprising:

40. Automatically instantiating the inference pipeline for the machine learning model further comprises: reading a feature pipeline definition for the machine learning model, the feature pipeline definition indicating instructions for pre-processing input data for the machine learning model; and generating a feature pipeline based on the feature pipeline definition; 37. The system of claim 36, comprising:

Images