Machine learning inference call for database query processing

Abstract

Techniques are described for making machine learning inference calls for database query processing. In some embodiments, a method of making machine learning inference calls for database query processing can include generating a first batch of machine learning requests based at least on a query to be performed on data stored in a database service, wherein the query identifies a machine learning service; sending the first batch of machine learning requests to an input buffer of an asynchronous request handler that generates a second batch of machine learning requests based on the first batch of machine learning requests; and obtaining a plurality of machine learning responses from an output buffer of the asynchronous request handler, the machine learning responses generated by the machine learning service in response to receiving the second batch of machine learning requests using a machine learning model.

Description

Background of the Invention

[0002] Machine learning is widely recognized as a potential major driver of future technologies. Organizations around the world are now seeking to use machine learning techniques to solve a wide variety of problems, such as optimizing their products, processes, customer experiences, and more. While the high-level concept of machine learning sounds simple – for example, feeding a computer training data and allowing it to automatically learn from that data to generate models that can make predictions on other data – implementing machine learning techniques in practice can be very difficult.

[0003] This difficulty stems in part from the underlying algorithmic and mathematical complexity of machine learning algorithms developed by leading academic researchers or individuals in the field. Additionally, generating, updating, and deploying useful models is challenging, potentially extremely time-consuming, resource-intensive, and complex. Furthermore, machine learning models tend to be highly specific to particular use cases and operating environments, meaning any change to the underlying environment or use case may necessitate a complete model rebuild. Moreover, building and deploying machine learning technologies differs significantly from traditional software engineering, requiring practices and architectures familiar to traditional software engineering development teams. While machine learning technologies have greatly benefited various organizations, their use demands substantial expertise that is not easily transferred to traditional data processing methods using relational databases and other data storage devices. Attached Figure Description

[0004] Various embodiments of this disclosure will be described with reference to the figures, in which:

[0005] Figure 1 This is a diagram illustrating the environment for machine learning inference calls used for database query processing according to some implementation schemes.

[0006] Figure 2 This is a diagram illustrating an asynchronous request handling procedure based on some implementation schemes.

[0007] Figure 3 This is a diagram illustrating an environment where machine learning inference calls are made according to some implementation schemes to perform database query processing using a local machine learning model.

[0008] Figure 4 This is a diagram illustrating alternative environments for performing machine learning inference calls based on some implementation schemes to use local machine learning models for database query processing.

[0009] Figure 5 This is a diagram illustrating an example user interface for machine learning inference calls according to some implementation schemes.

[0010] Figure 6 This is a flowchart illustrating the operation of a method for making machine learning inference calls for database query processing according to some implementation schemes.

[0011] Figure 7 It is a block diagram illustrating the operating environment for training and hosting machine learning models according to some implementation schemes.

[0012] Figure 8 The diagram illustrates an example provider network environment based on some implementation schemes.

[0013] Figure 9 This is a block diagram of an example provider network offering storage services and hardware virtualization services to customers, based on some implementation schemes.

[0014] Figure 10 It is a block diagram illustrating an example computer system that can be used in some implementation schemes. Detailed Implementation

[0015] This disclosure relates to methods, apparatus, systems, and non-transitory computer-readable storage media for using machine learning inference calls to perform database query processing. According to some embodiments, machine learning inference calls can be integrated into database queries to enable the use of machine learning techniques without requiring users to have specialized machine learning knowledge. In some embodiments, machine learning calls can be integrated into database queries, such as Structured Query Language (SQL) queries or other popular query languages ​​for processing structured data, without requiring significant changes to applications, database services, etc.

[0016] Figure 1 This is a diagram illustrating the environment for machine learning inference calls used for database query processing according to some implementation schemes. For example... Figure 1As shown, provider network 100 may include database service 112. Users can have structured data stored in one or more database instances 110 of database service 112. This data can be added to the database service by the user from user device 102, or it can be added from services of provider network 100 or from other services outside provider network 100. This data can be analyzed to obtain information useful to the user. Part of this analysis may include using machine learning techniques to perform inference on the data. For example, text data can be extracted from images stored in database service 112, the text data can be analyzed to identify sentiment associated with fragments of text data, and / or other specialized models can be used to perform inference on the user's data to obtain information about the data. However, as discussed, the use of machine learning techniques often requires specialized knowledge and is not well integrated into data management services (e.g., database service 112). Implementations address these problems by providing techniques for making machine learning inference calls for database query processing.

[0017] Provider Network 100 (or “cloud” provider network) enables users to utilize one or more of various types of computing-related resources, such as computing resources (e.g., executing virtual machine (VM) instances and / or containers, executing batch jobs, executing code without server provisioning), data / storage resources (e.g., object storage devices, block-level storage devices, data archive storage devices, databases and database tables, etc.), network-related resources (e.g., configuring virtual networks that include clusters of computing resources, content delivery networks (CDNs), domain name services (DNS)), application resources (e.g., databases, application building / deployment services), access policies or roles, identity policies or roles, machine images, routers, and other data processing resources. These and other computing resources can be provided as services, such as hardware virtualization services that can execute computing instances, storage services that can store data objects, etc. Users of Provider Network 100 (or “customers”) may utilize one or more user accounts associated with a customer account, but these terms may be used interchangeably to some extent depending on the use case. Users can interact with provider network 100 across one or more intermediate networks 106 (e.g., the Internet) via one or more interfaces 104 (e.g., through application programming interface (API) calls) via a console implemented as a website or application. Interface 104 may be part of or serve as a front end of the control plane 102 of provider network 100, which includes “back-end” services that support and enable services that can be provided more directly to customers.

[0018] For example, a cloud provider network (or simply "the cloud") typically refers to a large amount of accessible virtualized computing resources (such as computing, storage, and networking resources, applications, and services). The cloud can provide convenient, on-demand network access to a large, shared pool of configurable computing resources that can be programmatically provisioned and released in response to client commands. These resources can be dynamically provisioned and reconfigured to adapt to variable loads. Therefore, cloud computing can be viewed as applications delivered as services over publicly accessible networks (e.g., the Internet, cellular networks) and the hardware and software in the cloud provider's data centers that deliver these services.

[0019] Typically, provider network traffic and operations can be broadly categorized into two types: control plane operations performed above the logical control plane and data plane operations performed above the logical data plane. The data plane represents the movement of user data through a distributed computing system, while the control plane represents the movement of control signals through the distributed computing system. The control plane typically includes one or more control plane components distributed across and implemented by one or more control servers. Control plane traffic typically includes administrative operations such as system configuration and management (e.g., resource placement, hardware capability management, diagnostic monitoring, system status information). The data plane includes customer resources implemented on the provider network (e.g., compute instances, containers, block storage volumes, databases, file storage devices). Data plane traffic typically includes non-administrative operations such as transferring customer data to and from customer resources. Control plane components are typically implemented on a separate set of servers from the data plane servers, and control plane traffic and data plane traffic can be sent via separate / different networks.

[0020] To provide these and other computing resource services, provider network 100 typically relies on virtualization technology. For example, virtualization technology can be used to enable users to control or utilize computing instances (e.g., VMs using guest operating systems (O / S) that operate with a hypervisor that may or may not operate on top of an underlying host O / S; containers that may or may not operate within a VM; instances that can execute on "bare metal" hardware without an underlying hypervisor), where one or more computing instances can be implemented using a single electronic device. Therefore, users can directly utilize computing instances hosted by the provider network (e.g., provided by a hardware virtualization service) to perform various computing tasks. Alternatively, users can indirectly utilize computing instances by submitting code to be executed by the provider network (e.g., via an on-demand code execution service), which in turn utilizes the computing instance to execute the code—typically without the user having any control or knowledge of the underlying computing instance involved.

[0021] like Figure 1 As shown, a request can be sent to database service 112 to perform a query on data stored in one or more database instances 110. In some embodiments, the request may originate from user device 102 (as shown at 1A) or from service 108 of provider network 100 (e.g., serverless functionality or other services) (as shown at 1B). In various embodiments, a “serverless” functionality may include code that can be executed on demand, provided by a user or other entity (e.g., the provider network itself). The serverless functionality may be maintained within provider network 100 via on-demand code execution services and may be associated with a specific user or account or may generally be accessible to multiple users / accounts. The serverless functionality may be associated with a Uniform Resource Locator (URL), Uniform Resource Identifier (URI), or other reference used to invoke the serverless functionality. The serverless functionality may be executed by a computing instance (e.g., a virtual machine, container, etc.) when triggered or invoked. In some embodiments, the serverless functionality may be invoked via an Application Programming Interface (API) call or a specially formatted Hypertext Transfer Protocol (HTTP) request message. Therefore, users can define the serverless functions that can be executed on demand without requiring them to maintain dedicated infrastructure to perform these functions. Instead, serverless functions can be executed on demand using resources maintained by the provider network 100. In some implementations, these resources can be kept in a "ready" state (e.g., with a pre-initialized runtime environment configured to perform serverless functions), thereby allowing near real-time execution of serverless functions.

[0022] The request may originate from client 104A executing on user device 102 or client 104B of service 108, which may interface with database service 112 through one or more interfaces (e.g., application programming interface (API), text interface, graphical user interface (GUI), or other interfaces). The request may include database queries, such as SQL (or other query language) statements. While SQL statements are typically used to describe implementations, this is for illustrative purposes and not intended to be limiting. Implementations may similarly utilize alternative query languages. Database instance 110 may process the queries included in the request. In various implementations, the database service may be updated to identify inference requests included in database queries. In some implementations, database service 112 may be updated to identify API calls published to APIs of machine learning-enabled service 120. Machine learning-enabled service 120 may include one or more pre-trained models that can be used to perform inference on user data. Models may be trained for various inference tasks (e.g., sentiment analysis, text tagging, object detection, etc.) that can be used by multiple users.

[0023] In some implementations, a user can train a custom model or provide their own model, which is then hosted by machine learning service 116 as a hosted model 118. These hosted models can be used to perform user-specific inference tasks or user-specific tasks based on the user's own training data. In these implementations, a user can create function or model invoke commands, which the database service will identify during query execution. For example, a user can have a hosted model 118 that can be used to perform fraud detection on data stored in database service 112. To perform inference using the model, the user can, for example, identify when the model is being invoked in a query by specifying user-defined function commands to the database service.

[0024] create model My_FraudDetection

[0025] from [storage location]

[0026] returns[type] (

[0028] [inputs to the model] );

[0030] Users can then use the model in a database query to perform inference on data in database service 112. For example, the query could include: Select My_FraudDetection([inputs]) from [data source], where the managed model My_FraudDetection is invoked based on inputs to data from a data source (e.g., one or more database tables, specific rows of one or more database tables, etc.). During query processing, the database analyzer can identify the My_FraudDetection invocation within the select statement and determine the data associated with the invocation to provide to the machine learning service for inference using the model. For example, [inputs] could include statements identifying one or more columns of a specific database table, or specific rows and columns of a specific database table, data from multiple database tables, etc. Similarly, if an API associated with a machine learning-enabled service is identified during query processing, the database analyzer can identify the API within the query and determine the data associated with the API to provide to the machine learning-enabled service for inference using a pre-trained model.

[0031] At point 2, data to be provided to the machine learning service or a machine learning-enabled service can be provided to asynchronous request handler 114. If each record identified as associated with a machine learning call is individually passed to the machine learning service or the machine learning-enabled service, the resulting latency (e.g., due to various network calls and actual inference time incurred by calling another service) will lead to a poor user experience. Instead, asynchronous request handler 114 can be used to decouple query processing of database instance 110 from inference performed by machine learning service 116 or machine learning-enabled service 120.

[0032] An asynchronous request handler can receive data to be inferred in an input buffer. This allows the database service to send machine learning requests in batches, where each batch can include multiple records up to the size of the input buffer. Once the data determined by the asynchronous request handler has been added to the input buffer, the asynchronous request handler 114 can create micro-batches of data from the input buffer based on which service was invoked in the query, for sending to the machine learning service or machine learning-enabled service shown as numbers 3A and 3B. The size of the micro-batch can vary from service to service because each service can be configured to receive a different maximum number of records at a time. For example, the API provided by a given service can limit the number of requests that can be included in a batch. The micro-batch size and the input buffer batch size can be different, and the asynchronous request handler can generate micro-batches from requests in its input buffer. In some implementations, the micro-batch size can be smaller than the input buffer batch size, in which case the asynchronous request handler can generate multiple micro-batches until all machine learning requests from the input buffer have been sent to the invoked external service (e.g., the machine learning service or machine learning-enabled service). In some implementations, the micro-batch size may be greater than or equal to the input buffer batch size, in which case the micro-batch may include all requests included in the input buffer.

[0033] In some implementations, a single query may not generate enough machine learning requests to fill the input buffer of the asynchronous request handler 114. In such cases, the asynchronous request handler may receive machine learning requests generated by multiple queries, including queries from different users and / or as part of different processes being executed by a database service.

[0034] In response to receiving a micro-batch of machine learning requests, machine learning service 116 or machine learning-enabled service 120 (depending on which service is called in the query) can perform inference on the records included in the micro-batch and generate a response for each record. The responses can be added to the output buffer of asynchronous request handler 114 at 4A or 4B. The asynchronous request handler can monitor the output buffer and add flags or other data indicating that a complete set of responses has been received for the sent micro-batch of requests. The database service can monitor the output buffer and retrieve responses from it when a flag is identified, as shown at 5. In some embodiments, while the asynchronous request handler is processing requests from multiple users and / or transactions, each database instance can monitor the output buffer for its specific responses and retrieve only those responses corresponding to the requests sent by that instance. In some embodiments, each response can identify the request, database instance, user, and / or transaction associated with that response. Query processing can be completed using responses from the machine learning service and / or machine learning-enabled service by the database instance, and the query results can be returned at 6A or 6B.

[0035] Figure 2 This is a diagram illustrating an asynchronous request handler according to some implementation schemes. When database instance 110 processes a query, it can identify a query execution plan to execute the query. A given query can be executed in many different ways, and each way can provide different performance characteristics. The query optimizer can optimize the query execution plan for the query based on one or more performance requirements of the given query. In some implementations, during query execution, database processor 208 can create a virtual operator 200 that is capable of executing the query execution plan to add threads in which machine learning requests can be sent and can receive responses without blocking the main query processing thread. In some implementations, the query optimizer can change the evaluation order of predicates in the query to reduce the number of records that need to be called by the virtual operator 200 for machine learning.

[0036] Virtual operator 200 can identify records that need to be sent in batches to a machine learning service or a machine learning-enabled service, the batch being equal to the size of the input buffer 202 of asynchronous request handler 114. In some embodiments, virtual operator 200 can be implemented as a temporary data structure (e.g., a temporary file, notepad, or other data structure) that can be used to perform at least a portion of a query to identify records to be sent to the machine learning service or a machine learning-enabled service. For example, the query can specify merging data from multiple tables in a database service, and then a portion of the records in the merged data can be identified for sending to the machine learning service or a machine learning-enabled service. By using virtual operators, machine learning requests can be identified and sent to the asynchronous request handler in parallel with other parts of the query processing. At point 1, a batch of machine learning requests (e.g., including records, model endpoints / APIs, etc.) can be sent to input buffer 202. In some embodiments, different input buffers can be maintained for each machine learning service and machine learning-enabled service to which machine learning requests can be sent. Each input buffer can be implemented as a queue or other data structure to which virtual operators can add requests. Batch processing program 204 can generate appropriately sized micro-batches for the service being called. For example, at point 2, batch processing program 204 can divide the input batch from input buffer 202 into multiple micro-batches for sending to the called service. At point 3, each micro-batch can then be sent to the called external service (e.g., machine learning service 116 or machine learning-enabled service 120). As discussed, in some embodiments, the input batch size can be smaller than the batch size associated with the called external service. In the example described, a micro-batch can include all machine learning requests from the input batch.

[0037] When a machine learning response is generated, the external service can add the result to the output buffer 206 of the asynchronous request handler 114, as shown at number 4. When each microbatch has been fully processed, the external service can add a flag or other indicator indicating that processing of the microbatch is complete to the output buffer. In some implementations, alternatively, the external service can add a flag or other indicator to the output buffer once all machine learning requests associated with a given thing have been completed. While obtaining machine learning requests and responses in separate threads, the database processor 208 can simultaneously execute the query execution plan. When query execution reaches a machine learning service call (e.g., an API call, user-defined functionality, etc.), the database processor 208 can access the output buffer 206 at number 5 to obtain the machine learning response. If the response has not yet been populated in the output buffer, processing can wait until a flag (or multiple flags) indicating processing completion is set in the output buffer.

[0038] Figure 3 This diagram illustrates an environment for making machine learning inference calls to perform database query processing using a local machine learning model, according to some implementation schemes. As discussed, when using machine learning service 116 and managed model 118, a user can define the functionality associated with managed model 118 so that the database instance can identify that the machine learning service is being called. At point 1, database analyzer 200 (e.g., SQL analyzer or other analyzer) can receive user-defined functionality statements. In some implementations, to reduce the number of network calls required by the database service, at point 2, a request for the model identified in the user-defined functionality statement can be sent to the machine learning service. This request may include the performance, hardware, or other characteristics of the database instance. The machine learning service can compile a copy of the model for the database instance and, at point 3, return the model to the database instance. In some implementations, compiling model 306 can be implemented in shared library 304.

[0039] At point 4, a model schema 302 can be generated that maps the requested machine learning model to the compiled model 306. Subsequently, at point 5, the database instance 110 that requested the model can receive the query. The database processor can use model schema 302 to identify the corresponding compiled model 306 in the model library 304 and direct the machine learning request to the compiled model via asynchronous request handler 302, as shown at point 6. The processing of machine learning requests and responses can generally follow the above description. Figure 2The process is performed locally, but instead of sending the mini-batch requests to an external service via network calls, the mini-batch requests are sent locally to the compiled models 306 in the model library 304. This reduces the number of network calls required to the number of models being used in a given situation.

[0040] Figure 4 This is a diagram illustrating alternative environments for performing machine learning inference calls to use local machine learning models for database query processing, based on some implementation schemes. Figure 4 In this implementation scheme, the database service can be implemented as multiple nodes, including database instance 110 (e.g., the head node) and multiple nodes 400A-400N. Data stored in the database instance can be distributed across the multiple storage nodes. Numbers 1-3 can be referenced above. Figure 3 As discussed, however, at point 3, model deployer 402 receives the compiled model. Model deployer 402 can obtain the compiled model and deploy copies of the model to each storage node 400A-400N at point 4.

[0041] At point 5, a model pattern 302 can be generated that maps the called machine learning model to the compiled model 306. Then, at point 6, the database instance 110 calling the model can receive the query. At point 7, queries can be executed in parallel on one or more storage nodes based on the location of the stored data. While processing queries in parallel, inference can also be performed in parallel on each storage node using the compiled model 306 identified by model pattern 302. In some implementations, each node may also include an asynchronous request handler that can deliver multiple batches of requests to each compiled model located on its corresponding storage node. The processing of machine learning requests and responses can generally follow the above description. Figure 2 The process is performed locally, but instead of sending mini-batch requests to external services via network calls, the mini-batch requests are sent locally to compiled models 306 in model library 304. This reduces the number of network calls required to the number of models being used in a given situation.

[0042] Figure 5 This is a diagram illustrating an example user interface for machine learning inference calls based on some implementation schemes. For example... Figure 5As shown, multiple user interfaces (UIs) 500 can be implemented to enable multiple inference calls within a given database query. For example, UI 502 can use the ML_service function (or other user-qualified functions) to call a machine learning service on text input (e.g., via a selection statement that identifies, for example, a column named "verify" from a database table named "review_table"). Alternatively, UI 504 can be used to perform inference on any data type (e.g., a file "file_name" (e.g., data storage name, URI, URL, or other location identifier) ​​stored at "storage_location" from a dataset (e.g., a database table)). Alternatively, UI 506 can call a user-qualified function "ML_function," as previously discussed, which can be user-qualified to call a specific model (e.g., model_name) to perform inference on records from a dataset based on one or more input values. In some implementations, the model can be called directly (as shown at UI 508) instead of using the user-qualified function shown in UI 506. In some implementations, a view-based UI 510 can be used to invoke the model for a specific record from a predefined view V1 (e.g., such as...). Figure 5 The inference is performed on the records included in Table T1 shown, or other records defined in the predicate statement.

[0043] Figure 6 This is a flowchart illustrating operation 600 of a method for performing machine learning inference calls for database query processing according to some embodiments. Some or all of operation 600 (or other processes or variations thereof described herein and / or combinations thereof) are executed under the control of one or more computer systems configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) jointly executed by hardware or a combination thereof on one or more processors. The code is stored, for example, in the form of a computer program on a computer-readable storage medium, which includes instructions executable by one or more processors. The computer-readable storage medium is non-transitory. In some embodiments, one or more (or all) of operation 600 are executed by other diagrams such as database instance 110, asynchronous request handler 114, etc.

[0044] Operation 600 includes, at box 602, executing at least a portion of a query on data stored in a database service using a temporary data structure to generate an initial batch of machine learning requests, wherein the query identifies the machine learning service. In some embodiments, the temporary data structure may be a virtual operator created by a database processor to execute all or part of the query. In some embodiments, the query plan identified to execute all or part of the query may be optimized to reduce the number of machine learning calls required to process the query. In some embodiments, the query is a Structured Query Language (SQL) query. In some embodiments, the SQL query identifies the machine learning service using an Application Programming Interface (API) call to the machine learning service. In some embodiments, the machine learning service publishes the API to perform inference using a machine learning model in response to requests received from multiple users. In some embodiments, the query identifies the machine learning service using an endpoint associated with a machine learning model hosted by the machine learning service.

[0045] Operation 600 also includes generating a second batch of machine learning requests at block 604 based on the first batch of machine learning requests and based on the machine learning service. In some embodiments, the first batch of machine learning requests may be added to the input buffer of an asynchronous request handler. As discussed, the asynchronous request handler may manage machine learning requests to be sent to the machine learning service or a machine learning-enabled service. In some embodiments, the second batch of machine learning requests is sent to the machine learning service via at least one network. In some embodiments, the size of the second batch differs from the size of the first batch, and the size of the second batch is associated with the machine learning service. For example, the machine learning service may have a maximum batch size that limits the number of requests that can be sent to the machine learning service in batches. In some embodiments, the first batch of machine learning requests includes machine learning requests generated in response to multiple queries received from multiple different users.

[0046] In some implementations, operation 600 may further include sending a request for a machine learning model to a machine learning service; receiving a machine learning model from the machine learning service, the machine learning model being compiled by the machine learning service for use in a database service, and wherein a second batch of machine learning requests is sent to the machine learning model hosted by the database service. In some implementations, operation 600 may further include storing copies of the machine learning model across multiple nodes of the database service, wherein machine learning requests generated by a specific node of the database service during query processing are sent to the copies of the machine learning model stored on that specific node.

[0047] Operation 600 also includes obtaining multiple machine learning responses at box 606, said machine learning responses being generated by the machine learning service using a machine learning model in response to receiving a second batch of machine learning requests. In some implementations, as discussed, said multiple machine learning responses may be added to the output buffer of an asynchronous request handler. The database processor may obtain the machine learning responses from the output buffer and use said responses to complete the processing of the query.

[0048] In some implementations, operation 600 may include receiving a request at a database service, wherein the request includes a Structured Query Language (SQL) query to be executed on at least a portion of a dataset in the database service, and wherein the request identifies a machine learning service used when processing the SQL query; creating a virtual operator to execute at least a portion of the SQL query; generating a first batch of machine learning requests based at least on a portion of the SQL query executed by the virtual operator; sending the first batch of machine learning requests to an input buffer of an asynchronous request handler, which generates a second batch of machine learning requests based on the first batch of machine learning requests; obtaining a plurality of machine learning responses from an output buffer of the asynchronous request handler, the machine learning responses being generated by the machine learning service using a machine learning model in response to receiving the second batch of machine learning requests; and generating a query response based on the machine learning responses.

[0049] In some implementations, generating the first batch of machine learning requests based on SQL queries also includes determining a query execution plan that minimizes the number of records associated with each machine learning request. In some implementations, the machine learning service adds a flag to the output buffer when the second batch of machine learning requests has been processed.

[0050] Figure 7 This is a block diagram illustrating an operating environment for training and hosting machine learning models according to some implementation schemes. The operating environment includes an end-user device 102, a model training system 700, a model hosting system 702, a training data storage device 760, a training metric data storage device 765, a container data storage device 770, a training model data storage device 775, and a model prediction data storage device 780.

[0051] The machine learning service 116 described herein may include one or more of these entities, such as model hosting system 702, model training system 702, etc.

[0052] In some implementations, a user interacts with a model training system 702 via a user device 102 to provide data that enables the model training system 702 to train one or more machine learning models, as described elsewhere herein. A machine learning model can generally be viewed as one or more equations “trained” using a set of data. In some implementations, the model training system 702 provides ML functionality as a web service, and therefore messaging between the user device 102 and the model training system 702 (or provider network 100) and / or between components of the model training system 702 (or provider network 100) can use HTTP messages to transmit data in machine-readable file formats such as Extensible Markup Language (XML) or JavaScript Object Notation (JSON). In some implementations, providing access to various functions as a web service is not limited to communication exchanged via the World Wide Web, but more generally refers to services that can communicate with other electronic devices via computer networks.

[0053] User device 102 can interact with model training system 702 via front end 729. For example, user device 102 can provide training requests to front end 729, which includes a container image (or multiple container images or identifiers of one or more locations storing container images), an indicator of input data (e.g., the address or location of the input data), one or more hyperparameter values ​​(e.g., values ​​indicating how the algorithm will operate, how many algorithms will run in parallel, how much data will be split into clusters, etc.), and / or information describing the computing machine on which the machine learning model is trained (e.g., graphics processing unit (GPU) instance type, central processing unit (CPU) instance type, amount of memory to be allocated, type of virtual machine instance used for training, etc.).

[0054] In some implementations, the container image may include one or more layers, each representing executable instructions. Some or all of the executable instructions together represent an algorithm that defines a machine learning model. The executable instructions (e.g., the algorithm) may be written in any programming language (e.g., Python, Ruby, C++, Java, etc.). In some implementations, the algorithm is pre-generated and obtained by the user via user device 102 from an algorithm repository (e.g., a web-accessible marketplace, a data storage device provided by a machine learning training service, etc.). In some implementations, the algorithm is entirely or partially generated by the user (e.g., user-provided code that modifies or configures existing algorithm code).

[0055] In some implementations, instead of providing a container image (or its identifier) ​​in the training request, user device 102 may provide an algorithm written in any programming language in the training request. Model training system 702 then encapsulates the algorithm into a container (optionally containing additional code, such as a "base" ML algorithm supplemented with user-provided code), and finally loads the container into virtual machine instance 722 for training a machine learning model, as described in more detail below. For example, a user may develop an algorithm / code via user device 102 using an application (e.g., an interactive web-based programming environment) and provide the algorithm / code—perhaps as part of (or referenced in) a training request—to model training system 702, where the algorithm / code itself may be containerized or used with existing containers, such as those with machine learning frameworks.

[0056] In some implementations, instead of providing the container image in the training request, user device 102 provides an indicator of the container image (e.g., an indication of the address or location where the container image is stored) in the training request. For example, the container image may be stored in container data storage device 770, and this container image may have been previously created / uploaded by the user. Model training system 702 can retrieve the container image from the indicated location and create a container using the retrieved container image. The container is then loaded into virtual machine instance 722 for training a machine learning model, as described in more detail below.

[0057] In some implementations, model training system 702 can use information provided by user device 102 to train a machine learning model in one or more pre-built virtual machine instances 722. Specifically, model training system 702 includes a single physical computing device or multiple physical computing devices interconnected using one or more computing networks (not shown), wherein the physical computing devices host one or more virtual machine instances 722. Model training system 702 can handle the acquisition and configuration of computing power (e.g., containers, instances, etc., described in more detail below) based on information provided by user device 102 describing the computing machines on which the machine learning model is trained. Model training system 702 can then use the computing power to train the machine learning model, as described in more detail below. Model training system 702 can automatically scale up and down based on the total amount of training requests received from user device 102 via front end 729, thereby alleviating the burden on the user of worrying about over-utilization (e.g., acquiring too few computing resources and encountering performance problems) or under-utilization (e.g., acquiring more computing resources than needed to train the machine learning model, and thus duplicating resources).

[0058] In some implementations, virtual machine instance 722 is used to perform the task. For example, the task may include training a machine learning model. Figure 7 As shown, each virtual machine instance 722 includes an operating system (OS) 724, a language runtime 726, and one or more ML training containers 730. Typically, an ML training container 730 is a logical unit created within a virtual machine instance using resources available on the instance and can be used to isolate the execution of a task from other processes (e.g., task execution) occurring within the instance. In some implementations, an ML training container 730 is formed by one or more container images and a top container layer. Each container image may also include one or more image layers, where each image layer represents executable instructions. As described above, some or all of the executable instructions together represent an algorithm that defines a machine learning model. Changes made to the ML training container 730 (e.g., creating a new file, modifying an existing file, deleting a file, etc.) are stored in the top container layer. If the ML training container 730 is deleted, the top container layer is also deleted. However, the container images that form part of the deleted ML training container 730 may remain unchanged. The ML training container 730 can be implemented as, for example, a Linux container (LXC), a Docker container, etc.

[0059] In some implementations, the ML training container 730 may include individual runtimes 734, code 737, and dependencies 732 required by code 737. Runtime 734 may be defined by one or more executable instructions that form at least a portion of a container image for forming the ML training container 730 (e.g., executable instructions in the container image defining an operating system and / or a runtime running within a container formed by the container image). Code 737 includes one or more executable instructions that form at least a portion of the container image for forming the ML training container 730. For example, code 737 includes executable instructions in the container image representing an algorithm defining a machine learning model that may reference (or utilize) code or libraries according to dependency 732. Runtime 734 is configured to execute code 737 in response to an instruction to begin training the machine learning model. Execution of code 737 results in the generation of model data, as described in more detail below.

[0060] In some implementations, code 737 includes executable instructions representing algorithms that define different machine learning models. For example, code 737 includes: a set of executable instructions representing a first algorithm defining a first machine learning model; and a second set of executable instructions representing a second algorithm defining a second machine learning model. In some implementations, virtual machine instance 722 executes code 737 and trains all machine learning models. In some implementations, virtual machine instance 722 executes code 737 to select one of the machine learning models for training. For example, virtual machine instance 722 may identify the type of training data indicated by the training request and select a machine learning model corresponding to the identified training data type for training (e.g., executing executable instructions representing algorithms defining the selected machine learning model).

[0061] In some implementations, runtime 734 is the same as runtime 726 used by virtual machine instance 722. In some implementations, runtime 734 is different from runtime 726 used by virtual machine instance 722.

[0062] In some implementations, the model training system 702 uses one or more container images included in the training request (or container images retrieved from the container data storage device 770 in response to a received training request) to create and initialize an ML training container 730 in a virtual machine instance 722. For example, the model training system 702 creates an ML training container 730 that includes a container image and / or a top container layer.

[0063] Before initiating the training process, in some implementations, the model training system 702 retrieves training data from a location indicated in the training request. For example, the location indicated in the training request could be a location within the training data storage device 760. Therefore, the model training system 702 retrieves training data from the indicated location within the training data storage device 760. In some implementations, the model training system 702 does not retrieve training data before initiating the training process. Instead, the model training system 702 streams training data from the indicated location during the training process. For example, the model training system 702 may first retrieve a portion of the training data and provide the retrieved portion to a virtual machine instance 722 that is training the machine learning model. Once the virtual machine instance 722 has applied and used the retrieved portion, or once the virtual machine instance 722 will use all the retrieved portions (e.g., the buffer storing the retrieved portions is nearly empty), the model training system 702 may retrieve a second portion of the training data and provide the second retrieved portion to the virtual machine instance 722, and so on.

[0064] In some implementations, to perform machine learning model training, virtual machine instance 722 executes code 737 stored in ML training container 730. For example, code 737 includes some or all of the executable instructions that form a container image of the initialized ML training container 730. Therefore, virtual machine instance 722 executes some or all of the executable instructions that form the container image of the initialized ML training container 730 to train the machine learning model. Virtual machine instance 722 executes some or all of the executable instructions based on the hyperparameter values ​​included in the training request. As an illustrative example, virtual machine instance 722 trains the machine learning model by identifying the values ​​of certain parameters (e.g., coefficients, weights, centroids, etc.). The identified values ​​depend on the hyperparameters that define how training is performed. Therefore, virtual machine instance 722 can execute executable instructions to initiate a machine learning model training process, wherein the training process is run using the hyperparameter values ​​included in the training request. Execution of executable instructions may include virtual machine instance 722 applying training data retrieved by model training system 702 as input parameters to some or all of the instructions being executed.

[0065] In some implementations, executing executable instructions causes virtual machine instance 722 (e.g., ML training container 730) to generate model data. For example, ML training container 730 generates model data and stores the model data in its file system. The model data includes characteristics of the machine learning model being trained, such as the number of layers in the machine learning model, the hyperparameters of the machine learning model, the coefficients of the machine learning model, the weights of the machine learning model, etc. Specifically, the generated model data includes values ​​that define the characteristics of the machine learning model being trained. In some implementations, executing executable instructions modifies ML training container 730 such that model data is written to the top container layer of ML training container 730 and / or the container image forming part of ML training container 730 is modified to include the model data.

[0066] Virtual machine instance 722 (or model training system 702 itself) extracts generated model data from ML training container 730 and stores the generated model data in training model data storage device 775 as entries associated with virtual machine instance 722 and / or the machine learning model being trained. In some embodiments, virtual machine instance 722 generates a single file including model data and stores the single file in training model data storage device 775. In some embodiments, virtual machine instance 722 generates multiple files during the training of the machine learning model, each file including model data. In some embodiments, each model data file includes the same or different model data information (e.g., one file identifies the structure of the algorithm, another file includes a series of coefficients, etc.). Once training is complete, virtual machine instance 722 may package multiple files into a single file and store the single file in training model data storage device 775. Alternatively, virtual machine instance 722 may store multiple files in training model data storage device 775. Virtual machine instance 722 stores files in training model data storage device 775 during and / or after the training process is complete.

[0067] In some implementations, virtual machine instance 722 periodically stores model data files in training model data storage device 775 while the training process is in progress. Therefore, model data files can be stored in training model data storage device 775 at different times during the training process. Each set of model data files corresponding to a specific time, or each set of model data files existing in training model data storage device 775 up to a specific time, can serve as a checkpoint representing a different version of the partially trained machine learning model during different stages of the training process. Therefore, before training is complete, a user can submit a deployment and / or execution request via user device 102 in a manner described below to deploy and / or execute a version of the partially trained machine learning model (e.g., a machine learning model trained up to a certain stage of the training process). The version of the partially trained machine learning model can be based on some or all of the model data files stored in training model data storage device 775.

[0068] In some implementations, virtual machine instance 722 executes code 737 stored in multiple ML training containers 730. For example, the algorithm included in the container image may be in a format that allows the training process to be parallelized. Therefore, model training system 702 may create multiple copies of the container image provided in the training request and have virtual machine instance 722 load each copy of the container image into a separate ML training container 730. Virtual machine instance 722 can then execute the code 737 stored in the ML training containers 730 in parallel. Virtual machine instance 722 may further provide configuration information (e.g., information indicating that N ML training containers 730 are jointly training a machine learning model and that the specific ML training container 730 receiving the configuration information is the Xth ML training container 730 of the N) to each ML training container 730, the configuration information being included in the resulting model data. In some implementations, by parallelizing the training process, model training system 702 can significantly reduce training time.

[0069] In some implementations, multiple virtual machine instances 722 execute code 737 stored in multiple ML training containers 730. For example, the resources used to train a particular machine learning model may exceed the limitations of a single virtual machine instance 722. However, the algorithms included in the container image may be in a format that allows the training process to be parallelized. Therefore, the model training system 702 may create multiple copies of the container image provided in the training request, initialize multiple virtual machine instances 722, and have each virtual machine instance 722 load a copy of the container image into one or more separate ML training containers 730. The virtual machine instances 722 can then execute the code 737 stored in the ML training containers 730 in parallel. The model training system 702 can further provide configuration information (e.g., information indicating that N ML training containers 730 are jointly training a machine learning model and that the specific ML training container 730 receiving the configuration information is the Xth ML training container 730 out of N, information indicating that M virtual machine instances 722 are jointly training a machine learning model and that the specific ML training container 730 receiving the configuration information will be initialized in the Yth virtual machine instance 722 out of M, etc.) to each ML training container 730, and the configuration information can be included in the resulting model data. As described above, in some embodiments, by parallelizing the training process, the model training system 702 can significantly shorten the training time.

[0070] In some implementations, the model training system 702 includes multiple physical computing devices, and two or more of these physical computing devices host one or more virtual machine instances 722 that execute code 737. Therefore, in addition to occurring on different virtual machine instances 722 and / or ML training containers 730, parallelization can also occur on different physical computing devices.

[0071] In some implementations, the model training system 702 includes an ML model evaluator 728. The ML model evaluator 728 can monitor the virtual machine instance 722 while a machine learning model is being trained, thereby obtaining generated model data and processing the obtained model data to generate model metrics. For example, model metrics may include quality metrics such as the error rate of the machine learning model being trained, the statistical distribution of the machine learning model being trained, the latency of the machine learning model being trained, and the confidence level of the machine learning model being trained (e.g., a confidence level indicating the accuracy of the machine learning model being trained). The ML model evaluator 728 can obtain model data and evaluation data of the machine learning model being trained from the training data storage device 760. The evaluation data is separate from the data used to train the machine learning model and includes input data and expected outputs (e.g., known results), and therefore the ML model evaluator 728 can use the model data to constrain the machine learning model and execute the machine learning model by providing the input data as input. Then, the ML model evaluator 728 can compare the output of the machine learning model with the expected output and determine one or more quality metrics of the machine learning model being trained based on the comparison (e.g., the error rate could be the difference or distance between the machine learning model output and the expected output).

[0072] In some implementations, the ML model evaluator 728 periodically generates model metrics during the training process and stores them in a training metric data storage device 765. While the machine learning model is being trained, a user can access and retrieve model metrics from the training metric data storage device 765 via user device 102. The user can then use the model metrics to determine whether to adjust the training process and / or stop the training process. For example, model metrics may indicate that the machine learning model is performing poorly (e.g., error rate is above a threshold, statistical distribution is not the expected or desired distribution (e.g., not a binomial, Poisson, geometric, normal, Gaussian, etc.), execution latency is above a threshold, confidence level is below a threshold) and / or its performance is gradually deteriorating (e.g., quality metrics are deteriorating over time). In response, in some implementations, the user can transmit a request to modify the machine learning model being trained to the model training system 702 via user device 102 (e.g., transmit a modification request). The request may include a new or modified container image, a new or modified algorithm, new or modified hyperparameters, and / or new or modified information describing the computing machine on which the machine learning model is trained. The model training system 702 can modify the machine learning model accordingly. For example, the model training system 702 may cause the virtual machine instance 722 to optionally delete the existing ML training container 730, create and initialize a new ML training container 730 using some or all of the information included in the request, and execute code 737 stored in the new ML training container 730 to restart the machine learning model training process. As another example, the model training system 702 may cause the virtual machine instance 722 to modify the execution of code stored in the existing ML training container 730 based on data provided in the modification request. In some embodiments, a user may transmit a request to stop the machine learning model training process to the model training system 702 via user device 102. The model training system 702 may then instruct the virtual machine instance 722 to delete the ML training container 730 and / or delete any model data stored in the training model data storage device 775.

[0073] As described below, in some embodiments, the model hosting system 700 uses model data stored in the training model data storage device 775 to deploy a machine learning model. Alternatively, or additionally, the user device 102 or another computing device (not shown) can retrieve the model data from the training model data storage device 775 to implement the learning algorithm in an external device. As an illustrative example, a robotic device may include sensors applied to capture input data. The user device 102 can retrieve the model data from the training model data storage device 775 and store the model data in the robotic device. The model data defines the machine learning model. Therefore, the robotic device can provide the captured input data as input to the machine learning model, thereby producing an output. The robotic device can then perform actions (e.g., forward movement, arm raising, sound generation, etc.) based on the resulting output.

[0074] Although Figure 7 The virtual machine instances 722 are shown as a single group of virtual machine instances 722, but some embodiments of this application separate virtual machine instances 722 that are actively assigned to perform tasks from those that are not actively assigned to perform tasks. For example, virtual machine instances 722 that are actively assigned to perform tasks are grouped into an "active pool," while virtual machine instances 722 that are not actively assigned to perform tasks are placed in a "warm pool." In some embodiments, the virtual machine instances 722 in the warm pool may be pre-initialized in response to training requests to have an operating system, language runtime, and / or other software (e.g., rapid initialization of a machine learning model trained in the ML training container 730) to enable them to quickly perform tasks.

[0075] In some embodiments, the model training system 102 includes a processing unit, a network interface, a computer-readable media driver, and an input / output device interface, all of which can communicate with each other via a communication bus. The network interface provides connectivity to one or more networks or computing systems. Therefore, the processing unit can receive information and instructions from other computing systems or services (e.g., user device 102, model hosting system 700, etc.). The processing unit can also communicate with the memory of the virtual machine instance 722 and further provide output information to an optional display via the input / output device interface. The input / output device interface can also accept input from optional input devices. The memory may contain computer program instructions (grouped into modules in some embodiments) that the processing unit executes to implement one or more aspects of this disclosure.

[0076] In some implementations, the model hosting system 700 includes a single physical computing device or multiple physical computing devices interconnected using one or more computing networks (not shown), wherein the physical computing device hosts one or more virtual machine instance 742. The model hosting system 700 can handle the acquisition and configuration of computing power (e.g., containers, instances, etc.) based on the execution requirements of a trained machine learning model. The model hosting system 700 can then use the computing power to execute the machine learning model, as described in more detail below. The model hosting system 700 can automatically scale up and down based on the total number of execution requests received from the user device 102 via the front end of the model hosting system 700, thereby alleviating the burden on the user of worrying about over-utilization (e.g., acquiring too few computing resources and encountering performance problems) or under-utilization (e.g., acquiring more computing resources than required to run the machine learning model and thus duplicating resources).

[0077] In some implementations, virtual machine instance 742 is used to perform tasks. For example, the task may include executing a machine learning model. Figure 7 As shown, each virtual machine instance 742 includes an operating system (OS) 744, a language runtime 746, and one or more ML scoring containers 750. The ML scoring container 750 is similar to the ML training container 730 in that it is a logical unit created within the virtual machine instance using resources available on the instance and can be used to isolate the execution of a task from other processes occurring within the instance (e.g., task execution). In some implementations, the ML scoring container 750 is formed by one or more container images and a top container layer. Each container image also includes one or more image layers, where each image layer represents executable instructions. As described above, some or all of the executable instructions together represent an algorithm that defines a machine learning model. Changes made to the ML scoring container 750 (e.g., creating a new file, modifying an existing file, deleting a file, etc.) are stored in the top container layer. If the ML scoring container 750 is deleted, the top container layer is also deleted. However, the container images that form part of the deleted ML scoring container 750 may remain unchanged. The ML scoring container 750 can be implemented as, for example, a Linux container.

[0078] In some implementations, each ML scoring container 750 includes a runtime 754, code 756, and dependencies 752 required by code 756 (e.g., supporting software such as libraries). Runtime 754 may be defined by one or more executable instructions forming at least a portion of a container image (e.g., executable instructions in the container image defining an operating system and / or a runtime running within a container formed by the container image for forming the ML scoring container 750). Code 756 includes one or more executable instructions forming at least a portion of a container image for forming the ML scoring container 750. For example, code 756 includes executable instructions in the container image representing an algorithm defining a machine learning model, which may reference dependency 752. Code 756 may also include model data representing characteristics of the defined machine learning model, as described in more detail below. Runtime 754 is configured to execute code 756 in response to an instruction to begin executing a machine learning model. Execution of code 756 results in the generation of output (e.g., predicted results), as described in more detail below.

[0079] In some implementations, runtime 754 is the same as runtime 746 used by virtual machine instance 742. In some implementations, runtime 754 is different from runtime 746 used by virtual machine instance 742.

[0080] In some implementations, the model hosting system 700 uses one or more container images included in the deployment request (or container images retrieved from the container data storage device 770 in response to a received deployment request) to create and initialize an ML scoring container 750 in a virtual machine instance 742. For example, the model hosting system 700 creates an ML scoring container 750 that includes a container image and / or a top container layer.

[0081] As described above, in some implementations, user device 102 may submit deployment and / or execution requests to model hosting system 700 via a front end. The deployment request causes model hosting system 700 to deploy a trained machine learning model to virtual machine instance 742. For example, the deployment request may include an identifier for an endpoint (e.g., an endpoint name, such as an HTTP endpoint name) and an identifier for one or more trained machine learning models (e.g., the location of one or more model data files stored in training model data storage device 775). Optionally, the deployment request may also include an identifier for one or more container images stored in container data storage device 770.

[0082] Upon receiving a deployment request, the model hosting system 700 initializes one or more ML scoring containers 750 in one or more managed virtual machine instances 742. In an implementation where the deployment request includes an identifier of one or more container images, the model hosting system 700 forms the ML scoring container 750 from the identified container images. For example, the container image identified in the deployment request could be the same container image used to form an ML training container 730 for training the machine learning model corresponding to the deployment request. Therefore, the code 756 of the ML scoring container 750 includes one or more executable instructions in the container image representing an algorithm defining the machine learning model. In an implementation where the deployment request does not include an identifier of a container image, the model hosting system 700 forms the ML scoring container 750 from one or more container images stored in a container data storage device 770 suitable for executing the identified trained machine learning model. For example, a suitable container image could be a container image including executable instructions representing an algorithm defining the identified trained machine learning model.

[0083] In some implementations, the model hosting system 700 further forms an ML scoring container 750 by retrieving model data corresponding to an identified trained machine learning model. For example, a deployment request may identify the location of a model data file stored in a training model data storage device 775. In implementations where a single model data file is identified in a deployment request, the model hosting system 700 retrieves the identified model data file from the training model data storage device 775 and inserts the model data file into a single ML scoring container 750, the model data file forming part of code 756. In some implementations, the model data file is archived or compressed (e.g., formed by encapsulating individual files). Therefore, the model hosting system 700 unarchives or decompresses the model data file to obtain multiple individual files and inserts the individual files into the ML scoring container 750. In some implementations, the model hosting system 700 stores the model data file in the same location where the model data file is stored in the ML training container 730 where the model data file was generated. For example, the model data file is first stored in the top container layer of the ML training container 730 with a certain displacement, and then the model hosting system 700 stores the model data file in the top container layer of the ML scoring container 750 with the same displacement.

[0084] In an implementation that identifies multiple model data files in a deployment request, the model hosting system 700 retrieves the identified model data files from the training model data storage device 775. The model hosting system 700 can insert the model data files into the same ML scoring container 750, into different ML scoring containers 750 initialized in the same virtual machine instance 742, or into different ML scoring containers 750 initialized in different virtual machine instances 742. As an illustrative example, a deployment request can identify multiple model data files corresponding to different trained machine learning models because the trained machine learning models are related (e.g., the output of one trained machine learning model is used as the input of another trained machine learning model). Therefore, a user may need to deploy multiple machine learning models to ultimately receive a single output that depends on the outputs of multiple machine learning models.

[0085] In some implementations, the model hosting system 700 associates the initialized ML scoring containers 750 with endpoints identified in the deployment request. For example, each of the initialized ML scoring containers 750 may be associated with a network address. The model hosting system 700 may map the network addresses to the identified endpoints, and the model hosting system 700 or another system (e.g., a routing system, not shown) may store the mappings. Therefore, the user device 102 can use the endpoints to reference the trained machine learning model stored in the ML scoring containers 750. This allows the network address of the ML scoring containers 750 to be changed without requiring the user to manipulate the user device 102 to alter how the user references the trained machine learning model.

[0086] Once the ML scoring container 750 is initialized, it is ready to execute the trained machine learning model. In some embodiments, the user device 102 transmits an execution request via a front end to the model hosting system 700, wherein the execution request identifies an endpoint and includes input (e.g., a set of input data) destined for the machine learning model. The model hosting system 700 or another system (e.g., a routing system, not shown) can receive the execution request, identify the ML scoring container 750 corresponding to the identified endpoint, and route the input to the identified ML scoring container 750.

[0087] In some implementations, virtual machine instance 742 executes code 756 stored in the identified ML scoring container 750 in response to an execution request received by model hosting system 700. Specifically, the execution of code 756 causes executable instructions corresponding to the algorithm within code 756 to read the model data file stored in the ML scoring container 750, use the inputs included in the execution request as input parameters, and generate the corresponding output. For example, the algorithm may include coefficients, weights, layers, cluster centroids, etc. The executable instructions corresponding to the algorithm in code 756 can read the model data file to determine the values ​​of coefficients, weights, layers, cluster centroids, etc. The executable instructions may include input parameters, and the inputs included in the execution request can be supplied by virtual machine instance 742 as input parameters. Given the machine learning model characteristics and input parameters, virtual machine instance 742 can execute the executable instructions to produce output.

[0088] In some implementations, virtual machine instance 742 stores the output in model prediction data storage device 780. Alternatively, virtual machine instance 742 transmits the output via a front end to user device 102 that has submitted the execution results.

[0089] In some implementations, the execution request corresponds to a group of related trained machine learning models. Therefore, the ML scoring container 750 can transfer its output to a second ML scoring container 750 initialized in the same virtual machine instance 742 or a different virtual machine instance 742. The virtual machine instance 742 that initializes the second ML scoring container 750 can then execute second code 756 stored in the second ML scoring container 750, providing the received output as input parameters to the executable instructions in the second code 756. The second ML scoring container 750 also includes a model data file stored therein, and the executable instructions in the second code 756 read the model data file to determine values ​​that define the characteristics of the machine learning model. Execution of the second code 756 produces a second output. Then, the virtual machine instance 742 that initializes the second ML scoring container 750 can transmit the second output to the model prediction data storage device 780 and / or user device 102 via the front end (e.g., when no more trained machine learning models are needed to generate the output), or transmit the second output to the third ML scoring container 750 initialized in the same or different virtual machine instance 742 (e.g., when output from one or more additional trained machine learning models is needed), and the above-mentioned process can be repeated with respect to the third ML scoring container 750.

[0090] Although Figure 7The virtual machine instances 742 are shown as a single group of virtual machine instances 742, but some embodiments of this application separate virtual machine instances 742 that are actively assigned to perform tasks from those that are not actively assigned to perform tasks. For example, virtual machine instances 742 that are actively assigned to perform tasks are grouped into an "active pool," while virtual machine instances 742 that are not actively assigned to perform tasks are placed in a "warm pool." In some embodiments, virtual machine instances 742 in the warm pool may be pre-initialized in response to deployment requests and / or execution requests to have an operating system, language runtime, and / or other software required to quickly perform tasks (e.g., rapid initialization of the ML scoring container 750, rapid execution of code 756 in the ML scoring container, etc.).

[0091] In some embodiments, the model hosting system 700 includes a processing unit, a network interface, a computer-readable media drive, and an input / output device interface, all of which can communicate with each other via a communication bus. The network interface provides connectivity to one or more networks or computing systems. Therefore, the processing unit can receive information and instructions from other computing systems or services (e.g., user device 102, model training system 702, etc.). The processing unit can also communicate with the memory of the virtual machine instance 742 and further provide output information to an optional display via the input / output device interface. The input / output device interface can also accept input from optional input devices. The memory may contain computer program instructions (grouped into modules in some embodiments) that the processing unit executes to implement one or more aspects of this disclosure.

[0092] In some implementations, the operating environment supports many different types of machine learning models, such as multi-armed gambling machine models, reinforcement learning models, ensemble machine learning models, deep learning models, etc.

[0093] Figure 7 The model training system 702 and model hosting system 700 illustrated herein are not intended to be limiting. For example, the model training system 702 and / or model hosting system 700 may also have a greater number of devices than Figure 7 The diagrams illustrate operations within a limited or extensive computing environment. Therefore, for Figure 7The illustrations of model training system 702 and / or model hosting system 700 herein are to be considered illustrative and not limited to this disclosure. For example, model training system 702 and / or model hosting system 700, or various components thereof, may implement various web service components, hosted computing environments, or “cloud” computing environments and / or peer-to-peer network configurations to implement at least a portion of the processes described herein. In some embodiments, model training system 702 and / or model hosting system 700 may be implemented directly in hardware or software executed by hardware devices and may include, for example, one or more physical or virtual servers implemented on physical computer hardware configured to execute computer-executable instructions to perform the various features described herein. The one or more servers may be geographically distributed or geographically co-located, for example, located in one or more access points (POPs) or regional data centers.

[0094] Front-end 729 processes all training requests received from user device 102 and provisiones virtual machine instance 722. In some implementations, front-end 729 acts as a front door to all other services provided by model training system 702. Front-end 729 processes requests and ensures that the requests are properly authorized. For example, front-end 729 may determine whether to authorize the user associated with the training request to initiate the training process.

[0095] Similarly, the front end processes all deployment and execution requests received from user device 102 and provisiones virtual machine instance 742. In some implementations, the front end acts as a front door to all other services provided by model hosting system 700. The front end processes requests and ensures that the requests are properly authorized. For example, the front end may determine whether to authorize the user associated with a deployment or execution request to access the indicated model data and / or execute the indicated machine learning model.

[0096] Training data storage device 760 stores training data and / or evaluation data. Training data can be data used to train a machine learning model, and evaluation data can be data used to evaluate the performance of the machine learning model. In some embodiments, the training data and evaluation data share common data. In some embodiments, the training data and evaluation data do not share common data. In some embodiments, the training data includes input data and expected output. Although the training data storage device 760 is depicted as being located outside the model training system 702 and the model hosting system 700, this is not intended to be limiting. For example, in some embodiments not shown, the training data storage device 760 is located inside at least one of the model training system 702 or the model hosting system 700.

[0097] In some implementations, training metric data storage device 765 stores model metrics. While training metric data storage device 765 is depicted as located outside of model training system 702 and model hosting system 700, this is not intended to be limiting. For example, in some implementations not shown, training metric data storage device 765 is located inside at least one of model training system 702 or model hosting system 700.

[0098] Container data storage device 770 stores container images that can be retrieved by various virtual machine instances 722 and / or 742, such as container images used to form ML training container 730 and / or ML scoring container 750. While container data storage device 770 is depicted as residing outside model training system 702 and model hosting system 700, this is not intended to be limiting. For example, in some embodiments not shown, container data storage device 770 is located inside at least one of model training system 702 and model hosting system 700.

[0099] Training model data storage device 775 stores model data files. In some embodiments, some model data files consist of a single file, while others are encapsulated from multiple individual files. Although the training model data storage device 775 is depicted as being located outside the model training system 702 and the model hosting system 700, this is not intended to be limiting. For example, in some embodiments not shown, the training model data storage device 775 is located inside at least one of the model training system 702 or the model hosting system 700.

[0100] In some embodiments, the model prediction data storage device 780 stores the output (e.g., execution results) generated by the ML scoring container 750. While the model prediction data storage device 780 is depicted as being located outside the model training system 702 and the model hosting system 700, this is not intended to be limiting. For example, in some embodiments not shown, the model prediction data storage device 780 is located inside at least one of the model training system 702 and the model hosting system 700.

[0101] Although the model training system 702, model hosting system 700, training data storage device 760, training metric data storage device 765, container data storage device 770, training model data storage device 775, and model prediction data storage device 780 are illustrated as separate components, this is not intended to be limiting. In some embodiments, any or all of these components may be combined to perform the functions described herein. For example, any or all of these components may be implemented by a single computing device or by multiple different computing devices (e.g., computer servers) that are logically or physically grouped together to serve as a server system. Any or all of these components may communicate via a shared internal network, and the aggregate system (e.g., also referred to herein as a machine learning service) may communicate with one or more of the user devices 102 via one or more networks 106.

[0102] Figure 7 Various example user devices 102 (including desktop computers, laptop computers, and mobile phones) are illustrated herein, each provided by way of illustration. Typically, user device 102 can be any computing device, such as a desktop computer, laptop computer, tablet computer, personal computer, wearable computer, server, personal digital assistant (PDA), hybrid PDA / mobile phone, mobile phone, e-book reader, set-top box, voice command device, camera, digital media player, etc. In some embodiments, model training system 702 and / or model hosting system 700 provide user device 102 with one or more user interfaces, command line interface (CLI), application programming interface (API), and / or other programming interfaces for submitting training requests, deployment requests, and / or execution requests. In some embodiments, user device 102 can execute standalone applications that interact with model training system 702 and / or model hosting system 700 to submit training requests, deployment requests, and / or execution requests.

[0103] In some implementations, network 106 includes any wired network, wireless network, or a combination thereof. For example, network 106 may be a personal area network (PAN), local area network (LAN), wide area network (WAN), over-the-air broadcast network (e.g., for radio or television), cable network, satellite network, cellular telephone network, or a combination thereof. As another example, network 106 may be a publicly accessible network among linked networks that may be operated by various different parties, such as the Internet. In some implementations, network 106 may be a private or semi-private network, such as a corporate or university intranet. Network 106 may include one or more wireless networks, such as a Global System for Mobile Communications (GSM) network, a Code Division Multiple Access (CDMA) network, a Long Term Evolution (LTE) network, or any other type of wireless network. Network 106 may use protocols and components to communicate via the Internet or any of the aforementioned types of networks. For example, protocols used by network 106 may include HTTP, HTTP Secure (HTTPS), Message Queuing Telemetry Delivery (MQTT), CoAP, etc. The protocols and components of communication via the Internet or any of the aforementioned types of communication networks are well known to those skilled in the art and therefore will not be described further herein.

[0104] Figure 8 The diagram illustrates an example provider network (or "service provider system") environment according to some implementation schemes. Provider network 800 may provide resource virtualization to customers via one or more virtualization services 810, which allow customers to purchase, lease, or otherwise obtain instances 812 of virtualized resources (including, but not limited to, computing and storage resources) implemented on devices within the provider network or one or more data centers. A local Internet Protocol (IP) address 816 may be associated with resource instance 812; the local IP address is the internal network address of resource instance 812 on provider network 800. In some implementations, provider network 800 may also provide customers with public IP addresses 814 and / or ranges of public IP addresses (e.g., Internet Protocol version 4 (IPv4) or Internet Protocol version 6 (IPv6) addresses) that can be obtained from provider 800.

[0105] Conventionally, provider network 800, via virtualization service 810, can allow service provider customers (e.g., customers operating one or more client networks 850A-850C including one or more customer devices 852) to dynamically associate at least some public IP addresses 814 assigned or allocated to the customer with specific resource instances 812 assigned to the customer. Provider network 800 can also allow customers to remap public IP addresses 814 previously mapped to a virtualized computing resource instance 812 assigned to the customer to another virtualized computing resource instance 812 also assigned to the customer. Using the virtualized computing resource instances 812 and public IP addresses 814 provided by the service provider, customers of the service provider (e.g., operators of customer networks 850A-850C) can, for example, implement customer-specific applications and present these applications on an intermediate network 840 (e.g., the Internet). Then, other network entities 820 on the intermediate network 840 can generate traffic destined for the public IP address 814 published by the customer networks 850A-850C; the traffic is routed to the service provider's data center and, at the data center, routed via the network layer to the local IP address 816 of the virtualized computing resource instance 812, which is currently mapped to the public IP address 814. Similarly, response traffic from the virtualized computing resource instance 812 can be routed back to the intermediate network 840 leading to the source entity 820 via the network layer.

[0106] As used herein, a local IP address refers to an internal or “private” network address, such as that of a resource instance within a provider network. Local IP addresses may be within the address block reserved by the Internet Engineering Task Force (IETF) Request for Comment (RFC) 1918 and / or have the address format specified in IETF RFC 4193 and are variable within the provider network. Network traffic originating outside the provider network is not directly routed to local IP addresses; instead, traffic uses public IP addresses mapped to the local IP addresses of the resource instance. A provider network may include networking devices or appliances that provide Network Address Translation (NAT) or similar functionality to perform mappings from public IP addresses to local IP addresses (and vice versa).

[0107] A public IP address is a variable network address on the Internet assigned to a resource instance by a service provider or a customer. For example, traffic routed to a public IP address via 1:1 NAT translation is used to forward that traffic to the corresponding local IP address of the resource instance.

[0108] The provider's network infrastructure can assign certain public IP addresses to specific resource instances; these public IP addresses may be referred to as standard public IP addresses or simply standard IP addresses. In some implementations, the mapping of standard IP addresses to the local IP addresses of resource instances is the default startup configuration for all resource instance types.

[0109] At least some public IP addresses can be assigned to or obtained by customers of Provider Network 800; customers can then assign their assigned public IP addresses to specific resource instances assigned to them. These public IP addresses may be referred to as customer public IP addresses or simply customer IP addresses. Instead of being assigned to resource instances by Provider Network 800 as in the case of standard IP addresses, customer IP addresses can be assigned by the customer, for example, via an API provided by the service provider. Unlike standard IP addresses, customer IP addresses are assigned to customer accounts by the respective customer and can be remapped to other resource instances as needed or as required. Customer IP addresses are associated with customer accounts rather than specific resource instances, and the customer controls the IP address until the customer chooses to release it. Unlike regular static IP addresses, customer IP addresses allow customers to mask resource instance or availability zone failures by remapping their public IP addresses to any resource instance associated with their customer account. For example, customer IP addresses enable customers to plan around problems with their resource instances or software by remapping their customer IP addresses to alternative resource instances.

[0110] Figure 9 This is a block diagram of an example provider network providing storage services and hardware virtualization services to customers according to some implementation schemes. Hardware virtualization service 920 provides multiple computing resources 924 (e.g., VMs) to customers. Computing resources 924 may, for example, be leased or rented to customers of provider network 900 (e.g., customers implementing customer network 950). One or more local IP addresses may be provided for each computing resource 924. Provider network 900 may be configured to route packets from the local IP addresses of computing resources 924 to public Internet destinations and from public Internet sources to the local IP addresses of computing resources 924.

[0111] Provider network 900 enables client network 950, for example, coupled to intermediate network 940 via local area network 956, to implement virtual computing system 992 via hardware virtualization service 920 coupled to intermediate network 940 and provider network 900. In some embodiments, hardware virtualization service 920 may provide one or more APIs 902 (e.g., web service interfaces), through which client network 950 may access the functionality provided by hardware virtualization service 920, for example, via console 994 (e.g., web-based applications, standalone applications, mobile applications, etc.). In some embodiments, each virtual computing system 992 at provider network 900 and client network 950 may correspond to computing resources 924 leased, rented, or otherwise provided to client network 950.

[0112] From an instance of virtual computing system 992 (e.g., via console 994) and / or another client device 990, a client can access the functionality of storage service 910, for example, via one or more APIs 902, to access and store data from storage resources 918A-918N of virtual data storage device 916 provided by provider network 900 (e.g., folders or "buckets", virtualized volumes, databases, etc.). In some embodiments, a virtualized data storage gateway (not shown) may be provided at client network 950, which may locally cache at least some data (e.g., frequently accessed or critical data) and may communicate with storage service 910 via one or more communication channels to upload new or modified data from local cache memory to maintain the main data storage device (virtualized data storage device 916). In some implementations, users can install and access volumes of virtual data storage devices 916 via storage service 910, which serves as a storage virtualization service, through virtual computing system 992 and / or on another client device 990, and these volumes can be presented to users as local (virtualized) storage devices 998.

[0113] Although Figure 9 Not shown, but virtualization services can also be accessed from resource instances within provider network 900 via API 902. For example, a customer, appliance service provider, or other entity can access virtualization services from within a corresponding virtual network on provider network 900 via API 902 to request the allocation of one or more resource instances within that virtual network or another virtual network.

[0114] Explanatory System

[0115] In some implementations, a system implementing some or all of the techniques described herein may include a general-purpose computer system that includes or is configured to access one or more computer-accessible media, such as... Figure 10 The computer system 1000 is illustrated in the figure. In the illustrated embodiment, the computer system 1000 includes one or more processors 1010, which are coupled to system memory 1020 via an input / output (I / O) interface 1030. The computer system 1000 also includes a network interface 1040 coupled to the I / O interface 1030. Although Figure 10 The computer system 1000 is shown as a single computing device, but in various embodiments, the computer system 1000 may include a single computing device or any number of computing devices configured to cooperate as a single computer system 1000.

[0116] In various embodiments, computer system 1000 may be a single-processor system including one processor 1010 or a multiprocessor system including several processors 1010 (e.g., two, four, eight, or another suitable number). Processor 1010 may be any suitable processor capable of executing instructions. For example, in various embodiments, processor 1010 may be a general-purpose processor or an embedded processor implementing any of a variety of instruction set architectures (ISAs) (e.g., x86, ARM, PowerPC, SPARC, or MIPS ISA or any other suitable ISA). In a multiprocessor system, each of processors 1010 may typically, but not necessarily, implement the same ISA.

[0117] System memory 1020 can store instructions and data accessible by processor 1010. In various embodiments, system memory 1020 can be implemented using any suitable memory technology, such as random access memory (RAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory. In the illustrated embodiment, program instructions and data (such as those methods, techniques, and data described above) implementing one or more desired functions are shown stored in system memory 1020 as database service code 1025 and data 1026.

[0118] In one embodiment, I / O interface 1030 may be configured to coordinate I / O traffic between processor 1010, system memory 1020, and any peripheral devices (including network interface 1040 or other peripheral interfaces) within the device. In some embodiments, I / O interface 1030 may perform any necessary protocol, timing, or other data transformations to convert data signals from one component (e.g., system memory 1020) into a format suitable for use by another component (e.g., processor 1010). For example, in some embodiments, I / O interface 1030 may include support for devices attached via various types of peripheral buses (e.g., variants of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard). For example, in some embodiments, the functionality of I / O interface 1030 may be divided into two or more separate components, such as a northbridge and a southbridge. Furthermore, in some embodiments, some or all of the functionality of I / O interface 1030 (e.g., an interface to system memory 1020) may be directly incorporated into processor 1010.

[0119] For example, network interface 1040 can be configured to allow computer system 1000 to communicate with other devices 1060 attached to one or more networks 1050 (e.g., Figure 1 The network interface 1040 exchanges data with other computer systems or devices illustrated in the diagram. For example, in various embodiments, the network interface 1040 can support communication via any suitable wired or wireless general-purpose data network (e.g., Ethernet network type). Additionally, the network interface 1040 can support communication via telecommunications / telephone networks (e.g., analog voice networks or digital fiber optic communication networks), via storage area networks (SANs) (e.g., Fibre Channel SANs), or via I / O of any other suitable type of network and / or protocol.

[0120] In some embodiments, computer system 1000 includes one or more offload cards 1070 (including one or more processors 1075 and possibly one or more network interfaces 1040), which are connected using I / O interfaces 1030 (e.g., a bus implementing a version of the Peripheral Component Interconnect – Fast (PCI-E) standard or another interconnect (e.g., QuickPath Interconnect (QPI) or UltraPath Interconnect (UPI)). For example, in some embodiments, computer system 1000 may serve as a host electronic device hosting computing instances (e.g., as part of a hardware virtualization service), and one or more offload cards 1070 act as a virtualization manager capable of managing the computing instances running on the host electronic device. For example, in some embodiments, offload card 1070 may perform computing instance management operations, such as pausing and / or resuming computing instances, starting and / or terminating computing instances, performing memory transfer / copy operations, etc. In some implementations, these management operations may be performed by the offload card 1070 in conjunction with a hypervisor (e.g., a request from the hypervisor), which is executed by other processors 1010A-1010N of the computer system 1000. However, in some implementations, the virtualization manager implemented by the offload card 1070 may accommodate requests from other entities (e.g., from the computing instance itself) and may not cooperate with any separate hypervisor (or service).

[0121] In some embodiments, system memory 1020 may be an embodiment of a computer-accessible medium configured to store the program instructions and data described above. However, in other embodiments, program instructions and / or data may be received, transmitted, or stored on different types of computer-accessible media. Generally, computer-accessible media may include non-transitory storage media or memory media such as magnetic or optical media, for example, a disc or DVD / CD coupled to computer system 1000 via I / O interface 1030. Non-transitory computer-accessible storage media may also include any volatile or non-volatile media, such as RAM (e.g., SDRAM, Double Data Rate (DDR) SDRAM, SRAM, etc.), read-only memory (ROM), etc., which may be included as system memory 1020 or another type of memory in some embodiments of computer system 1000. Furthermore, computer-accessible media may include transmission media or signals (e.g., electrical signals, electromagnetic signals, or digital signals) delivered via communication media (e.g., networks and / or wireless links), which may be implemented, for example, via network interface 1040.

[0122] The various implementation schemes discussed or proposed herein can be implemented in a wide variety of operating environments, in some cases of which may include one or more user computers, computing devices, or processing devices capable of operating any of a number of applications. User devices or client devices may include any of a number of general-purpose personal computers, such as desktop or laptop computers running standard operating systems, and cellular, wireless, and handheld devices running mobile software and capable of supporting multiple networking and messaging protocols. The system may also include multiple workstations running various commercially available operating systems and any of other known applications for purposes such as development and database management. These devices may also include other electronic devices, such as virtual terminals, thin clients, gaming systems, and / or other devices capable of communicating via a network.

[0123] Most implementations utilize at least one network familiar to those skilled in the art, employing any of a variety of widely available protocols to support communication, such as Transmission Control Protocol / Internet Protocol (TCP / IP), File Transfer Protocol (FTP), Universal Plug and Play (UPnP), Network File System (NFS), Universal Internet File System (CIFS), Extensible Messaging and Field Protocol (XMPP), AppleTalk, etc. The network can include, for example, a Local Area Network (LAN), a Wide Area Network (WAN), a Virtual Private Network (VPN), the Internet, an intranet, an extranet, the Public Switched Telephone Network (PSTN), an infrared network, a wireless network, and any combination thereof.

[0124] In implementations utilizing a web server, the web server can run any of various server or middleware applications, including HTTP servers, File Transfer Protocol (FTP) servers, Common Gateway Interface (CGI) servers, data servers, Java servers, business application servers, etc. The server may also be able to execute programs or scripts in response to requests from user devices, for example, by executing one or more web applications, which can be implemented in any programming language (e.g., ...). One or more scripts or programs written in C, C#, or C++, or any scripting language (such as Perl, Python, PHP, or TCL, and combinations thereof). The server may also include a database server, including but not limited to database servers available from Oracle(R), Microsoft(R), Sybase(R), IBM(R), etc. The database server may be relational or non-relational (e.g., "NoSQL"), distributed or non-distributed, etc.

[0125] The environments disclosed herein may include the various data storage devices and other memories and storage media discussed above. These may reside in various locations, such as locally within one or more computers (and / or residing within one or more computers) or across a network on storage media remotely from any or all of the computers. In a particular set of embodiments, information may reside in a storage area network (SAN) familiar to those skilled in the art. Similarly, any necessary files for performing functions belonging to a computer, server, or other network device may be stored locally and / or remotely, as appropriate. Where the system includes computerized devices, each of these devices may include hardware elements that can be electrically coupled via a bus, including, for example, at least one central processing unit (CPU), at least one input device (e.g., mouse, keyboard, controller, touchscreen, or keypad), and / or at least one output device (e.g., display device, printer, or speaker). The system may also include one or more storage devices, such as disk drives, optical storage devices, and solid-state storage devices (e.g., random access memory (RAM) or read-only memory (ROM)), as well as removable media devices, memory cards, flash cards, etc.

[0126] The apparatus may also include a computer-readable storage medium reader, communication devices (e.g., modems, network cards (wireless or wired), infrared communication devices, etc.), and the working memory described above. The computer-readable storage medium reader may be connected to or configured to receive a computer-readable storage medium, which represents a remote, local, fixed, and / or removable storage device and storage medium for temporarily and / or more permanently containing, storing, transmitting, and retrieving computer-readable information. Systems and various apparatuses will also typically include multiple software applications, modules, services, or other elements residing within at least one working memory device, including operating systems and applications such as client applications or web browsers. It should be understood that alternative implementations can have many variations from the implementations described above. For example, custom hardware may be used and / or specific elements may be implemented in hardware, software (including portable software, such as applets), or both hardware and software. Furthermore, connections to other computing devices, such as network input / output devices, may be employed.

[0127] Storage media and computer-readable media used for portions containing code may include any suitable media known or used in this art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented in any way or technique to store and / or transmit information (e.g., computer-readable instructions, data structures, program modules or other data), including RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage devices, magnetic cassettes, magnetic tapes, disk storage devices or other magnetic storage devices, or any other media that can be used to store the desired information and can be accessed by system devices. Based on this disclosure and the teachings provided herein, those skilled in the art will understand other ways and / or methods of implementing various embodiments.

[0128] Various implementation schemes have been described in the foregoing description. Specific configurations and details have been stated for illustrative purposes to provide a thorough understanding of the implementation schemes. However, those skilled in the art will also understand that the implementation schemes can be practiced without specific details. Furthermore, well-known features may be omitted or simplified to avoid obscuring the implementation schemes being described.

[0129] This document uses parenthesized text and boxes with dashed borders (e.g., large dashes, small dashes, dotted lines, and dots) to illustrate optional operations of adding additional features to some embodiments. However, the notation should not be construed as implying that these are the only options or optional operations, and / or that boxes with solid borders are not optional in some embodiments.

[0130] Reference numbers with suffix letters (e.g., 918A-918N) can be used to indicate the existence of one or more instances of the mentioned entity in various embodiments, and when multiple instances exist, each need not be identical but may share some general characteristics or function in a common manner. Furthermore, the specific suffix used is not intended to imply the existence of a specific number of entities unless specifically indicated otherwise. Therefore, in various embodiments, two entities using the same or different suffix letters may or may not have the same number of instances.

[0131] The use of terms such as "an implementation scheme," "implementation scheme," or "example implementation scheme" indicates that the implementation scheme may include specific features, structures, or characteristics, but each implementation scheme may not necessarily include such specific features, structures, or characteristics. Furthermore, the wording does not necessarily refer to the same implementation scheme. Additionally, when a specific feature, structure, or characteristic is described in connection with an implementation scheme, it is assumed that, whether explicitly described or not, the influence of such feature, structure, or characteristic on the description in conjunction with other implementation schemes is within the knowledge of a person skilled in the art.

[0132] Furthermore, in the various embodiments described above, unless otherwise specifically stated, it is intended that antonymous conjunctions (e.g., the phrase "at least one of A, B, or C") be understood to mean A, B, or C, or any combination thereof (e.g., A, B, and / or C). Thus, antonymous conjunctions are not intended and should not be construed as implying that a given embodiment requires the existence of at least one of A, at least one of B, or at least one of C.

[0133] At least some implementations of the disclosed technology may be described in light of the following terms:

[0134] 1. A computer-implemented method, the computer-implemented method comprising:

[0135] A request is received at a database service, wherein the request includes a Structured Query Language (SQL) query to be executed on at least a portion of a dataset in the database service, and wherein the request identifies a machine learning service to be used when processing the SQL query;

[0136] Create virtual operators to execute at least a portion of the SQL query;

[0137] The first batch of machine learning requests is generated based at least on the portion of the SQL query executed by the virtual operator;

[0138] The first batch of machine learning requests is sent to the input buffer of the asynchronous request handler, which generates a second batch of machine learning requests based on the first batch of machine learning requests.

[0139] Multiple machine learning responses are obtained from the output buffer of the asynchronous request handler, the machine learning responses being generated by the machine learning service using a machine learning model in response to receiving the second batch of machine learning requests; and

[0140] The query response is generated based on the machine learning response.

[0141] 2. The computer-implemented method as described in Clause 1, wherein generating the first batch of machine learning requests based at least on the SQL query further includes:

[0142] Determine a query execution plan that minimizes the number of records associated with the machine learning request.

[0143] 3. A computer-implemented method as described in any one of Clauses 1 or 2, wherein when the second batch of machine learning requests has been processed, the machine learning service adds a flag to the output buffer.

[0144] 4. A computer-implemented method, the computer-implemented method comprising:

[0145] Use temporary data structures to perform at least a portion of queries on data stored in a database service to generate the first batch of machine learning requests, wherein the queries identify the machine learning service;

[0146] A second batch of machine learning requests is generated based on the first batch of machine learning requests and based on the machine learning service; and

[0147] Multiple machine learning responses are obtained, which are generated by the machine learning service using a machine learning model in response to receiving the second batch of machine learning requests.

[0148] 5. The computer-implemented method as described in Clause 4, wherein the query is a Structured Query Language (SQL) query.

[0149] 6. The computer-implemented method as described in Clause 5, wherein the SQL query identifies the machine learning using an application programming interface (API) call to the machine learning service.

[0150] 7. A computer-implemented method as described in Clause 6, wherein the machine learning service publishes the API to perform inference using the machine learning model in response to requests received from multiple users.

[0151] 8. A computer-implemented method as described in any one of Clauses 4 to 7, wherein the query identifies the machine learning service using an endpoint associated with the machine learning model hosted by the machine learning service.

[0152] 9. A computer-implemented method as described in any one of Clauses 4 to 8, wherein the second batch of machine learning requests is sent to the machine learning service via at least one network.

[0153] 10. The computer-implemented method as described in any one of clauses 4 to 9, wherein the computer-implemented method further comprises:

[0154] The request for the machine learning model is sent to the machine learning service;

[0155] The machine learning model is received from the machine learning service, and the machine learning model is compiled by the machine learning service for use with the database service; and

[0156] The second batch of machine learning requests is sent to the machine learning model hosted by the database service.

[0157] 11. The computer-implemented method as described in Clause 10, further comprising:

[0158] A copy of the machine learning model is stored across multiple nodes of the database service, wherein a machine learning request generated by a specific node of the database service during the query processing is sent to the copy of the machine learning model stored on that specific node.

[0159] 12. A computer-implemented method as described in any one of Clauses 4 to 11, wherein the size of the second batch is different from the size of the first batch, and wherein the size of the second batch is associated with the machine learning service.

[0160] 13. A computer-implemented method as described in any one of Clauses 4 to 12, wherein the first batch of machine learning requests includes machine learning requests generated in response to multiple queries received from multiple different users.

[0161] 14. A system comprising:

[0162] Machine learning services, the machine learning services being implemented by a first or more electronic devices; and

[0163] A database service, implemented by a second or more electronic devices, includes instructions that, when executed, cause the database service to:

[0164] At least a portion of the query is performed on the data stored in the database service using a temporary data structure to generate the first batch of machine learning requests, wherein the query identifies the machine learning service;

[0165] A second batch of machine learning requests is generated based on the first batch of machine learning requests and based on the machine learning service; and

[0166] Multiple machine learning responses are obtained, which are generated by the machine learning service using a machine learning model in response to receiving the second batch of machine learning requests.

[0167] 15. The system as described in Clause 14, wherein the query is a Structured Query Language (SQL) query.

[0168] 16. The system as described in Clause 15, wherein the SQL query identifies the machine learning service using an application programming interface (API) call to the machine learning service.

[0169] 17. The system as described in Clause 16, wherein the machine learning service publishes the API to perform inference using the machine learning model in response to requests received from multiple users.

[0170] 18. The system of any one of Clauses 14 to 17, wherein the query uses an endpoint associated with the machine learning model hosted by the machine learning service to identify the machine learning service.

[0171] 19. The system of any one of Clauses 14 to 18, wherein the second batch of machine learning requests is sent to the machine learning service via at least one network.

[0172] 20. The system of any one of clauses 14 to 19, wherein the system further comprises:

[0173] The request for the machine learning model is sent to the machine learning service;

[0174] The machine learning model is received from the machine learning service, and the machine learning model is compiled by the machine learning service for use in the database service;

[0175] The second batch of machine learning requests is sent to the machine learning model hosted by the database service; and

[0176] A copy of the machine learning model is stored across multiple nodes of the database service, wherein a machine learning request generated by a specific node of the database service during the query processing is sent to the copy of the machine learning model stored on that specific node.

[0177] Therefore, the specification and drawings are to be regarded in an illustrative rather than restrictive sense. However, it will be understood that various modifications and changes may be made to the specification and drawings without departing from the broader spirit and scope of this disclosure as set forth in the claims.

Claims

1. A computer-implemented method comprising: using, by a database instance of a database service, a temporary data structure to perform at least a portion of a query on data stored in the database service to generate a first batch of machine learning requests, wherein the query identifies a machine learning service, wherein performing at least the portion of the query comprises determining data to be provided to the machine learning service, and wherein each machine learning request comprises the determined data to be provided to the machine learning service; sending, by the database instance, the first batch of machine learning requests to an asynchronous request handler of the database service, wherein a size of the first batch of machine learning requests is equal to an input buffer size of the asynchronous request handler; generating, based on the first batch of machine learning requests and based on the machine learning service, a second batch of machine learning requests, wherein a size of the second batch of machine learning requests is associated with the machine learning service; receiving, by the asynchronous request handler, a plurality of machine learning responses generated by the machine learning service using a machine learning model in response to receiving the second batch of machine learning requests; and obtaining, by the database instance, the plurality of machine learning responses from an output buffer of the asynchronous request handler.

2. The computer-implemented method of claim 1, wherein the query is a structured query language (SQL) query.

3. The computer-implemented method of claim 2, wherein the SQL query identifies the machine learning using an application programming interface (API) call to the machine learning service.

4. The computer-implemented method of claim 3, wherein the machine learning service publishes the API to perform inference using the machine learning model in response to requests received from a plurality of users.

5. The computer-implemented method of any one of claims 1 to 4, wherein the query identifies the machine learning service using an endpoint associated with the machine learning model hosted by the machine learning service.

6. The computer-implemented method of any one of claims 1 to 5, wherein the second batch of machine learning requests is sent to the machine learning service via at least one network.

7. The computer-implemented method of any one of claims 1 to 6, further comprising: sending a request for the machine learning model to the machine learning service; receiving the machine learning model from the machine learning service, the machine learning model compiled by the machine learning service for use by the database service; and wherein the second batch of machine learning requests is sent to the machine learning model hosted by the database service.

8. The computer-implemented method of claim 7, further comprising: ​ storing a copy of the machine learning model in a plurality of nodes of the database service, wherein a machine learning request generated by a particular node of the database service during the query processing is sent to the copy of the machine learning model stored on the particular node.

9. The computer-implemented method of any one of claims 1 to 8, wherein a size of the second batch is different than a size of the first batch, and wherein the size of the second batch is associated with the machine learning service.

10. The computer-implemented method of any one of claims 1 to 9, wherein the first batch of machine learning requests comprises machine learning requests generated in response to a plurality of queries received from a plurality of different users.

11. A system comprising: a machine learning service implemented by a first one or more electronic devices; and a database service implemented by a second one or more electronic devices, the database service comprising instructions that, when executed, cause the database service to: execute, by a database instance of database service, at least a portion of a query on data stored in the database service using a temporary data structure to generate a first batch of machine learning requests, wherein the query identifies the machine learning service, wherein executing at least the portion of the query comprises determining data to be provided to the machine learning service, and wherein each machine learning request comprises the determined data to be provided to the machine learning service; send, by the database instance, the first batch of machine learning requests to an asynchronous request handler of the database service, wherein a size of the first batch of machine learning requests is equal to an input buffer size of the asynchronous request handler; generate a second batch of machine learning requests based on the first batch of machine learning requests and based on the machine learning service, wherein a size of the second batch of machine learning requests is associated with the machine learning service; receive, by the asynchronous request handler, a plurality of machine learning responses generated by the machine learning service in response to receiving the second batch of machine learning requests using a machine learning model; and obtain, by the database instance, the plurality of machine learning responses from an output buffer of the asynchronous request handler.

12. The system of claim 11, wherein the query is a structured query language (SQL) query.

13. The system of claim 12, wherein the SQL query identifies the machine learning using an application programming interface (API) call to the machine learning service.

14. The system of claim 13, wherein the machine learning service publishes the API to perform inference using the machine learning model in response to requests received from a plurality of users.

15. The system of any one of claims 11 to 14, wherein the query identifies the machine learning service using an endpoint associated with the machine learning model hosted by the machine learning service.

Images