Caching in machine learning model hosting services

Abstract

Techniques are described for caching in a machine learning (ML) model hosting service. In accordance with host usage data provided from each host in the first set of hosts, ML model usage data is aggregated, the ML model usage data including, for a particular ML model, a number of inferred requests for the particular ML model. A priority order of inference requests for hosts in the second set of hosts that serve the particular ML model is calculated. A set of ML models to be loaded to a particular host of the second set of hosts is determined based on the ML model usage data and the priority order. The particular host is caused to load the set of ML models. The router is updated to direct ML model inference requests among the second set of hosts.

Description

Background Technology

[0001] Machine learning (ML) is a discipline that trains computer systems to perform various tasks by repeatedly exposing them to training data. Using the trained model, often referred to as "inference," involves the model receiving new data not in its training dataset and providing outputs based on its learned parameters. Attached Figure Description

[0002] Various examples according to this disclosure will be described with reference to the following figures.

[0003] Figure 1 This is a diagram illustrating, based on some examples, an environment used to host machine learning models within a cloud provider's network.

[0004] Figure 2A and 2B This is a graph illustrating the relationship between load and wait times for models with and without cache warm-up, based on some examples.

[0005] Figure 3 This is a diagram illustrating hashing techniques used to determine model host priority based on some examples.

[0006] Figure 4 This is a diagram illustrating cache warm-up based on some examples.

[0007] Figure 5 This is a flowchart illustrating the operations of a method for caching in a machine learning model hosting service, based on some examples.

[0008] Figure 6 The illustration shows an exemplary cloud provider network environment based on some examples.

[0009] Figure 7 This is a block diagram of an exemplary cloud provider network that provides storage services and hardware virtualization services to customers, based on some examples.

[0010] Figure 8 This is a block diagram illustrating an exemplary computing device that can be used in some examples. Detailed Implementation

[0011] This disclosure relates to methods, apparatus, systems, and non-transitory computer-readable storage media for caching in machine learning (ML) model hosting services. According to some examples, a group of cloud provider network hosts (sometimes collectively referred to as endpoints) execute ML models to serve inference requests. In dynamic environments such as cloud provider network environments, various events can cause changes in the composition of those hosts, such as adding new hosts to expand the host group or application software updates. Introducing a new host that does not serve inference requests and therefore does not load any models can result in (at least initially) high cache miss rates (up to 100% if the entire group of hosts changes). These miss rates gradually decrease over time as hosts load models from network storage to local storage and from local storage to execution environment storage; however, misses still degrade the end-user experience during model refill cycles due to the additional latency required to load the requested models. This document discloses techniques to mitigate and potentially eliminate these performance losses by preloading models to hosts, such that when a host is introduced, the host receives a request for a potentially preloaded model.

[0012] Figure 1 This diagram illustrates an environment for hosting machine learning models within a cloud provider network, based on several examples. The cloud provider network's model hosting service 110 provides ML model hosting services. The cloud provider network 100 (also referred to herein as a provider network, service provider network, etc.) provides users with the ability to utilize one or more of various types of compute-related resources, such as compute resources (e.g., executing virtual machine (VM) instances and / or containers, executing batch jobs, executing code without provisioning servers), data / storage resources (e.g., object storage, block-level storage, data archive storage, databases and database tables, etc.), network-related resources (e.g., configuring virtual networks (containing compute resource groups), content delivery networks (CDNs), domain name services (DNS)), application resources (e.g., databases, application build / deployment services), access policies or roles, identity policies or roles, machine images, routers, and other data processing resources. These and other compute resources can be provided as services, such as hardware virtualization services that can execute compute instances, storage services that can store data objects, etc. Users (or “customers”) of a cloud provider network 100 may use one or more user accounts associated with a customer account, but these terms may be used interchangeably to some extent depending on the use case. Cloud provider networks are sometimes “multi-tenant” because they may use the same physical computing infrastructure to provide services to multiple different customers.

[0013] Users can interact with cloud provider network 100 across one or more intermediate networks (e.g., the Internet) via one or more interfaces, such as through application programming interface (API) calls, via a console implemented as a website or application, etc. An API is an interface and / or communication protocol between a client and a server, such that if a client issues a request in a predefined format, the client should receive a response in a specific format or initiate a defined action. In a cloud provider network scenario, an API provides a gateway for customers to access cloud infrastructure by allowing customers to obtain data from or initiate actions within the cloud provider network, thereby enabling the development of applications that interact with resources and services hosted within the cloud provider network. APIs can also enable different services within the cloud provider network to exchange data with each other. An interface may be part of the control plane of cloud provider network 100 or act as a front end of the control plane, which includes "back-end" services that support and implement services that can be provided more directly to customers.

[0014] Therefore, a cloud provider network (or simply "the cloud") generally refers to a large pool of accessible virtualized computing resources, such as compute, storage, and networking resources, applications, and services. The cloud provides convenient, on-demand network access to a shared pool of configurable computing resources that can be programmatically provisioned and deployed in response to client commands. These resources can be dynamically provisioned and reconfigured to adapt to variable loads. Thus, cloud computing can be viewed as both applications delivered as a service over a publicly accessible network (e.g., the internet, cellular networks) and the hardware and software in the cloud provider's data centers that provide those services.

[0015] To provide these and other computing resource services, cloud provider network 100 typically relies on virtualization technology. For example, virtualization technology can provide users with the ability to control or use computing resources (e.g., “computing instances,” such as VMs using guest operating systems (O / S) that may or may not operate on top of the underlying host O / S; containers that may or may not operate within VMs; computing instances that can execute on “bare metal” hardware without an underlying hypervisor), where one or more computing resources can be implemented using a single electronic device. Thus, users can directly use computing resources hosted by the provider network (e.g., provided by a hardware virtualization service) to perform various computing tasks. Alternatively, users can indirectly use computing resources by submitting code to be executed by the provider network (e.g., via an on-demand code execution service), which in turn uses one or more computing resources to execute the code, typically without the user having any control or knowledge of the underlying computing instances involved.

[0016] As described herein, one type of service that a provider network can offer may be referred to as a "managed computing service," which executes code or provides computing resources to its users within a managed configuration. Examples of managed computing services include, for example, hardware virtualization services, container services, on-demand code execution services, and so on. These services are collectively referred to as computing services 108.

[0017] Hardware virtualization services (known in various implementations as elastic computing services, virtual machine services, compute cloud services, compute engines, or cloud computing services) enable users of a cloud provider's network 100 to provision and manage computing resources, such as virtual machine instances. Virtual machine technology can use a physical server, for example, a hypervisor to run an equivalent form of many servers (each of which is called a virtual machine), which may at least partially run on an off-grid card of the server (e.g., a card connected to the physical CPU via PCI or PCIe), and other components of the virtualization host may be used for some virtualization management components. This off-grid card of the host may contain one or more CPUs that are not available for user instances but are dedicated to instance management tasks, such as virtual machine management (e.g., hypervisor), I / O virtualization of network-attached storage volumes, local migration management tasks, instance health monitoring, etc.). Virtual machines are often referred to as compute instances or simply "instances." As used herein, provisioning a virtual computing instance typically involves: reserving resources (e.g., compute and storage resources) of the underlying physical computing instance for the client (e.g., from a pool of available physical computing instances and other resources); installing or launching the required software (e.g., an operating system); and making the virtual computing instance available to the client to perform tasks specified by the client.

[0018] Container services, such as container orchestration and management services (referred to as container services, cloud container services, container engines, or container cloud services in various implementations), allow users within a cloud provider's network to instantiate and manage containers. In some examples, a container service can be a Kubernetes-based container orchestration and management service (referred to as Kubernetes container services, Azure Kubernetes service, IBM Cloud Kubernetes service, Kubernetes engine, or Kubernetes container engine in various implementations). As discussed in this article, containers package code and all its dependencies, allowing applications (also called tasks, swarms, or clusters in various container services) to run quickly and reliably across various computing environments. A container image is a standalone executable package containing everything needed to run an application process: code, runtime, system tools, system libraries, and settings. A container image becomes a container at runtime. Therefore, a container is an abstraction of the application layer (meaning each container emulates a different software application process). Although each container runs an isolated process, multiple containers can share a common operating system, for example, by starting within the same virtual machine. In contrast, a virtual machine is an abstraction of the hardware layer (meaning each virtual machine emulates a physical machine capable of running software). While multiple virtual machines can run on a single physical machine, each virtual machine typically has its own copy of the operating system, along with the application and its associated files, libraries, and dependencies. Some containers can run on instances running container agents, and some can run on bare metal servers or on off-board cards of the server.

[0019] Another type of managed computing service can be an on-demand code execution service (referred to in various examples as Function Compute Service, Function Service, Cloud Function Service, Function as a Service, or Serverless Computing Service), which enables users of the cloud provider network 100 to execute their code on cloud resources without having to select or manage the underlying hardware resources used to execute the code. For example, a user can use the on-demand code execution service by uploading their code and using one or more APIs to request service identification, provisioning, and management of any resources required to run the code. Therefore, in various examples, a “serverless” function may comprise on-demand executable code provided by a user or other entity—such as the provider network itself. Serverless functions can be maintained within the provider network through the on-demand code execution service and may be associated with a specific user or account or typically accessible by multiple users / accounts. Serverless functions may be associated with a Uniform Resource Locator (URL), Uniform Resource Identifier (URI), or other reference that can be used to invoke the serverless function. Serverless functions can be executed via computing resources such as virtual machines or containers when triggered or invoked. In some examples, serverless functions can be invoked via application programming interface (API) calls or specially formatted Hypertext Transfer Protocol (HTTP) request messages. Therefore, users can define serverless functions that can be executed on demand without requiring them to maintain dedicated infrastructure for executing these functions. Alternatively, resources maintained by the cloud provider's network 100 can be used to execute serverless functions on demand. In some examples, these resources can be maintained in a "ready" state (e.g., with a pre-initialized runtime environment configured to execute serverless functions), allowing for near real-time execution of serverless functions.

[0020] The cloud provider network 100 may also provide data storage through storage services 109 (such as block storage services and object storage services). Block storage services can provide virtual volumes that can be attached to or otherwise accessed by computing resources provided by computing service 108. For example, an instance may have an attached volume with a storage allocation of tens or hundreds of gigabytes, in which applications running on the instance can read and write files in a directory structure. Object storage services typically store objects (including files) in data structures such as key-value stores and generally do not have a fixed storage allocation. Other storage services 109 may include databases, etc.

[0021] One type of service that can provide model hosting service 110 is a multi-model endpoint (MME). An MME provides end users with access to multiple ML models. MMEs are typically constructed in a distributed manner, allowing them to serve multiple concurrent ML model inference requests (including requests for the same ML model). An exemplary MME 130, identified as “ABC,” is shown. An MME can be considered part of model hosting service 110. MME 130 contains a number of hosts 132A to 132N, which may be referred to differently as groups, clusters, or generally as sets. Hosts 132 are typically virtualized as virtual machines, containers, etc., located on top of a cloud provider's network infrastructure. Exemplary host 132A contains a cache manager 134 application and host storage devices 136 (e.g., disks / flash drives). In this example, the model is executed within a virtualized environment of a host (e.g., a container) with storage 138. (Note that a host may have multiple model hosting environments. For example, host 132A could be another environment with different model storage to execute models for different clients' MMEs.)

[0022] Storage service 109 may provide a network repository (or simply a "network storage device") 190 for model 198. Network storage device 190 may be considered part of model hosting service 110. Network storage device 190, host storage device 136, and memory 138 form an exemplary memory hierarchy 199. Given a sufficiently large number of models, each host can load only a smaller subset of the total number of models into host memory 136 and an even smaller subset of models into memory 138 to perform inference tasks. If host 132 receives a request to perform an inference operation on a model that is not loaded into memory 138, cache manager 134 may load the model from host storage device 136 into memory 138 (and evict another model to free up space if necessary). If a model does not exist in host storage device 136, cache manager 134 may retrieve the model from network storage device 190. Loading models from network storage device to host storage device and / or from local storage device to memory incurs latency penalties. Serving requests for models that are loaded into memory is faster than serving requests for models that the host needs to load from local storage, which in turn is faster than serving requests for models that the host needs to load from network storage.

[0023] Model hosting service 110 includes a data store 120 that can be supported by storage service 109. Model hosting service 110 can maintain a registry 124 of MMEs and associated hosts. For example, an entry for MME “ABC” may have an associated set of host identifiers that identify each of its associated hosts. Host identifiers may be resource identifiers, network addresses, etc. Each host may have an associated status indicator. An exemplary set of status indicators includes “active” (indicating that the host is serving an inference request), “new” (indicating that a host is to be added to the set of hosts serving the inference request), and “terminated” (indicating that a host is to be removed from the set of hosts serving the inference request).

[0024] exist Figure 1 In the diagram, the circles marked (1) to (8) illustrate an exemplary process of introducing a new host with a preloaded model into the MME. Note that during this process, from the client's perspective, the identity of MME 130 (e.g., MME "ABC") remains unchanged after the introduction of host 142. Initially, as shown in circle (1), endpoint router 180 routes inbound model inference requests to a set of hosts forming MME 130. Examples of endpoint router 180 include network devices, network appliances, or software-based network applications performed by computing service 108. In some examples, endpoint router 180 is formed by multiple independent nodes. Endpoint router 180 may be considered as part of model hosting service 110.

[0025] Endpoint router 180 (or its nodes) uses routing algorithm 116 to calculate which host to direct the inbound model inference request to. Below is an exemplary routing algorithm. Routing algorithm 116 can use a hash function to calculate the priority order of host service requests. The hash function can take as input a combination of each of the model identifiers and "active" host identifiers (such as their network addresses) contained in the request. Algorithm 116 can use the numerical ordering of those hash function outputs (e.g., from lowest to highest) as a priority order to determine where to route the request. An example of this routing strategy is as follows: Figure 3 As shown. Endpoint router 180 may continue to route requests to hosts in the order indicated, based on the inbound request volume for the model.

[0026] For example, when a request for specific content arrives, the endpoint router 180 receiving the request may start a timer (e.g., a two-minute timer). If the router finds more than X requests for that content within the two-minute timer, it will begin routing them sequentially to a second host (e.g., X = 5). If the router sees another X requests, it will begin routing them sequentially to a third host, and so on. When the timer expires, the router will continue routing the requests sequentially to the first host (and start a new timer).

[0027] In this way, endpoint router 180 routes requests for a specific model to the same host in the same order. As a byproduct, hosts such as host 132A are more likely to persist the model prioritized by endpoint router 180 to memory 138 (or, if not, host storage 136). In this way, certain models are "stickier" (or have affinity) to certain hosts. The per-model host order is referred to herein as a "caching chain".

[0028] As shown in circle (2), Model Hosting Service 110 receives an MME update request, which, as described above, can be used to update the MME software or to change the configuration of endpoint hosts (e.g., add or remove hosts). The request can be initiated by an administrator of the cloud provider network 100 as part of automated maintenance operations, customer guidance, etc. In this example, the request updates MME ABC, and the update includes adding hosts that Model Hosting Service 110 has started as a group of hosts 142A to 142M to be added to MME 130. Hosts 142A to 142M may have an associated “new” status identifier in registry 124. The endpoint updater 112 application of Model Hosting Service 110 manages the process of updating the MME.

[0029] At circle (3), endpoint updater 112 receives per-host model usage data. In some examples, host 132's cache manager sends per-host model usage data to model hosting service 110 in response to a request or as part of a periodic usage report. The usage data may be stored in model usage metric 122 of data storage 120 along with other calculated usage data.

[0030] In some examples, the cache manager may sell an API that returns usage data. An exemplary API includes a `listAllModels()` call that exposes which models are loaded into the model execution environment memory (e.g., memory 138 of host 132A) and optionally host storage (e.g., host storage 136 of host 132A). Usage data may further include usage information such as a "call count" or the number of times a model was called within a last time period (e.g., 60 minutes). Usage data may further include a "last access" time indicator, such as a timestamp indicating when a model was last accessed. The last access time may be appended to the call count data to account for periodic but peak traffic (e.g., models used more frequently than some models but outside the call count time window).

[0031] At circle (4), endpoint updater 112 calculates a new set of cache chains for hosts that will serve model inference requests in the updated MME ABC. For example, endpoint updater 112 may use the hash function described above to calculate a cache chain for each combination of models in a set of models identified in the usage data and each host associated with an MME ABC having an “active” or “new” status identifier (omitting any identifiers including a “termination” identifier that will be phased out).

[0032] Endpoint updater 112 can aggregate model usage data across all hosts 132 to determine the individual popularity of each model. For example, endpoint updater 112 can sum the call counts for each model to determine the total number of times the model has been accessed across all hosts 132.

[0033] Using the cache chain and aggregated usage data, endpoint updater 112 can identify models to be preloaded onto host 142. In some examples, the models to be preloaded may include identifiers of a first set of models to be loaded into host storage and execution environment memory (e.g., memory 148 of host 142A) and a second set of lower-priority models to be loaded into host storage (e.g., host storage 146 of host 142A). Reference Figure 3 and 4 Provides additional details on hash chain computation and model preloading options.

[0034] At circle (5), endpoint updater 112 instructs each cache manager of host 142 (e.g., cache manager 144 of host 142A) to load one or more models. For example, a cache manager may issue a loadModelAtContainer() call with parameters identifying one or more models, which, when called, causes the cache manager to load the identified models in the group from network storage device 190 to host storage and execution environment storage. In some examples, a cache manager may issue a loadModelInStorage() call with parameters identifying one or more models, which, when called, causes the cache manager to load the identified models in the group from network storage device 190 to host storage. Given that both the amount of memory in the execution environment and the model size can vary, pre-caching operations may result in the host having a different number of loaded models.

[0035] At circle (6), the cache manager of host 142 performs a load operation, thereby loading the identified model into the host storage device and / or execution environment memory. In some examples, once the load operation is complete, the cache manager of host 142 may return such an instruction to endpoint updater 112. Once endpoint updater 112 receives the instruction from each of the cache managers, the process may continue to circle (7). In other examples, endpoint updater 112 may occasionally query the cache manager to check the status of the loaded model. Once the status of the loaded model matches the status of the requested model across host 142, the process may continue to circle (7).

[0036] At circle (7), endpoint updater 112 can update the “new” status of host 142 to “active” in registry 124 and send the host identifier of each of the active hosts to endpoint router 180.

[0037] At circle (8), endpoint router 180 continues to route model inference requests among the updated set of hosts using the aforementioned routing algorithm 116. Since the cache chain calculation performed by endpoint updater 112 matches the host priority order of endpoint router 180, endpoint router 180 directs requests for a given model to those hosts that are more likely to have preloaded the model.

[0038] Please note that in some examples, one or more of hosts 132A to 132N may still be included as part of the MME130U. In this case, the cache chain calculation may include identifiers for hosts 132 and 142 that will continue to serve inference requests. In other examples, the switch from host 132 to host 142 may be part of a blue-green deployment model, where when host 142 is introduced, that host stops serving inference requests (e.g., the router no longer routes requests that will be served by host 132).

[0039] Computing service 108 can provide computing support for the components shown, including model hosting service 110, endpoint updater 112, MME 130 / 130U, host 132, 142 and endpoint router 180 (e.g., such components can be provisioned using resources provided by hardware virtualization service, on-demand code execution service, container service, etc.).

[0040] Figure 2A and 2B This is a graph illustrating the relationship between load and wait times for models with and without cache preheating, based on some examples. As shown in the figure, Figure 2A The diagram illustrates the performance loss associated with introducing a host without pre-warming the cache, based on some examples. Specifically, it plots the number of models loaded into the MME (solid line, left vertical axis) and the MME latency (dashed line, right vertical axis) relative to time. As shown in cache miss event 202, when the underlying host of the MME is swapped out without pre-warming at time 201, the number of loaded models drops to zero, and each model load results in a performance penalty. Figure 2B The graph shows the same curve, but the data is taken from a scenario where the host is introduced into the cache for preheating. As shown, the performance loss associated with host switching at time 203 is significantly reduced (if not completely eliminated).

[0041] Figure 3This diagram illustrates a hashing technique used to determine the priority of host models, based on some examples. In this example, hash function 304 is used to compute the priority order among five hosts (HOST_1...HOST_5) for a given model (MODEL_A). The hash function takes a host identifier and a model identifier as input. The host identifier can be a network address or other identifier that identifies the host within the MME. The model identifier can be a filename (e.g., "model_a.tar.gz") or other identifier that identifies the model loaded into the MME. The host identifier and model identifier can be concatenated to form the hash function input. For a model, exemplary hash function input 302 contains the model and each host. Hash function output 306 is indicated. Within the hash space, the ordered output of the hosts results in a cache chain 308 (the hash output of HOST_2 starts, followed by HOST_5, HOST_4, HOST_1, and HOST_3 in sequence). Using hash functions in this way allows individual routers to direct inference requests to those hosts that are more likely to load the requested model.

[0042] Figure 4 This is a diagram illustrating cache warm-up based on some examples. For a more specific description, please refer to a more detailed example. Figure 1 The operations at circles (3) to (6). Here, the MME 430, which initially contained four hosts 432A to 432D with the identifier "DEF", is updated to hosts 442A to 442D (with identifiers such as network addresses IP1', IP2', IP3', and IP4', respectively). As mentioned above, the number of models that can be loaded onto a particular host will vary because the model execution environment may have varying amounts of memory and the size of the models may vary. To keep the example simple, each host has memory that can store three models, and the currently loaded models are illustrated (e.g., host 432A loads models 4, 2, and 3).

[0043] At circle (3), endpoint updater 112 obtains usage data from each of hosts 432. Here, host 432A sends usage data identifying models 2, 3, and 4; host 432B sends usage data identifying models 2, 3, and 5; and so on. Each host may also send the time when a given model was last accessed.

[0044] At circle (4A), the endpoint updater aggregates the received usage data to form per-model usage data for the MME “DEF”. Here, model 4 was invoked 21 times during the time period, reported to be present on three hosts, and most recently accessed at a time “XY”; while model 1 was invoked 11 times during the time period, reported to be loaded on one host, and most recently accessed at a time “XX”.

[0045] At circle (4B), endpoint updater 112 calculates the new host priority order or cache chain for the model. For example, using Figure 3 The hashing strategy shown allows endpoint updater 112 to determine the hash function output order of model 1 and the identifier of host 442 (e.g., IP1', IP2', IP3', and IP4'), thereby performing repeated calculations on models 2 through 5. An exemplary cache chain is shown in the figure (e.g., the cache chain for model 5 is IP1'→IP4'→IP2'→IP3').

[0046] At circle (4C), endpoint updater 112 determines one or more models to be loaded into each of hosts 442. Various strategies can be used to place models based on per-model usage data and cache chains. In some examples, models can be placed such that the number of replicas of the highest priority model (here, model 4 with 21 calls and 3 replicas) is assigned before specifying the number of replicas of the next priority model (here, model 2 with 17 calls and 3 replicas). The result of this greedy approach is illustrated (model 4 is assigned to the first three hosts IP1', IP2', and IP4' in its cache chain; model 2 is assigned to the first three hosts IP1', IP3', and IP4' in its cache chain, and so on).

[0047] In other examples, endpoint updater 112 may use a round-robin approach, where, starting with the highest priority model, one copy of each model is placed in the next available host slot in priority order (up to the number of copies of the model) before re-starting dispatch. In this approach, model 4 would be dispatched to IP2' as the first host in its cache chain, models 2, 3, and 5 would be dispatched sequentially to IP1' (where IP1' is the first host in its respective cache chain), and model 1 would be dispatched to IP2'. Returning to model 4, it would be dispatched to IP4' as the next host in its cache chain, and so on.

[0048] In a further example, endpoint updater 112 may distribute an equal number of model copies across hosts, the number being independent of the number of copies existing across the original MME, thereby filling any remaining space with higher-priority models. Those skilled in the art will appreciate that other methods are also possible.

[0049] At circle 5, endpoint updater 112 instructs each of the hosts 442 (e.g., a cache manager) to load the determined model (e.g., models 4, 2, and 3 for IP1'). At circle 6, the hosts load their respective models from network storage device 190. Using the pre-loaded models, the router can be updated to change the hosts associated with MME “DEF” from IP1, IP2, IP3, and IP4 to IP1', IP2', IP3', and IP4'.

[0050] Figure 5 This is a flowchart illustrating operation 500 of a method for caching in a machine learning model hosting service, according to some examples. Some or all of operation 500 (or other processes, variations, and / or combinations thereof described herein) are executed under the control of one or more computing devices configured with executable instructions and are implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that commonly execute on one or more processors. The code is stored, for example, in the form of a computer program including instructions executable by one or more processors on a computer-readable storage medium. The computer-readable storage medium is non-transitory. In some examples, one or more (or all) of operation 500 are executed by endpoint updater 112 of model hosting service 110 in other figures.

[0051] Operation 500 includes, at box 502, summarizing machine learning (ML) model usage data based on host usage data provided by each host in the first group of hosts, wherein for a specific ML model, the ML model usage data includes the number of inference requests for that specific ML model. Operation 500 further includes, at box 504, calculating the priority order of the hosts in the second group of hosts serving the inference requests for the specific ML model. Operation 500 further includes, at box 506, determining a set of ML models to be loaded onto a specific host in the second group of hosts based on the ML model usage data and the priority order. Operation 500 further includes, at box 508, causing the specific host to load a set of ML models. Operation 500 further includes, at box 510, updating the router to route ML model inference requests among the second group of hosts.

[0052] Figure 6The illustration illustrates an exemplary provider network (or "service provider system") environment based on some examples. Provider network 600 may provide resource virtualization to customers via one or more virtualization services 610, which allow customers to purchase, lease, or otherwise obtain instances 612 of virtualized resources (including, but not limited to, computing and storage resources) implemented on devices within one or more provider networks in one or more data centers. A local Internet Protocol (IP) address 616 may be associated with resource instance 612; the local IP address is the internal network address of resource instance 612 on provider network 600. In some examples, provider network 600 may also provide public IP addresses 614 and / or ranges of public IP addresses (e.g., Internet Protocol version 4 (IPv4) or Internet Protocol version 6 (IPv6) addresses) available to customers from provider 600.

[0053] Typically, provider network 600 may, via virtualization service 610, allow service provider customers (e.g., customers operating one or more customer networks 650A to 650C (or “client networks”) containing one or more client devices 652) to dynamically associate at least some public IP addresses 614 assigned to them with specific resource instances 612 assigned to them. Provider network 600 may also allow customers to remap public IP addresses 614 previously mapped to one virtualized computing resource instance 612 assigned to them to another virtualized computing resource instance 612 also assigned to them. Using the virtualized computing resource instances 612 and public IP addresses 614 provided by the service provider, service provider customers, such as operators of customer networks 650A to 650C, may, for example, implement customer-specific applications and present these applications on an intermediate network 640 (such as the Internet). Then, other network entities 620 on the intermediate network 640 can generate traffic destined for a public IP address 614 published by customer networks 650A to 650C; the traffic is routed to the service provider's data center, and at the data center, it is routed via the network substrate to the local IP address 616 of the virtualized computing resource instance 612 currently mapped to the destination public IP address 614. Similarly, response traffic from the virtualized computing resource instance 612 can be routed back to the intermediate network 640 via the network substrate to reach the source entity 620.

[0054] As used herein, a local IP address refers to an internal or “private” network address of a resource instance within, for example, a provider network. Local IP addresses may be within an address block reserved by the Internet Engineering Task Force (IETF) Request for Comments (RFC) 1918 and / or have an address format specified by IETF RFC 4193, and may vary within the provider network. Network traffic originating outside the provider network is not directly routed to a local IP address; instead, traffic uses a public IP address mapped to the local IP address of the resource instance. A provider network may include networking devices or appliances that provide Network Address Translation (NAT) or similar functionality to perform mappings from public IP addresses to local IP addresses and from local IP addresses to public IP addresses.

[0055] A public IP address is a variable network address on the Internet assigned to a resource instance by a service provider or customer. For example, traffic routed to a public IP address via 1:1 NAT translation is used to forward the traffic to the appropriate local IP address of the resource instance.

[0056] Some public IP addresses may be assigned to specific resource instances by the provider's network infrastructure; these public IP addresses may be referred to as standard public IP addresses, or simply standard IP addresses. In some examples, the mapping of standard IP addresses to the local IP addresses of resource instances is the default startup configuration for all resource instance types.

[0057] At least some public IP addresses may be assigned to or obtained by customers of Provider Network 600; customers may then assign their assigned public IP addresses to specific resource instances assigned to them. These public IP addresses may be referred to as customer public IP addresses, or simply customer IP addresses. Instead of being assigned to resource instances by Provider Network 600 as in the case of standard IP addresses, customer IP addresses may be assigned to resource instances by the customer, for example, via an API provided by the service provider. Unlike standard IP addresses, customer IP addresses are assigned to customer accounts and may be remapped to other resource instances by the respective customer as needed. Customer IP addresses are associated with customer accounts, not specific resource instances, and the customer controls the IP address until the customer chooses to release it. Unlike regular static IP addresses, customer IP addresses allow customers to mask resource instance or availability zone failures by remapping their public IP addresses to any resource instance associated with their account. For example, customer IP addresses enable customers to resolve resource instance or software issues by remapping their public IP addresses to alternative resource instances.

[0058] Figure 7This is a block diagram of an exemplary provider network environment that provides storage services and hardware virtualization services to customers, based on some examples. Hardware virtualization service 720 provides customers with multiple computing resources 724 (e.g., computing instances 725 such as VMs). Computing resources 724 may be provided as a service to customers of provider network 700 (e.g., customers implementing customer network 750). Each computing resource 724 may have one or more local IP addresses. Provider network 700 may be configured to route packets from the local IP addresses of computing resources 724 to public internet destinations, and to route packets from public internet sources to the local IP addresses of computing resources 724.

[0059] Provider network 700 may provide a client network 750, for example, coupled to intermediate network 740 via local network 756, with the ability to implement virtual computing systems 792 via hardware virtualization service 720 coupled to intermediate network 740 and provider network 700. In some examples, hardware virtualization service 720 may provide one or more APIs 702, such as web service interfaces, through which client network 750 may access the functionality provided by hardware virtualization service 720, for example, via console 794 of client device 790 (e.g., web-based application, standalone application, mobile application, etc.). In some examples, at provider network 700, each virtual computing system 792 at client network 750 may correspond to computing resources 724 that are leased, rented, or otherwise provided to client network 750.

[0060] From instances of virtual computing system 792 and / or another client device 790 (e.g., via console 794), clients can access the functionality of storage service 710, for example, via one or more APIs 702, to access data from and store data in storage resources 718A to 718N of virtual data repositories 716 (e.g., folders or "buckets," virtualized volumes, databases, etc.) provided by provider network 700. In some examples, a virtualized data storage gateway (not shown) may be located at client network 750, which may locally cache at least some data (e.g., frequently accessed data or critical data) and communicate with storage service 710 via one or more communication channels to upload new or modified data from the local cache, thereby maintaining the primary data storage (virtualized data storage 716). In some examples, users can mount and access volumes of virtualized data storage 716 via storage service 710, which acts as a storage virtualization service, and these volumes may appear to the user as local (virtualized) storage devices 798.

[0061] Although Figure 7 Not shown, but also accessible via API 702 from resource instances within provider network 700. For example, a customer, appliance service provider, or other entity may access the virtualization service via API 702 from a corresponding virtual network on provider network 700 to request allocation of one or more resource instances within said virtual network or another virtual network.

[0062] Explanatory System

[0063] In some examples, a system implementing some or all of the techniques described herein may comprise a general-purpose computer system, such as Figure 8 The illustrated computing device 800 (also referred to as a computing system or electronic device) includes or is configured to access one or more computer-accessible media. In the illustrated example, the computing device 800 includes one or more processors 810 coupled to system memory 820 via an input / output (I / O) interface 830. The computing device 800 further includes a network interface 840 coupled to the I / O interface 830. Although Figure 8 The computing device 800 is shown as a single computing device, but in various examples, the computing device 800 may contain a single computing device or any number of computing devices configured to work together as a single computing device 800.

[0064] In various examples, computing device 800 may be a single-processor system containing one processor 810 or a multiprocessor system containing several processors 810 (e.g., two, four, eight, or another suitable number). Processor 810 may be any suitable processor capable of executing instructions. For example, in various examples, processor 810 may be a general-purpose or embedded processor implementing any of a variety of instruction set architectures (ISAs), such as x86, ARM, PowerPC, SPARC, or the MIPS ISA or any other suitable ISA. In a multiprocessor system, each of the processors 810 may, but does not necessarily, implement the same ISA.

[0065] System memory 820 may store instructions and data accessible to processor 810. In various examples, system memory 820 may be implemented using any suitable memory technology, such as random access memory (RAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory. In the illustrated example, program instructions and data that implement one or more desired functions (such as the methods, techniques, and data described above) are shown stored in system memory 820 as model hosting service code 825 (e.g., executable to fully or partially implement model hosting service 110) and data 826.

[0066] In some examples, I / O interface 830 may be configured to coordinate I / O traffic between processor 810, system memory 820, and any peripheral devices within the device (including network interface 840 and / or other peripheral interfaces (not shown)). In some examples, I / O interface 830 may perform any necessary protocol, timing, or other data transformations to convert data signals from one component (e.g., system memory 820) into a format suitable for use by another component (e.g., processor 810). In some examples, for example, I / O interface 830 may include support for devices attached via various types of peripheral buses (such as variants of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard). In some examples, for example, the functionality of I / O interface 830 may be split into two or more separate components, such as a northbridge and a southbridge. Additionally, in some examples, some or all of the functionality of I / O interface 830 (such as an interface to system memory 820) may be directly incorporated into processor 810.

[0067] For example, network interface 840 may be configured to allow computing device 800 to communicate with other computing devices 860 attached to one or more networks 850 (such as, e.g., Figure 1 The network interface 840 can exchange data with other computer systems or devices shown. In various examples, for instance, the network interface 840 may support communication via any suitable wired or wireless general-purpose data network (such as various types of Ethernet networks). Additionally, the network interface 840 may support communication via telecommunications / telephone networks (such as analog voice networks or digital fiber optic communication networks), via storage area networks (SANs) (such as Fibre Channel SANs), and / or via any other suitable type of network and / or protocol.

[0068] In some examples, computing device 800 includes one or more offload cards 870A or 870B (containing one or more processors 875 and possibly one or more network interfaces 840), which are connected using I / O interfaces 830 (e.g., a version of the Peripheral Component Interconnect Fast (PCI-E) standard or another interconnect bus such as Fast Path Interconnect (QPI) or Hyper Path Interconnect (UPI). For example, in some examples, computing device 800 may act as a host electronic device hosting computing resources such as computing instances (e.g., operating as part of a hardware virtualization service), and one or more offload cards 870A or 870B act as a virtualization manager that manages the computing instances running on the host electronic device. As an example, in some examples, offload cards 870A or 870B may perform computing instance management operations such as pausing and / or unpausing computing instances, starting and / or terminating computing instances, performing memory transfer / copy operations, etc. In some examples, these management operations may be performed by the offload card 870A or 870B in cooperation with a hypervisor (e.g., upon request from the hypervisor) executed by other processors 810A to 810N of the computing device 800. However, in some examples, the virtualization manager implemented by the offload card 870A or 870B may adapt to requests from other entities (e.g., from the computing instance itself) and may not cooperate with (or serve) any individual hypervisor.

[0069] In some examples, system memory 820 may be an example of a computer-accessible medium configured to store program instructions and data as described above. However, in other examples, program instructions and / or data may be received, transmitted, or stored on different types of computer-accessible media. Generally, computer-accessible media may comprise any non-transitory storage medium or memory medium, such as magnetic or optical media, for example, a disk or DVD / CD coupled to computing device 800 via I / O interface 830. Non-transitory computer-accessible storage media may also comprise any volatile or non-volatile medium that may be included in some examples of computing device 800 as system memory 820 or another type of memory, such as RAM (e.g., SDRAM, Double Data Rate (DDR) SDRAM, SRAM, etc.), read-only memory (ROM), etc. Furthermore, computer-accessible media may comprise transmission media or signals transmitted via communication media (such as networks and / or wireless links), such as electrical signals, electromagnetic signals, or digital signals, such as those implemented via network interface 840.

[0070] The various examples discussed or presented herein can be implemented in a wide variety of operating environments. In some cases, such operating environments may include one or more user computers, computing devices, or processing devices that can be used to operate any of several applications. User devices or client devices may include any of several general-purpose personal computers, such as desktop or laptop computers running standard operating systems, and cellular, wireless, and handheld devices running mobile software and capable of supporting several networking and messaging protocols. Such systems may also include several workstations running a variety of commercially available operating systems and other known applications for purposes such as development and database management. These devices may also include other electronic devices, such as virtual terminals, thin clients, gaming systems, and / or other devices capable of communicating via a network.

[0071] Most examples use at least one network familiar to those skilled in the art to support communication using any of a wide range of widely available protocols such as Transmission Control Protocol / Internet Protocol (TCP / IP), File Transfer Protocol (FTP), Universal Plug and Play (UPnP), Network File System (NFS), Public Internet File System (CIFS), Extensible Messaging and Field Protocol (XMPP), AppleTalk, etc. The network may include, for example, a Local Area Network (LAN), a Wide Area Network (WAN), a Virtual Private Network (VPN), the Internet, an intranet, an extranet, the Public Switched Telephone Network (PSTN), an infrared network, a wireless network, and any combination thereof.

[0072] In examples using a web server, the web server can run any of a variety of server or middleware applications, including HTTP servers, File Transfer Protocol (FTP) servers, Common Gateway Interface (CGI) servers, data servers, Java servers, business application servers, etc. The server may also be able to respond to requests from user devices, such as by executing one or more web applications that can be implemented in any programming language (such as Java®, C, C#, or C++) or any scripting language (such as Perl, Python, PHP, or TCL) and combinations thereof, to execute programs or scripts. The server may also include a database server, including but not limited to commercially available database servers from Oracle®, Microsoft®, Sybase®, IBM®, etc. The database server can be relational or non-relational (e.g., "NoSQL"), distributed or non-distributed, etc.

[0073] The environment disclosed herein can include a variety of data storage devices and other memories and storage media as discussed above. These can reside in a variety of locations, such as on (and / or in) storage media local to one or more computers within a computer, or on storage media of any or all computers located remotely across a network. In a particular set of examples, information can reside in a storage area network (SAN) familiar to those skilled in the art. Similarly, any necessary files for performing functions belonging to a computer, server, or other network device can be stored locally and / or remotely, depending on the circumstances. In the case of a system containing computerized devices, each such device may include hardware elements electrically coupled via a bus, such as at least one central processing unit (CPU), at least one input device (e.g., mouse, keyboard, controller, touchscreen, or keypad), and / or at least one output device (e.g., display device, printer, or speaker). Such a system may also include one or more storage devices, such as hard disk drives, optical storage devices, and solid-state storage devices such as random access memory (RAM) or read-only memory (ROM), as well as removable media devices, memory cards, flash memory cards, etc.

[0074] Such devices may also include a computer-readable storage medium reader, a communication device (e.g., a modem, a (wireless or wired) network card, an infrared communication device, etc.), and working memory as described above. The computer-readable storage medium reader may be connected to or configured to receive a computer-readable storage medium, which represents a remote, local, fixed, and / or removable storage device and storage medium for temporarily and / or more permanently accommodating, storing, transmitting, and retrieving computer-readable information. Systems and various devices will also typically include several software applications, modules, services, or other elements residing within at least one working memory device, including operating systems and applications such as client applications or web browsers. It should be understood that alternative examples may have numerous variations different from those described above. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software (including portable software such as applets), or both. Furthermore, connectivity to other computing devices, such as network input / output devices, may be employed.

[0075] Storage media and computer-readable media used to contain code or code portions may include any suitable media known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile media, removable and non-removable media implemented in any way or technology to store and / or transmit information (such as computer-readable instructions, data structures, program modules or other data), including RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc-read-only memory (CD-ROM), digital universal disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and is accessible by system devices. Based on this disclosure and the teachings provided herein, those skilled in the art will appreciate other ways and / or methods for implementing the various examples.

[0076] In the foregoing description, various examples have been described. Specific configurations and details have been elaborated for illustrative purposes to provide a thorough understanding of the examples. However, it will also be apparent to those skilled in the art that the examples can be practiced without these specific details. Furthermore, well-known features may be omitted or simplified to avoid obscuring the described examples.

[0077] In this document, parenthesized text and boxes with dashed borders (e.g., large dashes, small dashes, dot dashes, and dots) are used to illustrate optional aspects for adding additional features to some examples. However, this notation should not be interpreted as implying that these are only options or optional operations, and / or that in some examples, boxes with solid borders are not optional.

[0078] Reference numerals with suffix letters (e.g., 718A to 718N) can be used to indicate that one or more instances of the mentioned entity may exist in various examples, and when multiple instances exist, each instance need not be identical, but may share some general characteristics or function in a common form. Furthermore, unless explicitly indicated to the contrary, the use of a particular suffix does not imply the existence of a specific number of entities. Thus, in various examples, two entities using the same or different suffix letters may have or not have the same number of instances.

[0079] References to "an example," "example," etc., indicate that the described example may contain a particular feature, structure, or characteristic, but each example may not necessarily contain the stated particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same example. Additionally, when a particular feature, structure, or characteristic is described in conjunction with an example, it should be assumed that, whether explicitly described or not, implementing such a feature, structure, or characteristic in conjunction with other examples is within the knowledge of those skilled in the art.

[0080] Furthermore, in the various examples described above, unless otherwise specifically indicated, the disjunctive linguistic intent of phrases such as “at least one of A, B, or C” is understood to mean A, B, or C, or any combination thereof (e.g., A, B, and / or C). Similarly, the linguistic intent of phrases such as “at least one or more of A, B, and C” (or “one or more of A, B, and C”) is understood to mean A, B, or C, or any combination thereof (e.g., A, B, and / or C). Therefore, disjunctive language is neither intended nor should be understood to imply that a given example requires the existence of at least one of A, at least one of B, and at least one of C.

[0081] As used herein, the term "based on" (or similar) is an open-ended term used to describe one or more factors that influence a determination or other action. It should be understood that the term does not exclude additional factors that may influence a determination or action. For example, a determination may be based solely on the listed factors or on said factors and one or more additional factors. Therefore, if action A is "based on" B, it should be understood that B is a factor influencing action A, but this does not preclude the action from also being based on one or more other factors, such as factor C. However, in some cases, action A may be entirely based on B.

[0082] Unless otherwise expressly stated, articles such as “a / an” should generally be interpreted as including one or more of the described items. Therefore, phrases such as “a device configured to…” or “computing device” are intended to include one or more of the described devices. Such one or more described devices may be collectively configured to perform the described operations. For example, “a processor configured to perform operations A, B, and C” may include a first processor configured to perform operation A working together with a second processor configured to perform operations B and C.

[0083] Furthermore, the words “may” or “can” are used in a permissive sense (i.e., implying a possibility) rather than a mandatory sense (i.e., implying a requirement). The words “include,” “including,” and “includes” are used to indicate an open relationship and therefore imply inclusion but not limitation. Similarly, the words “have,” “having,” and “has” also indicate an open relationship and therefore imply having but not limitation. Terms such as “first,” “second,” “third,” etc., as used herein, serve as markers for the nouns that follow them and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless otherwise explicitly indicated. Similarly, the values ​​of such numerical markers are generally not used to indicate the required quantity of a particular noun in a claim set forth herein, and therefore the element “fifth” generally does not imply the presence of four other elements unless those elements are explicitly included in the claim or their presence is otherwise sufficiently clear.

[0084] At least some embodiments of the disclosed technology may be described in light of the following terms: 1. A computer-implemented method, comprising: Machine learning (ML) model endpoints are hosted across the first group of hosts, where routers prioritize model ML inference requests among the first group of hosts based on hash functions. Start a second group of hosts, where the hosts in the second group have software updates that are not included in the hosts in the first group; From the host usage data provided by the cache manager application running on each of the first group of hosts, the ML model usage data is aggregated, and for a specific ML model, the ML model usage data includes the number of inference requests for the specific ML model and the time when the specific ML model was last requested; The hash function is used to calculate the priority order of each ML model for the hosts in the second group of hosts; Based on the ML model usage data and the priority order of each ML model of the hosts in the second group of hosts, a set of ML models to be loaded into the first host in the second group of hosts is determined. This causes the first host to load the set of ML models; and The router is updated to guide ML model inference requests within the second group of hosts.

[0085] 2. The computer-implemented method according to Clause 1, wherein the host in the first group of hosts and the second group of hosts is a virtual machine, and wherein the router is a routing application.

[0086] 3. A computer-implemented method according to any one of Clauses 1 to 2, wherein the hash function generates output in a hash space based on an ML model identifier and a host identifier, and wherein the priority order of the hosts of the first ML model is based on the order of the hash function outputs of the ML model identifier of the first ML model and each host identifier of the hosts in the second group of hosts in the hash space.

[0087] 4. A computer-implemented method, comprising: Based on host usage data provided by each host in the first group of hosts, aggregate machine learning (ML) model usage data, which for a particular ML model includes the number of inference requests for that particular ML model; Calculate the priority order of the hosts in the second group of hosts for inference requests of the specific ML model; Based on the ML model usage data and the priority order, a set of ML models to be loaded onto a specific host in the second group of hosts is determined; This causes the specific host to load the set of ML models; and Update the router to route ML model inference requests among the second group of hosts.

[0088] 5. The computer-implemented method according to Clause 4, wherein the host in the first group of hosts and the second group of hosts is a virtual machine, and wherein the router is a routing application.

[0089] 6. The computer-implemented method according to Clause 4, wherein a hash function is used to compute the priority order, the hash function generating an output in a hash space based on an ML model identifier and a host identifier, and wherein the priority order of the hosts of a particular ML model is based on the order of the hash function outputs of the ML model identifier of the particular ML model and each host identifier of the hosts in the second group of hosts in the hash space.

[0090] 7. The computer-implemented method according to Clause 6, wherein the router uses the hash function to identify the host in the second group of hosts to which the received ML model inference request is to be routed.

[0091] 8. A computer-implemented method according to any one of Clauses 4 to 7, wherein each of the first group of hosts executes a cache manager application to load an ML model (1) from a network storage location to a host storage location, and (2) from the host storage location to the host's memory associated with the environment in which the ML model inference is performed, and the computer-implemented method further includes requesting host use data from the cache manager application of the first group of hosts.

[0092] 9. The computer-implemented method according to Clause 8, wherein the environment in which the ML model inference is performed is a container.

[0093] 10. The computer-implemented method according to Clause 8, wherein the cache manager application of the particular host loads the set of ML models from the network storage location into the memory of the particular host associated with the environment in which the set of ML models is executed.

[0094] 11. A computer-implemented method according to any one of clauses 4 to 7, wherein prior to the router update, the first group of hosts is part of an ML model hosting endpoint, and after the router update, the second group of hosts forms the ML model hosting endpoint, and the computer-implemented method further includes receiving a request to apply a software update to at least one of the ML model hosting endpoint or hosts in the first group of hosts.

[0095] 12. A computer-implemented method according to any one of clauses 4 to 7, wherein after the router is updated, the router no longer directs ML model inference requests to the first group of hosts.

[0096] 13. A system comprising: First or more computing devices for implementing routers in a multi-tenant provider network; and A second or more computing devices are configured to implement a machine learning (ML) model hosting service in the multi-tenant provider network, the ML model hosting service comprising instructions that, when executed, cause the ML model hosting service to: Based on host usage data provided by each host in the first group of hosts, aggregate ML model usage data, which for a specific ML model includes the number of inference requests for that specific ML model; Calculate the priority order of the hosts in the second group of hosts for inference requests of the specific ML model; Based on the ML model usage data and the priority order, a set of ML models to be loaded onto a specific host in the second group of hosts is determined; This causes the specific host to load the set of ML models; and The router is updated to guide ML model inference requests within the second group of hosts.

[0097] 14. The system according to Clause 13, wherein the hosts in the first group of hosts and the second group of hosts are virtual machines, and wherein the router is a routing application.

[0098] 15. The system according to Clause 13, wherein a hash function is used to compute the priority order, the hash function generating an output in a hash space based on an ML model identifier and a host identifier, and wherein the priority order of the hosts of a particular ML model is based on the order of the hash function outputs of the ML model identifier of the particular ML model and each host identifier of the hosts in the second group of hosts in the hash space.

[0099] 16. The system according to Clause 15, wherein the router uses the hash function to identify the host in the second group of hosts to which the received ML model inference request is to be routed.

[0100] 17. The system according to any one of Clauses 13 to 16, wherein each of the first group of hosts executes a cache manager application to load an ML model (1) from a network storage location to a host storage location, and (2) from the host storage location to the host's memory associated with the environment in which the ML model inference is performed, and the system further includes requesting host use data from the cache manager application of the first group of hosts.

[0101] 18. The system according to Clause 17, wherein the cache manager application of the particular host loads the set of ML models from the network storage location into the memory of the particular host associated with the environment in which the set of ML models is executed.

[0102] 19. The system according to any one of Clauses 13 to 16, wherein prior to the router update, the first group of hosts was part of the ML model hosting endpoint, and after the router update, the second group of hosts forms the ML model hosting endpoint, and further includes receiving a request to apply a software update to at least one of the ML model hosting endpoint or hosts in the first group of hosts.

[0103] 20. The system according to any one of clauses 13 to 16, wherein after the router is updated, the router no longer directs ML model inference requests to the first group of hosts.

[0104] The specification and drawings should therefore be considered illustrative rather than restrictive. However, it will be apparent that various modifications and alterations may be made therein without departing from the broader scope of this disclosure as set forth in the claims.

Claims

1. A computer-implemented method, comprising: Based on host usage data provided by each host in the first group of hosts, aggregate machine learning (ML) model usage data, which for a particular ML model includes the number of inference requests for that particular ML model; Calculate the priority order of the hosts in the second group of hosts for inference requests of the specific ML model; Based on the ML model usage data and the priority order, a set of ML models to be loaded onto a specific host in the second group of hosts is determined; This causes the specific host to load the set of ML models; as well as Update the router to route ML model inference requests among the second group of hosts.

2. The computer-implemented method according to claim 1, wherein the host in the first group of hosts and the second group of hosts is a virtual machine, and wherein the router is a routing application.

3. The computer-implemented method of claim 1, wherein a hash function is used to compute the priority order, the hash function generating an output in a hash space based on an ML model identifier and a host identifier, and wherein the priority order of the hosts of a particular ML model is based on the order of the hash function outputs of the ML model identifier of the particular ML model and each host identifier of the hosts in the second group of hosts in the hash space.

4. The computer-implemented method of claim 3, wherein the router uses the hash function to identify the host in the second group of hosts to which the received ML model inference request is to be routed.

5. The computer-implemented method according to any one of claims 1 to 4, wherein each of the first group of hosts executes a cache manager application to load an ML model (1) from a network storage location to a host storage location, and (2) from the host storage location to the host's memory associated with the environment in which the ML model inference is performed, and the computer-implemented method further includes requesting host use data from the cache manager application of the first group of hosts.

6. The computer-implemented method of claim 5, wherein the cache manager application of the particular host loads the set of ML models from the network storage location into the memory of the particular host associated with the environment in which the set of ML models are executed.

7. The computer-implemented method according to any one of claims 1 to 4, wherein prior to the router update, the first group of hosts is part of the ML model hosting endpoint, and after the router update, the second group of hosts forms the ML model hosting endpoint, and the computer-implemented method further includes receiving a request to apply a software update to at least one of the ML model hosting endpoint or hosts in the first group of hosts.

8. The computer-implemented method according to any one of claims 1 to 4, wherein after updating the router, the router no longer directs ML model inference requests to the first group of hosts.

9. A system comprising: A first or more computing devices are used to implement a router in a multi-tenant provider network; and A second or more computing devices are configured to implement a machine learning (ML) model hosting service in the multi-tenant provider network, the ML model hosting service comprising instructions that, when executed, cause the ML model hosting service to: Based on host usage data provided by each host in the first group of hosts, aggregate ML model usage data, which for a specific ML model includes the number of inference requests for that specific ML model; Calculate the priority order of the hosts in the second group of hosts for inference requests of the specific ML model; Based on the ML model usage data and the priority order, a set of ML models to be loaded onto a specific host in the second group of hosts is determined; This causes the specific host to load the set of ML models; and The router is updated to guide ML model inference requests within the second group of hosts.

10. The system of claim 9, wherein the hosts in the first group of hosts and the second group of hosts are virtual machines, and wherein the router is a routing application.

11. The system of claim 9, wherein a hash function is used to compute the priority order, the hash function generating an output in a hash space based on an ML model identifier and a host identifier, and wherein the priority order of the hosts of a particular ML model is based on the order of the hash function outputs of the ML model identifier of the particular ML model and each host identifier of the hosts in the second group of hosts in the hash space.

12. The system of claim 11, wherein the router uses the hash function to identify the host in the second group of hosts to which the received ML model inference request is to be routed.

13. The system according to any one of claims 9 to 12, wherein each of the first group of hosts executes a cache manager application to load an ML model (1) from a network storage location to a host storage location, and (2) from the host storage location to the host's memory associated with the environment in which the ML model inference is performed, and the system further includes requesting host use data from the cache manager application of the first group of hosts.

14. The system of claim 17, wherein the cache manager application of the particular host loads the set of ML models from the network storage location into the memory of the particular host associated with the environment in which the set of ML models are executed.

15. The system of any one of claims 9 to 12, wherein prior to the router update, the first group of hosts is part of the ML model hosting endpoint, and after the router update, the second group of hosts forms the ML model hosting endpoint, and further includes receiving a request to apply a software update to at least one of the ML model hosting endpoint or hosts in the first group of hosts.

Images