Dynamic endpoint management for heterogeneous machine learning models

Abstract

Dynamic endpoint management is performed for heterogeneous machine learning models. Placement events for machine learning models associated with managed network endpoints are detected. These managed network endpoints can provide access to machine learning models via requests received from clients of the machine learning service to invoke different specified machine learning models. Computational resources are selected from associated computing resources based on a determination that the computing resources meet the resource requirements of the machine learning model, and the machine learning model is placed on the selected computing resource.

Description

Background Technology

[0001] Machine learning models and data-driven systems are increasingly being used to aid decision-making across a wide range of applications. These applications have provided benefits such as increased accuracy, improved productivity, and cost savings. This trend is the result of a combination of factors, including ubiquitous connectivity, the ability to collect, aggregate, and process massive amounts of fine-grained data using cloud computing, and improved access to increasingly sophisticated machine learning models that can analyze this data. Attached Figure Description

[0002] Figure 1 This is a logic block diagram illustrating dynamic endpoint management for heterogeneous machine learning models according to some embodiments.

[0003] Figure 2 This is a logical block diagram illustrating an example provider network that can implement machine learning services according to some embodiments, the machine learning services implementing dynamic endpoint management for heterogeneous machine learning models.

[0004] Figure 3 This is a logic block diagram illustrating an interaction for dynamic endpoint management of heterogeneous machine learning models according to some embodiments.

[0005] Figure 4 This is a logical block diagram illustrating the interactions for creating managed network endpoints according to some embodiments.

[0006] Figure 5 It is a logical block diagram according to some embodiments for monitoring managed network endpoints used for dynamic endpoint management of heterogeneous machine learning models.

[0007] Figure 6 This is a logic block diagram of a rebalancing event according to some embodiments.

[0008] Figure 7 This is a logic block diagram of a reduction event based on some embodiments.

[0009] Figure 8 This is a logical block diagram of extended events according to some embodiments.

[0010] Figure 9 It is a logic block diagram based on a finely tuned model placement according to some embodiments.

[0011] Figure 10 It is a logic block diagram, according to some embodiments, of loading an incremental model into memory to generate inferences on a fine-tuned machine learning model.

[0012] Figure 11 This is a logical block diagram of a load-aware routing technology for managed network endpoints, according to some embodiments.

[0013] Figure 12 This is a high-level flowchart illustrating various methods and techniques for dynamic endpoint management of heterogeneous machine learning models according to some embodiments.

[0014] Figure 13 This is a high-level flowchart illustrating various methods and techniques for detecting scaling and rebalancing events according to some embodiments.

[0015] Figure 14 This is a high-level flowchart illustrating various methods and techniques for placing fine-tuned machine learning models according to some embodiments.

[0016] Figure 15 This is a high-level flowchart illustrating various methods and techniques for generating inferences using fine-tuned machine learning models, according to some embodiments.

[0017] Figure 16 This is a high-level flowchart illustrating various methods and techniques for load-aware routing of managed network endpoints according to some embodiments.

[0018] Figure 17 This is a high-level flowchart illustrating various methods and techniques for selecting load-aware routing strategies according to some embodiments.

[0019] Figure 18 This is a high-level flowchart illustrating various methods and techniques for deploying new models to managed network endpoints with zero downtime, according to some embodiments.

[0020] Figure 19 Example systems for implementing the various methods, techniques and systems described herein are shown according to some embodiments.

[0021] Although embodiments have been described herein by way of example with reference to several examples and illustrative drawings, those skilled in the art will recognize that the embodiments are not limited to the described embodiments or drawings. It should be understood that the drawings and detailed description thereof are not intended to limit the embodiments to the specific forms disclosed, but rather are intended to cover all modifications, equivalents, and alternatives falling within the spirit and scope described in the appended claims. The headings used herein are for organizational purposes only and are not intended to limit the scope of this specification or claims. As used throughout this application, the word “may” is used in a permissive sense (e.g., meaning possible) rather than a mandatory sense (e.g., meaning must). Similarly, the words “include,” “including,” and “includes” mean including but not limited to.

[0022] It should also be understood that although the terms first, second, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the invention, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact. Detailed Implementation

[0023] Machine learning is the discipline of training computer systems to recognize patterns through repeated exposure to training data. In unsupervised learning, self-organizing algorithms learn previously unknown patterns in a dataset without any provided labels. In supervised learning, such training data includes inputs labeled with “ground truth” outputs corresponding to those inputs (automatically or by a human annotator). To evaluate / validate the performance of a trained model, a portion of the training dataset is typically excluded from the training process. Using a trained model in production is often referred to as “inference” or “prediction,” during which the model receives new data not in its training dataset and provides outputs based on its learned parameters. By using new training data to improve previously learned parameters of a production model and deploying new production models for inference, the training and validation process can be repeated periodically or intermittently to mitigate the decline in model accuracy over time. For example, a computer vision machine learning model can be trained using a training dataset of mirror data and can also perform inference to perform various computer vision tasks such as mirror classification, object detection, or mirror regression.

[0024] As more systems, services, and applications integrate various features and operations based on inferences made by machine learning models, the use of multiple machine learning models integrated for different tasks within a client, system, or service has increased. For example, generative machine learning models (sometimes referred to as generative artificial intelligence (AI)) are being integrated into machine learning (ML) applications to support the performance of various tasks, such as custom helpers, text summarization, and AI-driven graphic design. However, these models require powerful accelerators, such as GPUs or dedicated hardware, to function well. Because such models are deployed in production, managing the number and complexity of models integrated into a single application can be extremely challenging. Furthermore, without managing the infrastructure used to host these models to maximize the utilization of accelerated compute instances, power waste, performance degradation, and various other technical challenges can occur.

[0025] To maximize hardware utilization, improve resilience, increase availability, and address other technical concerns, containerization technologies that enable operating system virtualization and orchestration can be implemented to share and manage hardware across multiple workloads. However, building and maintaining such infrastructure can be costly and technically challenging. While some past infrastructures as service solutions supported running multiple workloads on a specific set of resources (e.g., CPUs and a single GPU instance), such solutions only supported scaling multiple models as a single unit. This type of coarse-grained control cannot account for many different scenarios where workload variations, infrastructure, health, or other factors may require finer-grained management of the models and the computing resources deployed on them. For example, FM (Factorization Function) that generates varying numbers of tokens (e.g., to provide generative text or other outputs) can be integrated into applications that require consistent performance, regardless of the size of the generated tokens. The inference latency of one request can vary significantly from one request to another depending on the number of tokens the FM must generate. Therefore, there is a strong need for technologies that can adequately allocate workloads to resources in a way that maximizes utilization while still achieving consistent performance.

[0026] This article describes various techniques for dynamic endpoint management of heterogeneous machine learning models. Dynamic endpoint management allows client applications to access multiple different machine learning models using a single network endpoint. When adding a machine learning model to a network endpoint, different host systems with different hardware or other performance capabilities can be specified, such as hosts optimized for generative AI with multiple GPUs and other specialized hardware (e.g., systolic array hardware) to achieve the desired performance. Dynamic endpoint management can be performed against network endpoints, automatically managing containers (or other virtualization units) for optimal utilization, performance, and availability, and containers can be configured to scale up / down based on traffic. Models with intermittent traffic patterns can scale to zero, and the lifecycle of each model can be configured individually through an interface (e.g., via a specified scaling policy).

[0027] In various embodiments, optimal placement strategies can be implemented, including optimized placement of fine-tuned machine learning models at the host system to improve inference performance by co-locating related machine learning models to avoid various latency penalties. Furthermore, as discussed below, load-aware routing techniques for heterogeneous machine learning models accessed via public network endpoints can be implemented to intelligently route inference requests by keeping track of the availability of currently served requests and instances serving new requests, achieving higher throughput. Moreover, these routing techniques can support continuous streaming responses returned from models, enabling applications to leverage models associated with managed network endpoints to build interactive applications such as chatbots and virtual assistants at scale. Therefore, those skilled in the art will understand the various improvements to computer and machine learning-related technologies implemented through the various embodiments described in detail below.

[0028] Figure 1 This is a logical block diagram illustrating dynamic endpoint management of heterogeneous machine learning models according to some embodiments. Machine learning service 110 may be a standalone service providing machine learning model hosting and management services or a service implemented as part of a provider network (e.g., similar to...). Figure 2 The machine learning service 210, in addition to hosting, can provide many different features, such as model training and development features and integration with other provider network services. The machine learning service 110 can implement managed network endpoints 130. Managed network endpoints 130 can support many different machine learning models (e.g., thousands), such as models 134a, 134b, 134c, and 134d placed on different hosts, such as hosts 132a, 132b, and 132c. Models can be replicated, such that model copies are copies of machine learning models deployed on specific hosts. For example, model 134d has model copies on hosts 132a and 132c. These models can be added to managed network endpoints 130 via one or more requests to machine learning service 110 (not shown), as described below. Figure 3 The example request is discussed.

[0029] In various embodiments, the machine learning service 110 may implement dynamic endpoint management 120 to perform various management tasks 124 concerning the managed network endpoint 130. Some tasks may be implemented to handle workloads that are serving inference requests 104, which are received via the managed network endpoint and routed to a copy of the invoked model. Other management tasks may involve preparing or configuring the managed network endpoint (e.g., for future work, such as placing a copy of a new model associated with the managed network endpoint and deploying chunked batch updates or new model versions). Dynamic endpoint management 120 may implement various management objectives 160 in determining when and what management tasks to perform. These management objectives may include performance objectives 162, high utilization objectives 164, and availability objectives 166. Each of these objectives may specify what management actions to perform and when to perform them. As part of performing many of these management tasks, objective-based model placement techniques 170, as described below, are employed. Figure 5-9 This is discussed in detail in 12-13. Based on the monitoring of metrics 122 collected for hosts 132 and models 134 to detect various events that need to be performed, such as replica rebalancing and replica or host scaling, some management tasks can be reactive, as discussed below. Figure 13 This is discussed in detail. Because dynamic endpoint management 120 supports receiving model-specific scaling strategies and resource requirements 102 (e.g., via the following discussion on...). Figure 3 The described interface allows for the fulfillment of client application requirements (e.g., performance for a specific model) and utilization or availability concerns (e.g., scaling based on model-specific strategies) when performing management tasks.

[0030] The following text is about Figure 11 , 16 As discussed in detail on 17, load-aware routing technology can be implemented as indicated at 140. For example, upon receiving an inference request, instead of randomly directing the inference request to host 132 with a copy of the invoked model, inference load-aware routing 140 can use various workload information, such as the number of in-flight requests to be selected among different hosts 132. In this way, inference requests can be optimally distributed.

[0031] Please note that the preceding description is a logical diagram of a machine learning service that includes host, model, and dynamic endpoint management, and therefore should not be construed as a limitation on other embodiments of the machine learning system.

[0032] This specification continues with a general description of a network of providers implementing various services, including machine learning services, which can implement the creation of local computing resources to perform machine learning tasks. Various examples of different components or arrangements of components that can implement dynamic endpoint management for heterogeneous machine learning models are then discussed. Many different methods and techniques for implementing dynamic endpoint management for heterogeneous machine learning models are then discussed, some of which are illustrated in the accompanying flowcharts. Finally, a description of example computing systems on which various components, modules, systems, devices, and / or nodes can be implemented is provided. Various examples are provided throughout this specification.

[0033] Figure 2 This is a logical block diagram illustrating an example provider network that can implement machine learning services according to some embodiments, the machine learning services implementing dynamic endpoint management for heterogeneous machine learning models. In one embodiment, provider network 200 may be a dedicated or closed system, or may be set up by an entity such as a company or public sector organization to provide clients 250 with one or more services (such as various types of cloud-based storage) accessible via the Internet and / or other networks.

[0034] Provider network 200 may be implemented in a single location or may include a number of data centers that host and distribute various resource pools required for the infrastructure and storage services provided by provider network 200, such as a collection of physical and / or virtualized computer servers, storage devices, networking equipment, etc. (e.g., as described below regarding...). Figure 8 The computing system 1000 described herein. The provider network 200 can be formed into multiple zones, in which case a zone is a separate geographical area where cloud provider cluster data is concentrated. Each zone may include two or more availability zones connected to each other via a dedicated high-speed network (e.g., fiber optic communication connection). An availability zone (also referred to as an availability domain, or simply a "zone") is a segregated failure domain comprising one or more data center facilities having separate power, separate networking, and separate cooling from facilities in another availability zone. Preferably, availability zones within a zone are located far enough apart from each other that the same natural disaster should not simultaneously take down more than one availability zone. Customers can connect to the availability zones of the provider network 200 via a publicly accessible network (e.g., the Internet, cellular communication network).

[0035] The regions are connected to a global network, which includes dedicated networking infrastructure (e.g., fiber optic connections controlled by the cloud provider) connecting each region to at least one other region. Provider network 200 can deliver content from points of presence outside these regions but networked with them, via edge locations and regional edge caching servers. Edge locations can be extensions of the cloud provider network beyond the traditional region / AZ context. For example, an edge location could be a data center positioned to provide capacity to a group of customers within certain latency requirements, a set of servers provided to a customer's location, or a set of servers provided within (or forming part of) a cellular communication network, each of which can be at least partially controlled by the control plane of a nearby AZ or region. This partitioning and geographical distribution of computing hardware enables provider network 200 to provide customers with low-latency resource access globally with high fault tolerance and stability.

[0036] In various embodiments, provider network traffic and operations can be broadly categorized into two types: control plane operations carried over the logical control plane and data plane operations carried over the logical data plane. The data plane represents the movement of user data through a distributed computing system, while the control plane represents the movement of control signals through a distributed computing system. The control plane typically includes one or more control plane components distributed across and implemented by one or more control servers. Control plane traffic typically includes administrative operations such as system configuration and management (e.g., resource placement, hardware capacity management, diagnostic monitoring, system status information). The data plane includes customer resources implemented on the cloud provider network (e.g., compute instances, containers, block storage volumes, databases, or file storage). Data plane traffic typically includes non-administrative operations such as transferring customer data to and from customer resources. Some control plane components (e.g., first-tier control plane components, such as the control plane for virtualized computing services) are typically implemented on a separate set of servers from the data plane servers, while other control plane components (e.g., second-tier control plane components, such as analytics services) can share virtualization servers with the data plane, and control plane traffic and data plane traffic can be sent through separate / different networks.

[0037] In one embodiment, provider network 200 may be implemented in a single location, or may include a number of data centers that host and distribute various resource pools required for the infrastructure and services provided by provider network 200, such as a collection of physical and / or virtualized computer servers, storage devices, networking equipment, etc. (e.g., hereinafter referred to as...). Figure 19The computing system 2000 described herein. In some embodiments, the provider network 200 may implement various computing resources or services, such as machine learning service 210, storage service 230, computing service 270 and / or any other type of network-based service 240 (which may include virtual computing services and various other types of storage, database or data processing, analysis, communication, event processing, visualization, data cataloging, data ingestion (e.g., ETL) and security services).

[0038] In various embodiments, Figure 2 The components shown can be implemented directly within computer hardware as instructions that can be executed directly or indirectly by computer hardware (e.g., a microprocessor or computer system) or a combination of these technologies. For example, in one embodiment, Figure 2 The components can be implemented by a system comprising multiple compute nodes (or simply nodes), each of which can be similar to Figure 19 The computer system embodiments shown and described below. In various embodiments, the functionality of a given system or service component (e.g., a component of machine learning service 210) may be implemented by a specific node or may be distributed across several nodes. In some embodiments, a given node may implement the functionality of more than one service system component (e.g., more than one data storage component).

[0039] Machine learning service 210 may implement interface 211 to allow clients (e.g., client 250 or clients implemented internally within provider network 200, such as client applications hosted on another provider network service like event-driven code execution services or virtual computing services) to train and deploy machine learning models (e.g., neural networks or various other types of machine learning models). For example, interface 211 may implement a development interface for training machine learning models and a management interface for deploying machine learning models via both network endpoint 224a and managed network endpoint 226a. For example, machine learning service 210 may implement interface 211 (e.g., a graphical user interface, a programming interface implementing an application programming interface (API), and / or a command-line interface) that allows clients to submit, edit, or otherwise implement various model development, deployment, host system recommendations, or other management requests. For example, interface 211 may include a development and deployment environment interface that provides training scripts or other code editors with various development tools for creating, submitting, and / or monitoring machine learning pipelines with training jobs and / or monitoring jobs. In some embodiments, this development and management environment may be a graphical interface, and in some embodiments, it may provide an interface for generating past results for other models. Similarly, the management interface may provide various graphical user interface features for creating and managing accounts, studio groups, licenses, or various other features of machine learning service 210. (See below for more information.) Figure 3 As discussed, interface 211 can support various deployment requests, including requests to create and configure network endpoints associated with the model, such as network endpoint 224a and managed network endpoint 226a.

[0040] Machine learning service 210 may implement control plane 212 to perform various control operations for implementing the features of machine learning service 210. For example, the control plane may monitor the health and performance of requests at different components, such as training as part of model development and the execution of machine learning models as part of model deployment. For example, if a node or other component fails, a request fails, or other interruption occurs, control plane 212 may restart the job to complete the request (e.g., instead of sending a failure response to the client). In various embodiments, control plane 212 may arbitrate, balance, select, or dispatch requests to different nodes in some embodiments. For example, control plane 212 may receive a request interface 211, which may be a programmable interface, and identify nodes that can be used to begin processing requests.

[0041] In various embodiments, control plane 212 may include components supporting the management of both different types of endpoints, namely network endpoint 224a and managed network endpoint 226a, which may be manually managed by users of machine learning service 210. The managed network endpoints may be similar to managed network endpoint 130, and as described below regarding... Figure 3-18 Further details will be discussed. For example, endpoint management 215 can be similar to dynamic endpoint management 120 to implement various technologies for performing management tasks based on management objectives 160. For example, as described below... Figure 4 In detail, model placement 216 can be used to make placement decisions about adding new models to one or more locations to meet resource requirements, availability requirements, etc. Endpoint monitoring 219 can detect events that trigger the execution of various management tasks, such as replica scaling, host scaling 218, and replica rebalancing 217, as discussed below. Figure 5 , 12 As discussed in detail in section 13. Control plane 212 can also implement model registry 213 and endpoint / model deployment 214 to handle requests to create network endpoints 224a and 226a, and store relevant endpoint information in model registry 213. For example, Figure 4 The creation of the managed network endpoints is illustrated. In some embodiments, endpoint / model deployment 214 may also participate in batch model updates (or chunked models) in a rolling manner, together with endpoint management 215.

[0042] Although not shown, machine learning service 210 can implement development environment management to develop, configure, program, define, and / or otherwise execute training jobs on various machine learning models using datasets, such as datasets in storage service 230 in one or more host system types. These training jobs may include various configurations, sizes, and quantities of one or more corresponding processing devices for training, such as GPUs and other hardware (e.g., the amount and speed of memory) and / or software capabilities. In some embodiments, machine learning service 210 can provide various virtual machines, instances, containers, images, or other applications on these training nodes, which can implement various machine learning training frameworks (e.g., TensorFlow, PyTorch, MXNet, and XGBoost). On these frameworks, development environments can be used to specify or otherwise describe machine learning models and execute them. Various tests or other development operations can also be performed on the machine learning models. In some embodiments, various files, configuration information, and other data used for machine learning model development can be organized into projects (or other collections) and stored, versioned, or otherwise managed by the model development environment management (e.g., as a collection of one or more files or data objects in storage service 230).

[0043] Data plane 220 may include various features or artifacts for training and development, or deployed machine learning models 224b and 226b, as shown, accessible via corresponding network endpoints 224a and 226a. A network endpoint may be a network address, identifier, or other locator associated with the set of resources, namely host system 272 and model 224. A network endpoint may be the target of a request to invoke the hosted model 224 (e.g., in some embodiments, an API request to generate inference). Routing layer 222 may implement various networking components, systems, or services, including those described above. Figure 1 And the following text about Figure 11 The discussed load-aware routing is for the managed network endpoint 226a. Route 222 can identify the target network endpoint in a request and then dispatch the request to the appropriate host system for further processing (e.g., generating inferences).

[0044] In some embodiments, other provider network services, such as computing service 270 and data storage service 230, may be used for machine learning service 210. In other embodiments, these services may be implemented as internal systems of machine learning service 210. Computing service 270 may implement various host systems 272, which are, for example, instances of virtual computing services and host one or more containers on said instances. These containers / instances may be deployed on different physical computer systems capable of accessing different hardware components, thereby providing different performance capabilities. For example, different types or configurations of resources (including varying amounts of processing capacity, memory, storage, and / or dedicated hardware such as GPUs and Tensor Processing Units (TPUs), systolic arrays, or various other types of hardware-based accelerators for machine learning computations) may be assigned or otherwise obtained from computing service 270, and then assigned or otherwise obtained from machine learning models deployed to the assigned host systems and associated with network endpoints (and various software or other applications, such as those that may be implemented in containers, that support receiving requests for inference and returning inferences using one or more models).

[0045] In some embodiments, other services 240 may include a container registry service for both storing and providing two machine learning service 210 containers (e.g., an ML development environment laptop server image, an ML development environment kernel image, and an ML computing resource image for deployment), as discussed below.

[0046] Data storage service 230 may implement different types of data storage areas for storing, accessing, and managing data on behalf of client 250 as a network-based service, enabling client 250 to operate the data storage system in a cloud or network computing environment. In some embodiments, data storage service 230 may include object or file data storage areas for placing, updating, and retrieving data objects or files, one or more of which may be used to provide data storage for supporting machine learning service 210. For example, various machine learning models 232 may be stored and retrieved from data storage service 230 and may be loaded onto a host system according to various techniques discussed below, and may be associated with network endpoint 224 or managed network endpoint 226a (described as models 224b and 226b). Data storage service 230 may be a file system service, allowing different data objects of different formats or data types to be stored as storage areas of a corresponding file system associated with an account or user of machine learning service 210. In at least some embodiments, data storage service 230 may be considered a data lake. For example, an organization may generate many different kinds of data, which are stored in one or more sets of data objects in data storage service 230. The data objects in the collection can include related or homogeneous data objects, such as database partitions of sales data, and unrelated or heterogeneous data objects, such as mirrored data files (e.g., digital photos or video files), audio files, and website log files. The data storage service 230 can be accessed through a programming interface (e.g., an API) or a graphical user interface.

[0047] Generally, client 250 can encompass any type of client that can submit web-based requests to provider network 200 via network 260, including requests for machine learning service 210 (e.g., requests to begin executing a machine learning task). For example, a given client 250 may include a suitable version of a web browser, or may include a plugin module or other type of code module that can execute as an extension of or within an execution environment provided by the web browser. In some embodiments, this application may include sufficient capabilities to generate and process web-based service requests without necessarily implementing fully browser-supported protocol support (e.g., protocol support for a suitable version of Hypertext Transfer Protocol (HTTP)) for all types of web-based data. That is, client 250 can be an application that can directly interact with provider network 200. In some embodiments, client 250 may generate web-based service requests based on a representative state transfer (REST) ​​style web-based service architecture, a document- or message-based web-based service architecture, or another suitable web-based service architecture.

[0048] In some embodiments, client 250 may provide access to provider network 200 to other applications in a manner transparent to them. In one embodiment, client 250 may transmit network-based service requests (e.g., access requests to configure or perform machine learning tasks) via network 260. In various embodiments, network 260 may encompass any suitable combination of networking hardware and protocols necessary to establish network-based communication between client 250 and provider network 200. For example, network 260 may typically encompass various telecommunications networks and service providers that jointly implement the Internet. In one embodiment, network 260 may also include private networks, such as local area networks (LANs) or wide area networks (WANs), and public or private wireless networks. For example, given client 250 and provider network 200, both may be assigned within an enterprise with their own internal networks. In this embodiment, network 260 may include the hardware (e.g., modems, routers, switches, load balancers, proxy servers, etc.) and software (e.g., protocol stacks, accounting software, firewalls / security software, etc.) necessary to establish networking links between given client 250 and the Internet, and between the Internet and provider network 200. Note that in some embodiments, client 250 may use a private network instead of the public Internet to communicate with provider network 200.

[0049] Figure 3 This is a logical block diagram illustrating interactions for dynamic endpoint management of heterogeneous machine learning models according to some embodiments. Interface 211 may support various interactions for creating, configuring, and otherwise managing resources of machine learning service 210, including managed network endpoints. For example, one or more requests to create managed network endpoints 310 may be supported. Various characteristics or parameters of request 310 may include information for dynamic endpoint management. For example, the request may create a new managed network endpoint that enables the machine learning service to establish various networking rules or components to guide requests to invoke machine learning models to be deployed associated with the managed network endpoint. For example, routers, load balancers, or other networking components may be updated to include new network endpoints. Metadata may be created, such as at model registry 213, and network endpoints may be defined as managed network endpoints (rather than unmanaged network endpoints, such as...). Figure 2 The instructions for network endpoint 224a in the document.

[0050] Other characteristics of the managed network endpoints can be included in Request 310. For example, one or more models can be added. (See above regarding...) Figure 1The discussion may specify particular computational requirements (e.g., numbering CPUs, GPUs, memory, or other hardware, including various accelerator devices). Scaling strategies specific to each model may also be specified (e.g., minimum number of replicas, maximum number of replicas, rate at which replicas can scale up or down, etc.). Other performance objectives, such as availability objects, are achieved, though not depicted, by placing replicas of the model in multiple availability zones of the provider network 200.

[0051] Interface 211 can support one or more requests to update the managed network endpoint 320. For example, request 320 may include a request to add or remove a model. In some embodiments, these requests may be to replace the model with an updated version, which can trigger a bulk (or chunked) replacement action that deploys the new model as a copy on a new (or existing) host system associated with the managed network endpoint before removing / removing the current model copy. In this way, client applications that invoke the model (but not necessarily a specific version of the model) experience zero downtime. The following discussion... Figure 18 Further discussion of such techniques is provided. Similarly, request 320 may include adding, updating, or removing computational requirements and scaling policies for the model. For example, an updated scaling policy may be used to add (or remove) support for scaling down to zero copies. In some embodiments, removing a scaling policy or computational requirement may allow machine learning service 210 to apply service-determined scaling policies and computational requirements (e.g., default policies or computational requirements or dynamically determined scaling policies or computational requirements based on the model's historical usage or predictive usage).

[0052] While requests 310 and 320 (and other similar requests) can be considered control plane 212 requests, data plane 220 requests can also be received through interface 211. For example, requests relating to managed network endpoints 330 can be received. These requests can specify the machine learning model to be used to generate inferences. In some embodiments, the requests can have a request type (e.g., a request associated with a streaming interaction, a sticky session, or other interaction that may be expected to be handled differently by directing the request to the same host / model as the previous request 330).

[0053] The interaction with interface 211 described above is provided as an example only. Similar features can be performed using other combinations of requests with the same or different parameters. For example, a request to create a managed network endpoint can be separate from a request to add a model, which can be separate from a request to specify scaling policies or computational requirements.

[0054] Figure 4This is a logical block diagram illustrating the interaction for creating a managed network endpoint according to some embodiments. A request to create a managed network endpoint is received, as indicated at 402. This request may be processed by endpoint / model deployment 214, which may assess initial resource needs based on computing requirements and / or other performance goals, and assign multiple endpoint hosts from computing service 250 404. This may include specifying a specific host type (e.g., a specific instance type) capable of accessing specific hardware or other computing resources that meet specified computing requirements (e.g., sufficient GPUs, memory, or specific hardware accelerators) for different models. Computing service 250 may provide endpoint host resources, as indicated at 406 for the managed network endpoint 420.

[0055] Model placement 215 can indicate which model 408 to place. Model placement 216 can evaluate available host systems (e.g., by accessing the metadata of associated host instances 430a and 430b in the model registry or another metadata store (not shown) in the machine learning service 210). Model placement 216 can apply placement techniques that start with minimum placement requirements and are based on the above description. Figure 1 The discussion focuses on performance, efficient utilization, and availability, working towards more optimized placement considerations. For example, model placement 216 can first consider the specified computational requirements and ensure that at least the host instance can meet the model's computational requirements (e.g., to achieve the model's performance goals). Other considerations, such as availability (e.g., whether the model needs to be placed in a specific location, such as an availability zone) and / or whether the model can be placed on an underutilized host instance (e.g., to improve the utilization of said host instance without overloading it).

[0056] As indicated at 412, model placement 216 can involve duplicate placement, including whether to place multiple copies of the model. For example, model 433a can have copies on both host instances 430a and 430b. Other models, such as model 435a, can have a single copy. Figure 4 As depicted, replicas can be placed for each inference container, such as single replicas of inference containers 432a and 434a. In some embodiments, multiple models can be executed within a single container. As indicated at 416, data storage service 230 can provide artifacts of models to be loaded into their respective host instances / containers for execution. Model placement 216 can update model registry 213 to include the placement of replicas of managed network endpoints 420. Other information may be included, such as scaling policies or computational requirements applicable to each model (this other information may be used for subsequent placement and management tasks of managed network endpoints 420).

[0057] The above text is about Figure 4 The illustrated interaction can be similar to that performed to add a new model to a managed network endpoint that is already in operation. Model placement 216 can perform a similar placement evaluation and select a host instance. If a suitable host is unavailable (e.g., does not meet computation requirements), endpoint / model deployment 214 may be assigned a task to assign a host that does meet the computation requirements from compute service 250 and associate that host with the managed network endpoint 420. Similar interactions can also be performed for chunked or batch operations used to deploy multiple copies of a new model or to update multiple copies of an existing model to a new version. For example, a new version of the deployed model can be placed as multiple copies in a host instance in the network endpoint before removing an existing copy of the current version of the deployed model.

[0058] In contrast to those tasks discussed above that can be triggered by requests to perform different actions on the managed network endpoints, some management tasks can be triggered by monitoring or other automated evaluations of the managed network endpoints. Figure 5 This is a logical block diagram, according to some embodiments, for monitoring managed network endpoints used for dynamic endpoint management of heterogeneous machine learning models. Endpoint monitoring 219 can be implemented as part of endpoint management 215 to proactively address potential failures or other performance issues and maintain or improve performance, high utilization, and availability targets of the models associated with the managed network endpoints.

[0059] Replica / instance metric 502 can report various performance and utilization metrics for individual replicas of the model and their corresponding host instances. Some metrics may include various compute resource utilization metrics, such as CPU utilization, GPU utilization, memory utilization, retention rate, disk or other storage utilization, and inference performance metrics, such as the number of calls per replica, the number of call errors, and replica latency. In some embodiments, these metrics may be aggregated based on the model (e.g., the number of calls per model, average CPU utilization, GPU utilization, or other resource utilization). At least some of these replica / instance metrics 502 may be published by users of the machine learning service 210 or shared with users of the machine learning service (e.g., using provider network metrics services that can display or otherwise visualize the metrics, including metric monitoring services that can trigger alerts or other notifications based on received metrics).

[0060] Endpoint monitoring 219 can implement model replica rebalancing 217, which examines the placement and performance of model replicas 533a and 535a within inference containers 532a and 534a of host instance 530a to ensure efficient utilization of host 530a of the managed network endpoint 520. Model replica rebalancing 217 can apply various criteria to identify and determine whether one or more replicas should be moved to other host instances 530a of the managed network endpoint 520. For example, model replica rebalancing can identify underutilized host instances 530a. Utilization metrics 502 can be compared to a minimum utilization threshold. If a host instance is below the minimum utilization threshold, it can be identified as underutilized. Similar analysis can be performed on overutilized instances, where the workload used to perform inference requests may cause performance degradation. This overutilization condition can sometimes be referred to as "hot." Other unhealthy or poorly placed instances can be indicated by performance metrics such as the number of errors or inference latency. Therefore, when various criteria (e.g., thresholds) for analyzing these metrics are met, a rebalancing event can be triggered for performance.

[0061] To handle these detected rebalancing events, model replica rebalancing 217 may result in replica placement 504 to move replicas. In some embodiments, model placement 216 may return possible placements (as indicated at 506), which model replica rebalancing 217 may confirm before initiation. In this way, model replica rebalancing 217 can determine whether a placement improves the situation (e.g., whether moving a replica to another host instance makes the host system overutilized). In some scenarios, rebalancing events may not be performed due to a lack of alternative placement locations. In some embodiments, rebalancing events may work in conjunction with host scaling 218, which may allow the addition of new hosts and then perform rebalancing to move replicas after the host instance is added to the managed network endpoint.

[0062] Endpoint monitoring 219 can implement model replica / host scaling 218. Model replica / host scaling 218 can evaluate replica / instance metrics 502 relative to a scaling policy specified for the model. For example, if the model scaling policy specifies a threshold or condition for when additional replicas of the model should be added (or removed), model replica / host scaling 218 can detect scaling events that trigger scaling actions. For example, if the number of requests for the model exceeds a threshold number (or some other criterion, such as the average latency of model requests) within a certain time period, another (or more) replicas can be added according to the scaling policy for the model (e.g., the scaling policy may specify the rate at which replicas are added and the maximum number of replicas for the managed network endpoints). Similarly, the scaling policy can indicate when the number of replicas can be reduced based on various criteria regarding replica metrics 502. When a new replica is added to a managed network endpoint, replica scaling can be derived from model placement 216, resulting in replica placement 504.

[0063] In addition to model / replica scaling, host scaling can also be performed. For example, model / replica host scaling 218 can determine when underutilized host instances 530a can be removed from the managed network endpoint 520. In some scenarios, this can also trigger rebalancing 217. Similarly, if the managed network endpoint 520 is experiencing high workload and host instance 530 is experiencing heat that cannot be mitigated by replica rebalancing, an event to add a number of host systems can be triggered. Like replica scaling, host scaling may be subject to scaling policies. In this way, autoscaling technology does not ignore other concerns (e.g., cost) when scaling to high (or low).

[0064] The following discussion illustrates different example rebalancing and scaling scenarios. The examples shown do not depict all possible rebalancing and scaling actions that can be taken relative to the managed network endpoints. Figure 6This is a logical block diagram of a rebalancing event according to some embodiments. The managed network endpoint 620 may include host instances 630a, 630b, 630c, and 630d. Model replicas 632a and 632d may be hosted at instance 630a, model replica 632c may be hosted at host instance 630b, model replicas 632a, 632b, and 632e may be hosted at host instance 630c, and model replica 632a may be hosted at host instance 630d. Endpoint monitoring 219 may identify that host instance 630d is underutilized. Alternatively, endpoint monitoring 219 may identify that host instance 630c is overutilized (both conditions may also be true). In this scenario, a rebalancing can be performed, as indicated at 630. In this way, model replica 632e can be moved to host instance 630d. To make the move, host instance 630c may unload model 632e or otherwise cease performing inference requests for the model. You can update route 222 (for example, via the following text about...) Figure 11 (See model registry 213 for detailed discussion). Host instance 630d may implement a service agent, container, or other application (not shown) that can access model artifacts (and containers, if needed) from other services (e.g., storage service 230) to load model 632e and begin executing inference requests for model 632e.

[0065] Figure 7 This is a logical block diagram of a reduction event according to some embodiments. The managed network endpoint 720 may include host instances 730a, 730b, 730c, and 730d. Model replicas 732a and 732d may be hosted at instance 730a, model replica 732c may be hosted at host instance 730b, model replicas 732a and 732b may be hosted at host instance 730c, and model replica 732e may be hosted at host instance 730d. Endpoint monitoring 219 may identify that host instance 730e is underutilized. For example, model 732e may not have received inference requests during a previous time period (e.g., within 24 hours). In this scenario, removal of both the instance and the replica can be performed, as indicated at 730. In this way, model replica 732e is no longer actively hosted at the managed network endpoint 720 (even though it is still associated with the managed network endpoint 720). The removal of host instance 730d can be performed by releasing the host back to compute service 250 (e.g., unassigning or terminating the instance). Route 222 can be updated (e.g., as described below regarding...). Figure 11 The model registry (213) is discussed in detail to indicate that host instance 730d and model 732e are no longer available.

[0066] In at least some embodiments, removing a model copy, leaving a model without a currently hosted copy, can be considered a "scale-down to zero" characteristic, which may have to be implicitly authorized by a scaling policy for that model. If not authorized, model 732e can be moved to another host instance instead of being removed. The scaling policy for a model allowed to scale down to zero can also specify when and how it can return to hosting (e.g., upon receiving an inference request or upon receiving a larger number of inference requests, after a cooldown period, etc.). If an inference request for a model without a copy comes in, the scaling policy can also indicate how to handle the request (e.g., queue until the model is added back to the managed network endpoint or fail due to an error indicating that the model does not exist and when it exists / when it will reappear at the network endpoint).

[0067] Figure 8 This is a logical block diagram of an extended event according to some embodiments. The managed network endpoint 820 may include host instances 830a, 830b, 830c, and 830d. ​​Model replicas 832a and 832d may be hosted at instance 830a, model replica 832c may be hosted at host instance 830b, model replicas 832a, 832b, and 832e may be hosted at host instance 830c, and model replica 832a may be hosted at host instance 830d. ​​Endpoint monitoring 219 may identify that another replica of model 832c is needed. For example, the number of requests served by a replica of model 832c exceeds a threshold for expansion (or the average latency of the requests is inferred to exceed a latency threshold). It can be determined that other instances in the managed network endpoints do not have sufficient computing resources to meet the computing requirements specified for model 832c. In this scenario, in addition to adding a replica of model 832c, instances may be added, as indicated at 830. If instances are available, the instances may be added as replicas only. A new host instance 830d can be assigned from compute service 250 and associated with the managed network endpoint 820. Host instance 830d can then be instructed to load model 832c. Routes 222 can be updated (e.g., via the following text regarding...). Figure 11 (See model registry 213 for detailed discussion). Host instance 830d may implement a service agent, container, or other application (not shown) that can access model artifacts (and containers, if needed) from other services (e.g., storage service 230) to load model 832c and begin executing inference requests for model 832c.

[0068] Some machine learning model types can be tailored to managed network endpoints and offer additional optimization opportunities in other scenarios (e.g., unmanaged network endpoint 224a). Fine-tuned machine learning models are a type of machine learning model that can provide additional placement and inference performance optimization. Fine-tuning can refer to the technique of adapting features (e.g., weights) of a previously trained machine learning model to additional training data, which can "tune" the performance of the trained machine learning model or otherwise adapt it to a specific purpose or scenario included in the additional training data. For example, a computer vision model performing object classification can typically be tuned to identify specific categories of objects, such as traffic signs, in mirrored data. However, there may be scenarios where fine-tuning of a trained machine learning model is desired, but access restrictions do not support or allow modification of the trained machine learning model itself.

[0069] For example, some machine learning models are developed as a result of significant technical effort and resource costs. Appropriate datasets may need to be curated, and the architecture of the machine learning model may need to be designed to deliver high performance. Some of these models can be very large, using, for example, billions of parameters, allowing them to be adapted to a wide range of use cases and tasks, such as text, image generation, and aggregation. These models, sometimes referred to as “foundational models,” can perform well without any adaptation. However, in many scenarios, better performance can be achieved by fine-tuning the model for a specific use case. Given the technical effort and resource costs involved in developing and training these machine learning models, model providers may impose access restrictions on the model's content (e.g., the weights of the model parameters), because if the model's content were accessible, the model provider would have to relinquish proprietary model information.

[0070] For a fine-tuned machine learning model that does not change the initially trained model (which may be referred to as the "base" model), a combination of an "incremental" model that implements tuning aspects and the base model can be used to produce a version of the fine-tuned model. In some embodiments, different incremental models can be used to deploy many different versions of the same base model.

[0071] In some embodiments, various different types of fine-tuning techniques can be performed to produce these incremental models, such as Parametric Efficient Fine-Tuning (PEFT). PEFT refers to a set of fine-tuning techniques that do not require updating all model weights. Instead, only a subset of the weights is updated. A significant component of the PEFT method is fine-tuning only a small number of (additional) model parameters. Below are some examples of PEFT techniques.

[0072] LoRA (Low-Rank Adaptation) is a technique in which pre-trained weights from a provided machine learning model are frozen, and a smaller set of incremental weights is trained using a tuned dataset. During inference, the results of the incremental weights are added to the frozen weights. LoRA can achieve better results than incremental fine-tuning and can be fine-tuned much faster.

[0073] AdaLoRA: LoRA, but with an adaptive learning rate that is adjusted based on curvature information of the lost landscape.

[0074] Prefix tuning: The idea behind prefix tuning is to optimize the continuous vector preceding the language model input. This vector, also known as the "prefix," guides the model's generation process. Prefix tuning only adjusts the prefix, leaving the rest of the model parameters unchanged.

[0075] P-tuning: A set of trainable parameters (P) is introduced as additional lexical units at the beginning of the input sequence. These parameters are learned during the fine-tuning process and are task-specific.

[0076] Cue word tuning: A mechanism for learning “soft cue words” to tune a frozen language model, thereby enabling the execution of specific downstream tasks for labeled instances.

[0077] RLHF: Using reinforcement learning to “teach” a model using a reward model tuned based on human feedback data.

[0078] The placement of the base model and incremental models can be optimized so that they co-locate on the same host system. This placement improves the performance of inference requests (e.g., no network hops between inference generation steps of different models) and can be further optimized by sharing the base model with multiple different versions of a common base model, which are tuned differently using different incremental models. These techniques can be applied beyond managed network endpoints. For example, unmanaged network endpoints or other machine learning systems or services may want to place the base model and incremental models together to achieve this performance improvement. Because the base model can be very large, using a single copy with multiple incremental models achieves significant resource savings in both storage and computational resources (when compared to copies of the base model paired with each incremental model). The savings become even greater when a single base model is used with hundreds or thousands of incremental models that tune the base model for different tasks. Figure 9 It is a logic block diagram based on a finely tuned model placement according to some embodiments.

[0079] like Figure 9As depicted, model placement 216 can implement fine-tuned model placement optimization 901 to handle requests to place fine-tuned models 902. Placement requests can be for either a base model or an incremental model. Fine-tuned model placement 901 can access model registry 213 to identify the relevant model 904. For example, an incremental model may include an indication that the model is an incremental model as part of the model metadata in registry 213, as well as a model identifier of the base model for which the incremental model is fine-tuned. Therefore, when model placement does make a placement decision for a fine-tuned model, any relevant models can be considered (e.g., placing the incremental model alongside an already placed base model, thereby identifying and obtaining the incremental model to be placed when the base model is received for placement). In some embodiments, other placement considerations can still be made relative to computational requirements, performance, efficient utilization, and availability (as described above regarding...). Figure 1 (Discussed).

[0080] For example, such as Figure 9 As depicted, host instance 930a may include inference container 932a, which may perform inference using one of a base model 933 and an incremental model 934 (multiple incremental models). In some embodiments, as indicated at 916, the identified incremental and base models may be available from data storage service 230.

[0081] Further performance improvements can be achieved by co-locating multiple incremental models with the base model when inferring generation time. Figure 10This is a logical block diagram, according to some embodiments, of loading an incremental model into memory to generate inference for a fine-tuned machine learning model. Container 1010 can receive requests to invoke an endpoint of the specified fine-tuned ML model, as indicated at 1002. The specified version of the ML model can be generated using one of the incremental models 1022a, 1022b, 1022c, 1022d, and 1022e of the memory-loaded incremental model 1020. Because the incremental model is loaded into memory, there is little or no downtime for switching between different versions of the fine-tuned model. Instead, an incremental value can be computed at 1030 according to the different model tuning techniques discussed above, and the incremental value can be combined with the base model computed value generated at 1040 to complete inference generation and provide inference 1004 at 1050. This technique provides several performance improvements, such as reduced latency when generating inference (e.g., the in-memory incremental model can be quickly applied to generate input values). Furthermore, concurrent requests for different fine-tuned versions of the model can be handled. For example, when performing the base model calculation 1040, the same input can be used to generate different incremental values ​​using different incremental models, thereby allowing the base value to be reused to produce different tuned versions of inference using different sets of incremental weights.

[0082] As mentioned above Figure 1 The discussion focuses on implementing load-aware routing techniques for managed network endpoints. In this way, routing decisions can be made across multiple replicas to optimize the throughput of inferred requests and prevent unnecessary rebalancing and scaling actions. Therefore, load-aware load balancing for managed network endpoints improves the performance of managed network endpoints by enabling them to utilize existing model replicas and hosts more efficiently. Figure 11 This is a logical block diagram of a load-aware routing technology for managed network endpoints, according to some embodiments.

[0083] Router 1160 may be implemented as part of routing layer 222. In some embodiments, router 1160 may be assigned to one(s) managed network endpoints. Router 1160 may utilize model deployment cache 1162, which may store information about model replicas and host instances associated with the managed network endpoints. For example, managed network endpoint 1110 may include multiple host instances, such as host instances 1120, 1130, and 1140. These host instances may host multiple model replicas, such as model replicas 1150a, 1150b, 1150c, and 1150g. The host instances may also include corresponding service host agents 1122, 1132, and 1142, which may report various performance metrics and process requests to dispatch inference requests to the appropriate model replicas.

[0084] The managed network endpoints, such as endpoint 1110, are dynamic. As discussed in detail above, various moves, expansions, reductions, and rebalancings can occur. While router 1160 can maintain a local cache 1162, the model registry 213 can be used as a truth source for the endpoints because endpoint management 219 can update the model registry using various changes 1101. Therefore, router 1160 can periodically request and obtain model placement information 1164 to update cache 1162 (or when the cached information is incorrect or lost for a particular copy).

[0085] The following text is about Figure 16 and 17 As discussed in detail, load-aware routing technology can utilize various workload metrics or information about hosts and model replicas to make routing decisions. Upon receiving a call for a specified ML model, as indicated at 1102, if the model exists in the model deployment cache 1162, the cache can be accessed and used to determine and select the host instance of the replica of the model invoked by the hosting provider. Similar to model 1150a, multiple host instances can be considered. Workload information, such as that discussed above, can be taken into account. Figure 5 Various performance metrics are discussed (e.g., resource utilization, inference performance, etc.). Different selection strategies can be implemented, one of which can be specified in a configuration request (e.g., 320) for the managed network endpoint 1110. In some embodiments, an in-flight inference request (e.g., an in-progress inference request that has not yet returned to the client) as indicated at 1105 can be used to select between instance hosts of a replica of the storage model. (See also: ...) Figure 17 The discussion suggests that sticky sessions or other associations can be maintained between specific clients and host instances to ensure that streaming sessions or other types of interactions (e.g., stateful interactions) involving multiple inferences / responses based on previous responses / inferences can be supported without having to replay or obtain state information to continue.

[0086] Despite already having a network of providers implementing machine learning services Figure 2-11 It has been described and explained, but Figure 2-11 The various components shown and described can be readily applied to other machine learning systems capable of implementing network endpoint management for heterogeneous machine learning models. Therefore, Figure 2-11 This is not intended to be limited to other embodiments.

[0087] Figure 12This is a high-level flowchart illustrating various methods and techniques for creating local computing resources to perform machine learning tasks, according to some embodiments. As indicated at 1210, placement events of machine learning models associated with a managed network endpoint can be detected. As discussed above, the managed network endpoint can provide access to the different machine learning models by receiving a request from a client of the machine learning service to invoke a specified different machine learning model among different machine learning models including the machine learning model. Placement events can be triggered / requested by various actions concerning the managed network endpoint. For example, a request to add a new model (or add a new version of a model) can trigger a placement event. The above and below relate to... Figure 13 The automated management operations performed on the managed network endpoints, such as scaling and rebalancing, can trigger placement events.

[0088] As indicated at 1220, in some embodiments, computing resources can be selected from computing resources associated with the managed network endpoints to host machine learning models, the selection being at least in part based on a determination that the computing resources meet the resource requirements of the machine learning model. For example, selection techniques may first consider the specified computing requirements and ensure that at least the computing resources (e.g., host instances) can meet the model's computing requirements (e.g., to achieve the model's performance goals). Other considerations, such as availability (e.g., whether the model needs to be placed in a specific location, such as an availability zone) and / or whether the model can be placed on underutilized computing resources (e.g., to improve the utilization of the host instances without overloading the computing resources), may also be evaluated. In some embodiments, optimization techniques may be used to select among multiple options (e.g., if more than two hosts can meet the computing requirements). For example, binning techniques (e.g., best fit, next fit, etc.) may be used. Other placement optimization techniques, including machine learning placement techniques or simulations of the proposed placement, may be implemented alternatively or additionally to determine their impact on subsequent placement options.

[0089] As indicated at 1230, in some embodiments, the machine learning model can be placed at a selected computing resource. For example, the host system can be instructed to obtain the machine learning model, and mapping information used for routing and / or other features (e.g., managed network endpoint management operations) can be updated to reflect the placement.

[0090] In some embodiments, there may be no available placement. An error indicating that no host is available for placement or other indications can be returned to, for example, the client (e.g., endpoint management 219), which can trigger other actions, such as adding a new host to the managed network endpoint.

[0091] Placement may involve various management operations on the network endpoints being managed. Figure 13 This is a high-level flowchart illustrating various methods and techniques for detecting scaling and rebalancing events according to some embodiments. As indicated at 1310, in some embodiments, hosts and replicas associated with managed network endpoints can be monitored for various management operations. As discussed above, various performance metrics of individual replicas, host systems (which may have multiple replicas), and aggregate metrics of all replicas of a model can be monitored and evaluated for different management tasks.

[0092] For example, as indicated at 1320, in some embodiments, an evaluation can be performed to determine whether a replica rebalancing event has been detected. Underutilized, overutilized, and unhealthy hosts can be detected, which may lead to the detection of a rebalancing event. If so, then as indicated at 1322, in some embodiments, one or more replicas can be moved to one or more different hosts associated with the managed network endpoint. (See above regarding...) Figure 12 The discussion can lead to placement decisions based on these corresponding moves.

[0093] As indicated at 1330, in some embodiments, an evaluation of performance metrics can be performed to determine which replicas of the machine learning model can be added. For example, a scaling policy for replicas can indicate that if replica usage meets some criteria (or performance meets some criteria, such as failing to achieve average latency or other performance goals), then one(s) replicas should be added to the managed network endpoint. If so, a determination regarding whether a new host is needed can also be performed, as indicated at 1332. If not, as indicated at 1342, in some embodiments, a host can be added to the managed network endpoint. If so, the replica can be placed at an existing host, as indicated at 1334.

[0094] As indicated at 1340, in some embodiments, performance metrics can be evaluated regarding whether the number of hosts needs to be expanded and the events in which hosts are added. For example, it may be necessary to mitigate overutilized hosts, and there may be no other hosts available, so new hosts may be added. Alternatively, replicas may be needed, and no hosts may be available. If this is the case, then as indicated at 1342, in some embodiments, hosts may be added to the managed network endpoints.

[0095] As indicated at 1350, in some embodiments, performance metrics are evaluated regarding whether a replica of a machine learning model can be removed. For example, a scaling policy for replicas may indicate that if a replica performs below certain criteria (or meets certain performance criteria, such as failing to achieve average latency or other performance goals), one(s) replica(s) should be removed from the managed network endpoint. A minimum number of replicas may be specified in the scaling policy, or scaling down to zero may be allowed. If so, as indicated at 1352, in some embodiments, a replica may be removed from the host associated with the managed network endpoint.

[0096] As indicated at 1360, in some embodiments, performance metrics may be evaluated regarding whether a host should be removed from the managed network endpoint. For example, underutilized or unhealthy hosts may be identified based on the number of inference requests performed, the resources utilized, or the latency of the inference requests. If so, then as indicated at 1362, in some embodiments, the host may be removed from the managed network endpoint.

[0097] As mentioned above Figure 9 The discussion also suggests that some finely tuned machine learning models can achieve further performance improvements through optimized placement. Figure 14 This is a high-level flowchart illustrating various methods and techniques for placing a fine-tuned machine learning model according to some embodiments. As indicated at 1410, a request to place a machine learning model on a host system of a machine learning service may be received. In some embodiments, the machine learning model may be a base model of the fine-tuned machine learning model.

[0098] As indicated at 1420, in some embodiments, different machine learning models can be identified as corresponding incremental models relative to a base model, wherein corresponding combinations of incremental models with the base model produce corresponding versions of fine-tuned machine learning models. For example, a registry or other metadata store for machine learning models can indicate available and relevant machine learning models that share or utilize the base model as incremental models for generating fine-tuned inferences. This metadata can implicitly link incremental models, or in some embodiments, similarity analysis or other types of searches can be performed, where potentially relevant incremental models can be identified for placement (and subsequently removed upon request if the user finds the incremental model irrelevant).

[0099] As indicated at 1430, in some embodiments, both the base model and the corresponding incremental model can be placed on a host system that generates a corresponding inference for a request to invoke a corresponding version of the corresponding version of the fine-tuned machine learning model. For example, instructions can be made to the host system to obtain both the base model and the identified incremental model.

[0100] Figure 15 This is a high-level flowchart illustrating various methods and techniques for generating inference using a fine-tuned machine learning model, according to some embodiments. As indicated at 1510, in some embodiments, a request may be received to generate inference using a specified version of the fine-tuned machine learning model. For example, the specified version of the fine-tuned model may be an identifier of the incremental model, or it may be a different identifier that links a particular incremental model to a particular base model.

[0101] As indicated at 1520, in some embodiments, it can be evaluated whether the incremental model identified for a specified version of the machine learning model is a different incremental model among different incremental models loaded in memory. For example, memory mapping or other metadata can identify the current incremental model. If not, in some embodiments, the incremental model can be added to the incremental models loaded in memory, as indicated at 1550. For example, a separate data storage area (e.g., storage service 230) can be accessed and the incremental model can be obtained.

[0102] As indicated at 1530, in some embodiments, incremental values ​​for a given input can be generated to generate inferences using the identified incremental model. For example, if the incremental model is a LORA-based incremental model, a memory-loaded incremental model can be used to perform matrix multiplication to stream the generated incremental values ​​for combination with base values ​​generated using the base model. Other incremental value computation techniques may depend on the above description regarding... Figure 9 The various incremental model types discussed.

[0103] As indicated at 1540, in some embodiments, the generated incremental values ​​can be used to complete the generation of inference using base values ​​generated by a base model, which, when combined with the identified incremental model, provides a specified version of a fine-tuned machine learning model. For example, base model values ​​can be computed according to instructions in a container implemented to generate inference and used to combine or otherwise utilize the generated incremental values ​​to complete the generation of inference.

[0104] Figure 16This is a high-level flowchart illustrating various methods and techniques for load-aware routing of managed network endpoints according to some embodiments. As indicated at 1610, in some embodiments, a request to generate inferences using a specified machine learning model from among those associated with the managed network endpoint can be received through the managed network endpoint. For example, the request may include an identifier of the managed network endpoint (e.g., a network address) and an identifier of the requested model.

[0105] As indicated at 1620, in some embodiments, the corresponding workloads of different hosts, which in some embodiments are associated with managed network endpoints, can be evaluated. For example, a cache of workload metrics (discussed below) can be accessed, or other real-time workload information (e.g., the number of in-flight inferred requests) can be obtained. Because the models and corresponding workloads and arrangements of hosts in the managed network endpoints may change frequently, new workload information and mapping information can be obtained if, for example, current information is determined to be outdated or otherwise incorrect.

[0106] As indicated at 1630, in some embodiments, based on evaluation, one of the different hosts can be selected to execute the request. Different selection strategies can be used, including selection strategies specified in requests specific to the managed network endpoint. See below for more information. Figure 17 In detail, one such technique may involve using a randomized weighted selection to initialize potential receivers, and then selecting one of the potential receivers as the chosen host.

[0107] As indicated at 1640, in some embodiments, the selected host may execute a request to generate inference using a corresponding copy of the specified machine learning model. Because the models and corresponding workloads and arrangements of the hosts in the managed network endpoints may change frequently, error handling can be implemented. If a request sent to the host fails or returns an error, a retry mechanism can be implemented. For example, new workload or mapping information can be obtained, and an alternative option can be made and tried.

[0108] Figure 17 This is a high-level flowchart illustrating various methods and techniques for selecting load-aware routing strategies according to some embodiments. As indicated at 1710, in some embodiments, the type of request to generate inferences for a specific machine learning model associated with a managed network endpoint can be determined. Some requests may be streaming requests or other types of requests that depend on the execution of multiple interactions. Thus, the determined type may be for sticky sessions (which may support multiple interactions between the same client and host system).

[0109] As indicated at 1720, if the determined type is a sticky session associated with the request, in some embodiments, the request may be sent to the host associated with the managed network point of the previously processed session, as indicated at 1770. For example, an indication of which host previously processed the session may be stored in a cache.

[0110] As indicated at 1730, in some embodiments, model replica mapping information of the host associated with the managed network endpoint can be accessed. For example, this model replica mapping information could be a local cache, as described above regarding... Figure 11 As indicated at 1740, in some embodiments, two (or more) hosts may be randomly selected based on a replica-based weighting of the specified machine learning model. Random weighting can allow multiple replicas of the same model to exist on the same host, enabling scenarios where randomization is distributed across the replicas (e.g., as opposed to randomization based solely on the host).

[0111] As indicated at 1750, in some embodiments, the inferred number of in-flight requests to generate copies of the selected two or more hosts can be determined. This information can be obtained from the hosts in real time (e.g., as part of heartbeats or other status communications sent to the router) or from previous workload reports sent from the hosts. As indicated at 1760, in some embodiments, a randomly selected host can be selected from a randomly selected host using a minimum number of in-flight requests.

[0112] To ensure that model migration does not affect client applications, deployment techniques that migrate to a new model that replaces the current model can be used to ensure zero downtime for clients. Figure 18 This is a high-level flowchart illustrating various methods and techniques for deploying new models to managed network endpoints with zero downtime, according to some embodiments. As indicated at 1810, in some embodiments, a request to add a new machine learning model to replace the current machine learning model associated with the managed network endpoint may be received. The request may be a batch, chunk, or group replacement request covering all copies of the current model (or only specific copies in other embodiments).

[0113] As indicated at 1820, in some embodiments, a copy of the new model corresponding to a copy of the current model can be placed on a new or existing host associated with the managed network endpoint. For example, a model registry can be used to identify the number and location of existing copies of the current model. Placement decisions can then be made for each copy of the new model and the instructions used for placement (e.g., using the instructions above regarding placement). Figure 12 (The technology discussed).

[0114] After placement, inference requests for the current model received at the managed network endpoint can be routed to a copy of the new model, as indicated at 1830. As indicated at 1840, in some embodiments, a copy of the current model can be removed from the managed network endpoint. For example, a request to remove the copy can be executed, similar to a request for a scaling-down operation.

[0115] The methods described herein can be implemented in various embodiments using any combination of hardware and software. For example, in one embodiment, the methods can be implemented in one or more computer systems (e.g., such as...). Figure 19 The method is implemented on or across one or more computer systems, including one or more processors that execute program instructions stored on one or more computer-readable storage media coupled to the processors. The program instructions can implement the functions described herein (e.g., implement the functions of various servers and other components of a network-based virtual computing resource provider as described herein). The various methods shown in the accompanying drawings and described herein represent example embodiments of the methods. The order of any method can be changed, and various elements can be added, reordered, combined, omitted, modified, etc.

[0116] The embodiments for dynamic endpoint management of heterogeneous machine learning models described herein can be executed on one or more computer systems that can interact with a variety of other devices. One such computer system consists of... Figure 19 As shown. In different embodiments, the computer system 2000 can be any device of various types, including but not limited to personal computer systems, desktop computers, laptop computers, notebook or netbook computers, mainframe computers, handheld computers, workstations, network computers, cameras, set-top boxes, mobile devices, consumer devices, video game consoles, handheld video game devices, application servers, storage devices, peripheral devices such as switches, modems, routers, or generally any type of computing device, computing node, compute node, or electronic device.

[0117] In the illustrated embodiment, computer system 2000 includes one or more processors 2010 coupled to system memory 2020 via input / output (I / O) interface 2030. Computer system 2000 further includes a network interface 2040 coupled to I / O interface 1030, and one or more input / output devices 1050, such as cursor control device 1060, keyboard 1070, and display 1080. Display 1080 may include a standard computer monitor and / or other display systems, technologies, or devices. In at least some embodiments, input / output device 1050 may also include a touch- or multi-touch-enabled device, such as a tablet or tablet computer, through which a user types input using a stylus device and / or one or more fingers. In some embodiments, embodiments may be implemented using a single instance of computer system 2000, while in other embodiments, multiple such systems or multiple nodes constituting computer system 2000 may host different portions or instances of the embodiments. For example, in one embodiment, some elements may be implemented through one or more nodes of the computer system 2000 that are different from those nodes that implement other elements.

[0118] In various embodiments, computer system 2000 may be a single-processor system including one processor 2010, or a multiprocessor system including several processors 2010 (e.g., two, four, eight, or another suitable number). Processor 2010 may be any suitable processor capable of executing instructions. For example, in various embodiments, processor 2010 may be a general-purpose or embedded processor implementing any of a variety of instruction set architectures (ISAs), such as x86, PowerPC, SPARC, or MIPS ISA, or any other suitable ISA. In a multiprocessor system, each processor in processor 2010 may typically, but not necessarily, implement the same ISA.

[0119] In some embodiments, at least one processor 2010 may be a graphics processing unit (GPU). A GPU can be considered a dedicated graphics rendering device for a personal computer, workstation, game console, or other computing or electronic device. Modern GPUs can be highly efficient in manipulating and displaying computer graphics, and their highly parallel architecture can make them more efficient than a typical CPU for a range of complex graphics algorithms. For example, a GPU may implement graphics primitive operations in a manner that makes performing multiple graphics primitive operations much faster than drawing directly to the screen using a host central processing unit (CPU). In various embodiments, graphics rendering may be implemented at least in part by program instructions that execute on one of such GPUs or in parallel on two or more such GPUs. The GPU may implement one or more application programmer interfaces (APIs) that allow programmers to invoke the functions of the GPU. Suitable GPUs are commercially available from vendors such as NVIDIA Corporation and ATI Technologies (AMD).

[0120] System memory 2020 may store program instructions and / or data accessible to processor 2010. In various embodiments, system memory 2020 may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory. In the illustrated embodiments, program instructions and data that perform the desired function, such as program instructions and data for performing interpretation tasks for computer vision tasks as described above, are shown stored in system memory 2020 as program instructions 2025 and data storage 2035. In other embodiments, program instructions and / or data may be received, transmitted, or stored on different types of computer-accessible media or on similar media separate from system memory 2020 or computer system 2000. Generally, non-transitory computer-readable storage media may include storage media or memory media, such as magnetic or optical media, for example, a disk or CD / DVD-ROM coupled to computer system 2000 via I / O interface 2030. Program instructions and data stored on a computer-readable medium can be transmitted via a transmission medium or signal (such as an electrical signal, electromagnetic signal, or digital signal), which can be transmitted via a communication medium such as a network and / or a wireless link, as implemented via a network interface 2040.

[0121] In some embodiments, I / O interface 2030 can coordinate I / O traffic between processor 2010, system memory 2020, and any peripheral devices within the device, including network interface 2040 or other peripheral interfaces such as input / output device 2050. In some embodiments, I / O interface 2030 can perform any necessary protocols, timing, or other data transformations to convert data signals from one component (e.g., system memory 2020) into a format suitable for use by another component (e.g., processor 2010). In some embodiments, I / O interface 2030 may include support for devices attached via various types of peripheral buses, such as the Peripheral Component Interconnect (PCI) bus standard or variants of the Universal Serial Bus (USB) standard. In some embodiments, the functionality of I / O interface 2030 may be divided into two or more separate components, such as a northbridge and a southbridge. Additionally, in some embodiments, some or all of the functionality of I / O interface 2030, such as the interface to system memory 2020, may be directly incorporated into processor 2010.

[0122] Network interface 2040 may allow data to be exchanged between computer system 2000 and other devices (such as other computer systems) attached to the network, or between nodes of computer system 2000. In various embodiments, network interface 2040 may support, for example, communication over wired or wireless general data networks (such as any suitable type of Ethernet network); communication over telecommunications / telephone networks (such as analog voice networks or digital fiber optic communication networks); communication over storage area networks (such as fiber optic SANs); or communication over any other suitable type of network and / or protocol.

[0123] In some embodiments, the input / output device 2050 may include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other devices adapted to input or retrieve data through one or more computer systems 2000. Multiple input / output devices 2050 may be present in the computer system 2000 or distributed across various nodes of the computer system 2000. In some embodiments, similar input / output devices may be separate from the computer system 2000 and may interact with one or more nodes of the computer system 2000 via wired or wireless connections, such as through a network interface 2040.

[0124] like Figure 19As shown, memory 2020 may include program instructions 2025 and data storage 2035, the program instructions implementing various methods and techniques as described herein, and the data storage containing various data accessible to the program instructions 2025. In one embodiment, program instructions 2025 may include software elements as described herein and as illustrated in the accompanying drawings. Data storage 2035 may include data that can be used in an embodiment. In other embodiments, other or different software elements and data may be included.

[0125] Those skilled in the art will understand that the computer system 2000 is merely illustrative and not intended to limit the scope of the technology described herein. Specifically, computer systems and apparatuses may include any combination of hardware or software capable of performing the indicated functions, including computers, personal computer systems, desktop computers, laptop computers, notebook or netbook computers, mainframe computers, handheld computers, workstations, network computers, cameras, set-top boxes, mobile devices, network devices, internet devices, PDAs, wireless telephones, pagers, consumer devices, video game consoles, handheld electronic game devices, application servers, storage devices, peripheral devices such as switches, modems, routers, or generally any type of computing or electronic device. The computer system 2000 may also be connected to other devices not shown, or alternatively may operate as a standalone system. Furthermore, in some embodiments, the functionality provided by the illustrated components may be combined in fewer components or distributed across other components. Similarly, in some embodiments, the functionality of some of the illustrated components may not be provided and / or other additional functionality may be available.

[0126] Those skilled in the art will also understand that although the items are shown as being stored in memory or storage during use, these items, or portions thereof, may be transferred between memory and other storage devices for memory management and data integrity purposes. Alternatively, in other embodiments, some or all of the software components may be executed in memory on another device and communicate with the illustrated computer system via inter-computer communication. Some or all of the system components or data structures may also be stored (e.g., as instructions or structured data) on a computer-accessible medium or portable article of manufacture for retrieval by a suitable driver, examples of which are described above. In some embodiments, instructions stored on a non-transitory computer-accessible medium separate from computer system 2000 may be transmitted to computer system 2000 via a transmission medium or signal (such as electrical, electromagnetic, or digital signals transmitted via communication media such as networks and / or wireless links). Various embodiments may further include receiving, transmitting, or storing instructions and / or data implemented according to the foregoing description on a computer-accessible medium. Therefore, the invention can be practiced with other computer system configurations.

[0127] Note that any component of any distributed system embodiment or its components described in the distributed system embodiments herein may be implemented as one or more network services. In some embodiments, a network-based service may be implemented by software and / or hardware systems designed to support interoperable machine-to-machine interactions over a network. A network-based service may have an interface described in a machine-processable format such as a Network Services Description Language (WSDL). Other systems may interact with the network service in a manner specified by the description of the network-based service's interface. For example, a network-based service may describe various operations that other systems can invoke and may describe specific application programming interfaces (APIs) that, when requesting various operations, may be expected to be followed by other systems.

[0128] In various embodiments, a network-based service can be requested or invoked using a message that includes parameters and / or data associated with the network-based service request. Such messages can be formatted according to a specific markup language such as Extensible Markup Language (XML) and / or encapsulated using a protocol such as Simple Object Access Protocol (SOAP). To execute a network service request, the network-based service client can use an Internet-based application-layer transport protocol such as Hypertext Transfer Protocol (HTTP) to assemble a message including the request and deliver the message to an addressable endpoint (e.g., a Uniform Resource Locator (URL)) corresponding to the network service.

[0129] In some embodiments, web services may be implemented using representative state transfer (“RESTful”) technology instead of message-based technology. For example, web services implemented according to RESTful technology can be invoked by parameters included in HTTP methods such as PUT, GET, or DELETE, rather than being wrapped in SOAP messages.

[0130] Embodiments of this disclosure may be described in accordance with the following terms: Clause 1. A system comprising: Multiple computing devices, each including at least one processor and memory, implement a machine learning service, wherein the machine learning service is configured to: The managed network endpoint provides access to the multiple different machine learning models by receiving a request from one or more clients of the machine learning service to invoke a specified different machine learning model among multiple different machine learning models including the machine learning model hosted at one or more computing resources associated with the managed network endpoint. The following are monitored for the network endpoints under management: The event of rebalancing the multiple different machine learning models among the multiple computing resources; Events that require scaling the multiple computing resources or the multiple different machine learning models; A placement decision is made in response to the detection of an event requiring rebalancing or a scaling event, the placement decision selecting a computing resource from multiple computing resources to host one of multiple machine learning models, the selection being at least in part based on a determination that the computing resource meets the resource requirements of the machine learning model; and Place the machine learning model on the selected computing resource.

[0131] Clause 2. The system according to Clause 1, wherein the event to be rebalanced is detected, and wherein the machine learning model is moved from the other computing resource based on a performance metric of the selected computing resource or another computing resource among the plurality of computing resources.

[0132] Clause 3. The system according to Clause 1, wherein the event to be scaled is detected, and wherein the event to be scaled increases or decreases the amount of computing resources.

[0133] Clause 4. The system according to Clause 1, wherein the event to be scaled is detected, and wherein the event to be scaled increases or decreases the number of at least one copy of the plurality of different machine learning models.

[0134] Clause 5. A method comprising: The machine learning service detects placement events for machine learning models associated with managed network endpoints, wherein the managed network endpoints provide access to the multiple different machine learning models by receiving a request from one or more clients of the machine learning service to invoke a specified different machine learning model among multiple different machine learning models including the machine learning model. The machine learning service selects computing resources from a plurality of computing resources associated with the managed network endpoint to host the machine learning model, the selection being at least in part based on a determination that the computing resources meet the resource requirements of the machine learning model; and The machine learning service places the machine learning model at a selected computing resource in response to the placement event.

[0135] Clause 6. The method according to Clause 5, wherein the placement event is detected in response to a rebalancing event for rebalancing the plurality of different machine learning models among the plurality of computing resources, and wherein the machine learning model is moved from the other computing resource based on a performance metric of the selected computing resource or another computing resource among the plurality of computing resources.

[0136] Clause 7. The method according to Clause 5, wherein the placement event is detected in response to a scaling event for the plurality of computing resources associated with the managed network endpoint.

[0137] Clause 8. The method according to Clause 5, wherein the placement event is detected in response to a scaling event performed according to a scaling policy specified through the interface of the machine learning service.

[0138] Clause 9. The method according to Clause 8, wherein the scaling strategy specifies the machine learning model.

[0139] Clause 10. The method according to Clause 5, wherein the placement event is detected in response to a scaling event that extends from no copies of the machine learning model to at least one copy of the machine learning model.

[0140] Clause 11. The method according to any one of Clauses 5 to 10, wherein the resource requirements are specified through the interface of the machine learning service.

[0141] Clause 12. The method according to any one of Clauses 5 to 11, wherein the managed network endpoint is created in response to one or more requests received through the interface of the machine learning service to create the managed network endpoint and add the plurality of different machine learning models to the managed network endpoint.

[0142] Clause 13. The method according to any one of Clauses 5 to 12, wherein the placement event for the machine learning model is adding a copy of the machine learning model to the plurality of computing resources.

[0143] Clause 14. One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to perform: Detect placement events for a machine learning model associated with a network endpoint managed by a machine learning service, wherein the managed network endpoint provides access to the multiple different machine learning models by receiving a request from one or more clients of the machine learning service to invoke a specified different machine learning model among multiple different machine learning models including the machine learning model. Computing resources are selected from a plurality of computing resources associated with the managed network endpoint to host the machine learning model, the selection being at least in part based on a determination that the computing resources meet the resource requirements of the machine learning model; and This allows the machine learning model to be placed on the selected computing resources.

[0144] Clause 15. One or more non-transitory computer-readable storage media as described in Clause 14, wherein the placement event is detected in response to a rebalancing event for rebalancing the plurality of different machine learning models among the plurality of computing resources, and wherein the machine learning model is moved from the other computing resource based on a performance metric of the selected computing resource or another computing resource among the plurality of computing resources.

[0145] Clause 16. One or more non-transitory computer-readable storage media as described in Clause 14, wherein the placement event is detected in response to a scaling event of the plurality of computing resources associated with the managed network endpoint.

[0146] Clause 17. One or more non-transitory computer-readable storage media as described in Clause 14, wherein the placement event is detected in response to a scaling event performed according to a scaling policy specified through the interface of the machine learning service.

[0147] Clause 18. One or more non-transitory computer-readable storage media as described in Clause 14, wherein a placement event is detected in response to a scaling event to expand from no copy of the machine learning model to at least one copy of the machine learning model.

[0148] Clause 19. One or more non-transitory computer-readable storage media pursuant to any one of Clauses 14 to 18, wherein the resource requirements are specified through the interface of the machine learning service.

[0149] Clause 20. One or more non-transitory computer-readable storage media pursuant to any one of Clauses 14 to 19, wherein the placement event for the machine learning model is the addition of a copy of the machine learning model to the plurality of computing resources.

[0150] Clause 21. A system comprising: Multiple computing devices, each including at least one processor and memory, implement machine learning services. The machine learning service is configured as follows: The managed network endpoint provides access to the multiple different machine learning models by receiving a request from one or more clients of the machine learning service to invoke a specified different machine learning model among multiple different machine learning models including the machine learning model hosted at one or more computing resources associated with the managed network endpoint. For one of the at least multiple different machine learning models, the machine learning service is configured to: The at least one machine learning model is determined to be the base model of the fine-tuned machine learning model; Identify one or more different machine learning models among the plurality of different machine learning models that are corresponding incremental models relative to the base model, wherein a corresponding combination of the incremental model and the base model produces a corresponding version of the fine-tuned machine learning model; and This allows both the base model and the corresponding incremental model to be placed on the host system, wherein the host system generates a corresponding inference for a request to invoke a corresponding version of the corresponding version of the fine-tuned machine learning model.

[0151] Clause 22. The system as described in Clause 21, wherein the placement is caused in response to a scaling event or rebalancing event detected for the managed network endpoint.

[0152] Clause 23. The system according to any one of Clauses 21 to 22, wherein the host system is configured to: Receive a request to generate inference using one of the corresponding versions of the fine-tuned machine learning model at the host system; It is determined that the corresponding incremental model is loaded into the host system's memory; Generate incremental values ​​for the corresponding incremental model in response to the given input of the request; and The incremental value is used to complete the generation of the inference by combining the base value generated by the base model with the incremental value.

[0153] Clause 24. The system pursuant to any one of Clauses 21 to 23, wherein the managed network endpoint is created in response to one or more requests received through the interface of the machine learning service to create the managed network endpoint and add the plurality of different machine learning models to the managed network endpoint.

[0154] Clause 25. A method comprising: The machine learning service receives a request to place a machine learning model on the host system of the machine learning service, wherein the machine learning model is the base model of the fine-tuned machine learning model. The machine learning service identifies one or more different machine learning models as corresponding incremental models relative to the base model, wherein a corresponding combination of the incremental model and the base model produces a corresponding version of the fine-tuned machine learning model; and The machine learning service places both the base model and the corresponding incremental model on the host system, wherein the host system generates a corresponding inference for a request to invoke a corresponding version of the corresponding version of the fine-tuned machine learning model.

[0155] Clause 26. The method according to Clause 25, wherein the host system is associated with a managed network endpoint, and wherein the placement is in response to a scaling event request for increasing the number of copies of the machine learning model at the managed network endpoint.

[0156] Clause 27. The method according to any one of Clauses 25 to 26, wherein the host system is associated with a managed network endpoint, and wherein the placement is in response to a rebalancing event request for moving the machine learning model from the current host system to the managed network endpoint.

[0157] Clause 28. The method according to any one of Clauses 25 to 27, wherein the host system is associated with a managed network endpoint, and wherein a model registry is updated to include the placement of the base model and the corresponding incremental model on the host system, thereby routing subsequent requests for invoking a corresponding version of the corresponding version of the fine-tuned machine learning model to the host system.

[0158] Clause 29. The method according to any one of Clauses 25 to 27, wherein the host system is associated with a managed network endpoint, and wherein the placement is in response to a request to add the machine learning model to the managed network endpoint.

[0159] Clause 30. The method according to any one of Clauses 25 to 29 further comprises: Receive a request to generate inference using one of the corresponding versions of the fine-tuned machine learning model at the host system; It is determined that the corresponding incremental model is loaded into the host system's memory; Generate incremental values ​​for the corresponding incremental model in response to the given input of the request; and The incremental value is used to complete the generation of the inference by combining the base value generated by the base model with the incremental value.

[0160] Clause 31. The method according to any one of Clauses 25 to 27 and 30, wherein the host system is associated with a managed network endpoint, and wherein the placement is in response to a scaling event request to scale from no replica of the machine learning model to at least one replica of the machine learning model.

[0161] Clause 32. The method according to any one of Clauses 25 to 31, wherein different corresponding tuning datasets are used to train the one or more different machine learning models.

[0162] Clause 33. The method according to any one of Clauses 25 to 32, wherein the host system is one of a plurality of different host systems associated with a network endpoint managed by the machine learning service, wherein the base model and the plurality of incremental models are included in a plurality of different machine learning models associated with the managed network endpoint, and wherein the managed network endpoint is created in response to one or more requests received through the interface of the machine learning service to create the managed network endpoint and add the plurality of different machine learning models to the managed network endpoint.

[0163] Clause 34. One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to implement a machine learning service, the machine learning service implementing: Receive a request to place a machine learning model on the host system of the machine learning service, wherein the machine learning model is the base model of the fine-tuned machine learning model; Identify one or more different machine learning models as corresponding incremental models relative to the base model, wherein a corresponding combination of the incremental model and the base model produces a corresponding version of the fine-tuned machine learning model; and This allows both the base model and the corresponding incremental model to be placed on the host system, wherein the host system generates a corresponding inference for a request to invoke a corresponding version of the corresponding version of the fine-tuned machine learning model.

[0164] Clause 35. One or more non-transitory computer-readable storage media as described in Clause 34, wherein the host system is associated with a managed network endpoint, and wherein the placement is in response to a scaling event request for increasing the number of copies of the machine learning model at the managed network endpoint.

[0165] Clause 36. One or more non-transitory computer-readable storage media pursuant to any one of Clauses 34 to 35, wherein the host system is associated with a managed network endpoint, and wherein the placement is in response to a rebalancing event request for moving the machine learning model from the current host system to the managed network endpoint.

[0166] Clause 37. One or more non-transitory computer-readable storage media pursuant to any one of Clauses 34 to 36, wherein the host system is associated with a managed network endpoint, and wherein a model registry is updated to include the placement of the base model and the corresponding incremental model on the host system, thereby routing subsequent requests for invoking a corresponding version of one of the corresponding versions of the fine-tuned machine learning model to the host system.

[0167] Clause 38. One or more non-transitory computer-readable storage media pursuant to any one of Clauses 34 to 36, wherein the host system is associated with a managed network endpoint, and wherein the placement is in response to a request to add the machine learning model to the managed network endpoint.

[0168] Clause 39. One or more non-transitory computer-readable storage media pursuant to any one of Clauses 34 to 38, further comprising: Receive a request to generate inference using one of the corresponding versions of the fine-tuned machine learning model at the host system; It is determined that the corresponding incremental model is loaded into the host system's memory; Generate incremental values ​​for the corresponding incremental model in response to the given input of the request; and The incremental value is used to complete the generation of the inference by combining the base value generated by the base model with the incremental value.

[0169] Clause 40. One or more non-transitory computer-readable storage media pursuant to any one of Clauses 34 to 36 and 39, wherein the host system is one of a plurality of different host systems associated with a network endpoint managed by the machine learning service, wherein the base model and the plurality of incremental models are included in a plurality of different machine learning models associated with the managed network endpoint, and wherein the managed network endpoint is created in response to one or more requests received through the interface of the machine learning service to create the managed network endpoint and add the plurality of different machine learning models to the managed network endpoint.

[0170] Clause 41. A system comprising: Multiple computing devices, each including at least one processor and memory, implement machine learning services. The machine learning service is configured as follows: The managed network endpoint provides access to the multiple different machine learning models by receiving a request from one or more clients of the machine learning service to invoke a specified different machine learning model among multiple different machine learning models including the machine learning model hosted at one or more computing resources associated with the managed network endpoint. The machine learning service receives a request through its managed network endpoint to generate inference using a specified machine learning model from among a plurality of machine learning models associated with the managed network endpoint. Evaluate the corresponding workloads of different hosts for corresponding copies of the specified machine learning model at the routers used for the managed network endpoints, the different hosts being associated with the managed network endpoints; Based on the evaluation, the router selects a different host from the different hosts to execute the request; and The router forwards the request to generate the inference using the corresponding copy of the specified machine learning model to a selected host.

[0171] Clause 42. The system according to Clause 41, wherein, to select one of the different hosts for executing the request, the machine learning service is configured to apply a selection strategy specified through the interface of the machine learning service.

[0172] Clause 43. The system according to any one of Clauses 41 to 42, wherein the corresponding workload of the corresponding copy of the specified machine learning model on the different hosts includes the corresponding number of in-flight inference requests obtained from the different hosts.

[0173] Clause 44. The system pursuant to any one of Clauses 41 to 43, wherein the managed network endpoint is created in response to one or more requests received through the interface of the machine learning service to create the managed network endpoint and add the plurality of different machine learning models to the managed network endpoint.

[0174] Clause 45. A method comprising: The managed network endpoint of the machine learning service receives a request to generate inference using a specified machine learning model from among multiple machine learning models associated with the managed network endpoint. The machine learning service evaluates the workload of corresponding replicas of the specified machine learning model on different hosts associated with the managed network endpoints. Based on the evaluation, a different host from the different hosts is selected by the machine learning service to execute the request; and The request to generate the inference using the corresponding copy of the specified machine learning model is executed by one of the selected different hosts.

[0175] Clause 46. The method according to Clause 45 further comprises obtaining at least a portion of the corresponding workload from the model registry of the machine learning service to update the model deployment cache of the router used for the machine learning service.

[0176] Clause 47. The method according to Clause 46, wherein the evaluation of the corresponding model workload includes accessing the model deployment cache of the maintenance router, the model deployment cache including the corresponding workload of the corresponding copy of the specified machine learning model on the different hosts.

[0177] Clause 48. The method according to any one of Clauses 45 to 47, wherein the selection of the one different host among the different hosts for performing the request comprises a selection strategy specified by the application through the interface of the machine learning service.

[0178] Clause 49. The method according to any one of Clauses 45 to 48, wherein the selection of the one different host among the different hosts for performing the request comprises applying a weighted random selection to take into account more than one copy of the specified machine learning model hosted on the different hosts.

[0179] Clause 50. The method according to any one of Clauses 45 to 49, wherein the corresponding workload of a corresponding copy of the specified machine learning model on a different host includes a corresponding number of in-flight inference requests obtained from the different host.

[0180] Clause 51. The method according to any one of Clauses 45 to 50, wherein the selection of the one host for executing the request is further based on a determination of the association of the request with a sticky session.

[0181] Clause 52. The method according to any one of Clauses 45 to 51, after a previous attempt to send the request to another different host among the different hosts has failed, selects one of the different hosts.

[0182] Clause 53. The method according to any one of Clauses 45 to 52, wherein the managed network endpoint is created in response to one or more requests received through the interface of the machine learning service to create the managed network endpoint and add the plurality of different machine learning models to the managed network endpoint.

[0183] Clause 54. One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to implement a machine learning service, the machine learning service implementing: The machine learning service receives a request through its managed network endpoint to generate inference using a specified machine learning model from among a plurality of machine learning models associated with the managed network endpoint. Evaluate the workload of corresponding replicas of the specified machine learning model on different hosts associated with the managed network endpoints; Based on the evaluation, a different host is selected from the different hosts to execute the request; and This enables the execution of the request to generate the inference using the corresponding copy of the specified machine learning model.

[0184] Clause 55. One or more non-transitory computer-readable storage media as described in Clause 54, storing additional program instructions that, when executed on or across the one or more computing devices, cause the one or more computing devices to further implement obtaining at least a portion of the corresponding workload from the model registry of the machine learning service to update the model deployment cache of the router used for the machine learning service.

[0185] Clause 56. One or more non-transitory computer-readable storage media pursuant to any one of Clauses 54 to 55, wherein the selection of the one different host for executing the request comprises a selection strategy specified by the application through the interface of the machine learning service.

[0186] Clause 57. One or more non-transitory computer-readable storage media pursuant to any one of Clauses 54 to 56, wherein the selection of the one different host among the different hosts for performing the request comprises applying a weighted random selection to take into account more than one copy of the specified machine learning model hosted on the different hosts.

[0187] Clause 58. One or more non-transitory computer-readable storage media pursuant to any one of Clauses 54 to 57, wherein the respective workload of a respective copy of the specified machine learning model on a different host comprises a respective number of in-flight inference requests obtained from the different host.

[0188] Clause 59. One or more non-transitory computer-readable storage media according to any one of Clauses 54 to 58, wherein the selection of the one host for executing the request is further based on a determination of the association of the request with a sticky session.

[0189] Clause 60. One or more non-transitory computer-readable storage media pursuant to any one of Clauses 54 to 59, wherein the managed network endpoint is created in response to one or more requests received through the interface of the machine learning service to create the managed network endpoint and add the plurality of different machine learning models to the managed network endpoint.

[0190] The figures and various methods described herein represent example embodiments of the methods. These methods can be implemented in software, hardware, or a combination thereof. The order of the methods can be changed, and various elements can be added, reordered, combined, omitted, modified, etc.

[0191] As will be apparent to those skilled in the art who will benefit from this disclosure, various modifications and alterations can be made. This invention is intended to encompass all such modifications and alterations, and therefore, the above description is to be considered illustrative rather than restrictive.

Claims

1. A system comprising: Multiple computing devices, each including at least one processor and memory, implement a machine learning service, wherein the machine learning service is configured to: Detect placement events for a machine learning model associated with a managed network endpoint, wherein the managed network endpoint provides access to the multiple different machine learning models by receiving a request from one or more clients of the machine learning service to invoke a specified different machine learning model among multiple different machine learning models including the machine learning model; Computing resources are selected from a plurality of computing resources associated with the managed network endpoint to host the machine learning model, the selection being based at least in part on a determination that the computing resources meet the resource requirements of the machine learning model; and The machine learning model is placed on the selected computing resource to complete the response to the placement event.

2. The system of claim 1, wherein the placement event is detected in response to a rebalancing event for rebalancing the plurality of different machine learning models among the plurality of computing resources, and wherein the machine learning model is moved from the other computing resource based on a performance metric of the selected computing resource or another computing resource among the plurality of computing resources.

3. The system of claim 1, wherein the placement event is detected in response to an event that increases the scalability of the plurality of computing resources associated with the managed network endpoint.

4. The system of claim 1, wherein the placement event is detected in response to an event to be scaled, and wherein the event to be scaled increases or decreases the number of at least one copy of the plurality of different machine learning models.

5. A method comprising: The machine learning service detects placement events for machine learning models associated with managed network endpoints, wherein the managed network endpoints provide access to the multiple different machine learning models by receiving a request from one or more clients of the machine learning service to invoke a specified different machine learning model among multiple different machine learning models including the machine learning model. The machine learning service selects computing resources from a plurality of computing resources associated with the managed network endpoints to host the machine learning model, the selection being based at least in part on a determination that the computing resources meet the resource requirements of the machine learning model; as well as The machine learning service places the machine learning model at a selected computing resource in response to the placement event.

6. The method of claim 5, wherein the placement event is detected in response to a rebalancing event for rebalancing the plurality of different machine learning models among the plurality of computing resources, and wherein the machine learning model is moved from the other computing resource based on a performance metric of the selected computing resource or another computing resource among the plurality of computing resources.

7. The method of claim 5, wherein the placement event is detected in response to a scaling event for increasing the plurality of computing resources associated with the managed network endpoint.

8. The method of claim 5, wherein the placement event is detected in response to a scaling event performed according to a scaling policy specified through the interface of the machine learning service.

9. The method of claim 8, wherein the scaling strategy specifies the machine learning model.

10. The method of claim 5, wherein the placement event is detected in response to a scaling event indicating an intention to scale from no copies of the machine learning model to at least one copy of the machine learning model.

11. The method according to any one of claims 5 to 10, wherein the resource requirements are specified through the interface of the machine learning service.

12. The method of any one of claims 5 to 11, wherein the managed network endpoint is created in response to one or more requests received through the interface of the machine learning service to create the managed network endpoint and add the plurality of different machine learning models to the managed network endpoint.

13. The method of any one of claims 5 to 12, wherein the placement event for the machine learning model is adding a copy of the machine learning model to the plurality of computing resources.

14. One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to perform: Detect placement events for a machine learning model associated with a network endpoint managed by a machine learning service, wherein the managed network endpoint provides access to the multiple different machine learning models by receiving a request from one or more clients of the machine learning service to invoke a specified different machine learning model among multiple different machine learning models including the machine learning model. Computing resources are selected from a plurality of computing resources associated with the managed network endpoint to host the machine learning model, the selection being based at least in part on a determination that the computing resources meet the resource requirements of the machine learning model; as well as This allows the machine learning model to be placed on the selected computing resources.

15. The one or more non-transitory computer-readable storage media of claim 14, wherein the placement event is detected in response to a rebalancing event to detect a rebalancing to rebalance the plurality of different machine learning models among the plurality of computing resources, and wherein the machine learning model is moved from the other computing resource based on a performance metric of the selected computing resource or another computing resource among the plurality of computing resources.

Images