Capacity-based load balancing in shared resource pools

Abstract

A system provides capacity-based load balancing across model endpoints of a cloud-based artificial intelligence (AI) model. The system includes a consumption determination engine executable to determine a net resource consumption for processing tasks generated by the client application for input into the workload of the trained machine learning model. The system also includes a load balancer that determines a distribution of available resource capacity in a shared resource pool that includes computing resources at each of the plurality of model endpoints. Based on net resource consumption of the tasks and a distribution of available resource capacity in the shared resources, the load balancer allocates parallelizable tasks of the workload between the computing resources at the plurality of model endpoints.

Description

Background Technology

[0001] Large Language Models (LLMs) are machine learning models that can be used to perform various natural language processing tasks, such as generating and classifying text, answering questions, and translating text from one language to another. Transformer models represent a popular class of machine learning models used in various generative artificial intelligence (AI) and natural language processing applications. A transformer model is a neural network that learns context and meaning by tracking relationships in sequential data. Examples of transformer-based models include GPT (Generative Pre-trained Transformer), OPT (Open Pre-trained Transformer), and the Bloom language model (Big Open Science Open Access Multilingual). Transformer models are typically provided to end users as cloud-based software services.

[0002] Transformer models typically consume significant amounts of storage, and in some cases, they are too large to execute on a single processing node. The amount of storage utilized by each transformer model varies based on the model's unique characteristics, its specific purpose (e.g., its susceptibility to many different types of input parameters), the size of each workload being processed, and the size of the text generated by the model. The extremely high GPU (Graphics Processing Unit) utilization of these models makes them costly to operate and poses challenges to efficient resource management. Summary of the Invention

[0003] According to one implementation, a method provides load balancing of tasks across various model endpoints of a trained machine learning model. The method includes determining the net resource consumption of tasks used to process workloads generated by client applications and input to the trained machine learning model; determining the distribution of available resource capacity across each model endpoint; and allocating parallelizable tasks of the workload to computational resources across multiple model endpoints based on the net resource consumption of the tasks and the distribution of available resource capacity in a shared resource pool.

[0004] This summary provides a simplified overview of some concepts that will be further described in the detailed embodiments described below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0005] This article also describes and lists other implementation methods. Attached Figure Description

[0006] Figure 1 The illustration shows a client-side capacity-based load balancing example system that implements tasks provided to model endpoints of cloud-based artificial intelligence (AI) models.

[0007] Figure 2 The diagram illustrates an example load balancing action of another system that implements client-side capacity-based load balancing for cloud-based AI models.

[0008] Figure 3 The illustration shows an example operation for performing capacity-based load balancing between model endpoints of a cloud-based AI model.

[0009] Figure 4 The illustration shows an example schematic diagram of a processing device suitable for implementing various aspects of the disclosed technology. Detailed Implementation

[0010] In existing systems that deploy artificial intelligence (AI) models as cloud-based services, available resources are typically pooled at regional data centers and dynamically assigned to cloud tenants on demand. When a tenant requests a cloud-based model, the request is directed to a selected model endpoint that executes the model instance, such as a model endpoint corresponding to a data center in the same geographic region as the request origin. As used herein, a “model endpoint” refers to server hardware, typically implemented on one or more virtual machines or servers, configured to execute the computational logic of a cloud-based model. In one implementation, a model endpoint comprises a set of logical endpoints corresponding to one or more servers or one or more virtual machines executing on servers in a regional data center, all configured to execute the core logic of the same AI model. In some systems, a single server can operate multiple model endpoints for different model instances (e.g., the same model or different models). In one implementation, a model endpoint is a single instance of a model and the computational hardware supporting the execution of that instance.

[0011] When a processing request is received at a model endpoint, a certain amount of computing capacity is allocated from field hardware resources to each workload, and this allocated computing capacity is counted in the quota assigned to the requesting client application. As used herein, computing capacity refers to the computing hardware utilized during the processing of computing tasks, such as processors and memory. In various implementations of the disclosed technology, computing capacity is allocated in units of graphics processing units (GPUs) or central processing units and / or CPUs, which include processing units and internal memory (e.g., random access memory (RAM)). In some implementations, computing capacity also includes memory units external to the CPU and GPU for supporting CPU and / or GPU-driven processing tasks.

[0012] Because of this practice of allocating resources to workloads from the geographically nearest model endpoint, available capacity within a regional data center cannot be utilized by tenants in other geographic regions requesting the same type of computing tasks from the same AI model. For example, a workload requested by the first client application running in the western United States cannot be delegated to an endpoint in Asia that is fully capable of performing the requested task. This regional allocation practice results in constantly changing blocks of unused model capacity, also known as “fragmented capacity.”

[0013] Another source of capacity fragmentation in cloud-based AI model services is the assignment of workloads to model endpoints without considering the characteristics of each individual workload. For example, when selecting a model endpoint for a workload, the size of the workload or the size of the other workloads currently being executed by each other model endpoint of the requested cloud-based model is not taken into account. Therefore, it is entirely possible for the compute capacity at one model endpoint (e.g., individual servers or regional data centers) to saturate rapidly (e.g., be completely consumed), while other endpoints (even those receiving the same number of workloads) do not saturate similarly. For example, five large workloads might saturate 98% of the available GPU capacity at one model endpoint, while five smaller workloads might only saturate 15% of the available GPU capacity at another model endpoint with the same hardware and software characteristics. As used herein, compute resources are said to be consumed when they are allocated to support active workloads and therefore cannot be allocated to another requesting process. Allocation is typically achieved by reserving specific resources (e.g., GPUs, RAM sticks) with specific processes, making those specific resources unavailable to other processes.

[0014] The allocation system disclosed herein employs logic to discover and utilize available model endpoints anywhere in the world, and allocates computational tasks to selected model endpoints based on the characteristics of each workload, thereby ensuring that capacity is saturated approximately equally across all model endpoints executing model instances. In the various examples included herein, this logic is shown implemented on the client side, which can save significant costs for cloud-based service providers operating trained machine learning models by relying on hardware within the user's computing environment (e.g., the user's personal hardware or cloud hardware configured on behalf of the user) to perform tasks on behalf of the AI ​​model service. However, it is to be understood and appreciated that the disclosed logic for endpoint discovery and allocation can be implemented as part of a cloud-based service operating trained machine learning models, as a cloud-based service separate from the cloud-based AI model service, or on an edge device owned by the end user.

[0015] According to one implementation, the disclosed resource allocation method provides for creating a global pool of computing resources associated with different globally available model endpoints. All resources in the pool are globally discoverable and available for executing model workloads, regardless of the identity of the requesting process and / or the geographic location of the device executing the requesting process. Efficient utilization of these globally available computing resources is achieved through workload balancing logic executed on the client side (e.g., on devices or systems that support generating workloads and sending them to cloud-based AI models).

[0016] The workload balancing logic disclosed in this paper is "capacity-based" because it depends both on the current distribution of available compute capacity within the global resource pool and on the compute capacity expected to be consumed (e.g., allocated) by each individual workload delegated to resources in the global pool. This capacity expected to be allocated (reserved) for a task or workload is referred to herein as "net resource consumption" and can be understood as having both quantitative and temporal components. In one implementation, the net resource consumption of a task is given by multiplying the number of GPUs reserved to support task execution by the length of time these GPUs are "occupied" by the task and unable to execute other tasks.

[0017] The workload characteristics that affect the net resource consumption of a workload are referred to as resource consumption characteristics in this paper, such as characteristics that affect the total amount of computing resources required to perform a task or workload.

[0018] According to one implementation, the disclosed load balancing logic distributes parallelizable workload tasks across cloud-based endpoints of a cloud-based AI service (such as a converter model) to ensure that the net resource consumption of the assigned tasks is distributed across multiple endpoints within a shared resource pool according to a target allocation distribution. In one implementation, the target allocation distribution is based on (e.g., proportionally) a fractional distribution of available resource capacity among multiple endpoints within the shared resource pool. For example, the target allocation distribution ensures that the total (cumulative) net resource consumption of all tasks delegated from a client to the cloud-based AI service is fractionally distributed among the model endpoints in proportion to the distribution of available resource capacity at those endpoints, while taking into account dynamic fluctuations in the distribution of available capacity over time. In other implementations, the net resource consumption of the delegated tasks is distributed across endpoints according to other distribution logic based on available capacity, and not necessarily proportionally to that distribution. For example, tasks are distributed to fill unused capacity blocks exceeding a certain size, or distributions are created between blocks of a threshold size that reflect the distribution of available capacity within those blocks. In some cases, small capacity blocks are excluded from the distribution (e.g., tasks are not received). For various reasons, the target allocation distribution may not be perfectly proportional to the distribution of available capacity. For example, the size and number of parallelizable tasks in the workload may not be partitioned in a way that precisely matches the target distribution.

[0019] The capacity-based load balancing techniques disclosed herein can be used to ensure that cloud-based endpoints of AI models are always at approximately equal saturation levels, without any hotspots (e.g., locations where capacity is at or near 100% utilization) that could lead to request failures and job unreliability. While the disclosed resource allocation system can be implemented to improve resource utilization efficiency in various types of distributed cloud-based systems, specific efficiency improvements are achieved in systems providing cloud-based transformer models due to the significant resources consumed by instances of these models.

[0020] Figure 1 The illustration shows an example system 100 that implements capacity-based load balancing for tasks provided by multiple endpoints of a cloud-based AI model, offered by a peer-to-peer model service 107. In one implementation, the cloud-based AI model is a transformer model trained to perform natural language processing (NLP) tasks. System 100 includes an application 104 that generates workload 110 for execution by the cloud-based AI model. The system also includes a consumption determination engine 106 and a capacity-based load balancer 112 that performs actions to distribute the workload 110 among multiple different endpoints of the cloud-based AI model.

[0021] Figure 1The cloud-based AI model is supported by three distinct model endpoints, labeled A, B, and C. Each of the three model endpoints A, B, and C includes one or more servers executing model instances(s), such as models 112a, 112b, and 112c. Additionally, each of the three model endpoints includes various computing resources 120, 122, and 124 (e.g., GPUs and / or CPUs) available for performing computational tasks on the AI ​​model. For example, endpoint A includes a first subset of computing resources 120; endpoint B includes a second subset of computing resources 122; and endpoint C includes a third subset of computing resources 124. These subsets of computing resources 120, 122, and 124 are collectively referred to as “shared resource pool 102,” and resources from shared resource pool 102 are used to support different tasks (e.g., transformer model processing tasks) within the client-side workload pointing to model service 107. Shared resource pool 102 includes, for example, groups of servers supporting processing components, such as CPU groups, GPU groups, and the memory utilized by these components. Suppose that the various computing resources of the shared resource pool 102 are distributed along different model endpoints, and the computing resources at each endpoint are available to support tasks performed by (multiple) instances of a trained machine learning model that are executed at the endpoint where the resource is located.

[0022] In addition to model endpoints A, B, and C that execute instances of cloud-based AI models, model service 107 also includes an endpoint discovery mechanism 108, which acts as an intermediary for relaying information about the state of model endpoints A, B, and C back to a capacity-based load balancer 112, which performs load balancing on behalf of model service 107. In one implementation, endpoint discovery mechanism 108 is a cloud-based solution that communicates with the inference engine (not shown) executing on each of the multiple model endpoints A, B, and C in a shared resource pool 102. Through this communication, endpoint discovery mechanism 108 identifies the currently available resource capacity (e.g., unutilized GPU capacity) at each model endpoint. Upon request, this information is provided to capacity-based load balancer 112.

[0023] In addition to the components described above, system 100 also includes client application 104, consumption determination engine 106, and capacity-based load balancer 112. In one implementation, these components execute within a client computing system, which can be understood as including one or more cloud-based servers (e.g., virtual machines) and / or one or more personal computing devices configured on behalf of the client. In other implementations, one or more of consumption determination engine 106 and capacity-based load balancer 112 are located outside the client computing system.

[0024] Typically, client application 104 is the application that generates workloads for model service 107 to execute. In various implementations, client application 104 provides different functionalities and can be any of a variety of application types, including, for example, an email client, an application for composing text or presentation materials, a video chat application, a web plugin, etc. In some implementations, client application 104 executes locally on the end user's device. In other implementations, client application 104 is a web-based application that provides content for display on the end user's device.

[0025] In one example, client application 104 is a plug-in tool for a web browser that provides a chatbot service. By providing input to the plug-in tool's dialog window, the user can ask the chatbot questions for natural language processing by the LLM of model service 107 (e.g., "Show me a 7-day itinerary for visiting Paris"). In another example, client application 104 is a video chat application that includes user-optional options for generating transcripts of meetings and summarizing multiple sections of the meeting. Here, the summarization task is delegated to model service 107.

[0026] Client application 104 provides workload 110 to consumption determination engine 106, which in turn identifies the parallelizable tasks within the workload. As used herein, a “parallelizable task” refers to a workload task that can be executed concurrently and completely independently of each other. For example, if workload 110 is a batch processing request identifying multiple files to be processed identically, then the processing of each individual file is a parallelizable task because such processing can be performed without affecting the processing of any other individual files. It is important to understand that a batch processing request represents one of many different types of workloads that include parallelizable tasks. The logic used to identify parallelizable tasks within a workload uses rules to identify whether the workload comprises multiple parts that can be processed in parallel by different instances of the model, since the result of processing one part does not affect the processing of another part. In some implementations, the logic used to identify parallelizable tasks includes generating a hint submitted to model service 107 to suggest workload parts that can be parallelized. This hint is issued to one of the model instances, and the result is used to identify the workload parts to be parallelized.

[0027] In another example, workload 110 is a language translation task that requests the translation of a book from English to Chinese. Different paragraphs or chapters of the book can be translated in parallel because they have no logical dependencies on each other.

[0028] In one implementation, parallelizable tasks are different files processed within a batch job. These files are all designated as input to the batch job, and the same or substantially the same processing logic is performed on each different file. In other implementations, the consumption determination engine 106 includes logic for identifying parallelizable tasks for different workloads, such as based on characteristics of stored and / or learned AI models and / or requested specific model functions.

[0029] After identifying the parallelizable tasks of the requested workload, the consumption determination engine 106 identifies and evaluates resource consumption characteristics to estimate the net resource consumption 114 for each parallelizable task. In one implementation, workload 110 identifies workload inputs (e.g., files, parameters) and a model identifier (ID) that identifies the target model to be processed (e.g., a cloud-based AI model served by model service 107). For example, the model ID includes the name of a specific transformer model or an API used to reach model service 107. Based on the model identifier and workload inputs, the consumption determination engine 106 identifies the resource consumption characteristics of the workload corresponding to workload 110.

[0030] Typically, resource consumption characteristics are features that can be used to quantify the resource capacity (e.g., GPU utilization) required to perform a task or workload. Example capacity consumption characteristics include features of the input files identified in workload 110, such as the size of each file, the amount of data in each file, the data type in each file, and / or the amount of storage space required to read each file. In some implementations, the consumption determination engine 106 also determines the resource consumption characteristics of the target model, including, for example, the model's identity, its type or category (e.g., a generative pre-trained transformer (GBT)), the nature of the operations the model will perform, and / or the storage space usage of the target model when executing on the input data using the resource consumption characteristics identified in workload 110. Using this training dataset, the AI ​​model is able to determine the expected storage space usage of the AI ​​model when executing on those inputs, based on known inputs requested by a given workload request.

[0031] Based on some or all of the resource consumption characteristics of workload 110 described above, the consumption determination engine 106 outputs net resource consumption 114 for each parallelizable task of the workload. In various implementations, different metrics are used to quantify the net resource consumption of a given parallelizable task. It is further apparent from the following description that it is not necessary to define net resource consumption based on the computation time of a specific hardware component(s). Other units (such as a weighted measure of one or more of the amount of data processed by locks, memory or processor capacity, or the number of clock cycles for executing processing) are also sufficient. As further described below, these task-specific net resource consumptions help to fractionally divide the total workload tasks across model endpoints to match a known fractional distribution of available capacity.

[0032] In one implementation, the net resource consumption is initially determined based on the "tokens" processed by the transformer model, where a token is the basic unit of text or code that the large language model uses to process and generate language. For example, a given transformer model task might need to process 200 tokens, including 50 input tokens and 150 output tokens. In this case, the net resource consumption is given by the computational capacity required to process 150 input and output tokens, where the conversion between tokens and resource consumption (e.g., the amount of computational resources and time required to process one token) is known for the given model.

[0033] For example, suppose the first parallelizable task of workload 110 includes input (e.g., one or more text files or a specific text string) specifying the text to be processed as input and a parameter (e.g., "max_tokens") specifying the length or maximum length of the requested output. In some scenarios, client application 104 selects the output length parameter based on the type of the model and / or the characteristics of the task, such as selecting the output length parameter from a lookup table storing the maximum recommended output length (e.g., a value recommended by model service 107 or a value derived from historical / statistical length data of model service 107). In other implementations, the output length parameter is defaulted to a pre-specified maximum output corresponding to the selected model and / or the type of task the model is performing. Using the specified input text and the parameter specifying the maximum length of the requested output, consumption determination engine 106 predicts that processing a 20-word file will generate an output of 400 words. In this case, the net resource consumption of the first parallelizable task is initially represented as 420 tokens, with one token per word, and there is a known conversion between one token and the amount of given computational resources consumed when processing that token.

[0034] The consumption determination engine 106 repeats this analysis to determine the net resource consumption 114 for each different parallelizable task of workload 110.

[0035] The consumption determination engine 106 outputs the net resource consumption 114 associated with each parallelizable task in the workload to the capacity-based load balancer 112. Upon receiving this information, the capacity-based load balancer 112 requests capacity distribution information 126 from the endpoint discovery mechanism 108 of the model service 107. In response, the endpoint discovery mechanism 108 communicates with model endpoints A, B, and C and discovers the distribution of currently available resource capacity among the multiple endpoints in the shared resource pool 102 (e.g., by retrieving one or more measurements collected at each of the multiple endpoints). To simplify the example shown, it is assumed that the subsets of compute resources residing at each model endpoint (e.g., compute resources 120, 122, and 124) have the same size and the same characteristics. Figure 1 At the corresponding time, computing resource 120 at endpoint A has 50% available capacity; computing resource 122 at endpoint B has 75% available capacity; and computing resource 124 at endpoint C has 25% available capacity. This fractional distribution of available resource capacity within the shared resource pool 102 is included in the capacity distribution information 126 passed back to the capacity-based load balancer 112.

[0036] Upon receiving capacity distribution information 126, the capacity-based load balancer 112 determines the target allocation distribution 128 of parallelizable tasks within the requested workload. Figure 1 In this implementation, the target allocation distribution 128 is a consumption-based distribution (e.g., based on the net resource consumption per task), reflecting a fractional distribution of available resource capacity among multiple endpoints within the shared resource pool 102. More specifically, the capacity distribution information 126 illustrates a capacity distribution ratio of 50:75:25, which simplifies to a 2:3:1 ratio. In this scenario, the capacity-based load balancer 112 uses the same 2:3:1 ratio for the target allocation distribution 128. The capacity-based load balancer 112 then attempts to distribute the parallelizable tasks of the requested workload among endpoints A, B, and C, such that the net resource consumption of the entire workload is fractionally distributed in proportion to the capacity distribution ratio reflected in the capacity distribution information 126 (e.g., representing the distribution of currently available resource capacity among the model endpoints).

[0037] For example, if the requested workload requires 100 parallelizable tasks, and the net total net resource consumption of each individual task is 6000 words (e.g., each word is 1 consumption unit in the example above), then the capacity-based load balancer 112 attempts to delegate tasks between endpoints A, B, and C such that the net resource consumption of the tasks(s) delegated to endpoint A is approximately 2000 words, the net resource consumption of the tasks(s) delegated to endpoint B is approximately 3000 words, and the net resource consumption of the tasks(s) delegated to endpoint C is approximately 1000 words, thereby producing a 2:3:1 distribution ratio that reflects the distribution of available capacity among the model endpoints.

[0038] Then, the capacity-based load balancer 112 distributes parallelizable workload tasks across model endpoints A, B, and C based on the target allocation distribution 128. Due to the number of parallelizable tasks and the net resource consumption associated with each task in the workload, the capacity-based load balancer 112 may not distribute the parallelizable tasks of each workload in a way that precisely matches the target allocation distribution 128. However, given that the same logic is applied globally for hundreds, thousands, or even millions of requests, the overall effect of this logic is that all model endpoints within model service 107 are approximately equally saturated.

[0039] Figure 2 The illustration depicts an example load balancing action of another system 200 implementing client-side capacity-based load balancing for a cloud-based AI model. System 200 is shown as including a client computing platform 210 that stores and executes various software components, including a client application 204, a consumption determination engine 206, and a capacity-based load balancer 208. In one implementation, the client computing platform 210 includes cloud hardware and one or more edge devices (e.g., personal devices communicating with the cloud hardware). In another implementation, the client computing platform 210 is implemented entirely by edge device hardware.

[0040] In the example shown, client application 204 generates batch request 212, which requests the execution of a selected set of processing operations on each of a plurality of input files 216 (e.g., files labeled A, B, C…I). Batch request 212 includes a model ID 218, which uniquely identifies the target cloud-based model to receive batch request 212. In the example shown, model ID 218 is unique and used to identify an LLM named “TextSummarization.llm” trained to perform a text summarization task. In various implementations, model ID 218 identifies any of a plurality of cloud-based LLMs trained to perform different types of NLP tasks. In addition to specifying model ID 218 and a list of input files 216 to be processed, batch request 212 also specifies various model parameters that can be used by the target cloud-based model to identify and execute a specific set of request processing operations.

[0041] It is worth noting that batch request 212 is an example of a workload that includes parallelizable tasks. Each individual input file in input file 216 can be processed by the target cloud-based model in a manner independent of and potentially concurrent with the processing of any other files included in this list.

[0042] Batch request 212 is provided to consumption determination engine 206, which then performs effective actions to quantify the net resource consumption of each parallelizable task to be executed during the execution of batch request 212. In this example, each parallelizable task includes processing one of the input files A through I.

[0043] To determine the net resource consumption of the first input file (“A”), the consumption determination engine 206 determines various resource consumption characteristics related to the processing of file A, such as the size of file “A”, the type of data stored in file A (e.g., images, audio, text), the document category representing file A, the amount of storage space required to read file “A”, the type of operation requested from the target cloud-based model, the identity of the target cloud-based model, the category or type of the target cloud-based model, the size of the output requested or expected from the target cloud-based model, and the storage space occupancy of the target cloud-based model when performing processing on an input of a similar size to file A using the parameters specified in batch processing request 212. In one implementation, the consumption determination engine 206 generates the net resource consumption based on the size of the input data (e.g., the size of file A) and an estimated size of the data that will be output by the cloud-based model processing the input data (e.g., by setting an upper limit on the maximum output size). In scenarios where model inputs and outputs include images (e.g., individual images or a combination of images and text to be processed and / or generated), the number of input and output images, as well as the resolution of the input / output images, also affect the net resource consumption cost.

[0044] Based on some or all of the aforementioned resource consumption characteristics, the consumption determination engine 206 assigns the net resource consumption to each individual file named in the batch request 212. (As in combination...) Figure 1 The unit of net resource consumption can be arbitrary, as long as the metric facilitates a comparison of the relative capacity consumed by executing each of the different parallelizable tasks. In one implementation, the net resource consumption of file A is given based on the sum of (1) the total number of words in file A (e.g., the input to the model) and (2) the total number of words predicted as output by the model based on the processing of file A. For example, the net resource consumption of a task is represented by “tags,” where processing one word corresponds to one tag, and processing certain punctuation marks, non-English characters, and emojis corresponds to a variable (e.g., predefined) number of tags, respectively. In other implementations, the net resource consumption of a task is defined based on the predicted processing time of a given hardware component or set of hardware components.

[0045] For the purpose of illustrating the concept, Figure 2The diagram illustrates the net resource consumption for each input file in input file 216 (e.g., net resource consumption 220, including the consumption value associated with each input file A through I), where each net resource consumption is represented as a number 1, 2, or 3. Here, "2" corresponds to twice the net resource consumption of "1," and "3" corresponds to three times the net resource consumption of 1. These file-by-file resource consumption metrics, along with the corresponding input files for batch request 212 and the input parameters for batch request 212, are provided to the capacity-based load balancer 208.

[0046] Upon receiving batch request 212 and the net resource consumption 220 associated with each distinct file named in the batch request, the capacity-based load balancer 208 requests capacity distribution information from the target LLM's endpoint discovery mechanism 223. This capacity distribution information identifies the current fractional distribution of available capacity in the shared resource pool 202 associated with the target cloud-based model. In one implementation, the shared resource pool 202 is a pool of computing resources available at each of the multiple distinct model endpoints of the target cloud-based model.

[0047] In the example shown, shared resource pool 202 includes compute resources located at different model endpoints that serve different instances of the target cloud-based model (e.g., textsummarization.llm) named in batch request 212. In the example shown, shared resource pool 202 includes compute resources at three different model endpoints (endpoint A, endpoint B, and endpoint C). In one implementation, each of these endpoints corresponds to a different data center serving one or more instances of the LLM (textsummarization.llm), where the data center includes computer hardware that performs the compute tasks for those instances. In different implementations, the target LLM has any number of model endpoints.

[0048] Endpoint discovery mechanism 224 communicates with the inference engine (not shown) executing on each model endpoint in the target LLM to discover the current available compute capacity at each model endpoint. For example, endpoint discovery mechanism 224 obtains a current GPU utilization metric at each endpoint (e.g., determining which GPUs are available for allocation and which are not).

[0049] Endpoint discovery mechanism 224 provides capacity distribution information of model endpoints to capacity-based load balancer 208, which in turn provides capacity distribution information to... Figure 2The available capacity ratio is represented as 226. The available capacity ratio 226 indicates the fractional distribution of the total available compute capacity in the shared resource pool across each model endpoint in the model endpoints. In the example shown, 75% of the compute resources are available at endpoint A, 50% at endpoint B, and 25% at endpoint C. For simplicity, it is assumed that each model endpoint supports approximately the same total compute capacity. Therefore, the above percentages can be used to infer that the available compute capacity at endpoint B is twice that at endpoint C, and the available compute capacity at endpoint A is three times that at endpoint C, resulting in an available capacity ratio of 3:2:1 between A:B:C, as shown in the figure (equal to a fractional distribution where 50% of the total available capacity is at endpoint A, 33.33% at endpoint B, and 16.66% at endpoint C). In practical implementations, the total compute capacity may be unevenly distributed across the model endpoints, but the available capacity distribution ratio can still be determined similarly.

[0050] The capacity-based load balancer 208 uses capacity distribution information (e.g., available capacity ratio 226) to identify a target allocation distribution to distribute parallelizable tasks among model endpoints based on the corresponding net resource consumption of the model endpoints. In this case, the target allocation distribution is equal to the identified available capacity distribution characterized by a 3:2:1 ratio. The capacity-based load balancer 208 then distributes the individual parallelizable tasks of batch request 212 among the model endpoints such that the total net resource consumption of the workload is distributed according to the target allocation distribution. Therefore, the allocation of parallelizable tasks results in the total net resource consumption being distributed across the model endpoints in a manner that matches or otherwise proportional to the fractional distribution of available resource capacity among the multiple endpoints. Figure 2 In the figure, the distribution is illustrated by the consumption distribution ratio 230 that matches the available capacity ratio 226.

[0051] In the example shown, the total net resource consumption for processing files A through I is “18” (each file processing task is associated with a net resource consumption unit represented by a value of 1, 2, or 3, with higher values ​​indicating greater resource consumption). A capacity-based load balancer distributes these tasks according to a net resource consumption distribution consistent with a 3:2:1 available capacity ratio between endpoints A, B, and C. Here, processing of files D through H is delegated to endpoint A; processing of files A through C is delegated to endpoint B; and processing of file I is delegated to endpoint C. As illustrated by the value “1, 2, or 3” associated with each file, endpoint A receives 9 of the 18 total net resource consumption units, endpoint B receives 6 of the 18 net resource consumption units, and endpoint C receives the remaining 3. Therefore, the net resource consumption distribution is 9:6:3, which simplifies to 3:2:1 (as shown in consumption distribution ratio 230) – this final distribution is proportional to the available capacity ratio 226 between the model endpoints.

[0052] It is worth noting that the above process can be repeated for batch requests of any size (including those processing tens, hundreds, or even thousands of input files). As described above, performing load balancing based on endpoint capacity and task-specific resource consumption ensures that all model endpoints are saturated in a roughly equal manner, thereby maximizing the utilization of available computing resources and minimizing the time when available resources are not used.

[0053] In some implementations, the capacity-based load balancer 208 and / or consumption determination engine 206 are included in a software package provided by the same service provider as the management endpoint discovery mechanism 224. This service provider may differ from the service provider of the LLM that benefits from the disclosed load balancing practices. For example, load balancing capabilities are delivered as a third-party service package installed on the client computing platform 210, which can be used in conjunction with various different LLMs. From the service provider's perspective, performing load balancing (as shown and described) on the client computing platform 210 is significantly less costly than in a centralized, cloud-based location because the hardware costs for performing load balancing operations are borne by the end user, not the service provider.

[0054] In real-time applications, the capacity distribution information of model endpoints changes over time. The capacity-based load balancer 208 continuously monitors these changes and dynamically adjusts the available capacity ratio 226 to reflect them. In real-world scenarios where client workloads may take hours to run, the capacity-based load balancer 208 can adjust the available capacity ratio 226 multiple times in response to detected changes in model endpoint capacity. With each change, the capacity-based load balancer 208 dynamically adjusts the target allocation distribution and begins distributing workload tasks according to the changed target allocation distribution.

[0055] Figure 3 The illustration depicts an example operation 300 for performing capacity-based load balancing on a task delivered to a model endpoint of a cloud-based AI model. In one implementation, operation 300 is executed by software running within a client computing platform, such as on one or more virtual machines configured on behalf of a client user. The receiving operation 302 receives a request identifying a workload to be processed by the cloud-based AI model. For example, the request is a batch processing request generated by a locally executed client application, specifying a cloud-based transformer model by name, model type, and / or API call. The identifying operation 304 identifies parallelizable tasks of the workload. In the example where the workload is a batch processing request, each distinct input file processed (e.g., batch files) is a parallelizable task. In other implementations, parallelizable tasks are identified based on characteristics of the stored cloud-based AI model, such as characteristics identifying the processing operations required by the workload and / or the corresponding dependencies of such operations.

[0056] Operation 306 determines the resource consumption characteristics of the workload, such as the size of the (multiple) files or other input data to be processed, the known storage space usage of the cloud-based AI model, and / or the storage space usage when the model uses the parameter set specified in the request and executes on input data whose characteristics match the characteristics of the (multiple) input files. In some implementations, operation 306 also needs to estimate the size of the data that the cloud-based AI model will output in response to processing each or all of the input files. In some implementations, the resource consumption characteristics determined by operation 306 are used to estimate the net resource consumption of each parallelizable task in the parallelizable tasks of the workload.

[0057] Another determination operation 308 determines the distribution of available resource capacity (e.g., GPU compute capacity) across multiple model endpoints, each executing a separate instance of a cloud-based AI model. For example, determination operation 308 needs to communicate with a cloud-based tool that can access compute resource utilization information for each model endpoint and retrieve measurements of compute capacity collected at each model endpoint.

[0058] The allocation operation 310 distributes parallelizable tasks of the workload across model endpoints based on the resource consumption characteristics of the workload and the distribution of available resource capacity across model endpoints. In one implementation, distributing parallelizable tasks of the workload requires distributing parallelizable tasks across model endpoints such that the total net resource consumption of the workload is distributed in a manner that matches or is otherwise proportional to the fractional distribution of available resource capacity across multiple model endpoints.

[0059] Figure 4 An example schematic diagram of a processing device 640 suitable for implementing various aspects of the disclosed technology is illustrated. The processing device 400 includes one or more processor units 402, multiple memory devices 404, a display 406, and other interfaces 408 (e.g., buttons). Each of the multiple processor units 402 may include one or more CPUs, GPUs, etc.

[0060] The (multiple) memory devices 404 typically include volatile memory (e.g., RAM) and non-volatile memory (e.g., flash memory). An operating system 410 (such as the Microsoft Windows® operating system, the Microsoft Windows® Phone operating system, or a specific operating system designed for a gaming device) resides in the (multiple) memory devices 404 and can be executed by the (multiple) processor units 402, but it should be understood that other operating systems may also be used.

[0061] One or more applications 412 (e.g., Figure 1 Client application 104 Figure 1 The consumption of engine 106 is determined. Figure 1 Capacity-based load balancer 112 Figure 1 Endpoint discovery mechanism 108, or Figure 1 The LLM (Local Limiting Model) is loaded into memory device(s) 404 and executed by processor unit(s) 402 on operating system 410. Application 412 can receive input from each other and from various local input devices, such as microphone 434, input accessories 435 (e.g., keyboard, mouse, stylus, touchpad, gamepad, racing steering wheel, joystick), and camera 432. Additionally, application 412 can provide network connectivity (e.g., mobile phone networks, Wi-Fi®, Bluetooth®) by using additional communication transceivers 430 and antennas 438 to receive input from one or more remote devices (such as remote smart devices) over wired or wireless networks. Processing device 400 may also include one or more storage devices 428 (e.g., non-volatile memory). Other configurations may also be employed.

[0062] The processing device 400 also includes a power supply 416, which is powered by one or more batteries or other power sources and provides power to other components of the processing device 400. The power supply 416 can also be connected to an external power source (not shown) that replaces or recharges the built-in battery or other power source.

[0063] Processing device 400 may include various tangible computer-readable storage media and intangible computer-readable communication signals. Tangible computer-readable storage devices may be embodied by any available medium accessible to processing device 400, and include volatile and non-volatile storage media, removable and non-removable storage media. Tangible computer-readable storage media do not include intangible and transient communication signals, and include volatile and non-volatile, removable and non-removable storage media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Tangible computer-readable storage media include RAM, ROM, EEPROM, flash memory or other memory technologies, CDROM, digital universal disc (DVD) or other optical disc storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other tangible medium that can be used to store desired information and is accessible to processing device 400. In contrast to tangible computer-readable storage media, intangible computer-readable communication signals may be embodied as computer-readable instructions, data structures, program modules or other data residing in modulated data signals (such as carrier waves or other signal transmission mechanisms). The term "modulated data signal" refers to a signal in which one or more of its characteristics are set or altered to encode information in the signal. By way of example, intangible communication signals include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media).

[0064] Some implementations may include an article of art. The article of art may include a tangible storage medium (memory device) for storing logic. Examples of storage media may include one or more types of processor-readable storage media capable of storing electronic data, including volatile or non-volatile memory, removable or non-removable memory, erasable or non-erasable memory, writable or rewritable memory, etc. Examples of logic may include various software elements, such as software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, operation segments, methods, procedures, software interfaces, application programming interfaces (APIs), instruction sets, computational code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. In one implementation, for example, the article of art may store executable computer program instructions that, when executed by a computer, cause the computer to perform methods and / or operations according to the described implementation. Executable computer program instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, etc. Executable computer program instructions can be implemented according to predefined computer languages, methods, or syntax, and are used to instruct the computer to perform specific segments of operations. Instructions can be implemented using any suitable high-level, low-level, object-oriented, visual, compiled, and / or interpreted programming languages.

[0065] In some aspects, the techniques described herein relate to a system for improving the utilization of computing hardware distributed across multiple endpoints of a cloud-based artificial intelligence (AI) model, the system comprising: a consumption determination engine, stored in memory and executable to determine the net resource consumption of tasks for processing workloads generated by client applications and input into the cloud-based AI model; and a load balancer, stored in memory and executable to: determine the distribution of available resource capacity in a shared resource pool comprising computing resources at multiple model endpoints executing instances of the cloud-based AI model; and distribute parallelizable tasks of the workload across the computing resources at the multiple model endpoints based on the net resource consumption of the tasks and the distribution of available resource capacity in the shared resource pool.

[0066] In some respects, the techniques described herein relate to a system in which parallelizable tasks of a workload are distributed across multiple model endpoints according to a target allocation distribution based on a fractional distribution of the available resource capacity of the multiple model endpoints across a shared resource pool.

[0067] In some respects, the technique described herein relates to a system in which the target allocation distribution is the distribution of the net resource consumption of tasks in a workload across multiple model endpoints, which is proportional to the fractional distribution of available resource capacity across multiple model endpoints.

[0068] In some respects, the techniques described herein relate to a system in which a consumption determination engine is configured to: determine the resource consumption characteristics of a workload; and, based on the resource consumption characteristics, determine the net resource consumption of each parallelizable task in the parallelizable tasks of the workload.

[0069] In some respects, the techniques described in this paper relate to a system in which the cloud-based AI model is a transformer model, and the net resource consumption of each parallelizable task in the parallelizable task is determined based on the identity of the transformer model and the input set of the workload.

[0070] In some respects, the techniques described herein relate to a system in which the net resource consumption of each parallelizable task is determined at least in part based on the size of the data input to each parallelizable task and an estimated size of the data output obtained in response to the processing of the data input.

[0071] In some respects, the technology described herein relates to a system in which a client application, a consumption determination engine, and a load balancer are executed within a client computing platform that communicates with a model service that implements a cloud-based AI model.

[0072] In some aspects, the techniques described herein relate to a method comprising: determining the net resource consumption of tasks for processing workloads generated by client applications and input into cloud-based AI models; determining the distribution of available resource capacity in a shared resource pool, which includes computing resources at multiple model endpoints executing instances of the cloud-based AI model; and allocating parallelizable tasks of the workload among the computing resources at the multiple model endpoints based on the net resource consumption of the tasks and the distribution of available resource capacity in the shared resource pool.

[0073] In some respects, the techniques described herein relate to a method in which parallelizable tasks of a workload are distributed across multiple model endpoints according to a target allocation distribution based on a fractional distribution of the available resource capacity of the multiple model endpoints across a shared resource pool.

[0074] In some respects, the techniques described herein relate to a method in which the target allocation distribution is the distribution of the net resource consumption of tasks in a workload across multiple model endpoints, which is proportional to the fractional distribution of available resource capacity across multiple model endpoints.

[0075] In some respects, the techniques described herein relate to a method that also includes: determining the resource consumption characteristics of a workload; and determining the net resource consumption of each parallelizable task in a parallelizable task of the workload based on the resource consumption characteristics.

[0076] In some respects, the techniques described in this paper involve a method in which the cloud-based AI model is a transformer model, and the net resource consumption of each parallelizable task in the parallelizable task is determined based on the identity of the transformer model and the input set of the workload.

[0077] In some respects, the techniques described herein relate to a method in which the net resource consumption of each parallelizable task is determined at least in part based on the size of the data input to each parallelizable task and an estimated size of the data output obtained in response to the processing of the data input.

[0078] In some respects, the techniques described herein relate to a method in which the workload is a batch processing request, and the method further includes determining the net resource consumption associated with processing each of a plurality of different files.

[0079] In some aspects, the techniques described herein relate to one or more tangible computer-readable storage media that encode processor-executable instructions for performing computer processes including: determining the net resource consumption of tasks for processing workloads generated by client applications and input into cloud-based AI models; determining the distribution of available resource capacity in a shared resource pool, the shared resource pool comprising computing resources at multiple model endpoints executing instances of cloud-based AI models; and allocating parallelizable tasks of the workload among the computing resources at the multiple model endpoints based on the net resource consumption of the tasks and the distribution of available resource capacity in the shared resource pool.

[0080] In some respects, the techniques described herein relate to one or more tangible computer-readable storage media in which parallelizable tasks of a workload are distributed across multiple model endpoints according to a target allocation distribution based on a fractional distribution of the available resource capacity of the multiple model endpoints across a shared resource pool.

[0081] In some respects, the techniques described herein relate to one or more tangible computer-readable storage media, wherein the target allocation distribution is the distribution of the net resource consumption of tasks in a workload across multiple model endpoints, which is proportional to the fractional distribution of available resource capacity across multiple model endpoints.

[0082] In some respects, the techniques described herein relate to one or more tangible computer-readable storage media, wherein the computer process further includes: determining resource consumption characteristics of a workload; and determining, based on the resource consumption characteristics, the net resource consumption of each parallelizable task in a parallelizable task of the workload.

[0083] In some respects, the techniques described herein relate to one or more tangible computer-readable storage media, wherein the net resource consumption of each parallelizable task is determined at least in part based on the size of the data input to each parallelizable task and an estimated size of the data output obtained in response to the processing of the data input.

[0084] In some respects, the techniques described herein relate to one or more tangible computer-readable storage media, wherein the cloud-based AI model is a transformer model, and the net resource consumption of each parallelizable task in the parallelizable task is determined based on the identity of the transformer model and the input set of the workload.

[0085] In some aspects, the technology described herein relates to a system for improving the utilization of computing hardware distributed across multiple endpoints of a cloud-based artificial intelligence (AI) model, the system comprising: means for determining the net resource consumption of tasks for processing workloads generated by client applications and input into the cloud-based AI model; means for determining the distribution of available resource capacity in a shared resource pool, the shared resource pool including computing resources at multiple model endpoints executing instances of the cloud-based AI model; and means for distributing parallelizable tasks of the workload among the computing resources at the multiple model endpoints based on the net resource consumption of the tasks and the distribution of available resource capacity in the shared resource pool.

[0086] The logical operations described herein are implemented as logical steps in one or more computer systems. Logical operations can be implemented as (1) a processor-implemented sequence of steps executed in one or more computer systems and (2) interconnected machines or circuit modules within one or more computer systems. The implementation method is a matter of choice depending on the performance requirements of the computer system utilized. Therefore, the logical operations constituting the implementations described herein are referred to differently as operations, steps, objects, or modules. Furthermore, it should be understood that logical operations can be executed in any order unless otherwise expressly stated or the language of the claims inherently requires a specific order. The foregoing specification, examples, and data, along with the appendix, provide a complete description of the structure and use of the example implementations.

Claims

1. A system (200) (100) for improving the utilization of computing hardware distributed across multiple endpoints of a cloud-based service, the service operating a trained machine learning model, the system (200) (100) comprising: A resource consumption determination engine (206) (106), stored in memory and executable, determines the net resource consumption (114) for processing tasks generated by the client application (204) (104) and input into the trained machine learning model; and The load balancer, stored in the memory, can execute: Determine the distribution of available resource capacity in a shared resource pool (202) (102), the shared resource pool comprising computing resources (122) (120) at multiple model endpoints executing instances of the trained machine learning model; and Parallelizable tasks of the workload (110) are distributed among the computing resources (122) (120) at the plurality of model endpoints based on the net resource consumption (114) of the task and the distribution of available resource capacity in the shared resource pool (202) (102).

2. The system of claim 1, wherein the parallelizable tasks of the workload are distributed among the plurality of model endpoints according to a target allocation distribution based on a fractional distribution of the available resource capacity across the plurality of model endpoints within the shared resource pool.

3. The system of claim 2, wherein the target allocation distribution is the distribution of the net resource consumption of the tasks in the workload among the plurality of model endpoints, the distribution being proportional to the fractional distribution of the available resource capacity among the plurality of model endpoints.

4. The system of claim 1, wherein the consumption determination engine is configured to: Determine the resource consumption characteristics of the workload; and The net resource consumption of each parallelizable task in the parallelizable tasks for the workload is determined based on the resource consumption characteristics.

5. The system of claim 4, wherein the trained machine learning model is a transformer model, and the net resource consumption for each of the parallelizable tasks is determined based on the identity of the transformer model and the input set of the workload.

6. The system of claim 4, wherein the net resource consumption for each of the parallelizable tasks is determined at least in part based on the size of the data input to each of the parallelizable tasks and an estimated size of the data output in response to the processing of the data input.

7. The system of claim 1, wherein the load balancer determines the distribution of available resource capacity by requesting capacity measurements relating to the availability of computing resources supporting the execution of the model instance at the model endpoint from an endpoint discovery mechanism, the endpoint discovery mechanism being configured to retrieve the capacity measurements from the model endpoint.

8. A method for improving the utilization of computing hardware across multiple model endpoints distributed across a cloud-based service, the cloud-based service operating a trained machine learning model, the method comprising: Determine the net resource consumption (114) for processing tasks generated by the client application (204) (104) and used as input to the trained machine learning model. Determine the distribution of available resource capacity in a shared resource pool (202) (102), the shared resource pool including computing resources (122) (120) at the plurality of model endpoints executing the trained machine learning model; and Based on the distribution of the net resource consumption (114) of the task and the available resource capacity in the shared resource pool (202) (102), the workload (110) is distributed among the computing resources (122) (120) at the plurality of model endpoints as a parallelizable task.

9. The method of claim 8, wherein the parallelizable tasks of the workload are distributed among the plurality of model endpoints according to a target allocation distribution based on a fractional distribution of the available resource capacity across the plurality of model endpoints within the shared resource pool.

10. The method of claim 9, wherein the target allocation distribution is the distribution of the net resource consumption of the tasks in the workload among the plurality of model endpoints, the distribution being proportional to the fractional distribution of the available resource capacity among the plurality of model endpoints.

11. The method of claim 8, further comprising: Determine the resource consumption characteristics of the workload; The net resource consumption of each of the parallelizable tasks in the workload is determined based on the resource consumption characteristics. as well as Request a capacity measurement related to the availability of computing resources that support the execution of the model instance at the model endpoint.

12. The method of claim 11, wherein the trained machine learning model is a transformer model, and the net resource consumption for each of the parallelizable tasks is determined based on the identity of the transformer model and the input set of the workload.

13. The method of claim 11, wherein the net resource consumption for each of the parallelizable tasks is determined at least in part based on the size of the data input to each of the parallelizable tasks and an estimated size of the data output in response to the processing of the data input.

14. The method of claim 8, wherein the workload is a batch processing request, and the method further comprises determining the net resource consumption associated with processing each of a plurality of different files.

15. One or more tangible computer-readable storage media, said one or more tangible computer-readable storage media encoding processor-executable instructions for performing a computer process to improve the utilization of computing hardware distributed across multiple model endpoints of a cloud-based service, said cloud-based service operating a trained machine learning model, said computer process comprising: Determine the net resource consumption (114) for processing tasks generated by the client application (204) (104) and used as input to the trained machine learning model. Determine the distribution of available resource capacity in a shared resource pool (202) (102), the shared resource pool including computing resources (122) (120) at the plurality of model endpoints executing cloud-based AI models; and Based on the net resource consumption (114) of the task and the distribution of available resource capacity in the shared resources, the workload (110) is distributed among the computing resources (122) (120) at the plurality of model endpoints as a parallelizable task.

16. The one or more tangible computer-readable storage media of claim 15, wherein the parallelizable tasks of the workload are distributed among the plurality of model endpoints according to a target allocation distribution based on a fractional distribution of the available resource capacity across the plurality of model endpoints within the shared resource pool.

17. One or more tangible computer-readable storage media of claim 16, wherein the target allocation distribution is the distribution of the net resource consumption of the tasks in the workload among the plurality of model endpoints, the distribution being proportional to the fractional distribution of the available resource capacity among the plurality of model endpoints.

18. One or more tangible computer-readable storage media according to claim 15, wherein the computer process further comprises: Determine the resource consumption characteristics of the workload; as well as The net resource consumption of each of the parallelizable tasks in the workload is determined based on the resource consumption characteristics. as well as Request a capacity measurement related to the availability of computing resources that support the execution of the model instance at the model endpoint.

19. The one or more tangible computer-readable storage media of claim 18, wherein the net resource consumption for each of the parallelizable tasks is determined at least in part based on the size of the data input to each of the parallelizable tasks and an estimated size of the data output in response to the processing of the data input.

20. The one or more tangible computer-readable storage media of claim 18, wherein the trained machine learning model is a transformer model, and the net resource consumption for each of the parallelizable tasks is determined based on the identity of the transformer model and the input set of the workload.

Images