Delay-based resource allocation in a model-as-a-service platform

Abstract

A model-as-a-service (MaaS) platform performs cross-model resource allocation from a shared pool of GPU resources based on model-agnostic metrics generated by a metric normalizer. The metric normalizer receives model-specific benchmark metrics from model providers that define a relationship between resource utilization and token processing according to different model-specific tokenization schemes; receives token-based job metrics related to LLM processing tasks from one or more MaaS components; and determines model-agnostic metrics for a plurality of model pools executing instances of different large language models (LLMs) that generate and process text according to different model-specific tokenization schemes based on the model-specific benchmark metrics and the token-based job metrics. The MaaS platform further includes one or more resource allocation components that dynamically reallocate resources in the shared pool based on the model-agnostic metrics.

Description

Background Technology

[0001] Model-as-a-Service (MaaS) platforms are cloud-based artificial intelligence (AI) platforms that provide developers and enterprises with access to pre-built machine learning models, accessible via application programming interfaces (APIs) managed by a responsible AI layer. These models can be designed to perform a wide range of AI tasks, such as natural language processing (NLP), computer vision, speech recognition, sentiment analysis, recommender systems, and anomaly detection. MaaS simplifies the process of integrating AI capabilities into applications and is offered as a service to enterprises that do not wish to invest significant time and resources in creating and training AI models from scratch. The model services provided through MaaS platforms can be pre-trained, or, in some cases, allow platform users to bring their own data for training and inference. Summary of the Invention

[0002] In one implementation, a Model-as-a-Service (MaaS) platform dynamically allocates GPU resources from a shared pool of models. These model pools execute instances of different Large Language Models (LLMs), which generate and process text according to different model-specific lexicalization schemes. The MaaS platform includes a metric normalizer that: receives model-specific benchmark metrics from model providers, defining the relationship between resource utilization and lexical processing for different model-specific lexicalization schemes; receives lexical-based job metrics related to LLM processing tasks from one or more MaaS platform components; and determines model-independent metrics for multiple model pools based on the model-specific benchmark metrics and the lexical-based job metrics. The MaaS platform also includes one or more resource allocation components that dynamically reallocate resources from the GPU resource pool based on the model-independent metrics determined for the multiple model pools.

[0003] The present invention presents, in a simplified form, the selection of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0004] This article also describes and narrates other implementation methods. Attached Figure Description

[0005] Figure 1 The illustration shows an example system that includes a MaaS platform that performs demand-based graphics processing unit (GPU) resource allocation based on a model-independent large language model (LLM) performance metric generated by a metric normalizer.

[0006] Figure 2The illustration shows another example system including a MaaS platform with a throttling service that throttles incoming LLM processing requests based on a model-independent metric generated by a metric normalizer.

[0007] Figure 3 The illustration shows another example system including a MaaS platform that dynamically allocates GPU resources from a shared resource pool across various model pools based on model-agnostic performance metrics generated by a metric normalizer.

[0008] Figure 4 The illustration shows another example system including a MaaS platform that dynamically reallocates GPU resources from a shared resource pool across various model pools based on model-agnostic performance metrics generated by a metric normalizer.

[0009] Figure 5 The illustration shows an example schematic diagram of a processing device applicable to various aspects of the disclosed technology. Detailed Implementation

[0010] The challenges addressed by the techniques disclosed in this paper arise within MaaS platforms that provide Large Language Models (LLMs) as a Service, which can be configured to perform inference on behalf of end clients, such as enterprises. This MaaS platform provides GPU capacity to support each model deployed as a service within the platform, and this GPU capacity can be freely allocated to support different model instances in a manner independent of identifying the model utilizing the capacity. Within this platform, it is desirable to be able to dynamically reallocate GPU resources in response to dynamically observed changes in model performance. For example, it is desirable to be able to dynamically move resources from a resource pool supporting a first model instance to a resource pool supporting a second model instance, for example, in response to the second model exhibiting an increase in latency exceeding a threshold while the first model exhibits relatively low latency, or in response to detecting that the amount of processing resources being utilized by the second model exceeds its target utilization while the amount being utilized by the first model is significantly lower than its own target utilization. When reallocating GPU resources within a model, it is also desirable to be able to predict how much computing capacity will be gained or lost due to a given reallocation.

[0011] However, in current applications, it is not easy to readily assess how different LLM model instances compare latency or resource utilization at a given point in time. Metrics used to quantify LLM utilization and latency are often based on client load, measured in terms of “terms,” which are the inputs and outputs of the LLM associated with concurrently active requests. For example, LLM utilization can be measured based on the total input and output terms associated with requests processed by the model over a runtime interval (e.g., the past minute). LLM latency metrics similarly rely on terms, as they are typically dependent on term generation time, such as the average time required to generate the first term in response to a query, the average time between the first and last terms generated in response to a query, or the average time between each pair of consecutively generated terms. Term-based metrics (such as utilization and latency) cannot be readily compared across models because different models utilize different termination schemes.

[0012] As used in this paper, a “lexicalization scheme” refers to the lexicalization methods and vocabulary that influence how an LLM processes and generates text. Each LLM includes a lexer (e.g., a software component) that translates natural language text into a stream of lexical units based on a model-specific lexicalization scheme. A lexical is the basic unit for LLM text processing, where each lexical represents a segment of language, such as a single word, phrase, part of a word, or punctuation mark. The lexicalization scheme of each different LLM defines how natural language text is translated into lexical units for LLM processing (e.g., as input) and generation (e.g., as output). A pair of LLMs implementing different lexicalization schemes can receive the same input text stream and translate that text stream into sequences of lexical units of different lengths. For example, some lexicalization schemes assign one lexical unit to each distinct word, while others assign two or more lexical units for certain types of words (e.g., compound words or based on character count) and / or assign lexical units to certain types of punctuation marks. Therefore, a given text query such as "what is a nursery rhyme about alamb?" can be input into a model using the first lexicalization scheme as 8 individual words, while input into another model using the second lexicalization scheme (e.g., a scheme that assigns words to punctuation marks) as 9 individual words. Similarly, there are scenarios where the same text string output by two different models is processed into a first number of output words according to the lexicalization scheme of the first model, while it is processed into a different number of output words according to the lexicalization scheme of the other of the two models.

[0013] Based on the above, it is difficult to meaningfully compare latency measures based on lexical counts or lexical generation time. For example, if a given word is represented as one lexical by one model and as two lexicals by another model, the latency measure "time to the first lexical" does not represent the time required to generate an equivalent text fragment, even if both models ultimately output the same text string. Similarly, latency measures representing the average time between lexical units can differ even if two models spend the same total amount of time generating the same output text string corresponding to different numbers of lexical units.

[0014] Besides the possibility of allocating different numbers of tokens to the same text string, the memory consumed during processing a single token can also be variable across instances of different models and even across the same model supported by different GPU types. To illustrate this, assume model A is deployed in a computing environment with a specific type of GPU and GPU count, and its expected peak utilization is 4 million tokens. This means that when actual utilization reaches its peak at a given time interval (e.g., 1 minute or 5 minutes), the model's performance is known to degrade. Further assume model B is deployed in the same computing environment with an expected peak utilization of 1 million tokens. When model A reaches its peak utilization of 4 million tokens, this does not necessarily correspond to four times the GPU memory utilization observed when model B reaches its peak utilization of 1 million tokens. Similarly, reallocating a quarter of the available computing capacity for model A (e.g., reducing the peak utilization from 4 million tokens to 3 million tokens) does not necessarily double the peak utilization of model B from 1 million tokens to 2 million tokens.

[0015] For all the reasons mentioned above, existing measurements of LLM latency and LLM utilization do not facilitate meaningful cross-model comparisons of GPU resource utilization or latency. This poses a significant challenge when performing any type of demand-based dynamic resource allocation between different versions of the same model, or even instances of the same model deployed on different GPU architectures.

[0016] The techniques disclosed in this paper include a platform-level metric normalizer that takes lexical-based metrics from LLMs and model providers as input and generates model-independent metrics based on these inputs. These model-independent metrics help compare and quantify metrics of utilization and latency. These model-independent metrics provide a basis for performing demand-based resource allocation. As used herein, a “lexical-based metric” is a metric that depends on the lexicalization scheme of a given model, meaning that the lexical vocabulary of that scheme influences the value of the metric. For example, a lexical-based metric could be the number of lexicals, a throughput value identifying the number of lexicals processed per time interval, a lexical-time generation measurement, or even the memory utilization required to process a particular lexical via a particular lexicalization scheme. For the above reasons, two different LLMs utilizing the same amount of resources may report lexical-based metrics that quantify their respective resource utilization based on the number of lexicals processed, with one LLM reporting a significantly higher number of lexicals processed than the other. Therefore, lexical-based utilization measurements cannot be compared to determine which LLM actually has higher utilization.

[0017] In the following description, "LLM" is used to refer to a class of trained models that process and generate lexical units, including text (e.g., letters, numbers, symbols). While this class of trained models includes Natural Language Processing (NLP) models, it also includes multimodal models that can receive prompts that include various types of input (e.g., text, images, audio, and / or video data) and similarly generate various types of output that are not necessarily the same as the input type. For example, a multimodal LLM trained to perform image alteration could receive an image provided by a user (e.g., a picture of a panda eating grass) and a text prompt provided by the user requesting the image alteration (e.g., "alter this image to show the panda eating bamboo instead of grass"). In response, the multimodal LLM converts the binary pixel values ​​of the image to Base64, a set of binary-to-text encoding schemes that convert binary data into a printable sequence of characters. The LLM then translates the input text string and Base64 image representation into an input lexical sequence and processes the input lexical sequence in a manner consistent with traditional LLMs that receive and generate natural language text to produce an output lexical sequence. In this example, the output lexical sequence consists of the text in a modified Base64 image representation, which can be converted back to binary and displayed as the output image. Publicly available examples of multimodal LLMs include the MistralAI model and the Meta AI Large Language Model (LLaMa) model.

[0018] Therefore, although the various examples described below primarily involve LLMs that receive text strings as input and generate text strings as output, the techniques disclosed herein are considered for implementation within a Model-as-a-Service platform, for any type of LLM—including LLMs that receive text, image, audio, and / or video input and / or generate text, image, audio, and / or video output.

[0019] Figure 1 The illustration shows an example system 100, which includes a MaaS platform 102 that performs demand-based GPU resource allocation based on model-agnostic LLM performance metrics generated by a metric normalizer 104. During the loading process, the model service provider loads its LLMs into the MaaS platform 102. These LLMs are referred to as “platform LLMs” in the description below.

[0020] During the initial configuration, one or more different model pools (e.g., model pools 108, 110, and 112) are configured for each different platform LLM in the platform. As used herein, the term "model pool" refers to a networked computer system that includes one or more model endpoints, which may be located in different geographical locations, each endpoint hosting (executing) one or more model instances of the same model (LLM).

[0021] exist Figure 1 In the example, model pool 108 includes multiple endpoints (e.g., endpoints A through N), each hosting one or more instances of a Generative Pre-trained Transformer 4 (GPT-4) model; model pool 110 includes multiple endpoints, each hosting one or more instances of a popular large language model, Meta AI (LLaMA); and model pool 112 includes multiple endpoints hosting instances of a Big Science Open Access Multilingual (BLOOM) model. In one implementation, each endpoint within a model pool is created by or on behalf of an end user (e.g., during the initial load configuration process) to perform modeling on behalf of the end user. Gateway 140 of MaaS platform 100 performs user authentication on each incoming request and then forwards each successfully authenticated request to a model pool hosting the model type (e.g., GPT-4, Bloom) identified by the request. Each model pool further includes a routing layer 114 that routes incoming requests to model endpoints configured on behalf of the requesting user(s) and routes outgoing requests back to the corresponding user-configured endpoints.

[0022] As used herein, a "model endpoint" refers to server hardware, typically implemented on one or more virtual machines or servers configured to execute computational logic for training machine learning models. In one implementation, a model endpoint comprises a collection of logical endpoints corresponding to one or more servers or one or more virtual machines running on servers in a regional data center, all configured to execute the core logic of the trained machine learning model. In another implementation, a model endpoint comprises a single instance of a model and the computational hardware supporting the execution of that instance.

[0023] In one implementation, each model instance (e.g., model instance 1) within a different model instance runs inside a container that executes an agent that reports certain lexical-based job metrics 118 back to the metric normalizer 104. Examples of lexical-based job metrics 118 include job-specific latency metrics (e.g., quantifying the latency of a single processing job), job-specific utilization metrics (e.g., quantifying the resource utilization of a single job), and lexical count metrics that quantify the number of input and output lexicals processed to answer each received LLM query. Lexical-based job metrics 118 are metrics that quantify aspects of a single processing job, which are computed based on the lexicalization scheme of a given model and cannot be directly compared across models without some type of normalization to account for different lexicalization schemes (e.g., similar to currency with conversion rates). For example, a lexical-based job metric that identifies the number of tokens (e.g., the number of input and / or output tokens processed) in a single processing job can be understood as “dependent” on the lexicalization scheme of a given model, because (as discussed elsewhere in this paper) the same input / output string can correspond to different numbers of tokens in different lexicalization schemes. Similarly, a lexical-based job metric that quantifies the resource utilization of a single job “depends” on a specific lexicalization scheme by quantifying current or maximum memory utilization based on the input and output tokens processed by a given model instance. Likewise, a lexical-based job metric that quantifies latency “depends” on a specific lexicalization scheme because it is calculated based on the specific time intervals associated with tokens in a given lexicalization scheme. For example, common lexical-based latency metrics include Time to First Token (TTFT), which measures the speed at which an LLM can produce the first token in a response, and Time Between Tokens (TBT), which measures the consistency of an LLM model producing tokens at regular intervals.

[0024] It is worth noting that lexical-specific job metrics 118 (such as metrics that quantify utilization based on the number of lexicals processed or quantify latency based on the time taken to process lexicals) are not easily compatible across models implementing different lexicalization schemes because, as mentioned above, different lexicalization schemes can use different numbers of lexicals to embed the same text string, and the memory utilization associated with processing a single lexical varies between models and even between model instances deployed in different memory architectures (such as different GPU types).

[0025] In some implementations, such as those discussed in this article... Figure 2 In the example explored, metric normalizer 104 receives additional lexical-based job metrics from other components within MaaS platform 102, such as throttling service 121 (shown in gateway 140), which throttles user access to model resources based on allocated customer quotas. For example, throttling service 121 is shown as communicating input / output lexical counts 123 associated with different LLM processing requests to metric normalizer 104. Throttling service 121 determines how much of the customer-allocated quota was consumed for each granted request based on the corresponding output of metric normalizer 104.

[0026] A key function of the metric normalizer 104 is to transform or normalize the lexical-based metrics (e.g., 118, 123) received during active LLM operations into corresponding metrics that can be meaningfully compared across models. This transformation or normalization is facilitated by stored model-specific benchmark metrics 120.

[0027] In one implementation, the metric normalizer 104 experimentally determines model-specific benchmark metrics 120 by performing internal benchmarking and load testing on various models deployed on different GPU architectures on the MaaS platform. In other implementations, a model service provider (not shown) provides some or all of the model-specific benchmark metrics 120 to the MaaS platform 102.

[0028] In implementations of resource allocation based on resource utilization (e.g.) Figure 2 and Figure 3 Model-specific benchmark metric 120 defines the relationship between resource utilization and lexical processing for a specific model-specific lexicalization scheme in different model-specific lexicalization schemes. Model-specific benchmark metric 120 used by metric normalizer 104 determines model-independent metric 122 corresponding to lexical-based job metric 118.

[0029] As used herein, the term "model-independent metric" refers to a metric represented according to a model-independent unit type, which facilitates direct comparison across different models and GPU architectures without transformation or normalization. Utilization metrics that are strictly based on the lexical representation of the request or usage are not model-independent because, as mentioned above, the same number of lexical units can correspond to different amounts of GPU capacity when processed by different models. In contrast, a "model-independent utilization metric" is a model-independent utilization quantification that can be directly compared across different models without transformation or normalization.

[0030] A model-independent metric derived about one model instance can be directly compared with the same model-independent metric derived about another model instance (e.g., a different model type and / or a different GPU architecture) without performing any intermediate unit conversions, scaling, or normalization. An example of a model-independent metric is the "pre-defined throughput unit (PTU)," which is based on... Figure 2 It was described in detail, and regarding Figure 3 It was also cited. Another example of a model-independent metric is a latency metric that has been normalized across different lexicalization schemes, such as those concerning... Figure 4 To be discussed in more detail.

[0031] In various implementations, the model-independent metric 122 generated by the metric normalizer 104 is used to facilitate the allocation of different types of shared resources. (This is in the context of the paper...) Figure 2 In the discussed implementation, gateway 140 provides lexical-based usage requests to metric normalizer 104, these requests defining the requested number of model lexicals associated with a specific model instance. Metric normalizer 104 translates each lexical-based usage request in the lexical-based usage requests into a corresponding model-agnostic utilization metric, which represents the number of units of a model-agnostic unit type (e.g., GPU capacity) required to process the corresponding lexical-based usage request. The model-agnostic utilization metric computed for each processing request can be used by MaaS platform 302 to directly compare the sizes of different processing requests and perform centralized, model-agnostic request throttling to enforce memory utilization quotas that can be used across different platform LLMs. For example, an end user subscribes to a single compute quota tracked according to model-agnostic compute capacity units (e.g., PTU), which can be redeemed for compute tasks performed by any or more different platform LLMs. This example relates to... Figure 2 Detailed discussion and explanation.

[0032] In this article about Figures 3 to 4In other implementations discussed, model-independent metrics 122 include model-independent performance metrics (e.g., utilization or latency metrics) provided to the autoscaler 124 residing in the control plane 130 of the MaaS platform 102. In response to observed model behaviors, such as increased latency or high utilization of GPU resources currently allocated to a model pool, the autoscaler 124 automatically increases the number of GPU resources allocated from the shared resource pool 132 to a given model pool (e.g., model pool 108). In some implementations, dynamically "scaling up" GPU resources is achieved by reducing (removing) GPU resources allocated to other model pools in the MaaS platform 102.

[0033] Figure 2 Another example system 200 is illustrated, which includes a MaaS platform 202. This MaaS platform includes a throttling service 204 that throttles incoming LLM processing requests based on a model-independent metric generated by a metric normalizer 208. The MaaS platform 202 includes [related to...]. Figure 1 The description includes multiple identical or similar architectural software components, including various model pools 222, 224, and 226, each supporting different LLMs. Each model pool includes a model routing layer 220 for routing requests within the pool to selected model endpoints (e.g., model endpoint B), and each model endpoint further routes incoming requests to the corresponding model instance hosted by the model endpoint.

[0034] Each LLM processing request received at MaaS platform 202 is directed to gateway 218, which acts as the "front door" to the corresponding model pool. Gateway 218 includes a throttling service 204, which limits the number of LLM processing requests issued by each user endpoint (e.g., client application 214 on client computing platform 210) to various model pools within a specific time period. While throttling services are common in cloud-based shared resource systems, throttling services for LLM are typically lexical-based. In such systems, users pay for a subscription to a cloud-based model (e.g., a single LLM), and are granted a set lexical quota that can be redeemed from the LLM service within a runtime interval, such as 10,000 lexicals per minute.

[0035] The GPU capacity required to process a single term is highly variable across models, different versions of the same model, and versions of the same model deployed on different GPU architectures. Therefore, the 10,000-term quota limit corresponds to a limited and equivalent upper limit on memory utilization only among users submitting workloads with the same characteristics to the same model instance or the same version of the model deployed on the same GPU architecture. If terms from these existing LLM throttling services are exchangeable for processing tasks performed by different models, different users subscribing to the same quota limit will be allocated very different total compute capacity as a result of model selection. For example, 10,000 terms might correspond to 5 GB of memory per minute for the first model, while the same 10,000 terms for the second model might correspond to 12 GB of memory per minute.

[0036] To prevent inequalities in quota management and, to some extent, inefficiencies in processing and computation management, the metric normalizer 208 of the MaaS platform 202 converts token-specific memory utilization requests (e.g., a sample set of token-based job metrics) into model-independent units that quantify memory utilization. This is facilitated by intelligently derived and used model-specific benchmark metrics 206, which define the observed relationship between resource utilization and token processing for different tokenization schemes of different LLMs deployed on different GPU architectures.

[0037] In one implementation, the model service provider provides its corresponding model service, along with model-specific benchmark metrics 206 associated with various different types of GPU architectures that can be deployed to perform these services, to a metric normalizer 208. In other implementations, the metric normalizer 208 experimentally derives the model-specific benchmark metrics 206 by performing load tests associated with various internally tracked benchmarks. For example, tests are performed to quantify the computational load posed when handling workloads with different characteristics (e.g., different numbers of input and output tokens) for each model deployed on each different GPU architecture.

[0038] According to one implementation, model-specific benchmark metric 206 includes information (e.g., model, lookup table) that defines or can be used to derive the computational load induced by a selected workload with known input / output characteristics of a specific LLM process deployed on a given GPU architecture. As used herein, the term "computational load" refers to a measure of memory and compute utilization induced for processing a workload of a given workload shape and a selected level of concurrency. Figure 2In the figure, model-specific benchmark metric 206 is shown as storing the per-input lexical computational load 244 for each individual input lexical and the per-output lexical computational load 246 for each individual output lexical in a given workload, both of which are derived based on model-independent units.

[0039] In one implementation, different sets of model-specific benchmark metrics 120 are stored for each model supported on the platform and for each model supporting different GPU architectures. For example, model-specific benchmark metrics 206 may include a first set of metrics that can be used to determine the computational load caused by a GPT-4 model instance executing on a single NVIDIA 8100 GPU chip processing any single lexical, while other sets of the same model-specific benchmark metrics 206 are also provided for alternative GPU architectures supporting the same model.

[0040] Depending on the predetermined acceptable error margin for estimating the computational load for different use cases, the per-word computational load (244, 246) can be derived with varying degrees of granularity in different implementations. For many LLMs, input words have significantly lower computational loads compared to output words; therefore, estimating the computational loads for input words and output words separately can be more accurate. In one such implementation, it is assumed that the per-input word computational load 224 is equal for all input words of a given workload, and that the per-output word computational load 244 is equal for all output words of a given workload.

[0041] While computational load can vary based on lexical length (e.g., the number of characters in a given word), these length-based variations are relatively small compared to lexical units of the same type (e.g., input and output). Therefore, for service throttling purposes, they can be considered equivalent in some scenarios without introducing significant differences in the total amount of memory available to each different user. Still, in some implementations, per-lexical computational load (244, 246) is determined based on the characteristics of individual input and output lexical units within the workload. For example, model-specific benchmark metric 206 specifies that input lexical units at a first lexical index have a first computational load, while input characters at another lexical index (e.g., a different lexicalization scheme) have a second computational load, and / or model-specific benchmark metric 206 provides other types of rules from which per-lexical computational load can be easily determined.

[0042] For example, one implementation of the disclosed technique derives the computational load for each workload based on model-independent units (referred to herein as "pre-defined throughput units (PTUs)"). PTUs are defined by the MaaS provider as units representing lexical throughput and can be used to facilitate GPU utilization comparisons across models. A PTU represents a logical unit of GPU capacity, but the number of PTUs corresponding to a given GPU type (e.g., chip type) is not constant. Instead, the amount of throughput in each PTU is based on the workload and defined relative to the maximum lexical throughput supported by the LLM and GPU architectures used to process that workload.

[0043] It is worth noting that a GPU supporting a given LLM dedicates a portion of its GPU capacity to storing the LLM, with the remaining GPU capacity subsequently available to support model processing operations. PTU corresponds to the portion of the aforementioned "remaining GPU capacity" available to support the lexical throughput in a given model deployment. This capacity is defined based on the experimentally determined "maximum lexical throughput" (also referred to as "maximum utilization" in this paper), which is the maximum lexical throughput that can be dedicated to processing a workload for a specific model instance without compromising the quality or speed of lexical generation. In one implementation, PTU is defined as a fixed percentage equal to the maximum lexical throughput (also referred to as "maximum utilization" in this paper) for a given workload.

[0044] The maximum lexical throughput or maximum utilization for a selected workload describes the maximum amount of memory that can be allocated to processing for a specific model instance when executing multiple workloads with characteristics substantially similar to the selected workload without compromising lexical generation quality or speed. It is noteworthy that the maximum lexical throughput for a workload depends on many factors, including (1) the LLM processing the workload and the lexicalization scheme of the LLM; (2) the underlying GPU hardware supporting the LLM, including GPU type and count; and (3) the size characteristics of the workload, including the number of input and output lexicals. For the reasons described above, the maximum lexical throughput is highly variable, even among workloads of the same model. However, it is possible to statistically model the maximum lexical throughput for a given LLM and GPU architecture, taking into account various assumptions about the characteristics of the workloads being processed.

[0045] In one implementation, the maximum lexical throughput of a workload is identified by a probability distribution that identifies and references a relevant stored probability distribution from a plurality of pre-generated and stored probability distributions. The “relevant” probability distribution for the selected workload is, for example, a probability distribution modeling maximum utilization (“maximum lexical throughput”), which includes throughput measurements collected during the processing of workloads of similar input / output size by an LLM and GPU architecture corresponding to the target instance (e.g., an LLM identical to the target instance deployed within the same GPU architecture as the target instance). For example, a first probability distribution for a given LLM and supported GPU architecture is generated by: (1) performing an LLM on a set of concurrent workloads characterized by a common set of input / output features (e.g., consisting of 100 input lexical units and 500 output lexical units); (2) recording the maximum throughput observed before model performance begins to decline; and (3) repeating the experiment a statistically significant number of times (e.g., re-observing the maximum throughput for the same model while concurrently processing other workloads characterized by the same input / output features). Steps 1 through 3 above are repeated for workloads characterized by different sets of input / output features (e.g., a workload consisting of 50 input tokens and 300 output tokens; a workload with 100 input tokens and 100 output tokens; any other combination of input / output lengths) to generate additional probability distributions for the same LLM and GPU architecture. In this way, multiple probability distributions can be generated for each LLM and supported GPU architecture, where each individual probability distribution can be used to identify the probabilistically expected maximum token throughput when the LLM is used to concurrently process a set of workloads characterized by known input token sequence lengths and known output token sequence lengths.

[0046] As described above, in one implementation, PTU is defined as a portion determined (in the manner described above) equal to the observed maximum utilization (“maximum token throughput”) associated with a given workload. For example, for a given workload, the size of the PTU is set to be equal to 1% of the maximum token throughput associated with that workload, identified from a stored probability distribution. Depending on the type of LLM, the supported GPU architecture, and the workload characteristics of the model, a fixed 1% of the maximum utilization for a PTU can correspond to a highly variable token throughput unit—for example, an average of 1000 tokens / second or 100 tokens / second, where the throughput is divided across cue tokens and generated tokens, which are processed according to different throughput rates. From the above examples, it can be concluded that PTU represents a workload-specific unit with well-defined boundaries. Therefore, a single PTU corresponds to a number of tokens that varies based on the identity of the target LLM for the LLM processing task of an incoming client request, the GPU architecture supporting that target LLM, and even the characteristics of the workload.

[0047] In one implementation using PTU as a model-agnostic utilization metric, model-specific benchmark metric 206 includes modeling data describing the probability distributions of the maximum lexical throughput available per model instance running on various GPU architectures. Each probability distribution corresponds to the lexical throughput associated with a specific LLM and GPU architecture during a workload identified by certain common input / output characteristics (e.g., input lexical length and output lexical sequence length). Therefore, for any given model, GPU architecture, and workload scenario with known input / output characteristics, it is possible to identify the corresponding maximum lexical throughput (from the corresponding storage probability distribution) using model-specific benchmark metric 206, and further determine a score of the maximum lexical throughput represented by that workload, which can then be translated into the number of PTUs. Furthermore, by defining PTU based on maximum utilization, it becomes possible to use the storage of workload models and the maximum lexical throughput of those stored models as a workload-based PTU definition. This, in turn, makes it possible to directly compare the amount of computational capacity utilized across different LLMs deployed on different GPU architectures.

[0048] exist Figure 2In the example, the throttling service 204 limits the number of requests a user can concurrently submit to different LLM instances based on a customer-allocated quota 232, defined in units of a model-independent unit type. Units of the customer-allocated quota can be exchanged by the throttling service 204 for computational tasks performed by different LLMs. For example, a client can subscribe to a single quota and concurrently submit processing requests to different LLMs, with each request deducting a corresponding amount of units from that single quota. In one implementation, the throttling service 204 manages and tracks the "current resource utilization" for each different client point (e.g., client application 214) based on the PTU, where each client is allocated a maximum number of preset throughput units without time units (such as one minute, five minutes, or other time intervals). In other implementations, other model-independent GPU capacity units are used instead of PTUs.

[0049] Throttling service 204 receives token-based memory utilization requests and communicates with metric normalizer 208 to determine a corresponding estimated utilization 230 for each token-specific request. This estimated utilization 230 represents a defined computational load of the associated workload. Based on the estimated utilization 230, the current resource utilization determined for each client point, and the quota 232 allocated to the client—all determined in model-independent units—throttling service 204 determines whether to grant or deny each new LLM processing request.

[0050] exist Figure 2 In this process, client application 214 generates an LLM query 245 to submit to the target model instance in model pool 222. Client application 214 also generates a lease request 241 and submits it to gateway 218. The function of lease request 241 is to provision resources in model pool 222 to process LLM query 245. Lease request 241 identifies the target model instance (e.g., by specifying a specific model pool, endpoint, instance ID, or other identifying information) and further specifies the number of input tokens and the number of output tokens requested, wherein the number of output tokens requested sets an upper limit on the number of output tokens allowed to be generated by the corresponding model instance. The input token count and output token count in lease request 241 are determined based on the specific lexicalization scheme of the corresponding target model instance and the text based on LLM query 246.

[0051] Before determining whether to grant lease request 241, throttling service 204 communicates a lexical-based job metric 231 to metric normalizer 208. The lexical-based job metric 231 includes an identifier for the target model instance and the input / output lexical counts of the request. Metric normalizer 208 uses the lexical-based job metric 231 to determine a model-independent utilization metric. Figure 2This is shown as "Estimated Utilization 230". Estimated Utilization 230 represents the estimated total computational load associated with LLM query 246, which is given according to the model-independent unit type. In one implementation, Estimated Utilization 230 is determined by metric normalizer 208 as the number of provisioned throughput units (PTUs).

[0052] The throttling service 204 then determines the user's current utilization 242. In one implementation, the current utilization 242 represents the net resource utilization associated with LLM processing requests originating from the client computing platform 210 within a recent time period (such as the past 1 minute or 5 minutes). The current utilization 242 is determined based on model-independent token throughput units (such as PTU (as defined above)).

[0053] In one implementation, the throttling service 204 dynamically determines the current utilization 242 of the client point by querying a platform-level database (not shown) to retrieve model-agnostic utilization metrics (e.g., in PTU) for the most recent time interval. These model-agnostic utilization metrics quantify the client's total utilization over the most recent time interval. For example, the platform-level database stores model-agnostic utilization metrics that are published by the metric normalizer 208 based on the lexical-based job metrics reported back to the metric normalizer 208 for each model instance (see below for example regarding...). Figure 3 The discussion focuses on lexical-based job metrics 318. Examples include model instance transfer reports indicating the number of input and output lexicals processed by each client point, and metric normalizer 208 converting these lexical-based job metrics into model-independent utilization metrics (e.g., PTU utilized per job and per model instance), which are then published to a platform-level database. In another implementation, throttling service 204 determines the current utilization 242 for each client point in the client network independently, without referencing a platform-level database, such as by storing and aggregating utilization information included in the response packet header received at gateway 218 in association with each submitted LLM processing job.

[0054] Throttling service 204 limits the number of requests a user can concurrently submit to different LLM instances based on the customer's current utilization 242, model-independent estimated utilization 230, and the customer's allocated quota 232, all defined according to the unit type of the model-independent unit. Specifically, throttling service 204 determines whether the sum of current utilization 242 and estimated utilization 230, if utilized by the customer, would exceed the customer's allocated quota 232. If so, throttling service 204 rejects lease request 241, and client application 214 queues the request for resubmission at a later time. Otherwise, throttling service 204 grants the request and instructs gateway 218 to process the LLM query 246 associated with lease request 241.

[0055] Figure 3 Another example system 300 is illustrated, which includes a MaaS platform 302 that dynamically allocates GPU resources of a shared resource pool 320 among various model pools (e.g., model pool A, model pool B) based on model-agnostic performance metrics generated by a metric normalizer 308. The MaaS platform 302 includes [related to...]. Figure 1 and Figure 2 The described architecture comprises multiple identical or similar software components, including model pools (e.g., model pool A, model pool B) that each executes various instances of the corresponding LLM deployed at one or more endpoints. Each model pool supports a different LLM. Model instances within a model pool are executed by GPU resources belonging to a shared resource pool 320. Figure 3 In addition, the MaaS platform 302 also includes a control plane 330, which has an autoscalorie 312 that dynamically reallocates resources of the shared resource pool 320 within the model pool.

[0056] For example, Figure 3 The diagram illustrates two model pools—Model Pool A and Model Pool B. Model Pool A contains instances of a first LLM (Model A), while Model Pool B contains instances of a second, different LLM (Model B), where Model A and Model B utilize different lexicalization schemes. (The diagram is in the context of...) Figure 3 As shown in the resource allocation time, the first resource subset 322 from the shared resource pool 320 has been allocated as an instance supporting model A, while the second resource subset 324 from the shared resource pool 320 has been allocated as an instance supporting model pool B.

[0057] In one implementation, each model instance in the model instances is executed within a container that reports lexical-based job metrics 318 to a metric normalizer 314. In the example shown, each set of lexical-based job metrics 318 identifies the corresponding model and model instance for each of the multiple processing jobs, as well as the total number of input and output lexicals processed by the model instance during the corresponding job. Although Figure 3 Not shown, but in some implementations, the lexical-based job metric 318 can identify the specific text strings of input and output during each processing job, as a supplement to and / or alternative to the input / output lexical count. For example, upon completion of each processing job, the model instance transmits a report to the metric normalizer 314, which identifies the model instance, the input lexical sequence, and the output lexical sequence generated in response to the processing of the input lexical sequence.

[0058] Assuming that when each new model instance is initialized within the MaaS platform 302, the metric normalizer 314 receives an identifier for the new model instance and a description of the GPU architecture on which that new model instance is running. Therefore, given a unique model instance identifier for a set of lexical-based job metrics 318 (e.g., a pairing of model identifier – “Model A” – with instance identifier – “Instance 2”), the metric normalizer 314 can identify the specific GPU architecture currently used to execute the corresponding model instance (e.g., the type and number of GPUs). This capability allows the metric normalizer 314 to identify the applicable set of model-specific benchmark metrics 306, which can be used to compute memory utilization for each individual processing job reported in the lexical-based job metrics 318 in units of a model-independent unit type. The model-specific benchmark metric 306 defines the observed relationship between resource utilization and lexical processing for different lexicalization schemes of different LLMs deployed on different GPU architectures. In one implementation, the model-independent unit type is as described above regarding… Figure 2 The aforementioned PTU.

[0059] In the illustrated implementation, model-specific benchmark metric 306 is derived by metric normalizer 314 based on load tests performed on various internally defined benchmarks. In one implementation, model-specific benchmark metric 306 is stored in a platform-managed database for repeated lookups and for generating model-specific utilization metrics into model-independent utilization metrics. Model-specific benchmark metric 306 includes information that can be used to convert the lexical counts provided in the lexical-based job metric 318 into model-independent utilization metrics that can be easily compared across models. In one implementation, model-independent utilization metrics are generated in model-independent units (PTUs) representing the number of lexical throughputs. The PTUs described above are an example of a model-independent unit type. In one implementation, model-specific benchmark metric 306 includes different sets of model-specific benchmark metrics 306, where each set is applicable to a specific LLM and can be deployed to support a GPU architecture that supports that LLM.

[0060] In the implementation shown, each set of model-specific benchmark metrics 306 is specific to a model instance defined by an LLM identifier and the selected GPU architecture. Furthermore, model-specific benchmark metrics 306 are shown storing information including, or usable to derive, the per-input term computational load 327 for each individual input term in a given workload and the per-output term computational load 328 for each individual output term in the workload, both derived from model-independent units representing GPU capacity (e.g., PTUs described elsewhere in this document). In other implementations, model-specific benchmark metrics 306 store information usable to derive the computational load of the workload as a whole but not necessarily the computational load of individual terms within the workload. The derivation of the per-input term computational load 327 and the per-output term computational load 328 for each individual output term in one implementation is related to... Figure 2 The same or substantially similar descriptions are defined at the workload level.

[0061] Model-specific benchmark metric 306 also defines information for determining the maximum utilization 326 associated with each individual workload being processed. Maximum utilization 326 (also referred to elsewhere in this document as “maximum lexical throughput”) describes the maximum amount of memory that can be devoted to processing a particular model instance (e.g., as defined by LLM and GPU architectures) without compromising lexical generation quality or speed. In one implementation, maximum utilization 326 can vary depending on workload characteristics such as input cue length and the number of generated lexicals. Therefore, maximum utilization 326 for a given workload can be determined based on simulations run against workloads identified as similar to the LLM processing task of the incoming client request (e.g., workloads with the same or similar number of input and output lexicals).

[0062] about Figure 3 Other aspects of the undefined model-specific benchmark metric 306 can be assumed to be related to... Figure 2 The model-specific benchmark metric 206 is the same as or similar to that described.

[0063] The metric normalizer 308 calculates the maximum utilization for each model instance by aggregating the maximum utilization 326 corresponding to different workloads of a given model instance. It is assumed that GPU allocation within each model pool (e.g., model pool A and model pool B) is performed by an appropriate load balancing algorithm. For example, the load balancing algorithm ensures that resources within subset 322 are fairly allocated to active model instances in model pool A based on load. Due to this load balancing, various model instances within a model pool have corresponding maximum utilizations that are relatively close to each other (e.g., within + / -10% or some other known deviation). Therefore, the maximum utilization for a single model pool—referred to herein as the “maximum pool utilization”—is chosen in one implementation to be equal to the maximum utilization of the model instance within the pool with the highest peak load.

[0064] In one implementation, the maximum pool utilization rate derived for each model pool is used as the basis (e.g., a trigger point) for dynamically scaling up and down the GPU resources allocated to that model pool, as discussed in further detail below.

[0065] For each processing job identified in the lexical-based job metric 318, the metric normalizer 314 selects a relevant subset of the model-specific benchmark metric 306 (e.g., a metric derived based on the LLM and GPU architectures that match the model instance executing the processing job). The selected relevant subset in the model-specific benchmark metric 306 is used to determine the computational load of the corresponding processing job based on the unit of the model-independent unit type (e.g., PTU).

[0066] In one implementation, the computational loads calculated for different processing jobs targeting a model instance are aggregated to derive a model instance utilization metric that quantifies the GPU utilization of the model instance within the most recent time interval (e.g., the past minute). Then, the model instance utilization metrics determined for different model instances within each pool are combined to derive a model pool utilization metric 317 for each model pool. The model pool utilization metric 317 quantifies the resource utilization for the model pool as a whole. For example, the model pool utilization metric 317 represents the total GPU utilization of model pool A over the past minute.

[0067] In one implementation, model pool utilization metric 317 is defined as the number of PTUs. For example, model pool utilization metric 317 represents the number of PTUs jointly utilized by all processing jobs across all model instances of model pool A within the previous minute interval. In another implementation, model pool utilization metric 317 is given as a percentage of the maximum pool utilization (as defined above) for the corresponding model pool. For example, model pool utilization metric 317 for model pool A could be determined as "79% of the maximum pool utilization of model pool A" (where the maximum pool utilization is derived as described above).

[0068] exist Figure 3 In this context, the autoscaler 312 performs dynamic resource allocation by adding and / or removing GPU resources from subsets 322 and 324 of the GPUs allocated to each model pool in model pool A and model pool B in response to changes in observed model pool utilization metric 317 for each pool satisfying a predefined scaling criterion.

[0069] In one implementation, the goal of the autoscaler 312 in performing resource allocation is to ensure that the model pool utilization metric 317 for each model pool remains within a threshold target range defined relative to the corresponding maximum pool utilization determined for the same model pool. For example, the autoscaler 312 may implement logic to ensure that model pool A uses an average of 75% to 85% of the maximum pool utilization determined for model pool A over the most recent time interval (the past 1 minute or 5 minutes). In this example, a “shrink” criterion is met when the model pool utilization metric 317 of subset 322 drops below 75% of the determined maximum pool utilization for model pool A within a threshold time period (e.g., 30 minutes). Similarly, in the same example, a “scaling up” criterion is met when the model pool utilization metric 317 of subset 322 allocated to model pool A is equal to or exceeds 85% of the maximum pool utilization determined for model pool A.

[0070] In one implementation, GPU resources are dynamically allocated as resources become needed—meaning all resources become needed at that time—and / or removed from subsets 322 and 324 in finite, static increments. For example, each time the autoscaler 312 selects to "expand" GPU resources in subset 322, a predetermined number of GPUs are added to subset 322, and each time the autoscaler 312 selects to "shrink" GPU resources in subset 322, a predetermined number of GPUs are subtracted from subset 322. In this implementation, resources are expanded / shrunken in static, step-like increments, and each increment is triggered when the autoscaler 312 "observes" a value of the model pool utilization metric 317 that meets predefined expansion or shrinkage criteria.

[0071] In one implementation, subsets 322 and 324 of GPU resources are available within the memory mapping used by each model pool for that model pool. In this case, the autoscaler 312 “shrinks” the model pool by instructing the VM provider (not shown) to remove a subset of GPU resources from the memory mapping of the logical address used by the model pool. Similarly, the autoscaler 312 “expands” the model pool by instructing the VM provider to add a subset of GPU resources to the memory mapping used by the model pool. In an implementation where resources are dynamically reallocated between model pools, the autoscaler 312 can dynamically allocate GPU resources by removing a subset of GPU resources from the memory mapping used by the first model pool and by adding that subset of GPU resources to the memory mapping used by the second model pool.

[0072] In other implementations, GPU resources are allocated in variable increments within the model pool, these increments being determined to specifically meet discrete and particular utilization targets(s). For example, autoscaler 312 receives model pool utilization metric 317 as a percentage of the maximum utilization for model pool A and determines specific scaling-up or scaling-down increments sufficient to make model pool utilization metric 317 equal to the target percentage of the maximum utilization for model pool A.

[0073] exist Figure 3 In this context, the MaaS platform 302 is also shown as including a virtual machine (VM) provisioning layer 310 that implements GPU reallocation instructions from an autoscaler 312. For example, if the autoscaler 312 provides a reallocation instruction to the VM provisioning layer that identifies a specific number and / or type of GPUs to be moved from subset 322 (support model pool A) to subset 324 (support model pool B), the VM provisioning layer 310 identifies the VMs currently configured to utilize resource subset 322 and, as appropriate, reconfigures these VMs to achieve the requested GPU reallocation. For example, the VM provisioning layer 310 implements a “reduction” reallocation instruction by reducing the size of the GPU memory footprint available to a given VM (e.g., by removing logical addresses allocated to the VM and / or removing them from the memory mapping that maps those logical addresses to physical addresses) and / or by remapping the logical memory allocated to the VM to exclude certain GPU resources from the memory mapping of logical-to-physical GPU addresses used by the VM. In some cases, the VM provisioning layer 310 implements a “shrink” reallocation instruction by shutting down one or more VMs, thereby reducing the total number of VMs executing on behalf of a given model pool.

[0074] Similar to the above, VM provisioning layer 310 can implement “scaling up” reallocation instructions that increase the size of the GPU memory footprint available to a given VM, and / or include certain additional (reallocated) GPU resources in a new logical-to-physical memory mapping by remapping the GPU memory available to the VM. These “spin-up” instructions may require provisioning one or more additional VMs and configuring these VMs as containers to support the execution of associated model pool instances.

[0075] Because of the collection and storage of model-specific benchmark metrics 306, the metric normalizer 314 can support dynamic cross-model resource allocation in response to dynamically observed performance metrics for each model pool, such as model pool utilization metric 317. Since these performance metrics are—initially—model-agnostic, the autoscalorie 312 can compute efficient, specific resource allocation to achieve the target utilization within each model pool.

[0076] Figure 4 Another example system 400 is illustrated, which includes a MaaS platform 402 that dynamically reallocates GPU resources of a shared resource pool 420 among various model pools (e.g., model pool A, model pool B) based on model-agnostic performance metrics generated by a metric normalizer 408. Figure 3 The response is independent of the observed model. Utilization Compared to resource allocation operations performed in response to changes in metrics, system 400 responds to observed changes such as... Figure 4 One or more model-independent delay metrics are shown as "Model-Independent Delay Metric 417". Delay Dynamic resource allocation is performed based on changes in metrics.

[0077] As with other examples shown herein, it is assumed that model pool A supports instances of model A and model pool B supports instances of model B, where model A and model B are different LLMs that process text using different lexicalization schemes. In system 400, metric normalizer 414 receives lexical-based latency metrics 410 generated in association with various processing jobs performed by instances of model A and model B. Additionally, metric normalizer 414 receives lexical-based job metrics 409 for each job processed by each instance of model A and model B deployed within MaaS platform 402. Lexical-based job metrics 409 include at least a model instance identifier 411 and input / output lexical sequences 413, which identify the input and output lexical sequences processed by the corresponding model instance during the corresponding processing job.

[0078] In one implementation, each model instance executes within a container configured to transmit a lexical-based job metric 409 to a metric normalizer 408 upon completion of each processing job. Additionally, the container generates and sends a lexical-based latency metric 410 in association with some or all of the model instance's processing jobs.

[0079] The lexical-based latency metric 410 quantifies the latency observed during the execution of LLM processing tasks performed by different LLMs deployed in different model pools (e.g., model pool A, model pool B), which are configured to process lexicals according to different lexicalization schemes. In one implementation, the lexical-based latency metric 410 includes the Time to the Last Lexical (TTLT), which measures the rate at which the LLM model can produce the last lexical in the response. The example described below provides the use of TTLT to determine a model-independent per-lexical latency estimate that can be directly compared between models to evaluate relative latency. In other implementations, the lexical-based latency metric 410 may additionally or alternatively include the Time to the First Lexical (TTFT), which measures the rate at which the LLM model can produce the first lexical in the response, and / or the Time Between Lexicals (TBT), which measures the consistency of the LLM model in producing lexicals at regular intervals. As the load increases, TBT may increase, and the generation rate may decrease. The formula for calculating TBT is given below:

[0080] Because Model A and Model B implement different lexicalization schemes, there are scenarios where the same input string can be processed into different numbers of words. Similarly, if Model A and Model B are to generate the same output string, the same output string can be processed into different numbers of words. It is worth noting that different lexicalization schemes can also rely on different (LLM-specific) vocabularies, meaning that even when generated based on the same input string, the actual outputs returned by Model A and Model B may differ in wording.

[0081] Given these inter-model differences stemming from reliance on different lexicalization schemes, lexical-based latency metrics 410 (e.g., TBT, TTLT, and / or TTFT) cannot be used as a reliable means of evaluating the latency of different LLMs relative to each other. Figure 4 The various sub-components of the metric normalizer 414 address this shortcoming, helping to convert the lexical-based delay metric 410 into model-independent units representing delay. Figure 4 The value in the middle represents the model-independent delay metric 417.

[0082] Model-independent latency metric 417 may include the same or similar type of metric discussed with regard to lexical-based latency metric 410; however, model-independent latency metric 417 is generated in a manner that ensures cross-model consistency. For example, suppose an instance of model A reports a TTLT of 0.8 seconds and an instance of model B reports a TTLT of 1.1 seconds. These initially reported values ​​for lexical-based latency metric 410 do not necessarily mean that an instance of model B experiences higher latency than an instance of model A for the same output text (e.g., it could be because the generated lexical sequences are very different from each other). In some cases, an instance of model B may actually experience less latency than an instance of model A, even if an instance of model B reports a higher TTLT value, because model B has generated a higher quality response with a larger number of output lexicals to answer the same question.

[0083] In system 400, a lexical sequence length normalizer 416 performs lexical sequence normalization operations designed to adjust the lengths (in terms of the number of lexical units) of the input / output lexical sequences 413 to account for differences in lexical sequence lengths resulting from the use of different lexicalization schemes. For example, suppose model A and model B process the same input text string and generate output sequences with different numbers of lexical units due to differences in their respective lexicalization schemes. When the lexical sequence length normalizer 416 is used to normalize the lengths of these two corresponding output sequences (e.g., based on the total number of lexical units), this produces two different normalized length output lexical sequences 418 that are either the same length or more similar in length than the original two output sequences, with the remaining differences in lexical sequence lengths falling within a predefined error margin depending on the lexical transformation (normalization) technique employed in any given implementation.

[0084] In various implementations, the lexical sequence length normalizer 416 employs various techniques to normalize the length of the output lexical sequence. These techniques generally aim to reformat the output lexical sequence of each input / output lexical sequence 413 with a “master lexicalization scheme” using the vocabulary of the “master LLM”. As used herein, the term “master LLM” refers to any LLM selected by the metric normalizer 308 and used to normalize the length of the output lexical sequence as described herein. In various implementations, the master LLM can be any LLM used to provide the functionality of the master LLM described herein. Similarly, the term “master lexicalization scheme” refers to the lexicalization scheme used by the master LLM, regardless of the identity of the master LLM and / or the individual characteristics of the master LLM that can vary from one implementation to another.

[0085] There are two different methods to advance the aforementioned end goal: using a "master lexicalization scheme" to reformat the output lexical sequence of each input / output lexical sequence 413 with the vocabulary of the "master LLM". In the first method, used to generate the normalized-length output sequence 418, the original input string is lexicalized according to the master lexicalization scheme and "reprocessed" by the master LLM (e.g., if the output size has an original limit) to produce a corresponding output sequence of length (in terms of the number of lexical units) that has now been normalized to the common lexicalization scheme. For example, suppose the first pair of input / output lexical sequences 413 includes a first sequence embedding the input string "What is a popular nursery rhyme about a sheep?" and a second sequence embedding the output string "Mary had a little lamb." In the aforementioned "first method", the lexical sequence length normalizer 416 lexicalizes the input string using the master lexicalization scheme, and the result is input to the master LLM to produce the master lexicalized output sequence. In this case, the subject meta-output sequence is the actual output sequence of the main LLM that has been generated based on the same input text string.

[0086] In the second method, which generates a normalized length output sequence of 418, various techniques are employed to "translate" the original output lexical sequence into the most similar sequence in the lexical vocabulary of the selected lexical scheme. This differs from the "first method" mentioned above in that it attempts to reconstruct the output of the main LLM and the lexical scheme through a "smart translation" of the original output sequence, without actually calling the main LLM to reprocess the original query and generate a new output sequence. While less straightforward to implement, some solutions for this "translation" method are more cost-effective in terms of processing overhead.

[0087] In one implementation of the aforementioned "translation" method (e.g., the second one), a normalized length output sequence 418 is generated by translating each original output lexical sequence to the vocabulary of the master LLM (e.g., a vocabulary defined by the master lexicalization scheme), such as by using semantic similarity and / or the frequency of corresponding lexical units. This approach may require some training data and additional computational requirements, and may not guarantee perfect lexical alignment.

[0088] In another implementation of the aforementioned "translation" method, a neural machine translation (NMT) model trained to convert lemmas from the original lemma scheme into lemmas from a chosen main lemma scheme is used to generate a normalized length output sequence 418. Typically, NMT models are used to provide translation from one language to another, with popular publicly available NMTs including Google Translate. ® and Baidu Translate ®In the lexical sequence translation example proposed in this paper, (multiple) identical underlying NMT models can be trained on a set of output lexical sequences generated by different LLMs in response to the lexicalization and processing of the same input text string. This approach may require model fine-tuning and can introduce some errors or noise into lexical translation.

[0089] In another implementation of the aforementioned "translation" method, a normalized length output sequence 418 is generated by mapping the tokens of the original output sequence to corresponding tokens in the main lexicalization scheme using token embedding. For example, a token embedding model such as Word2Vec can be used to convert the tokens of the original LLM model into vector representations. Then, nearest neighbor search or clustering algorithms are used to find the corresponding tokens of the main LLM.

[0090] Various other methods can also be used to achieve the above translation, where the uniformity of the length of the corresponding normalized output sequence (e.g., a sequence generated based on the output of the same original input query) varies depending on the chosen normalization method. For example, suppose two LLMs are asked to create a 3-day travel itinerary for a trip to Paris. One model generates an itinerary of 304 words, and the other model outputs an itinerary of 411 words. Using the “first method” described above for normalization (e.g., regenerating the output sequence using the master LLM), the two resulting normalized length output sequences 418 are the same length. According to other methods, the resulting normalized length output sequences are more similar in length than they were originally (e.g., differing by + / - 5% or an error margin of other identifiers statistically guaranteed by the chosen normalization method and considered acceptable for the use of the model-independent delay metric 417).

[0091] The lexical converter 416 converts each normalized-length output sequence in the normalized-length output sequence 418 into a latency metric calculation engine 426, which also receives a corresponding set of lexical-based latency metrics 410 generated by the LLM that initially processed the corresponding query. In the example discussed below, the lexical-based latency metric 410 includes at least the time to the last lexical (TTLT), representing the total time taken by the original LLM to generate the original output sequence. Using the TTLT and the corresponding normalized-length output sequence, the latency metric calculation engine 426 generates a model-independent latency metric (e.g., model-independent latency metric 417).

[0092] In different implementations, normalized latency can be determined in different ways. In scenarios where multiple GPUs are deployed to support the raw LLM, it is desirable to first determine the number of tokens generated "per GPU" in the normalized output sequence. For example, suppose a given instance of model A runs on 24 GPUs, and the model generates a raw output token sequence of 800 words in 60 seconds. Further suppose the raw output sequence of 800 words has been normalized according to the above operation to generate a normalized length output sequence of 754 words. In this scenario, the latency metric calculation engine 426 then determines the "normalized tokens per GPU" by dividing the 754 tokens in the normalized length output sequence by 24 GPUs, obtaining 30 (754 / 24=30). Subsequently, the latency metric calculation engine 426 determines the normalized time per GPU for a given processing job by dividing the "normalized tokens per GPU" (e.g., 30) by the initially reported TTLT. For example, assuming the initially reported TTLT is 60 seconds, the normalized time per term is (30 terms) / (60 seconds), or 0.5 seconds per term. This "normalized time per term" represents a model-independent latency metric that can be easily compared to similarly determined latency metrics for other models.

[0093] In other implementations, other types of model-independent latency metrics can be derived using the same or similar techniques described above to provide alternative benchmarks for comparing latency and response quality across models (e.g., assuming that longer generation corresponds to higher quality).

[0094] In one implementation, model-independent latency metrics corresponding to each model pool are aggregated (e.g., averaged) to derive a pool-specific latency metric, which is then used to evaluate the overall latency of the model pool relative to the other model pools of the MaaS platform 402.

[0095] and Figure 3 Similarly, the MaaS platform 402 includes a control plane 424 with an autoscalorie 412 that performs dynamic resource allocation of GPU capacity in response to determining that model-independent latency metrics for computation of a given pool meet certain predefined criteria. Figure 4 In this context, the first subset 442 represents GPU resources allocated to model pool A, while the second subset 444 represents resources allocated to model pool B. The autoscaler 412 issues reallocation instructions to the VM provisioning layer 422, causing a reallocation of GPU resources in subsets 442 and 444, such as by moving resources between subsets or by transferring resources among other model pools supported by the MaaS platform 402.

[0096] In system 400, each GPU reallocation is performed by the VM provisioning layer 422 in response to a reallocation instruction issued by the autoscaler 412. The autoscaler 412 transmits a "scaling" reallocation instruction in response to determining that the model-independent latency metric 417 for that pool meets a "scaling criterion". For example, the scaling criterion is met when the average TTFT, TBT, or TTLT or other latency metric for a given model pool exceeds a set threshold.

[0097] In the same or another implementation, the autoscalor 412 transmits a "reduction" reallocation instruction to the supply layer 422 in response to determining that the model-independent latency metric 417 for the pool meets the "reduction criterion". For example, the reduction criterion is met when the average TTFT, TBT, or TTLT or other latency metric for a given model pool falls below a set threshold.

[0098] In one implementation, GPU resources are dynamically allocated to and removed from subsets 442 and 444 in limited, static increments. For example, each time the model-independent latency metric 417 for model pool A meets the expansion criterion, a set number of GPUs is added to subset 442, and each time the model-independent latency metric 417 for model pool A meets the reduction criterion, the same or a different static number of GPUs is subtracted from subset 442. Regarding... Figure 4 Other example features of the MaaS platform 402 not explicitly described and / or the functionality it provides may be related to... Figure 3 The same or similar MaaS platforms.

[0099] Figure 5 The illustration shows an example schematic diagram of a processing device 540 suitable for implementing various aspects of the disclosed technology. The processing device 500 includes one or more processor units 502, multiple memory devices 504, a display 506, and other interfaces 508 (e.g., buttons). Each of the multiple processor units 502 may include one or more CPUs, GPUs, etc.

[0100] The (multiple) memory devices 504 typically include volatile memory (e.g., RAM) and non-volatile memory (e.g., flash memory). An operating system 510, such as the Microsoft Windows® operating system, the Microsoft Windows® Phone operating system, or a specific operating system designed for gaming devices, resides in the (multiple) memory devices 504 and can be executed by the (multiple) processor units 502, although it should be understood that other operating systems may also be used.

[0101] One or more applications 512 (e.g., LLM, throttling service 204, metric normalizer 314, or autoscaler 312) are loaded into memory devices 504 and executed by processor units 502 on operating system 510. In some implementations, one or more of these applications are distributed applications loaded into the memory of multiple different processing devices across a network connection.

[0102] Application 512 can receive input from each other and from various local input devices, such as microphone 534, input accessories 535 (e.g., keyboard, mouse, stylus, touchpad, gamepad, racing wheel, joystick), and camera 632. Additionally, application 512 can receive input from one or more remote devices (such as remotely located smart devices) by providing network connectivity (e.g., mobile phone networks, Wi-Fi®, Bluetooth®) using additional communication transceivers 530 and antennas 538. Processing device 500 may also include one or more storage devices 528 (e.g., non-volatile memory). Other configurations are also possible.

[0103] The processing device 500 also includes a power supply 516, which is powered by one or more batteries or other power sources and provides power to other components of the processing device 500. The power supply 516 can also be connected to an external power source (not shown) that covers or charges the built-in battery or other power source.

[0104] Processing device 500 may include various tangible computer-readable storage media and intangible computer-readable communication signals. Tangible computer-readable storage can be embodied by any available medium (including both volatile and non-volatile storage media, removable and non-removable storage media) accessible to processing device 500. Tangible computer-readable storage media excludes intangible and transient communication signals and includes volatile and non-volatile, removable and non-removable storage media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Tangible computer-readable storage media include RAM, ROM, EEPROM, flash memory or other memory technologies, CDROM, digital versatile disc (DVD) or other optical disc storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other tangible medium that can be used to store desired information and can be accessed by processing device 600. In contrast to tangible computer-readable storage media, intangible computer-readable communication signals can embody computer-readable instructions, data structures, program modules, or other data residing in modulated data signals (such as carrier waves or other signal transmission mechanisms). The term "modulated data signal" refers to a signal whose characteristics are set or altered in a way that encodes information in the signal. For example, invisible communication signals include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media).

[0105] In some aspects, the technology described herein relates to a model-as-a-service platform comprising: a metric normalizer that: receives lexical-based latency metrics that quantify latency observed during the execution of LLM processing tasks performed by different large language models (LLMs) deployed in different model pools, the different LLMs being configured to process lexicals according to different lexicalization schemes; receives output lexical sequences generated during the execution of the LLM processing tasks; generates normalized-length output sequences by reformatting the output lexical sequences using a master lexicalization scheme; generates model-independent latency metrics based on these lexical-based latency metrics and the normalized-length output sequences; and an autoscalar that dynamically allocates graphics processing unit (GPU) resources among the different model pools based on model-independent latency metrics generated in association with the different model pools.

[0106] In some respects, the technology described herein relates to a model-as-a-service platform in which an autoscaler dynamically allocates GPU resources by removing a subset of GPU resources from a memory map utilized by a first model pool and by adding that subset of GPU resources to a memory map utilized by a second model pool, the allocation being performed in response to determining that the second model pool is experiencing higher latency than the first model pool.

[0107] In some respects, the techniques described herein relate to a model-as-a-service platform in which a metric normalizer is also configured to determine pool-specific latency metrics by aggregating a subset of model-independent latency metrics that correspond to LLM tasks executed within a first pool in the model pool, wherein an autoscaler dynamically reallocates a certain amount of GPU resources to the first pool in response to determining that the pool-specific latency metric for the first pool in the model pool exceeds a threshold.

[0108] In some respects, the techniques described herein relate to a model-as-a-service platform in which generating each normalized-length output sequence in a normalized-length output sequence further includes: creating a lexicalized input sequence by lexicalizing the input string of the corresponding LLM processing task according to the master lexicalization scheme; and having the master LLM process the lexicalized input sequence to regenerate the output lexical sequence in the master LLM's vocabulary.

[0109] In some respects, the techniques described in this paper relate to a model-as-a-service platform in which different LLMs include one or more multimodal LLMs, and the input string is a textual representation of image, audio, or video data.

[0110] In some respects, the techniques described in this paper relate to a model-as-a-service platform in which generating each normalized-length output sequence further includes translating the output lexical sequence of the corresponding LLM processing task into the most similar lexical sequence within the lexical vocabulary of the main lexicalization scheme.

[0111] In some respects, the techniques described herein relate to a model-as-a-service platform in which translating the output lexical sequence of a corresponding LLM processing task into the corresponding most similar lexical sequence further includes at least one of the following: using lexical embeddings to map the lexicals of the output lexical sequence to the corresponding lexicals of the main lexicalization scheme; or using a neural machine translator trained to convert the lexicals of the first lexicalization scheme into the lexicals of the main lexicalization scheme.

[0112] In some respects, the techniques described herein relate to a model-as-a-service platform in which the generation of model-independent latency metrics based on these lexical-based latency metrics and normalized-length output sequences further includes: determining the normalized number of lexical units per GPU by dividing a normalized-length output lexical unit sequence in the normalized-length output lexical unit sequence by the number of GPUs supporting the corresponding LLM processing task in the LLM processing task; and dividing the lexical generation time associated with the corresponding LLM task in the LLM task by the normalized number of lexical units per GPU.

[0113] In some respects, the techniques described herein relate to a method for dynamically allocating resources among model pools in a Model-as-a-Service platform, comprising: receiving lexical-based latency metrics that quantify latency observed during the execution of LLM processing tasks performed by different Large Language Models (LLMs) deployed in different model pools, the different LLMs being configured to process lexicals according to different lexicalization schemes; receiving output lexical sequences generated during the execution of the LLM processing tasks; generating normalized-length output sequences by reformatting the output lexical sequences using a master lexicalization scheme; generating model-independent latency metrics based on these lexical-based latency metrics and the normalized-length output sequences; and dynamically reallocating graphics processing unit (GPU) resources among the different model pools in response to determining that the model-independent latency metrics meet predefined criteria.

[0114] In some respects, the techniques described herein relate to a method in which dynamically allocating GPU resources further includes: removing a subset of GPU resources from a memory map utilized by a first model pool, and adding the subset of GPU resources to a memory map utilized by a second model pool, the allocation being performed in response to determining that the second model pool is experiencing higher latency than the first model pool.

[0115] In some respects, the techniques described herein relate to a method that also includes: determining pool-specific latency metrics by aggregating a subset of model-independent latency metrics that correspond to LLM tasks executed within a first pool in the model pool; and dynamically reallocating a certain amount of GPU resources to the first pool in response to determining that the pool-specific latency metric for the first pool in the model pool exceeds a threshold.

[0116] In some respects, the techniques described herein relate to a method in which generating each normalized-length output sequence in a normalized-length output sequence further includes: creating a lexicalized input sequence by lexicalizing the input string of the corresponding LLM processing task according to a master lexicalization scheme; and processing the lexicalized input sequence by the master LLM to regenerate the corresponding output lexical sequence in the output lexical sequence in the vocabulary of the master LLM.

[0117] In some respects, the technique described herein relates to a method in which generating each normalized-length output sequence in a normalized-length output sequence further includes translating the output lexical sequence corresponding to the LLM processing task into the most similar lexical sequence within the lexical vocabulary of the main lexicalization scheme.

[0118] In some respects, the techniques described herein relate to a method in which translating the output lexical sequence of a corresponding LLM processing task into the corresponding most similar lexical sequence further includes: using lexical embeddings to map the lexicals of the output lexical sequence to the corresponding lexicals of the main lexicalization scheme.

[0119] In some respects, the techniques described herein relate to a method in which translating the output lexical sequence of a corresponding LLM processing task into the corresponding most similar lexical sequence further includes using a neural machine translation trained to convert lexicals of a first lexicalization scheme into lexicals of a main lexicalization scheme.

[0120] In some respects, the techniques described herein relate to a method in which the generation of model-independent latency metrics based on these lexical-based latency metrics and normalized-length output sequences further includes: determining the normalized number of lexical units per GPU by dividing the normalized-length output lexical sequence by the number of GPUs deployed within the GPU architecture executing the corresponding LLM processing task in the LLM processing task; and dividing the lexical generation time associated with the corresponding LLM task in the LLM task by the normalized number of lexical units per GPU.

[0121] In some aspects, the techniques described herein relate to one or more tangible computer-readable storage media storing processor-executable operations for performing a computer process comprising: receiving lexical-based latency metrics that quantify latency observed during the execution of LLM processing tasks performed by different large language models (LLMs) deployed in different model pools, the different LLMs being configured to process lexicals according to different lexicalization schemes; receiving output lexical sequences generated during the execution of the LLM processing tasks; generating a normalized-length output sequence by reformatting each output lexical sequence in the output lexical sequence using a master lexicalization scheme; generating model-independent latency metrics based on these lexical-based latency metrics and the normalized-length output sequences; deriving pool-specific latency metrics by aggregating subsets of model-independent latency metrics corresponding to LLM tasks executed in a first pool in different model pools; and dynamically allocating additional graphics processing unit (GPU) resources to the first pool in response to determining that the pool-specific latency metrics for the first pool meet a scaling criterion.

[0122] In some respects, the techniques described herein relate to one or more tangible computer-readable storage media, wherein generating each normalized-length output sequence in the normalized-length output sequence further includes: creating a lexicalized input sequence by lexicalizing the input string of the corresponding LLM processing task according to the master lexicalization scheme; and processing the lexicalized input sequence by the master LLM to regenerate the output lexical sequence in the vocabulary of the master LLM.

[0123] In some respects, the techniques described herein relate to one or more tangible computer-readable storage media, wherein generating each normalized-length output sequence in the normalized-length output sequence further includes: translating the output lexical sequence of the corresponding LLM processing task into the most similar lexical sequence within the lexical vocabulary of the main lexicalization scheme.

[0124] In some respects, the techniques described herein relate to one or more tangible computer-readable storage media, wherein translating the output lexical sequence of a corresponding LLM processing task into the corresponding most similar lexical sequence further includes: mapping using lexical embeddings or mapping using a neural translation model, mapping the lexicals of the output lexical sequence to the corresponding lexicals of the main lexicalization scheme.

[0125] The logical operations described herein are implemented as logical steps in one or more computer systems. Logical operations can be implemented as (1) a sequence of processor-implemented steps executed in one or more computer systems, and (2) interconnected machines or circuit modules within one or more computer systems. The implementation method is a matter of choice, depending on the performance requirements of the computer system utilized. Therefore, the logical operations constituting the implementations described herein are referred to differently as operations, steps, objects, or modules. Furthermore, it should be understood that logical operations can be performed in any order unless otherwise expressly stated or a particular order is inherently required by the language of the claims. The foregoing description, examples, and data, together with the appendix, provide a complete description of the structure and use of the exemplary implementations.

Claims

1. A Model-as-a-Service platform, comprising: The metric normalizers are (414)(408)(314)(308)(208)(104), and the metric normalizers are: Receive a lexical-based latency metric (410) that quantifies the latency observed during the execution of an LLM processing task performed by different large language models (LLMs) deployed in different model pools (112) (110), the different LLMs being configured to process lexicals according to different lexicalization schemes; Receive the output lexical sequence generated during the execution of the LLM processing task; The normalized length output sequence (418) is generated by reformatting the output word sequence using a subject metamorphism scheme. A model-independent delay metric (417) is generated based on the lexical-based delay metric (410) and the normalized length output sequence (418); and An autoscaler (412) (312) (124) dynamically allocates graphics processing unit (GPU) resources among the different model pools based on the model-independent latency metric (417) generated in association with the different model pools (112) (110).

2. The Model-as-a-Service platform of claim 1, wherein the autoscaler dynamically allocates the GPU resources by removing a subset of GPU resources from a memory map utilized by a first model pool and by adding the subset of GPU resources to a memory map utilized by a second model pool, the allocation being performed in response to determining that the second model pool is experiencing higher latency than the first model pool.

3. The Model-as-a-Service platform according to claim 1, wherein the metric normalizer is further configured to: Pool-specific latency metrics are determined by aggregating a subset of the model-independent latency metrics that correspond to LLM tasks executed within a first pool in the model pool, wherein the autoscaler dynamically reallocates a certain amount of the GPU resources to the first pool in response to determining that the pool-specific latency metric for the first pool in the model pool exceeds a threshold.

4. The Model-as-a-Service platform according to claim 1, wherein generating each normalized length output sequence in the normalized length output sequence further comprises: A lexicalized input sequence is created by lexicalizing the input string of the corresponding LLM processing task according to the main lexicalization scheme. The lexicalized input sequence is processed by the main LLM to regenerate the output lexical sequence within the vocabulary of the main LLM.

5. The Model-as-a-Service platform of claim 4, wherein the different LLMs include one or more multimodal LLMs, and the input string is a text representation of image, audio, or video data.

6. The Model-as-a-Service platform according to claim 1, wherein generating each normalized length output sequence in the normalized length output sequence further comprises: The output lexical sequence of the corresponding LLM processing task is translated into the most similar lexical sequence within the lexical vocabulary of the main lexicalization scheme.

7. The Model-as-a-Service platform according to claim 6, wherein translating the output lexical sequence corresponding to the LLM processing task into the corresponding most similar lexical sequence further includes at least one of the following: Lexical embedding is used to map the lexical units of the output lexical sequence to the corresponding lexical units of the main lexicalization scheme; or Neural machine translation is used, trained to convert lexical units of the first lexicalization scheme into lexical units of the main lexicalization scheme.

8. The model-as-a-service platform according to claim 1, wherein generating the model-independent latency metric based on the lexical-based latency metric and the normalized length output sequence further comprises: The normalized number of terms per GPU is determined by dividing a normalized length output term sequence in the normalized length output term sequence by the number of GPUs supporting the corresponding LLM processing task in the LLM processing task; and Divide the lexical generation time associated with the corresponding LLM task in the LLM task by the normalized number of lexical units per GPU.

9. A method for dynamically allocating resources within a model pool in a Model-as-a-Service platform, the method comprising: Receive a lexical-based latency metric (410) that quantifies the latency observed during the execution of an LLM processing task performed by different large language models (LLMs) deployed in different model pools (112) (110), the different LLMs being configured to process lexicals according to different lexicalization schemes; Receive the output lexical sequence generated during the execution of the LLM processing task; The normalized length output sequence (418) is generated by reformatting the output word sequence using a subject metamorphism scheme. A model-independent delay metric (417) is generated based on the lexical-based delay metric (410) and the normalized length output sequence (418); and In response to determining that the model-independent latency metric (417) meets a predefined criterion, graphics processing unit (GPU) resources are dynamically reallocated among the different model pools (112) (110).

10. The method of claim 9, wherein dynamically allocating the GPU resources further comprises: A subset of GPU resources is removed from the memory map utilized by the first model pool, and the subset of GPU resources is added to the memory map utilized by the second model pool, the allocation being performed in response to determining that the second model pool is experiencing higher latency than the first model pool.

11. The method of claim 9, further comprising: Pool-specific latency metrics are determined by aggregating subsets of the model-independent latency metrics that correspond to LLM tasks executed within the first pool of the model pool. as well as In response to determining that the pool-specific latency metric for the first pool in the model pool exceeds a threshold, a certain amount of the GPU resources are dynamically reallocated to the first pool.

12. The method of claim 9, wherein generating each normalized length output sequence in the normalized length output sequence further comprises: A lexicalized input sequence is created by lexicalizing the input string of the corresponding LLM processing task according to the main lexicalization scheme. as well as The lexicalized input sequence is processed by the main LLM to regenerate the corresponding output lexical sequence in the output lexical sequence within the vocabulary of the main LLM.

13. The method of claim 9, wherein generating each normalized length output sequence in the normalized length output sequence further comprises: The output lexical sequence of the corresponding LLM processing task is translated into the most similar lexical sequence within the lexical vocabulary of the main lexicalization scheme.

14. The method of claim 13, wherein translating the output lexical sequence corresponding to the LLM processing task into the corresponding most similar lexical sequence further comprises: Lexical embedding is used to map the lexical units of the output lexical sequence to the corresponding lexical units of the main lexicalization scheme.

15. The method of claim 13, wherein translating the output lexical sequence corresponding to the LLM processing task into the corresponding most similar lexical sequence further comprises: Neural machine translation is used, trained to convert lexical units of the first lexicalization scheme into lexical units of the main lexicalization scheme.

16. The method of claim 9, wherein generating the model-independent delay metric based on the lexical-based delay metric and the normalized length output sequence further comprises: The normalized per-GPU token count is determined by dividing the normalized length output token sequence by the number of GPUs deployed within the GPU architecture executing the corresponding LLM processing task; and Divide the lexical generation time associated with the corresponding LLM task in the LLM task by the normalized number of lexical units per GPU.

17. One or more tangible computer-readable storage media, said one or more tangible computer-readable storage media storing processor-executable operations for performing a computer process, said computer process comprising: Receive a lexical-based latency metric (410) that quantifies the latency observed during the execution of an LLM processing task performed by different large language models (LLMs) deployed in different model pools (112) (110), the different LLMs being configured to process lexicals according to different lexicalization schemes; Receive the output lexical sequence generated during the execution of the LLM processing task; A normalized length output sequence (418) is generated by reformatting each output word sequence in the output word sequence using a subject metamorphism scheme. A model-independent delay metric (417) is generated based on the lexical-based delay metric (410) and the normalized length output sequence (418). Pool-specific latency metrics are derived by aggregating subsets (322) of the model-independent latency metrics (417) that correspond to LLM tasks executed in the first pool of the different model pools (112) (110). as well as In response to determining that the pool-specific latency metric for the first pool meets the scaling criteria, additional graphics processing unit (GPU) resources are dynamically allocated to the first pool.

18. One or more tangible computer-readable storage media according to claim 17, wherein generating each of the normalized length output sequences further comprises: A lexicalized input sequence is created by lexicalizing the input string of the corresponding LLM processing task according to the main lexicalization scheme. as well as The lexicalized input sequence is processed by the main LLM to regenerate the output lexical sequence within the vocabulary of the main LLM.

19. The one or more tangible computer-readable storage media of claim 17, wherein generating each normalized length output sequence in the normalized length output sequence further comprises: The output lexical sequence of the corresponding LLM processing task is translated into the most similar lexical sequence within the lexical vocabulary of the main lexicalization scheme.

20. The one or more tangible computer-readable storage media according to claim 19, wherein translating the output lexical sequence corresponding to the LLM processing task into the corresponding most similar lexical sequence further comprises: At least one of the following methods is used: mapping using lexical embedding or mapping using a neural translation model, to map the lexical units of the output lexical unit sequence to the corresponding lexical units of the main lexicalization scheme.

Images