Multi-model and multi-tenant-oriented key value cache scheduling method and device, equipment and medium

Abstract

The invention discloses a multi-model and multi-tenant-oriented key value cache scheduling method and device, equipment and a medium, and relates to the technical field of reasoning acceleration, and the method comprises the steps: obtaining a multi-stage stability portrait of a target large language model, so as to determine a hardware calculation node; the multi-level stability portrait comprises a pre-divided cache scheduling set of the large language model and a preset Top-K sampling reference parameter; the cache scheduling set comprises a head group and / or a layer group; the head group is obtained by aggregating attention heads according to the stability level, and the layer group is obtained by aggregating attention layers based on the stability level of the head group; the stability level is determined based on the stability score division of each attention head, and at least three level types are included; calculating a theoretical video memory occupation amount for key value caching to determine a parameter scaling factor of the request, and determining a Top-K sampling actual parameter; when reasoning processing is carried out, the generated target key value cache is stored in a layered mode, and dynamic scheduling is carried out in the decoding process.

Description

Technical Field

[0001] This invention relates to the field of inference acceleration technology, and in particular to a key-value cache scheduling method, apparatus, device, and medium for multi-model, multi-tenant systems. Background Technology

[0002] The Transformer architecture has become the mainstream architecture for large language models. In the inference phase, the model typically includes a prefill phase and a decoding phase. The prefill phase computes the query Q, key K, and value V vectors in parallel on the input sequence, storing K and V in a KV (key-value) cache. The decoding phase generates output token by token, requiring access to the historical KV cache for self-attention computation at each step. As the context length and generation length increase, the KV cache size grows approximately linearly with both, making memory pressure and access overhead key bottlenecks limiting the deployment efficiency of LLMs (Large Language Models). To alleviate the memory bottleneck, existing technologies propose a paginated attention mechanism. This mechanism measures temporal stability by statistically analyzing the overlap of the Top-K (a sampling strategy for controlling the output of the generative model) page sets of each attention head across different decoding steps, and classifies attention heads into stable and unstable categories, dynamically scheduling the key-value cache accordingly. However, the existing technologies mentioned above still have significant shortcomings in the following aspects: the stability distribution of different attention heads is a continuous spectrum, and the key-value cache management method based solely on stable / unstable binary classification is too absolute and cannot make a more refined trade-off between GPU memory and accuracy; existing methods are mostly designed for single models and single workers (an execution component used to efficiently perform large language model inference on different hardware platforms) under given hyperparameters, which makes it difficult to be compatible with scenarios of multiple concurrent requests and multiple models sharing GPU (i.e., Graphics Processing Unit), resulting in limited efficiency of key-value cache management; and the KV cache is managed only at the "head" granularity, leading to excessively fine metadata redundancy and scheduling granularity.

[0003] In summary, how to improve the efficiency of key-value cache management by implementing multi-model, multi-tenant key-value cache scheduling while ensuring task accuracy is an urgent problem to be solved. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a key-value cache scheduling method, apparatus, device, and medium for multi-model, multi-tenant systems, which can achieve multi-model, multi-tenant key-value cache scheduling while ensuring task accuracy, thereby improving the efficiency of key-value cache management. The specific solution is as follows: Firstly, this application provides a key-value cache scheduling method for multi-model, multi-tenant systems, including: After receiving several pending requests from each tenant, a multi-level stability profile of the target large language model corresponding to the pending requests is obtained to determine the hardware computing nodes corresponding to each of the pending requests. The multi-level stability profile includes a pre-divided cache scheduling set of the large language model and a preset Top-K sampling benchmark parameter corresponding to the cache scheduling set. The cache scheduling set includes a head group and / or a layer group. The head group is a set obtained by aggregating each attention head according to the stability level, and the layer group is a set obtained by aggregating each attention layer based on the stability level of the head group. The stability level is determined based on the stability score of each attention head, and the level types include at least three. The theoretical memory usage for key-value caching of the pending requests is calculated based on the multi-level stability profile. The parameter scaling factor of the several pending requests is determined based on the theoretical memory usage. The actual Top-K sampling parameters of the cache scheduling set are determined using the parameter scaling factor and the Top-K sampling benchmark parameters. During the inference processing of the request to be processed by the hardware computing node, the generated target key-value cache is stored in layers according to the multi-level stability profile, and the target key-value cache is dynamically scheduled according to the actual parameters of the Top-K sampling during the decoding process.

[0005] Optionally, the generation process of the multi-level stability profiles for various language models includes: Obtain the target inference task log of the large language model, and sample several inference processes from the target inference task log; For the inference process, the Top-K page index of each attention head in each decoding step is recorded to obtain the Top-K page set of the decoding step; the Top-K page is the key-value cache page obtained by sampling several key-value cache pages corresponding to the current decoding step using the Top-K technique; The stability score of each attention head is calculated based on the overlap of the Top-K page sets between different decoding steps, and the stability level is divided based on the stability score; Based on the stability level, the attention heads are aggregated to obtain the head group, and based on the stability level of the head group, the attention layers are aggregated to obtain the layer group. Set the Top-K sampling baseline parameters for each stability level.

[0006] Optionally, the calculation process for the stability score of each attention head includes: Calculate the intersection of the Top-K page sets between the first decoding step and the second decoding step to obtain the set overlap, and calculate the difference between the set overlap and the preset random expected overlap. Normalize the difference to obtain the corrected overlap. Within a preset time window, the corrected overlap is averaged or aggregated to determine the result as a stability score.

[0007] Optionally, the Top-K sampling reference parameters include a Top-K quantity reference value and a resampling period reference value; Accordingly, calculating the theoretical memory usage for key-value caching of the request to be processed based on the multi-level stability profile includes: Determine the request inference length characteristics of the request to be processed, and the model structure of the corresponding target large language model; Based on the request inference length characteristics and the model structure, calculate the theoretical number of key-value cache pages for the request to be processed at each of the stability levels; By using the theoretical number of key-value cache pages and the baseline value of the Top-K number, the theoretical value of the Top-K number for each cache scheduling set is calculated to determine the theoretical memory usage for key-value caching of the pending requests.

[0008] Optionally, determining the parameter scaling factor for the plurality of pending requests based on the theoretical video memory usage includes: The requests to be processed are divided according to their respective hardware computing nodes to obtain a set of requests to be processed. Determine the target hardware computing node corresponding to the set of requests to be processed, and determine the preset budget memory usage for key-value caching of the target hardware computing node; For the target pending request in the pending request set, the theoretical video memory usage is compared with the budgeted video memory usage to obtain the corresponding comparison result; If the comparison result indicates that the theoretical video memory usage is not greater than the budgeted video memory usage, then the parameter scaling factor of the target request to be processed is determined to be one. If the comparison result indicates that the theoretical memory usage is greater than the budgeted memory usage, then under the target constraint, the target optimization problem is solved, and the solution is determined as the parameter scaling factor of the target request to be processed.

[0009] Optionally, the step of storing the generated target key-value cache in a hierarchical manner based on the multi-level stability profile includes: If the stability level of the cache scheduling set is the first stability level, then the cache scheduling set containing several key-value cache pages and key-value page summary information of the first group-level Top-K pages are written into the node memory of the corresponding hardware computing node; the group-level Top-K pages are the operation results obtained by performing union or intersection operations on several Top-K page sets in the cache scheduling set; the key-value page summary information includes the minimum key vector and the maximum key vector of the key-value cache page; If the stability level of the cache scheduling set is the second stability level, then the first group-level Top-K pages and the key-value page summary information of the cache scheduling set are written into the node video memory, the key-value cache pages are written into the host memory, and the target history pages in the key-value cache pages are flushed to the external non-volatile memory; the target history page is the key-value cache page where the number of steps between the corresponding decoding step and the current decoding step is greater than a preset number of steps threshold; Wherein, the stability score of the second stability level is higher than the stability score of the first stability level; the target key-value cache is a key-value cache page of a plurality of the cache scheduling sets.

[0010] Optionally, the actual parameters of Top-K sampling include the actual value of the number of Top-K samples and the actual value of the resampling period; Accordingly, the dynamic scheduling of the target key-value cache based on the actual parameters of the Top-K sampling includes: If the current target cache scheduling set is determined to meet the preset resampling conditions, then the importance of several candidate key-value cache pages is scored using the key-value page summary information to obtain the corresponding score results; the preset resampling conditions include the number of steps between the current decoding step and the historical resampling decoding step reaching the actual value of the resampling period, and the historical resampling decoding step being the decoding step that was resampled last time; the candidate key-value cache pages are key-value cache pages stored in the host memory and the external non-volatile memory; A second set of top-K pages is generated based on the scoring results and the actual number of top-K pages. The first set of top-K pages is compared with the second set of top-K pages, and the target key-value cache is migrated and scheduled based on the comparison results.

[0011] Secondly, this application provides a key-value cache scheduling device for multi-model, multi-tenant systems, comprising: The profile acquisition module is used to acquire a multi-level stability profile of the target large language model corresponding to each pending request after receiving several pending requests from each tenant, so as to determine the hardware computing node corresponding to each of the several pending requests. The multi-level stability profile includes a pre-divided cache scheduling set of the large language model and a preset Top-K sampling benchmark parameter corresponding to the cache scheduling set. The cache scheduling set includes a head group and / or a layer group. The head group is a set obtained by aggregating each attention head according to the stability level, and the layer group is a set obtained by aggregating each attention layer based on the stability level of the head group. The stability level is determined based on the stability score of each attention head, and the level types include at least three. The parameter determination module is used to calculate the theoretical video memory usage of the requests to be processed for key-value caching based on the multi-level stability profile, determine the parameter scaling factor of the several requests to be processed based on the theoretical video memory usage, and determine the actual Top-K sampling parameters of the cache scheduling set using the parameter scaling factor and the Top-K sampling benchmark parameters. The cache scheduling module is used to perform hierarchical storage of the generated target key-value cache according to the multi-level stability profile during the inference processing of the request to be processed by the hardware computing node, and to dynamically schedule the target key-value cache according to the actual parameters of the Top-K sampling during the decoding process.

[0012] Thirdly, this application provides an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned key-value cache scheduling method for multi-model, multi-tenant environments.

[0013] Fourthly, this application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned key-value cache scheduling method for multi-model multi-tenancy.

[0014] In this application, after receiving several pending requests from each tenant, a multi-level stability profile of the target large language model corresponding to the pending requests is obtained to determine the hardware computing nodes corresponding to each of the pending requests. The multi-level stability profile includes a pre-divided cache scheduling set of the large language model and preset Top-K sampling benchmark parameters corresponding to the cache scheduling set. The cache scheduling set includes a head group and / or a layer group. The head group is a set obtained by aggregating each attention head according to its stability level, and the layer group is a set obtained by aggregating each attention layer based on the stability level of the head group. The stability level is based on the stability of each attention head. The scoring is determined, with at least three levels. The theoretical memory usage for key-value caching of the pending requests is calculated based on the multi-level stability profile. A parameter scaling factor for the pending requests is determined based on this theoretical memory usage. The Top-K sampling actual parameters of the cache scheduling set are then determined using the parameter scaling factor and the Top-K sampling benchmark parameters. During inference processing of the pending requests via the hardware computing node, the generated target key-value cache is stored hierarchically according to the multi-level stability profile. During decoding, the target key-value cache is dynamically scheduled based on the Top-K sampling actual parameters. As can be seen from the above, after receiving several pending requests from each tenant, this application first obtains a multi-level stability profile of the target large language model corresponding to the request, thereby determining the hardware computing node corresponding to each pending request. This multi-level stability profile covers a pre-divided cache scheduling set of the large language model, and a preset Top-K sampling benchmark parameter corresponding to the cache scheduling set. The cache scheduling set includes a head group formed by aggregating attention heads according to their stability levels, and / or a layer group formed by aggregating attention layers based on the stability levels of the head groups. The stability level is based on the stability rating of each attention head. The system is divided into at least three levels; then, based on the multi-level stability profile, the theoretical memory usage for key-value caching of the requests to be processed is calculated, and the parameter scaling factor for each request to be processed is determined accordingly. The parameter scaling factor is combined with the Top-K sampling benchmark parameter to obtain the actual Top-K sampling parameters of the cache scheduling set. Finally, during the inference processing of the requests on the hardware computing node, the generated target key-value cache is stored in layers according to the multi-level stability profile, and the target key-value cache is dynamically scheduled according to the actual Top-K sampling parameters during the decoding stage.In this way, through the process described above in this application, request scheduling and cache management based on the multi-level stability profile of the large language model can achieve accurate matching of hardware computing resources and efficient utilization of key-value cache. While ensuring the accuracy of long context and long generation tasks, it can effectively reduce the GPU memory occupation pressure during the inference process, improve the concurrent processing capability of the large language model for multi-tenant requests and the overall inference efficiency, and thus achieve multi-model multi-tenant key-value cache scheduling to improve the efficiency of key-value cache management while ensuring task accuracy. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0016] Figure 1 This application discloses a flowchart of a key-value cache scheduling method for multi-model, multi-tenant environments. Figure 2 This is a schematic diagram of the overall architecture of a key-value cache scheduling method for multi-model multi-tenancy disclosed in this application; Figure 3 This is a schematic diagram of a key-value cache scheduling device for multi-model multi-tenancy disclosed in this application; Figure 4 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] To alleviate the memory bottleneck, existing technologies have proposed a paging attention mechanism. This mechanism measures temporal stability by statistically analyzing the overlap of the Top-K page sets for each attention head across different decoding steps, and categorizes attention heads into stable and unstable types for dynamic scheduling of the key-value cache. However, these existing technologies still have significant shortcomings in the following aspects: the stability distribution of different attention heads exhibits a continuous spectrum, making key-value cache management solely based on stable / unstable binary classification rather absolute and unable to make finer trade-offs between memory and accuracy; existing methods are mostly designed for single models and single workers under given hyperparameters, making them incompatible with scenarios involving multiple concurrent requests and multiple models sharing the GPU, resulting in limited efficiency in key-value cache management; and managing the KV cache only at the "head" granularity leads to excessively fine-grained metadata redundancy and scheduling.

[0019] To overcome the aforementioned technical problems, this application provides a key-value cache scheduling method for multiple models and multiple tenants, which can improve the efficiency of key-value cache management by achieving key-value cache scheduling for multiple models and multiple tenants while ensuring task accuracy.

[0020] See Figure 1 As shown, this embodiment of the invention discloses a key-value cache scheduling method for multi-model, multi-tenant systems, including: Step S11: After receiving several pending requests from each tenant, obtain a multi-level stability profile of the target large language model corresponding to the pending requests to determine the hardware computing nodes corresponding to each of the several pending requests; the multi-level stability profile includes a pre-divided cache scheduling set of the large language model and a preset Top-K sampling benchmark parameter corresponding to the cache scheduling set; the cache scheduling set includes a head group and / or a layer group; the head group is a set obtained by aggregating each attention head according to the stability level, and the layer group is a set obtained by aggregating each attention layer based on the stability level of the head group; the stability level is determined based on the stability score of each attention head, and the level types include at least three.

[0021] In this embodiment, after receiving several pending requests from each tenant, a multi-level stability profile of the target large language model corresponding to the request is obtained to determine the hardware computing node, i.e., GPU, corresponding to each pending request. The multi-level stability profile includes a pre-divided cache scheduling set of the large language model and preset Top-K sampling benchmark parameters corresponding to the cache scheduling set. The cache scheduling set covers head groups formed by aggregating each attention head according to its stability level, and / or layer groups formed by aggregating each attention layer based on the stability level of the head group. The stability level is determined based on the stability score of each attention head, and the level types include at least three. The multi-level stability profile is stored in a stability profile library in the form of a table or tensor for online loading.

[0022] It should be noted that, such as Figure 2The diagram shows the overall architecture of a key-value cache scheduling method for multi-model, multi-tenant applications provided in this application. It includes a multi-tenant request queue, a multi-model inference service layer, a global key-value budget scheduler, and inference workers deployed on GPUs. The global KV budget scheduler is set at the cluster level and manages the KV cache quotas for different model instances and different tenant requests in a unified manner. It can simultaneously manage host and NVMe (Non-Volatile Memory Express) budgets, allocating three-level cache shares to each request to achieve "hot and cold" KV management. Internally, it includes a model stability profile library, a tenant and SLA management unit, a budget calculation and allocation unit, and a scheduling instruction issuance unit. The inference worker includes a multi-level stability profile loading module, a header / layer group joint scheduling and Top-K allocation module, and a hierarchical KV cache management module. These three components work together to achieve unified scheduling of multi-level stability-aware KV caches for multi-model services. The system comprises the following components: a model profile management unit, which stores the offline stability profiles of each model, the default Top-K strategy, and the minimum Top-K limit; a tenant and SLA (Service Level Agreement) management unit, which maintains the priorities, QPS (Queries Per Second) targets, and latency constraints of different tenants, and supports prioritizing different tenants of the same model; a budget calculation and adjustment unit, which updates the current active request set and re-solves or incrementally updates the KV budget allocation under request arrival and completion events; and a scheduling instruction issuance unit, which issues the Top-K quota, reordering cycle, and KV hierarchical layout strategy for each request to the corresponding Worker. It should be noted that this embodiment maintains group-level Top-K page metadata and block tables, along with corresponding Min-Max caches, in the GPU for scoring host sidepages without loading the KV content. It is understood that the key-value cache scheduling method for multi-model multi-tenancy in this application is a method that can be compatible with multi-request concurrency and multi-model shared GPU scenarios. The above scenarios include, but are not limited to, cloud service provider multi-tenant LLM API (Application Programming Interface) services, enterprise internal multi-scenario intelligent service integration, scientific research institution multi-model comparison and experimental platform, multi-modal intelligent terminal on edge computing device, etc.

[0023] It should be noted that the generation process of the multi-level stability profile of the major language models is as follows: The target inference task log of the major language model is obtained, and several inference processes are sampled from the target inference task log; for each inference process, the Top-K page index of each attention head in each decoding step is recorded to obtain the Top-K page set of the decoding step; the Top-K pages are key-value cache pages obtained by sampling several key-value cache pages corresponding to this decoding step using Top-K technology; the stability score of each attention head is calculated based on the overlap of the Top-K page sets between different decoding steps, and the stability level is divided based on the stability score; based on the stability level, each attention head is aggregated to obtain the head group, and each attention layer is aggregated based on the stability level of the head group to obtain the layer group; the Top-K sampling benchmark parameter is set for each stability level. That is, for each large language model deployed to the target, the target inference task log of the large language model is obtained, that is, a typical representative long context and long generation task log, and several inference processes are sampled from it. For each inference process, the Top-K page index of the key-value cache page of each layer and each attention head in each decoding step of the decoding stage is recorded, that is, the page index sequence in the Top-K page selection, so as to form the Top-K page set of the corresponding decoding step. The Top-K pages are key-value cache pages obtained by sampling several key-value cache pages corresponding to the current decoding step through Top-K technology. Then, based on the overlap of the Top-K page sets between different decoding steps, the stability of each attention head is calculated. The stability of each attention head is evaluated, and based on the evaluation, a multi-threshold or clustering algorithm is used to classify each attention head into at least three stability levels, such as Level 1: highly stable head, Level 2: moderately stable head, Level 3: low-stable head, and Level 4: unstable head (optional). Then, attention heads with similar stability curves are aggregated according to the stability level to obtain head groups. For example, several highly stable heads in the same layer form a head group. At the same time, the inter-layer correlation is determined based on the stability level of the head group. If the stability patterns of some head groups in multiple layers are similar, the attention layers are further aggregated to obtain layer groups so that the scheduling strategy can be shared between layers. Finally, a corresponding Top-K sampling benchmark parameter is set for each stability level.

[0024] It should be further noted that the calculation process for the stability score of each attention head is as follows: The intersection of the Top-K page sets between the first decoding step and the second decoding step is calculated to obtain the set overlap. The difference between the set overlap and the preset random expected overlap is calculated, and the difference is normalized to obtain the corrected overlap. Within a preset time window, the corrected overlap is averaged or aggregated to determine the result as the stability score. Here, the preset random expected overlap is the expected intersection size under randomly selected conditions. That is, for a given attention head, the intersection of the Top-K page set between the first decoding step and at least one offset, i.e., the adjacent or offset second decoding step, is calculated to obtain the set overlap. Then, the difference between the set overlap and the preset random expected overlap is calculated and the difference is normalized to obtain the corrected overlap. Subsequently, the corrected overlap is averaged or aggregated within a preset time window, and the result is determined as the stability score of the attention head. Specifically, within the preset time window, the corrected overlap is averaged to obtain the stability score within that window. Then, the results of different windows are averaged or quantiles are used to obtain the final stability score. In this way, the request and hardware computing node matching method based on multi-level stability profiles in this embodiment can combine the hierarchical aggregation features of the attention mechanism of large language models with sampling benchmark parameters to achieve precise scheduling of computing resources, thereby improving the concurrent processing efficiency and resource utilization of multi-tenant requests. From inference task log analysis to stability level classification, and then to cache scheduling set construction and sampling parameter configuration, the operational characteristics of the attention mechanism of large language models can be accurately characterized, improving the overall efficiency and resource utilization of model inference. Through the stability score calculation method that combines set intersection analysis, difference correction and time window operation, the interference of random fluctuation factors on the score results can be effectively filtered out, and the cache access stability of the attention head in different decoding steps can be accurately quantified, providing data support for subsequent steps.

[0025] Step S12: Calculate the theoretical video memory usage of the pending requests for key-value caching based on the multi-level stability profile, determine the parameter scaling factor of the several pending requests based on the theoretical video memory usage, and determine the actual Top-K sampling parameters of the cache scheduling set using the parameter scaling factor and the Top-K sampling benchmark parameters.

[0026] In this embodiment, the theoretical memory usage of the pending requests for key-value caching is calculated based on the multi-level stability profile. Based on this, the parameter scaling factor corresponding to the several pending requests is determined. Then, the Top-K sampling actual parameters of the cache scheduling set are calculated by combining the parameter scaling factor with the Top-K sampling benchmark parameters, and the Top-K sampling actual parameters are sent to the corresponding inference worker.

[0027] It should be noted that the Top-K sampling benchmark parameters include a Top-K quantity benchmark value and a resampling period benchmark value. Correspondingly, the processing flow for calculating the theoretical GPU memory usage for key-value caching of the request to be processed is as follows: determine the request inference length characteristics of the request to be processed and the corresponding model structure of the target large language model; calculate the theoretical number of key-value cache pages for the request to be processed at each stability level based on the request inference length characteristics and the model structure; calculate the theoretical Top-K quantity for each cache scheduling set using the theoretical number of key-value cache pages and the Top-K quantity benchmark value, thereby determining the theoretical GPU memory usage for key-value caching of the request to be processed. That is, the Top-K sampling benchmark parameters include a Top-K quantity benchmark value. Compared with the resampling period reference value The higher the stability, the smaller the Top-K number baseline value and the longer the reordering cycle baseline value; the lower the stability, the larger the Top-K number baseline value and the shorter the reordering cycle baseline value, or reordering every step. The corresponding theoretical memory usage calculation process is as follows: At a certain moment, for each request in the current batch, first determine the request inference length characteristics of the request to be processed, such as context length, estimated maximum generation length, and the model structure of the corresponding target large language model. Then, combine the request inference length characteristics and the model structure to calculate the theoretical number of key-value cache pages for the request to be processed under each stability level. Finally, through the theoretical number of key-value cache pages and the Top-K number baseline value, calculate the theoretical Top-K number value of each cache scheduling set, such as the number and size of the Top-K pages that each head group needs to reside in the GPU, thereby estimating the theoretical memory usage of a single request and the entire batch, and thus determining the theoretical memory usage of the request to be processed for key-value caching.

[0028] It should be further noted that the process of determining the parameter scaling factor of the plurality of pending requests based on the theoretical memory usage is as follows: The plurality of pending requests are divided according to their respective corresponding hardware computing nodes to obtain a plurality of pending request sets; the target hardware computing node corresponding to the pending request set is determined, and the budgeted memory usage for key-value caching of the target hardware computing node is determined; for the target pending request in the pending request set, the theoretical memory usage is compared with the budgeted memory usage to obtain a corresponding comparison result; if the comparison result indicates that the theoretical memory usage is not greater than the budgeted memory usage, then the parameter scaling factor of the target pending request is determined to be one; if the comparison result indicates that the theoretical memory usage is greater than the budgeted memory usage, then under the target constraint, the target optimization problem is solved, and the obtained solution is determined as the parameter scaling factor of the target pending request. That is, the several pending requests are divided according to their corresponding hardware computing nodes to obtain several sets of pending requests sharing the same GPU. Then, the target hardware computing node corresponding to each set of pending requests is determined, as well as the budgeted video memory usage of the key-value cache of the target hardware computing node. For a target request in the set of requests to be processed, the theoretical memory usage is compared with the budgeted memory usage. If the theoretical memory usage does not exceed the budgeted memory usage, then the parameter scaling factor of the target request is adjusted. If the theoretical memory usage exceeds the budgeted memory usage, then the objective optimization problem is solved under the objective constraint to ensure that the total memory usage does not exceed 1. The solution result is then used as the parameter scaling factor for the target request to be processed.

[0029] It should be noted that the objective optimization problem can be: maximizing cluster-weighted throughput, minimizing estimated decoding time, and maximizing concurrency while satisfying QoS (Quality of Service). The objective constraints include: the total key-value memory usage of all requests not exceeding the preset GPU key-value cache memory budget; the actual Top-K for any stable level not being lower than the preset minimum Top-K for that level; and maintaining a certain preset resource fairness or priority relationship based on tenant or model. It is understood that when prioritization is required, different allocations can be made to different requests based on tenant weight or latency requirements. In a simplified implementation, if the theoretical video memory usage exceeds the budgeted video memory usage, the scaling factor can be determined using the following formula: ; in, This represents the theoretical memory usage. In this embodiment, by dynamically adjusting parameters based on the theoretical memory usage, precise matching between cache sampling parameters and hardware resource capacity can be achieved. This ensures the performance of large language model inference while effectively controlling the memory consumption of the key-value cache, improving resource utilization and concurrent processing efficiency in multi-tenant request scenarios. Calculating memory usage by combining request characteristics, model structure, and sampling baseline parameters enables accurate prediction of key-value cache resource consumption, ensuring the rationality and efficiency of resource allocation during large language model inference. Determining the parameter scaling factor based on the difference between hardware node computing power and memory budget allows for dynamic adaptation of key-value cache resources during large language model inference, maximizing concurrent processing efficiency and inference performance for multi-tenant requests while meeting hardware resource constraints.

[0030] Step S13: During the inference processing of the request to be processed by the hardware computing node, the generated target key-value cache is stored in layers according to the multi-level stability profile, and during the decoding process, the target key-value cache is dynamically scheduled according to the actual parameters of the Top-K sampling.

[0031] In this embodiment, during the inference processing of the hardware computing node for the request to be processed, the generated target key-value cache is stored in layers according to the multi-level stability profile, and the inference worker dynamically schedules the target key-value cache according to the actual parameters of the Top-K sampling during the decoding process.

[0032] It should be noted that, to further reduce redundant storage among multiple heads, a joint scheduling strategy of head group and layer group is adopted. The layered storage includes GPU memory, host memory, and external non-volatile storage, i.e., NVMe. The processing flow for layered storage of the generated target key-value cache based on the multi-level stability profile is as follows: If the stability level of the cache scheduling set is the first stability level, then several key-value cache pages and key-value page summary information of the first group level Top-K pages in the cache scheduling set are written into the node memory of the corresponding hardware computing node; the group level Top-K pages are the operation results obtained by performing union or intersection operations on several Top-K page sets in the cache scheduling set; the The key-value page summary information includes the minimum key vector and the maximum key vector of the key-value cache page; if the stability level of the cache scheduling set is the second stability level, then the first group-level Top-K pages of the cache scheduling set and the key-value page summary information are written to the node video memory, the key-value cache pages are written to the host memory, and the target historical pages in the key-value cache pages are flushed to external non-volatile memory; the target historical page is a key-value cache page where the number of steps between the corresponding decoding step and the current decoding step is greater than a preset step threshold; wherein, the stability score of the second stability level is higher than the stability score of the first stability level; the target key-value cache is a number of key-value cache pages of the cache scheduling sets. That is, if the stability level of the cache scheduling set is the first stability level, such as low stability or even unstable, then the set containing several key-value cache pages of the first group level Top-K pages and key-value page summary information are written into the node memory of the corresponding hardware computing node, so as to keep more pages on the GPU at least in the most recent decoding steps to reduce frequent data migration. The key-value page summary information contains the minimum key vector and maximum key vector of the key-value cache page, that is, the Min-Max cache. If the stability level of the cache scheduling set is the second stability level with a stability score higher than the first stability level, such as high stability or medium stability, then the first group level Top-K pages of the set and the key-value page summary information are written into the node memory, and the key-value cache pages are written into the host memory. After the pages are filled, the target historical pages in the key-value cache pages, that is, old pages (e.g., pages that are more than a certain threshold away from the current decoding step), are flushed to external non-volatile memory. The host only retains pages that may be reused in the recent period of time. The target key-value cache is the key-value cache pages of several cache scheduling sets. It should be noted that the group-level Top-K pages are the result of performing a union or intersection operation on several Top-K page sets in the cache scheduling set, which reduces duplicate storage across headers. If a union strategy is used, duplicate pages can be reduced from being loaded repeatedly in the same group header; if an intersection strategy is used, the number of pages can be further reduced in extreme compression scenarios, at the expense of a small amount of precision.

[0033] Understandably, for heads belonging to stability levels 1 to n, a joint caching layout is implemented through head groups / layer groups: multiple attention heads within a head group share a single Top-K page set or a shared physical page block pool, maintaining only a lightweight logical mapping; group-level pages occupy only one physical block in the block table, and the head only accesses the corresponding key-value pair through the head group index and offset; when the group-level Top-K is updated, only the block mapping and some metadata of the group need to be updated, without updating each head in the group separately, reducing the complexity of block table updates and "dirty area" synchronization; at the layer group level, the same Top-K budget and reordering cycle can be used for multiple layers, for example, the same "medium stability level configuration" can be used for several middle layers, further simplifying the scheduling logic. For a single request, the first round of large-scale GPU-to-host key-value (KV) flushing is initiated immediately after pre-filling, migrating the historical pages of the stable header group to the host. During the decoding phase, the KV rearrangement behavior of different requests is triggered in staggered shifts, so that at any given time only a small number of requests are loading host-to-GPU pages, while other requests continue decoding and computation. Based on the real-time queue length and historical statistics, the batch size of each tenant and the secondary batch fusion strategy are dynamically adjusted to further balance TTFT (Time To First Token, the latency from input to output of the first token) and TPOT (Time Per Output Token, the average generation time of each output token).

[0034] It should be further noted that the actual Top-K sampling parameters include the actual value of the Top-K quantity and the actual value of the resampling period. Therefore, the process of determining the actual Top-K sampling parameters of the cache scheduling set includes: for each head group, the specific formula for setting the actual value of the Top-K quantity is as follows: ; in, The actual number of Top-K items; max represents taking the maximum value; This represents the minimum Top-K allowed for this level; `round` denotes rounding to the nearest integer. The reordering cycle is also adjusted as needed. If the size is too small and the Top-K is significantly compressed, the reordering cycle of the corresponding level is shortened, and the reordering frequency is increased to compensate for information loss and mitigate the impact of Top-K reduction on accuracy. Accordingly, the processing flow for dynamically scheduling the target key-value cache based on the actual Top-K sampling parameters is as follows: If the current target cache scheduling set is determined to meet the preset resampling conditions, then the importance of several candidate key-value cache pages is scored using the key-value page summary information to obtain the corresponding scoring results; the preset resampling conditions include the number of steps between the current decoding step and the historical resampling decoding step reaching the actual value of the resampling cycle, where the historical resampling decoding step is the decoding step that was resampled last time; the candidate key-value cache pages are key-value cache pages stored in the host memory and the external non-volatile memory; a second group of top-K pages is generated based on the scoring results and the actual value of the Top-K quantity; the first group of top-K pages is compared with the second group of top-K pages, and the target key-value cache is migrated and scheduled based on the comparison results. That is, if the current target cache scheduling set is determined to meet the preset resampling condition, that is, the number of steps between the current decoding step and the previous historical resampling decoding step reaches the actual value of the resampling period, then the importance score of several candidate key-value cache pages stored in the host memory and the external non-volatile memory is performed by using the key-value page summary information. Specifically, the dot product operation is performed on the current query vector and the key-value page summary information, and the sum of the maximum values ​​of each dimension is taken as the upper bound score of the key-value page. Based on the score result and the actual value of the Top-K number, a second group of top-K pages is generated, which is compared with the first group of top-K pages, and the target key-value cache is migrated and scheduled according to the comparison result. Specifically, based on the comparison results, promoted pages (not included in the first group-level Top-K pages but included in the second group-level Top-K pages) and degraded pages (not included in the second group-level Top-K pages but included in the first group-level Top-K pages) are determined. Host-to-GPU migration is triggered for promoted pages, and GPU-to-host or GPU-to-NVMe migration is triggered for degraded pages. Migration operations and decoding computations are performed in parallel and overlapped as much as possible through multiple CUDA streams (a type of serialized command queue). A dedicated CUDA kernel directly reads and writes fixed-page memory on the host machine, and low-priority CUDA streams are executed in parallel with the decoding computation stream, thereby reducing interference from host-GPU data transfer to forward inference. When the scoring results indicate that a page becomes highly important again, it is preferentially read from external non-volatile memory into the host cache and then loaded onto the GPU as needed. It is understood that during attention computation, only the key-value cache corresponding to the group-level Top-K pages is accessed.

[0035] It should be noted that when a new request arrives, the global scheduler calculates the estimated response time and resource overhead for that request on each candidate GPU based on the request's model, tenant and its SLA, the current number of active requests on each GPU, the remaining KV memory budget, and the historical queue length. It then selects the optimal GPU and pre-allocates its KV budget. When the number of active requests on a GPU changes or memory pressure increases sharply (i.e., memory usage exceeds a threshold), the scaling factor for each request is adjusted. This involves tightening or loosening the overall Top-K hierarchy, temporarily lowering the Top-K of higher stability headgroups for low-priority tenants, thereby freeing up memory for high-priority requests. In extreme cases, request migration may be triggered, such as during hot migration, migrating only a small number of Top-K pages and necessary metadata from stable groups. Additionally, the memory usage of low-priority requests can be reduced. This approach compresses the Top-K cache and migrates some low-priority requests to other GPUs or queues them. By combining model stability characteristics with tiered cache storage and dynamic scheduling, this embodiment achieves fine-grained management of key-value cache resources. While ensuring the inference accuracy of large language models, it effectively improves cache access efficiency and resource utilization, optimizing overall inference performance in multi-tenant request scenarios. Based on a tiered key-value cache storage strategy with differentiated stability levels, it can accurately allocate multi-level storage resources by combining the access frequency and importance of cache data with different stability levels. This ensures efficient reading of frequently accessed cache data while effectively reducing the pressure on node GPU memory. Dynamic cache scheduling, triggered by resampling cycles, filtered by importance scoring, and compared with group-level Top-K pages, enables precise flow of key-value cache data between different storage levels. This further optimizes the configuration and utilization of multi-level storage resources while ensuring cache data access efficiency during large language model inference.

[0036] As can be seen from the above, after receiving several pending requests from each tenant, this embodiment first obtains a multi-level stability profile of the target large language model corresponding to the request, thereby determining the hardware computing node corresponding to each pending request. This multi-level stability profile covers a pre-divided cache scheduling set of the large language model, and preset Top-K sampling benchmark parameters corresponding to the cache scheduling set. The cache scheduling set includes a head group formed by aggregating attention heads according to their stability levels, and / or a layer group formed by aggregating attention layers based on the stability levels of the head groups. The stability level is based on the stability of each attention head. The stability profile is divided into at least three levels; then, based on the multi-level stability profile, the theoretical memory usage for key-value caching of the requests to be processed is calculated, and the parameter scaling factor of each request to be processed is determined accordingly. The parameter scaling factor is combined with the Top-K sampling benchmark parameter to obtain the actual Top-K sampling parameters of the cache scheduling set. Finally, during the inference processing of the requests on the hardware computing node, the generated target key-value cache is stored in layers according to the multi-level stability profile, and the target key-value cache is dynamically scheduled according to the actual Top-K sampling parameters during the decoding stage. In this way, through the above-described process of the embodiments of this application, the header is divided into multiple stability levels, and different Top-K benchmarks and reordering cycles are set for different levels. This allows for cache management based on the stability levels of different headers, achieving a more refined trade-off between precision and video memory. By tightly coupling the Top-K configuration with the key-value video memory budget, and solving the global budget allocation problem to obtain the scaling factor for each request, the utilization of GPU resources is improved during multi-request concurrency. The introduction of header groups and layer groups leverages the similarity between attention headers to perform KV page reuse and joint scheduling at the group level, reducing redundant page storage across headers and lowering the management overhead of block tables and scheduling metadata, thus helping to improve the overall efficiency of scheduling and reordering behavior. This can be achieved on a single GPU. It can simultaneously manage multiple model instances and multi-tenant requests on GPU clusters, enabling unified planning of key-value memory across models. It supports differentiated resource allocation and preemption based on tenant weight or SLA, which is beneficial to improving overall throughput and QoS in cloud service environments. Request scheduling and cache management based on multi-level stability profiles of large language models can achieve accurate matching of hardware computing resources and efficient utilization of key-value caches. While ensuring the accuracy of long contexts and long generation tasks, it effectively reduces the memory usage pressure during inference, improves the concurrent processing capability and overall inference efficiency of large language models for multi-tenant requests, and thus achieves multi-model, multi-tenant key-value cache scheduling to improve the efficiency of key-value cache management while ensuring task accuracy.

[0037] As can be seen from the previous embodiment, this application discloses a key-value cache scheduling method for multiple models and multiple tenants, which can improve the efficiency of key-value cache management by achieving key-value cache scheduling for multiple models and multiple tenants while ensuring task accuracy. Next, taking the LLM API service provided by a cloud service provider, where tenants can call large language models of different sizes and functions through the API, as an example, we will provide a detailed explanation of the key-value cache scheduling method for multiple models and multiple tenants.

[0038] It should be noted that the target tenants include: Tenant A (Enterprise Customer, High Priority): using Llama 2 (an open-source large model, 70B) for long document analysis, requiring latency ≤500ms and QPS ≥10; Tenant B (Developer User, Medium Priority): using ChatGLM3 (a Chinese-English bilingual dialogue model, 6B) for intelligent dialogue, requiring latency ≤300ms and QPS ≥20; Tenant C (Free User, Low Priority): using Phi-2 (a language model, 2.7B) for simple question-and-answer, allowing latency ≤1s and QPS ≥5. The shared resources for these tenants include a cluster of 8 NVIDIA A100 GPUs (a GPU chip, 80GB of VRAM), with each GPU required to handle concurrent requests from multiple models simultaneously.

[0039] First, multi-model, multi-level stability profiles are constructed offline. Offline profiling is performed on each deployed model (Llama 2-70B, ChatGLM3-6B, Phi-2-2.7B): Typical long-context tasks for each model are selected (e.g., Llama2 processes 10240-token documents, ChatGLM3 processes 4096-token conversations), recording the Top-K page index sequence for each layer and attention head during the decoding phase. For each attention head, the corrected overlap of the Top-K page set of adjacent decoding steps (e.g., steps s and s+10) is calculated (the difference between the set overlap and the expected random overlap). The average is calculated over a preset time window (e.g., 50 decoding steps) to obtain a stability score. The scores are divided into four levels of stability: extremely stable (score ≥ 0.8), highly stable (0.6~0.8), moderately stable (0.4~0.6), and unstable (<0.4). Attention heads with similar stable patterns are aggregated into head groups (e.g., Llama...). Attention heads in layers 1-8 of 2-70B are aggregated into "extremely stable head group G1", and those in layers 24-32 are aggregated into "unstable head group G4". Layers are then associated to form layer groups (e.g., "bottom extremely stable layer group L1" and "top unstable layer group L4"). Top-K baseline values ​​and reordering periods are set for each stability level, as shown in Table 1 below. Table 1

[0040] Subsequently, during the online inference phase, global key-value budget scheduling and adaptive Top-K allocation are performed: stability profiles and tenant SLAs for all models are loaded (priority: A>B>C; latency / QPS constraints); the single-GPU key-value memory budget is set to 60GB (20GB reserved for model weights and computation); tenant A initiates 10 concurrent requests (Llama 2-70B, context length 8192 tokens), tenant B initiates 20 concurrent requests (ChatGLM3-6B, context length 2048 tokens), and tenant C initiates 5 concurrent requests (Phi-2-2.7B, context length 1024 tokens). The nominal memory usage of each request is estimated using the global scheduler (based on K_base): Llama 2-70B single request nominal usage: extremely stable header group (8 pages / group × 10 groups) + highly stable header group (32 pages / group × 8 groups) + ... ≈ 4GB; ChatGLM3-6B single request nominal usage ≈ 1.2GB; Phi-2-2.7B single request nominal usage ≈ 0.5GB; total nominal usage: 10×4 + 20×1.2 + 5×0.5 = 40 + 24 + 2.5 = 66.5GB > 60GB (single GPU budget), requiring a scaling factor α.

[0041] The scaling factors are calculated, with the optimization objective being to maximize weighted throughput (weight = tenant priority: A=3, B=2, C=1), constraining total video memory usage to ≤60GB and single request Top-K to ≥ minimum threshold (e.g., extremely stable header group K_min=4). The scaling factors are calculated as follows: α_A=1.0 (high priority, no compression), α_B=0.9, α_C=0.8. The actual Top-K is calculated as follows: for example, for ChatGLM3-6B, the extremely stable header group K_act=8×0.9=7, and the moderately stable header group K_act=64×0.9=58. Finally, joint scheduling and hierarchical storage of header / layer groups are performed: GPU memory layer: only the actual Top-K pages of each header group are retained (e.g., the full 128 pages of G4 group and 7 pages of G1 group in Llama 2-70B), total memory usage = 10×4 + 20×(1.2×0.9) + 5×(0.5×0.8) = 40 + 21.6 + 2 = 63.6GB (exceeding the budget, further adjusting the low-priority tenant α_C=0.7, the final total usage = 59.8GB); host memory layer: storing the full KV pages of extremely stable / highly stable header groups (e.g., the full pages of G1 group in Llama 2-70B), quickly scored by the GPU resident Min-Max vector; NVMe cold storage layer: storing extremely old KV pages (e.g., pages that are >500 from the current decoding step). In addition, dynamic adjustments can be made at runtime: when tenant A adds 5 requests, the global scheduler triggers budget reallocation: reducing tenant C's α_C=0.5 (extremely stable header group K_act=4), freeing up GPU memory to accommodate the new requests; during the decoding phase, page migration is triggered for the header group according to the reordering cycle (e.g., G1 group is reordered every 100 steps, loading high-scoring pages from host memory to the GPU), and the migration and decoding calculations are executed in parallel through multiple CUDA streams to avoid increased latency.

[0042] Accordingly, see Figure 3 As shown in the embodiments of this application, a key-value cache scheduling device for multi-model multi-tenancy is also provided, including: The profile acquisition module 11 is used to acquire a multi-level stability profile of the target large language model corresponding to the pending requests after receiving several pending requests from each tenant, so as to determine the hardware computing nodes corresponding to each of the several pending requests; the multi-level stability profile includes a pre-divided cache scheduling set of the large language model and a preset Top-K sampling benchmark parameter corresponding to the cache scheduling set; the cache scheduling set includes a head group and / or a layer group; the head group is a set obtained by aggregating each attention head according to the stability level, and the layer group is a set obtained by aggregating each attention layer based on the stability level of the head group; the stability level is determined based on the stability score of each attention head, and the level types include at least three types; The parameter determination module 12 is used to calculate the theoretical video memory usage of the request to be processed for key-value caching based on the multi-level stability profile, determine the parameter scaling factor of the several requests to be processed based on the theoretical video memory usage, and determine the actual Top-K sampling parameters of the cache scheduling set using the parameter scaling factor and the Top-K sampling benchmark parameters. The cache scheduling module 13 is used to perform hierarchical storage of the generated target key-value cache according to the multi-level stability profile during the inference processing of the request to be processed by the hardware computing node, and to dynamically schedule the target key-value cache according to the actual parameters of the Top-K sampling during the decoding process.

[0043] In some specific embodiments, the image acquisition module 11 may specifically include: The log acquisition unit is used to acquire the target inference task log of the large language model, so as to sample several inference processes from the target inference task log; The index recording unit is used to record the Top-K page index of each attention head in each decoding step for the inference process, so as to obtain the Top-K page set of the decoding step; the Top-K page is the key-value cache page obtained by sampling several key-value cache pages corresponding to the current decoding step through the Top-K technique; The scoring calculation unit is used to calculate the stability score of each attention head based on the overlap of the Top-K page sets between different decoding steps, so as to classify the stability level based on the stability score; An attention layer aggregation unit is configured to aggregate the attention heads based on the stability level to obtain the head group, and aggregate the attention layers based on the stability level of the head group to obtain the layer group. The parameter setting unit is used to set the Top-K sampling reference parameters for each stability level.

[0044] In some specific embodiments, the image acquisition module 11 may specifically include: The normalization unit is used to calculate the intersection of the Top-K page sets between the first decoding step and the second decoding step to obtain the set overlap, and to calculate the difference between the set overlap and the preset random expected overlap, and to normalize the difference to obtain the corrected overlap. The aggregation calculation unit is used to perform averaging or aggregation calculations on the corrected overlap within a preset time window, so as to determine the calculation result as a stability score.

[0045] In some specific implementations, the Top-K sampling reference parameters include a Top-K quantity reference value and a resampling period reference value; Accordingly, the parameter determination module 12 may specifically include: The structure determination unit is used to determine the request inference length feature of the request to be processed, and the model structure of the corresponding target large language model; The quantity calculation unit is used to calculate the theoretical number of key-value cache pages for the request to be processed at each of the stability levels based on the request inference length characteristics and the model structure. The theoretical value calculation unit is used to calculate the theoretical Top-K number of each cache scheduling set by using the theoretical number of key-value cache pages and the Top-K number baseline value, so as to determine the theoretical memory usage of the pending request for key-value caching.

[0046] In some specific embodiments, the parameter determination module 12 may specifically include: A node partitioning unit is used to partition the several requests to be processed according to their respective corresponding hardware computing nodes to obtain several sets of requests to be processed. The occupancy determination unit is used to determine the target hardware computing node corresponding to the set of requests to be processed, and to determine the budgeted video memory occupancy of the target hardware computing node for key-value caching. The memory usage comparison unit is used to compare the theoretical memory usage with the budgeted memory usage for a target request in the set of requests to be processed, so as to obtain the corresponding comparison result. A factor determination unit is used to determine the parameter scaling factor of the target request to be processed as one if the comparison result indicates that the theoretical video memory usage is not greater than the budgeted video memory usage. The problem-solving unit is used to solve the target optimization problem under the target constraint if the comparison result indicates that the theoretical memory usage is greater than the budgeted memory usage, so as to determine the obtained solution as the parameter scaling factor of the target request to be processed.

[0047] In some specific embodiments, the cache scheduling module 13 may specifically include: The information writing unit is configured to, if the stability level of the cache scheduling set is the first stability level, write a plurality of key-value cache pages and key-value page summary information of the cache scheduling set containing the first group-level Top-K pages into the node memory of the corresponding hardware computing node; the group-level Top-K pages are the operation results obtained by performing union or intersection operations on a plurality of Top-K page sets in the cache scheduling set; the key-value page summary information includes the minimum key vector and the maximum key vector of the key-value cache page; The history page flushing unit is used to, if the stability level of the cache scheduling set is the second stability level, write the first group-level Top-K pages and the key-value page summary information of the cache scheduling set into the node video memory, write the key-value cache pages into the host memory, and flush the target history page in the key-value cache pages to the external non-volatile memory; the target history page is the key-value cache page where the number of steps between the corresponding decoding step and the current decoding step is greater than a preset number of steps threshold; Wherein, the stability score of the second stability level is higher than the stability score of the first stability level; the target key-value cache is a key-value cache page of a plurality of the cache scheduling sets.

[0048] In some specific implementations, the actual Top-K sampling parameters include the actual value of the Top-K number and the actual value of the resampling period; Accordingly, the cache scheduling module 13 may specifically include: An importance scoring unit is used to score the importance of several candidate key-value cache pages based on the key-value page summary information if the current target cache scheduling set is determined to meet a preset resampling condition, so as to obtain the corresponding scoring result. The preset resampling condition includes the number of steps between the current decoding step and the historical resampling decoding step reaching the actual value of the resampling period, where the historical resampling decoding step is the decoding step that was resampled last time. The candidate key-value cache pages are key-value cache pages stored in the host memory and the external non-volatile memory. The page comparison unit is used to generate a second group of top-K pages based on the scoring results and the actual value of the number of top-K pages, compare the first group of top-K pages with the second group of top-K pages, and perform migration scheduling of the target key-value cache based on the comparison results.

[0049] Furthermore, embodiments of this application also disclose an electronic device, Figure 4 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the key-value cache scheduling method for multi-model multi-tenancy disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be a computer.

[0050] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0051] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0052] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the multi-model, multi-tenant key-value cache scheduling method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0053] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed key-value cache scheduling method for multi-model, multi-tenant environments. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0054] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0055] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0056] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0057] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0058] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A key-value cache scheduling method for multi-model, multi-tenant environments, characterized in that, include: After receiving several pending requests from each tenant, a multi-level stability profile of the target large language model corresponding to the pending requests is obtained to determine the hardware computing nodes corresponding to each of the pending requests. The multi-level stability profile includes a pre-divided cache scheduling set of the large language model and a preset Top-K sampling benchmark parameter corresponding to the cache scheduling set. The cache scheduling set includes a head group and / or a layer group. The head group is a set obtained by aggregating each attention head according to the stability level, and the layer group is a set obtained by aggregating each attention layer based on the stability level of the head group. The stability level is determined based on the stability score of each attention head, and the level types include at least three. The theoretical memory usage for key-value caching of the pending requests is calculated based on the multi-level stability profile. The parameter scaling factor of the several pending requests is determined based on the theoretical memory usage. The actual Top-K sampling parameters of the cache scheduling set are determined using the parameter scaling factor and the Top-K sampling benchmark parameters. During the inference processing of the request to be processed by the hardware computing node, the generated target key-value cache is stored in layers according to the multi-level stability profile, and the target key-value cache is dynamically scheduled according to the actual parameters of the Top-K sampling during the decoding process.

2. The key-value cache scheduling method for multi-model multi-tenancy according to claim 1, characterized in that, The generation process of the multi-level stability profiles of various language models includes: Obtain the target inference task log of the large language model, and sample several inference processes from the target inference task log; For the inference process, the Top-K page index of each attention head in each decoding step is recorded to obtain the Top-K page set of the decoding step; the Top-K page is the key-value cache page obtained by sampling several key-value cache pages corresponding to the current decoding step using the Top-K technique; The stability score of each attention head is calculated based on the overlap of the Top-K page sets between different decoding steps, and the stability level is divided based on the stability score; Based on the stability level, the attention heads are aggregated to obtain the head group, and based on the stability level of the head group, the attention layers are aggregated to obtain the layer group. Set the Top-K sampling baseline parameters for each stability level.

3. The key-value cache scheduling method for multi-model multi-tenancy according to claim 2, characterized in that, The calculation process for the stability score of each attention head includes: Calculate the intersection of the Top-K page sets between the first decoding step and the second decoding step to obtain the set overlap, and calculate the difference between the set overlap and the preset random expected overlap. Normalize the difference to obtain the corrected overlap. Within a preset time window, the corrected overlap is averaged or aggregated to determine the result as a stability score.

4. The key-value cache scheduling method for multi-model multi-tenancy according to claim 1, characterized in that, The Top-K sampling reference parameters include the Top-K quantity reference value and the resampling period reference value; Accordingly, calculating the theoretical memory usage for key-value caching of the request to be processed based on the multi-level stability profile includes: Determine the request inference length characteristics of the request to be processed, and the model structure of the corresponding target large language model; Based on the request inference length characteristics and the model structure, calculate the theoretical number of key-value cache pages for the request to be processed at each of the stability levels; By using the theoretical number of key-value cache pages and the baseline value of the Top-K number, the theoretical value of the Top-K number for each cache scheduling set is calculated to determine the theoretical memory usage for key-value caching of the pending requests.

5. The key-value cache scheduling method for multi-model multi-tenancy according to claim 1, characterized in that, The step of determining the parameter scaling factor for the plurality of pending requests based on the theoretical video memory usage includes: The requests to be processed are divided according to their respective hardware computing nodes to obtain a set of requests to be processed. Determine the target hardware computing node corresponding to the set of requests to be processed, and determine the preset budget memory usage for key-value caching of the target hardware computing node; For the target pending request in the pending request set, the theoretical video memory usage is compared with the budgeted video memory usage to obtain the corresponding comparison result; If the comparison result indicates that the theoretical video memory usage is not greater than the budgeted video memory usage, then the parameter scaling factor of the target request to be processed is determined to be one. If the comparison result indicates that the theoretical memory usage is greater than the budgeted memory usage, then under the target constraint, the target optimization problem is solved, and the solution is determined as the parameter scaling factor of the target request to be processed.

6. The key-value cache scheduling method for multi-model multi-tenancy according to any one of claims 1 to 5, characterized in that, The step of storing the generated target key-value cache in a hierarchical manner based on the multi-level stability profile includes: If the stability level of the cache scheduling set is the first stability level, then the cache scheduling set containing several key-value cache pages and key-value page summary information of the first group-level Top-K pages are written into the node memory of the corresponding hardware computing node; the group-level Top-K pages are the operation results obtained by performing union or intersection operations on several Top-K page sets in the cache scheduling set; the key-value page summary information includes the minimum key vector and the maximum key vector of the key-value cache page; If the stability level of the cache scheduling set is the second stability level, then the first group-level Top-K pages and the key-value page summary information of the cache scheduling set are written into the node video memory, the key-value cache pages are written into the host memory, and the target history pages in the key-value cache pages are flushed to the external non-volatile memory; the target history page is the key-value cache page where the number of steps between the corresponding decoding step and the current decoding step is greater than a preset number of steps threshold; Wherein, the stability score of the second stability level is higher than the stability score of the first stability level; the target key-value cache is a key-value cache page of a plurality of the cache scheduling sets.

7. The key-value cache scheduling method for multi-model multi-tenancy according to claim 6, characterized in that, The actual parameters of Top-K sampling include the actual value of the number of Top-K samples and the actual value of the resampling period. Accordingly, the dynamic scheduling of the target key-value cache based on the actual parameters of the Top-K sampling includes: If the current target cache scheduling set is determined to meet the preset resampling conditions, then the importance of several candidate key-value cache pages is scored using the key-value page summary information to obtain the corresponding score results; the preset resampling conditions include the number of steps between the current decoding step and the historical resampling decoding step reaching the actual value of the resampling period, and the historical resampling decoding step being the decoding step that was resampled last time; the candidate key-value cache pages are key-value cache pages stored in the host memory and the external non-volatile memory; A second set of top-K pages is generated based on the scoring results and the actual number of top-K pages. The first set of top-K pages is compared with the second set of top-K pages, and the target key-value cache is migrated and scheduled based on the comparison results.

8. A key-value cache scheduling device for multi-model, multi-tenant environments, characterized in that, include: The profile acquisition module is used to acquire a multi-level stability profile of the target large language model corresponding to each pending request after receiving several pending requests from each tenant, so as to determine the hardware computing node corresponding to each of the several pending requests. The multi-level stability profile includes a pre-divided cache scheduling set of the large language model and a preset Top-K sampling benchmark parameter corresponding to the cache scheduling set. The cache scheduling set includes a head group and / or a layer group. The head group is a set obtained by aggregating each attention head according to the stability level, and the layer group is a set obtained by aggregating each attention layer based on the stability level of the head group. The stability level is determined based on the stability score of each attention head, and the level types include at least three. The parameter determination module is used to calculate the theoretical video memory usage of the requests to be processed for key-value caching based on the multi-level stability profile, determine the parameter scaling factor of the several requests to be processed based on the theoretical video memory usage, and determine the actual Top-K sampling parameters of the cache scheduling set using the parameter scaling factor and the Top-K sampling benchmark parameters. The cache scheduling module is used to perform hierarchical storage of the generated target key-value cache according to the multi-level stability profile during the inference processing of the request to be processed by the hardware computing node, and to dynamically schedule the target key-value cache according to the actual parameters of the Top-K sampling during the decoding process.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the key-value cache scheduling method for multi-model multi-tenancy as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein, when the computer programs are executed by a processor, they implement the key-value cache scheduling method for multi-model multi-tenancy as described in any one of claims 1 to 7.

Images