A key-value cache management method and device, electronic equipment and storage medium

Abstract

This application provides a key-value cache management method, system, electronic device, and storage medium, applied to a large language model inference system. The key-value cache is distributed in a multi-level heterogeneous storage system, including near-end storage media and far-end storage media. The method includes: periodically collecting target indicators, the target indicators including at least one of the following: a first indicator representing bus transmission latency, a second indicator representing the proportion of far-end accesses, and a third indicator representing the overhead of direct memory access mapping; calculating a congestion index based on the target indicators; determining a target scheduling strategy for the key-value cache when optimization conditions are met; the optimization conditions include: the congestion index is greater than a first threshold, and / or, any target indicator is greater than a corresponding sub-indicator threshold; and executing the target operation corresponding to the target scheduling strategy. This application can improve the overall performance of the system and ensure the efficiency and stability of model inference.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a key-value cache management method, apparatus, electronic device, and storage medium. Background Technology

[0002] Large Language Models (LLMs) are widely used in natural language processing, intelligent question answering, code generation, and other fields. During autoregressive inference, the attention module of each layer of the model needs to access the key-value vectors corresponding to historical tokens. To avoid redundant computation, the system caches these vectors in a structure called a key-value cache (KV-Cache).

[0003] As the model size, context length, and number of concurrent sessions continue to increase, the size of the KV-Cache grows exponentially, placing extremely high demands on the storage system's capacity, affecting the stability of inference, and severely impacting the overall performance of the system. Summary of the Invention

[0004] In view of the above problems, embodiments of this application are proposed to provide a key-value cache management method that overcomes or at least partially solves the above problems, which can improve the overall performance of the system and ensure the efficiency and stability of model inference.

[0005] Accordingly, embodiments of this application also provide a key-value cache management device, an electronic device, and a storage medium to ensure the implementation and application of the above methods.

[0006] In a first aspect, embodiments of this application disclose a key-value cache management method applied to a large language model inference system. The key-value cache of the large language model inference system is distributed in a multi-level heterogeneous storage system, which includes a near-end storage medium and a far-end storage medium. The method includes: Periodically collect target metrics, which include at least one of the following: a first metric for representing bus transmission latency, a second metric for representing the proportion of remote accesses, and a third metric for representing the overhead of direct memory access mapping. Calculate the congestion index based on the target indicators; When the optimization conditions are met, a target scheduling strategy for the key-value cache is determined; the optimization conditions include: the congestion index is greater than a first threshold, and / or any target indicator is greater than the corresponding sub-indicator threshold; Execute the target operation corresponding to the target scheduling strategy.

[0007] Secondly, embodiments of this application disclose a key-value cache management device applied to a large language model inference system. The key-value cache of the large language model inference system is distributed in a multi-level heterogeneous storage system, which includes a near-end storage medium and a far-end storage medium. The device includes: A sensing module is used to periodically collect target indicators, which include at least one of the following: a first indicator for representing bus transmission delay, a second indicator for representing the proportion of remote accesses, and a third indicator for representing the overhead of direct memory access mapping. A strategy engine is used to calculate a congestion index based on the target metric; and to determine a target scheduling strategy for the key-value cache when optimization conditions are met; the optimization conditions include: the congestion index is greater than a first threshold, and / or any target metric is greater than the corresponding sub-metric threshold. An executor is used to execute the target operation corresponding to the target scheduling strategy.

[0008] Thirdly, embodiments of this application disclose an electronic device, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, which causes the processor to perform the steps of any of the key-value cache management methods described above.

[0009] Fourthly, embodiments of this application disclose a readable storage medium storing a program or instructions, which, when executed by a processor, can implement any of the key-value cache management methods described in the embodiments of this application.

[0010] The key-value cache management method, apparatus, electronic device, and storage medium provided in this application have the following advantages: The key-value cache management method provided in this application addresses the KV-Cache management problem in the Large Language Model (LLM) inference process by managing it at the operating system level. Specifically, it periodically collects target metrics and calculates a congestion index based on these metrics. The system's congestion state can be determined based on the calculated congestion index and the collected target metrics. When the system's congestion state meets optimization conditions, a target scheduling strategy for the key-value cache is determined, and the corresponding target operation is executed. The target metrics are related to KV-Cache access behavior and system bottlenecks. Based on these metrics, a system performance degradation trend can be identified earlier, and corresponding target scheduling strategies can be implemented before the problem escalates. These strategies include deciding how to allocate, migrate, and schedule KV-Cache resources, providing a stable, reliable, and efficient resource guarantee mechanism for LLM inference, improving overall system performance, and ensuring the efficiency and stability of model inference. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating the steps of an embodiment of the key-value cache management method of this application; Figure 2 This is a schematic diagram of the architecture of a large language model reasoning system according to this application; Figure 3 This is a schematic diagram illustrating the state transitions of the key-value cache management method executed in this application; Figure 4 This is a structural block diagram of an embodiment of a key-value cache management device according to this application; Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0012] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0013] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and are not limited in number; for example, a first object can be one or more. Furthermore, the term "and / or" in the specification and claims is used to describe the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. In the embodiments of this application, the term "multiple" refers to two or more, and other quantifiers are similar.

[0014] Figure 1 This diagram illustrates a flowchart of an embodiment of a key-value cache management method according to this application, applied to a large language model inference system. The key-value cache of the large language model inference system is distributed in a multi-level heterogeneous storage system, which includes near-end storage media and far-end storage media. The method may include the following steps: Step 101: Periodically collect target indicators, the target indicators including at least one of the following: a first indicator for representing bus transmission delay, a second indicator for representing the proportion of remote accesses, and a third indicator for representing the overhead of direct memory access mapping; Step 102: Calculate the congestion index based on the target index; Step 103: When the optimization conditions are met, determine the target scheduling strategy for the key-value cache; the optimization conditions include: the congestion index is greater than the first threshold, and / or any target indicator is greater than the corresponding sub-indicator threshold; Step 104: Execute the target operation corresponding to the target scheduling strategy.

[0015] The key-value cache management method disclosed in this application can be applied to large language model inference systems. The large language model inference system is a hardware and software environment system that deploys large language models and supports their inference operations.

[0016] Reference Figure 2 The diagram illustrates the architecture of a large language model reasoning system according to this application. Figure 2 As shown, the large language model inference system (hereinafter referred to as the system) comprises a business layer, an operating system layer, and a hardware resource layer. The business layer is the software layer oriented towards users and business logic, providing the inference framework and LLM services. The business layer relies on the unified resource interface provided by the operating system layer to implement the inference logic. The operating system layer acts as a bridge connecting the business layer and the underlying hardware; its core function is to abstract and schedule the hardware resource layer, shielding the heterogeneity of the underlying hardware and efficiently managing heterogeneous resources. The hardware resource layer provides the computing power and data storage carrier for the entire system, forming the physical foundation of the system.

[0017] The hardware resource layer may include a multi-level heterogeneous storage system, and the key-value cache (KV-Cache) of the large language model inference system is distributed across this multi-level heterogeneous storage system. The multi-level heterogeneous storage system includes near-end storage media and far-end storage media. Near-end storage media refers to storage media that are physically integrated with computing units (such as GPUs or specific CPU cores) or located within the same NUMA (Non-Uniform Memory Access) node, and can be directly accessed using ordinary read / write memory instructions. For example, the near-end storage media may include at least one of the following: GPU High Bandwidth Memory (GPU HBM) and DRAM (Dynamic Random-Access Memory) within the local NUMA node. Far-end storage media refers to storage media that are physically isolated from computing units, requiring longer paths and more complex protocol conversions for access, resulting in higher latency. For example, the remote storage medium may include at least one of the following: DRAM in the remote NUMA node, Compute Express Link Extended Memory (CXL Extended Memory), and Non-Volatile Memory Express Solid State Drive (NVMe SSD).

[0018] Host DRAM is core system memory, referring to memory directly managed by the CPU. It mainly includes DRAM within the local NUMA node (directly connected to the CPU) and DRAM within a remote NUMA node (requiring cross-CPU interconnect access). GPU HBM is independent device memory located on the GPU, accessible at high speed by the GPU computing cores; the CPU accesses GPU HBM via PCIe. CXL extended memory and NVMe SSDs are host-addressable extended storage, with performance lower than DRAM. They are connected via CXL / PCIe, serving as a supplementary storage tier to DRAM.

[0019] Furthermore, the large language model inference system can support parallel deployment of multiple instances. Multiple independent inference instances (hereinafter referred to as instances) share underlying hardware resources (including GPU, CPU, and multi-level heterogeneous storage media, etc.) and each is configured with its own dedicated resource quota. The dedicated resources include key-value cache (KV-Cache) independently managed by each instance and corresponding page lock quota, memory quota, etc. Each instance independently executes the key-value cache management method based on its own corresponding target indicators.

[0020] It should be noted that the descriptions in this application are all based on a single example. The large language model inference system collects target metrics on an instance-by-instance basis, and then determines the corresponding target scheduling strategy and executes the corresponding target operation for each instance, so as to improve the overall resource utilization of the system and reduce inference latency.

[0021] For example, suppose the system deploys m instances, such as m=3, denoted as instance A, instance B, and instance C. When n (assuming n=10) user requests (such as inference task requests) arrive simultaneously, the system distributes these 10 user requests to these 3 instances. During the process of each of these 3 instances processing the received user requests, this application analyzes the running status of each instance at the operating system level, makes corresponding target scheduling decisions based on the collected target indicators, and executes the target operations.

[0022] KV-Cache is managed in memory in units of memory pages. Each page contains contiguous key-value vector data, referred to as a KV page.

[0023] This application periodically collects target metrics at the operating system layer during the inference task execution of a large language model. These target metrics may include at least one of the following: a first metric reflecting bus transmission latency, a second metric reflecting the proportion of remote accesses, and a third metric reflecting the overhead of direct memory access mapping. These target metrics are the main bottleneck signals for KV-Cache performance jitter, and all target metrics can be obtained at the operating system layer. The storage, access, and transmission efficiency of the KV-Cache directly determines whether system congestion will occur and the severity of congestion. Therefore, this application calculates a congestion index based on the collected target metrics. The congestion index can reflect whether the system is in a congested state. For example, if the congestion index is greater than a first threshold, the system can be determined to be in a congested state. If any target metric is greater than its corresponding sub-metric threshold, the system can be determined to be in an abnormal state.

[0024] When the congestion index is greater than the first threshold (congestion state), and / or any target indicator is greater than the corresponding sub-indicator threshold (abnormal state), the optimization conditions are considered met, and optimizations need to be made to the storage, access, and transmission of KV-Cache to alleviate the congestion state and / or resolve the abnormal state.

[0025] The first metric may include PCIe Round-Trip Time (PCIe RTT), which refers to the average delay of data transmission to and from the PCIe link. This metric can be used to determine whether system congestion is related to bus transmission delay.

[0026] The second metric can include the remote NUMA node access ratio, such as denoted as numa_remote_ratio, which refers to the proportion of the number of operations (or data volume) accessing the memory of remote NUMA nodes to the total number of memory accesses (or the total data volume). This metric can be used to determine whether system congestion is related to an unreasonable KV page storage location.

[0027] The third metric may include the average DMA mapping time, such as dma_map_latency, which refers to the average time overhead for the system to complete a single memory address mapping in a Direct Memory Access (DMA) operation. This metric can be used to determine whether system congestion is related to the DMA mapping mode.

[0028] This application calculates a congestion index based on the stated target indicators. If the congestion index exceeds a first threshold, the large language model inference system is determined to be in a congested state. If any target indicator exceeds its corresponding sub-indicator threshold—for example, if the first indicator exceeds the first sub-indicator threshold, and / or the second indicator exceeds the second sub-indicator threshold, and / or the third indicator exceeds the third sub-indicator threshold—then the large language model inference system is determined to be in an abnormal state.

[0029] If the large language model inference system is determined to be in a congested state and / or an abnormal state, then the optimization conditions are met. At this time, a target scheduling strategy for the key-value cache can be determined, and the target operation corresponding to the target scheduling strategy can be executed to optimize the management of the KV-Cache, alleviate the congestion state and / or resolve the abnormal state of the system.

[0030] In one optional embodiment of this application, calculating the congestion index based on the target index may include: Step S11: Normalize the target index; Step S12: Obtain the weight parameters corresponding to each target indicator; the weight parameters are obtained through pre-training offline. Step S13: The normalized target indicators and their corresponding weight parameters are weighted and summed to obtain the congestion index.

[0031] Normalization is used to unify indicators with different dimensions to a comparable scale, so as to avoid a single dimension dominating the strategic judgment.

[0032] In one example, the congestion index I can be calculated using the following formula: I=α RTTnorm+β RemoteRatio+γ DMAmapI(1) In equation (1) above, I represents the congestion index; RTTnorm represents the normalized first indicator (such as PCIe RTT), and α is the weight parameter corresponding to the first indicator. RemoteRatio represents the normalized second indicator (such as numa_remote_ratio), and β is the weight parameter corresponding to the second indicator; DMAmapI represents the normalized third indicator (such as dma_map_latency), and γ is the weight parameter corresponding to the third indicator.

[0033] Furthermore, in equation (1) above, DMAmapI can also represent DMA mapping overhead. DMA mapping average execution time refers to the average execution time of a single DMA mapping operation, that is, the average execution time to complete one DMA mapping operation, including mapping and unmapping. DMA mapping overhead refers to the total time overhead incurred by all DMA mapping operations over a period of time. The calculation of DMA mapping overhead is based on the average DMA mapping execution time and can be combined with the number of DMA mapping and unmapping operations or the time distribution statistics (such as the P95 quantile).

[0034] For example, the DMA mapping overhead is calculated as follows: DMAmapI = dma_map_latency × total operation frequency (2) Wherein, DMAmapI represents DMA mapping overhead; dma_map_latency represents the third metric, namely the average DMA mapping time; total operation frequency refers to the number of times DMA mapping operations are performed within a certain period of time (such as within a monitoring window). One DMA mapping operation includes one mapping and one unmapping.

[0035] For example, a fixed window (e.g., 10–50 ms) is used as a period (i.e., a monitoring window). The fixed window is a time slice for the sensing module to sample and statistically analyze data, used to aggregate raw events and metrics.

[0036] Alternatively, the DMA mapping overhead can be calculated as follows: DMAmapI = P95 quantile of average DMA mapping time × total number of operations (3) The P95 quantile indicates that 95% of all DMA mapping operations have a time consumption lower than this value, while the remaining 5% have a time consumption higher than this value. This calculation method is suitable for complex scenarios with long sequences and high concurrency in LLM inference, and can more comprehensively reflect the impact of DMA mapping operations on system tail latency.

[0037] Here, α, β, and γ are the weight parameters corresponding to the first, second, and third indicators, respectively. The initial values ​​of α, β, and γ can be configured according to the hardware environment (such as PCIe version, DDR bandwidth, etc.); or, the initial values ​​of α, β, and γ can be obtained in advance through offline training.

[0038] For example, the offline training process is as follows: Multiple sets of target metrics and latency data are collected in an offline environment under various loads (such as different concurrency levels, context lengths, and hardware environments). Tail latency or throughput is used as the target, and methods such as regression, grid search, and Bayesian optimization are employed to find the optimal α, β, and γ values ​​for the target. In practice, linear regression can be used to fit the relationship between the congestion index I and the latency data, and then A / B testing can be used for fine-tuning to obtain initial values ​​suitable for the specific hardware environment.

[0039] The target scheduling strategy described herein is used to optimize the management of the KV-Cache, such as optimizing the storage, access, and transmission of the KV-Cache, in order to improve the overall system performance. This application does not limit the specific target scheduling strategy adopted.

[0040] For example, the target scheduling strategy may include at least one of the following: KV page migration strategy, DMA mapping strategy, KV page prefetch strategy, and page lock management strategy.

[0041] Key-value (KV) page migration strategies refer to the dynamic migration of KV page data between different tiers of storage media. For example, frequently accessed KV pages (called hot KV pages or hot pages) are migrated to high-bandwidth, low-latency near-end storage media (such as GPUHBM, local NUMA nodes, etc.); while infrequently accessed KV pages (called cold pages) are migrated to high-capacity, low-cost far-end storage media (such as CXL extended memory, NVMe SSDs, etc.); achieving a globally optimal balance between KV-Cache access performance and storage cost.

[0042] DMA mapping strategy refers to the decision-making mechanism for selecting the direct memory access mapping mode and adjusting parameters of the management device, aiming to balance DMA operation efficiency and system resource consumption. For example, DMA mapping performance can be optimized by dynamically switching different operating modes and adjusting mapping parameters.

[0043] Key-value (KV) page prefetching strategy refers to the decision-making mechanism that controls whether to enable prefetching function and prefetching behavior. It reduces access latency by predicting future access patterns and loading KV page data in advance.

[0044] Page lock management strategy refers to the decision-making mechanism that controls the physical memory page lock quota and allocation rules of the system, ensuring that critical data resides in memory while preventing memory resource exhaustion.

[0045] Optionally, this application combines congestion index and sub-index thresholds to determine a specific target scheduling strategy. In one optional embodiment of this application, determining the target scheduling strategy for the key-value cache may include one or more of the following: Based on the remaining local memory, a key-value page migration strategy is determined; the key-value page migration strategy is used to migrate hot pages to the near-end storage medium, or to migrate cold pages to the far-end storage medium. The DMA mapping strategy is determined based on the average DMA mapping time; the DMA mapping strategy is used to switch the DMA mapping mode to either large block aggregation mode or small block pipeline mode. The KV page prefetch strategy is determined based on the prefetch hit rate and / or the average DMA mapping time; the KV page prefetch strategy is used to pause or resume the prefetch function, or to adjust the prefetch step size. Based on the proportion of hot pages, a page lock management strategy is determined; the page lock management strategy is used to increase or decrease the page lock quota for hot pages.

[0046] In this application, "local" refers to a local NUMA node bound to the CPU / GPU of the current instance (such as instance A), or a NUMA node on the CPU side directly connected to the GPU.

[0047] If the congestion index is greater than a first threshold (e.g., denoted as I_high), this application determines the KV page migration strategy based on the remaining local memory and the hot and cold tags of the KV page.

[0048] For example, if the remaining local memory exceeds a preset capacity (e.g., denoted as H_threshold), it indicates that the remaining local memory capacity is sufficient. In this case, the KV page migration strategy is determined to include: migrating hot pages to the near-end storage medium to improve inference efficiency; or, if the remaining local memory does not exceed the preset capacity, it indicates that the remaining local memory capacity is insufficient. In this case, the KV page migration strategy is determined to include: migrating cold pages to the far-end storage medium to free up memory.

[0049] The preset capacity (H_threshold) is the safe lower limit of the remaining local memory. It can be set to a fixed value (such as 2GB) or a preset percentage of the total local memory (such as 5%–10%).

[0050] When I > I_high and there is sufficient local memory, it indicates that congestion mainly originates from remote access or link bottlenecks. Therefore, migrating hot pages to local storage to reduce remote access can effectively alleviate congestion. For example, hot pages can be migrated to the memory of the same NUMA node as the instance's compute unit. For instance, if instance A has an increased proportion of remote accesses and sufficient local memory, then instance A's hot pages can be migrated from remote storage to the memory of the same NUMA node as instance A's compute unit. This effectively reduces the number of round-trip transfers and time overhead between the PCIe bus and NUMA nodes, improves KV-Cache access efficiency, and reduces inference latency.

[0051] When I > I_high and local memory is insufficient, continuing to migrate hot pages to near-end storage will exacerbate memory pressure and worsen congestion. Therefore, cold pages should be migrated to far-end storage to prevent them from crowding out local memory.

[0052] If the congestion index is greater than a first threshold (I_high), this application determines a DMA mapping strategy based on the average DMA mapping time. The DMA mapping strategy is used to switch the DMA mapping mode to either a large-block aggregation mode or a small-block pipeline mode.

[0053] For example, in scenarios prioritizing high throughput, or when the system is congested and DMA mapping overhead is higher than a preset overhead, a large-block aggregation mode can be used. In scenarios prioritizing low latency, or when the system is congested but DMA mapping overhead is lower than a preset overhead, a small-block pipeline mode can be used.

[0054] For example, if the average DMA mapping time exceeds a preset duration, the DMA mapping strategy is determined by switching the DMA mapping mode to a large block aggregation mode. If the average DMA mapping time does not exceed the preset duration, but the PCIe RTT is higher than a preset latency and the tail latency increases, the DMA mapping strategy is determined by switching the DMA mapping mode to a small block pipeline mode to reduce the tail latency of a single transmission.

[0055] Furthermore, in scenarios where the user request size is smaller than a preset size, the token window length is smaller than a preset length, or the KV hot set is smaller than a preset size, a small-block pipeline mode can be adopted, which is more conducive to fast response. Here, the KV hot set refers to the set of KV pages (such as hot pages) whose access frequency exceeds a specified threshold within a certain period. The KV hot set size refers to the number of KV pages contained in the KV hot set.

[0056] Furthermore, when the system enters the recovery phase (such as when the congestion index I decreases but remains unstable), in order to avoid amplifying delay jitter with large block transmissions, the DMA mapping mode can be switched to the small block pipeline mode.

[0057] Furthermore, the triggering condition for the DMA mapping strategy is not limited to the congestion index being greater than the first threshold (I_high). The DMA mapping mode can also be switched when the congestion index is less than or equal to the first threshold (I_high), but dma_map_latency or PCIe RTT suddenly increases (such as the increase within a preset period being greater than the preset increase).

[0058] When the congestion index is greater than the first threshold (I_high), this application determines the KV page prefetching strategy based on the prefetch hit rate and / or the average DMA mapping time; the KV page prefetching strategy is used to suspend or resume the prefetching function, or to adjust the prefetching step size.

[0059] For example, when the congestion index is greater than a first threshold, the prefetching function can be paused or resumed based on the prefetch hit rate and DMA mapping overhead, and the prefetching step size can be adjusted based on the prefetch hit rate. Further, when the congestion index is less than or equal to the first threshold, the prefetching step size can be dynamically adjusted based on the hit rate.

[0060] For example, adjusting the prefetch step size based on the prefetch hit rate may include: if the prefetch hit rate is less than a preset hit rate, then reducing the prefetch step size or pausing the prefetch function.

[0061] Furthermore, the pause prefetch function may include: pausing cold page prefetch function and / or pausing low-priority session prefetch function.

[0062] For example, when the congestion index I is greater than the first threshold (I_high) and dma_map_latency is greater than the preset duration, or when the congestion index I is greater than the first threshold (I_high) and PCIe RTT is greater than the preset latency, it indicates that the system bandwidth is tight or the DMA mapping overhead is large, and the cold page prefetching function can be suspended. Similarly, when multiple concurrent sessions cause system resource congestion, and the resource usage of high-priority sessions is squeezed out by low-priority sessions, the prefetching function of the low-priority sessions can be suspended to ensure the execution performance of high-priority sessions.

[0063] In practice, the hot page prefetching function is usually not suspended. It may be temporarily suspended only when entering a bad case and needing to adopt an emergency mode.

[0064] The prefetch hit rate can be calculated as follows: Prefetch hit rate = number of pages actually accessed within the statistical time window after prefetching / total number of pages prefetched (4) Furthermore, when the congestion index I is less than the second threshold (I_low) and the prefetch hit rate rises above the preset hit rate, or dma_map_latency drops to the preset range, the prefetch function can be gradually restored. For example, the prefetch function for high-priority sessions and hot pages can be restored first, and then the prefetch function for cold pages and low-priority sessions can be restored.

[0065] When the congestion index is greater than the first threshold (I_high), this application determines the page locking strategy based on the proportion of hot pages. Alternatively, when the congestion index is less than or equal to the first threshold, but the growth rate of the KV hot set increases, the page locking strategy can also be determined based on the proportion of hot pages. Here, an increase in the growth rate of the KV hot set refers to a faster rate of expansion of the KV-Cache capacity marked as "hot pages" per unit time.

[0066] For example, the conditions for increasing the hot page lock quota include: the congestion index is greater than the first threshold (I_high) and the hot page ratio (hotset_ratio) is greater than the preset ratio, or the high-priority session latency increases. In this case, the page lock management strategy is to increase the hot page lock quota.

[0067] For example, the conditions for reducing the hot page lock quota include: the congestion index is less than the second threshold (I_low) and the page lock occupancy rate increases, or the latency optimization benefit brought by the page lock operation does not reach the preset standard. In this case, the page lock management strategy is determined to include: reducing the hot page lock quota.

[0068] It should be noted that page-locking policies are not limited to increasing or decreasing hot page lock quotas; they can also include tiered protection. For example, prioritizing the protection of high-priority sessions and hot page lock quotas, while prioritizing the reduction of low-priority sessions' page lock quotas.

[0069] Furthermore, the method may also include: performing real-time popularity statistics on the KV pages in the key-value cache, and dynamically updating the hot and cold tags of the KV pages based on the popularity statistics results.

[0070] This application achieves dynamic management of cross-level hot and cold stratification of KV-Cache through real-time heat statistics and dynamic tag updates, providing a reliable basis for the selection of target scheduling strategies. Heat statistics refer to the continuous statistical analysis of access behavior for all KV pages (statistical dimensions include access frequency, access interval, etc.). When the statistical results show that certain KV pages maintain high-frequency access within a continuous monitoring window (e.g., access frequency greater than the first frequency value), these KV pages are marked as hot pages; when the statistical results show that the access frequency of certain KV pages continuously decreases to below the second frequency value, and the inference position corresponding to them exceeds the prediction window of the KV page prefetching strategy, these KV pages are marked as cold pages. The prediction window refers to the time / sequence prediction window for future access intervals, which can be the same as or more conservatively than the token window length. After cross-level hot and cold stratification is completed, the system continuously monitors the heat changes of KV pages in each level of storage media and dynamically updates the hot and cold tags.

[0071] When determining the target scheduling strategy, key-value page data marked as hot pages can be preferentially deployed or migrated to near-end storage media such as GPUHBM and local NUMA nodes to obtain low-latency, high-bandwidth access performance; key-value page data marked as cold pages can be migrated to remote storage media such as remote NUMA nodes, CXL extended memory, or NVMe SSDs to save near-end storage resources and reduce overall storage costs.

[0072] In one optional embodiment of this application, the method may further include: Step S21: After executing the target operation, record the quantitative data of the benefit generated by the target operation for each target indicator; Step S22: Adjust the weight parameters corresponding to each target indicator based on the revenue quantification data.

[0073] Furthermore, this application provides an online feedback adjustment mechanism. In practical applications, after executing the target operation according to the selected target scheduling strategy, the weight parameters corresponding to each target indicator are continuously optimized based on the impact of the target operation. Through multiple iterations, adaptive adjustment is formed, so that the congestion index reflects the real bottleneck changes of the system and more effectively guides the selection of a better target scheduling strategy.

[0074] For example, after executing the target operation, the quantitative data of the benefit generated by the target operation for each target metric can be recorded. Quantitative benefit data refers to the quantitative change or rate of change of the target metric after executing the target operation compared to before the operation. If, after executing the target operation, a certain target metric does not generate a positive benefit (e.g., PCIeRTT does not decrease or the remote NUMA node access ratio does not decrease), the weight parameter corresponding to that target metric is automatically reduced, while the weight parameters corresponding to other target metrics are increased.

[0075] Positive returns refer to results where the quantified return data meets pre-defined positive change conditions, and this change has a beneficial impact on the system's performance, cost, or efficiency. The criteria for judging positive returns are related to the type of target metric. For example, for latency-related metrics (such as PCIe RTT, average DMA mapping time, inference latency, etc.), positive returns are manifested as a decrease in the metric's value; for utilization-related metrics (such as GPU HBM utilization, CPU core utilization, etc.), positive returns are manifested as an increase in the metric's value; and for cost-related metrics (such as storage cost, etc.), positive returns are manifested as a decrease in the metric's value.

[0076] In one optional embodiment of this application, the method may further include: If the congestion index is detected to be between the second threshold and the first threshold, and remains stable for a preset period, then the execution intensity of the target scheduling strategy is gradually reduced; or, If the congestion index is detected to be less than the second threshold and remains stable for a preset period, the target scheduling strategy is revoked and the default scheduling strategy is restored.

[0077] Optionally, the first threshold I_high is calculated as follows: I_high=μI+1.5σ(5) Optionally, the second threshold I_low is calculated as follows: I_low=μI+0.5σ(6) In this embodiment, the first threshold is a threshold value used to determine if the system has entered a congestion state. The second threshold is a threshold value used to determine if the system has entered a recovery state or a normal state. For example, when the congestion index I is greater than the first threshold I_high, the system is determined to have entered a congestion state. When the congestion index I is between the second threshold I_low and the first threshold I_high and remains stable, the system is determined to have entered a recovery state. When the congestion index I is less than the second threshold I_low and remains stable, the system is determined to have entered a normal state.

[0078] μ and σ are the mean and standard deviation of the congestion index I within the monitoring window (monitoring period). μ and σ are obtained from recent monitoring window statistics, and can be initially taken as empirical values ​​and updated online.

[0079] In one optional embodiment of this application, the execution of the target operation corresponding to the target scheduling strategy may include: Step S31: Convert the target index into a state vector of the large language model inference system; Step S32: Input the state vector into a preset rule model, and output the control action corresponding to the target scheduling strategy through the rule model; Step S33: Send the command corresponding to the control action to the actuator, carrying the action type and action parameters, and have the actuator execute the control action.

[0080] The rule model can be a reinforcement learning (RL) model. For example, the RL model is designed as follows: State space: S = [state vector 1, state vector 2, ..., state vector m, TokenWindow]; Control action: A = {action parameter 1, action parameter 2, ..., action parameter n}; Reward function: R.

[0081] The state space is a set describing the current states of the system. Each state vector represents a state of the system; for example, a state vector can be a normalized target metric. For instance, the state vector may include one or more of the following: normalized PCIe round-trip latency (PCIe RTT), remote NUMA node access ratio (numa_remote_ratio), average DMA mapping latency (dma_map_latency), PCIe bandwidth utilization (PCIe_BW), and local free memory (LocalFreeMem). TokenWindow is the current or predicted decoding window length. The current or predicted decoding window length refers to the length of the actual window in the current decoding stage or the predicted window in the next stage.

[0082] A control action (hereinafter referred to as an action) refers to the set of all actions derived from the target scheduling strategy determined based on the current system state. Action parameters are the parameters used when executing an action. For example, the action parameters may include one or more of the following: the number of migrated pages in the KV page migration strategy, the mapping mode in the DMA mapping strategy, the prefetch step size in the KV page prefetch strategy, and the page lock quota in the page lock management strategy. Action parameters can be dynamically determined based on the state vector, combined with indicators such as the total number of hot pages, remaining bandwidth (the difference between actual used bandwidth and theoretical maximum bandwidth), and page lock occupancy rate (the ratio of used page lock quota to total page lock quota).

[0083] The reward function is a scalar feedback signal that evaluates the quality of an action, used to adjust the parameters of the next action, thereby achieving adaptive adjustment of the action parameters. For example, the reward function can calculate whether tail latency has decreased, whether the congestion index I has decreased, whether throughput (such as the number of tokens generated per second or the number of requests processed per second) has increased, and changes in page occupancy. Tail latency refers to the latency of the slowest portion of requests in the system response time distribution, usually expressed as percentiles, such as P95, P99, or P99.9.

[0084] In one example, the RL model is designed as follows: State space: S = [PCIe_BW, RemoteRatio, LocalFreeMem, TokenWindow]; Action: A = {Number of migrated pages M, Prefetch step size P, Lock page quota L}; Reward function: R = (TokenLatency) λ MigrationOverheadR.

[0085] Where PCIe_BW is the PCIe bandwidth utilization; RemoteRatio is the normalized second indicator; LocalFreeMem is the remaining local memory; and TokenWindow is the current or predicted decoding window length.

[0086] λ is a non-negative weighting coefficient used to balance the importance of various penalty terms (such as TokenLatency and MigrationOverheadR) in the reward function. For example, as λ increases, more emphasis is placed on minimizing MigrationOverheadR; as λ decreases, more emphasis is placed on minimizing TokenLatency; when λ=0, MigrationOverheadR is completely ignored, and only TokenLatency is minimized.

[0087] TokenLatency is the latency per token; MigrationOverheadR is the additional overhead caused by KV page migration. Single-token latency refers to the average time spent generating each token during the generation phase, which can be expressed as: total generation phase time / number of tokens generated. MigrationOverheadR can be defined as the migration overhead ratio, for example: KV page migration time / total inference time, or the additional latency caused by KV page migration (post-migration latency). Baseline delay (baseline delay) / baseline delay, calculated by window. Baseline delay refers to the delay before the migration is performed.

[0088] For example, suppose the target operation is executed, and its action parameter increases the number of migrated pages M, causing the congestion index I to decrease and the migration overhead (MigrationOverheadR) to meet the condition (e.g., less than the preset overhead). Then, in the next window, the number of migrated pages M can be increased further. As another example, suppose the target operation is executed, and its action parameter increases the page lock quota L, causing the page lock occupancy rate to increase and the tail latency benefit to be insignificant (e.g., less than the preset benefit). Then, in the next window, the page lock quota L can be decreased. Furthermore, suppose the target operation is executed, and its action parameter uses a prefetch step size P, causing a decrease in the prefetch hit rate and PCIe congestion. Then, in the next window, the prefetch step size P can be decreased or the prefetch function can be paused.

[0089] Reference Figure 3 This diagram illustrates the state transitions during the execution of the key-value cache management method in this application. Figure 3 As shown, when the congestion index I is detected to be between the second threshold and the first threshold and remains stable, i.e., the system is in a recovery state, the target scheduling policy can be gradually revoked, such as gradually reducing the execution intensity of the target scheduling policy. This includes gradually reducing the number of migrated pages, restoring the default DMA mode, releasing some locked pages, restoring the default prefetch policy, and rebalancing hot and cold pages. Furthermore, if the congestion index I is detected to be less than the second threshold I_low for N consecutive windows, and none of the target indicators exceed their limits, it indicates that the system has entered a new stable state, and the target scheduling policy can be revoked, restoring the default scheduling policy. If the congestion index I is detected to rise again above the first threshold or if any target indicator exceeds a sub-indicator threshold, the system returns to the congestion / abnormal state, the target scheduling policy is redefined, and the corresponding target operations are executed.

[0090] The default scheduling strategy refers to the system's default key-value page migration strategy, default DMA mapping mode, key-value page prefetching strategy, and page lock management strategy. For example, the default DMA mode refers to the general DMA mapping mode pre-set by the driver or system. Typically, in low-latency priority scenarios, the default DMA mode can be a small-block pipeline mode; in high-throughput priority scenarios, the default DMA mode can be a large-block aggregation mode.

[0091] This application does not limit the methods for collecting target metrics. For example, instrumentation, hardware counter reading, and kernel interface calls can be used for collection or statistics. For instance, the PCIe RTT value within a period can be obtained by reading the round-trip latency counter of the PCIe configuration space. Another example is inserting instrumentation functions on the critical path of memory allocation and access in an LLM inference task to record the node affiliation of each memory access, thereby statistically analyzing the proportion of remote accesses within a period. Yet another example is inserting timing instrumentation functions on the critical path of DMA mapping for KV-Cache cross-media transfer to record the time consumed by each DMA mapping operation, thereby statistically analyzing the average DMA mapping time within a period.

[0092] In practical implementation, the large language model inference system of this application may include the following modules at the operating system layer: a perception module, a policy engine, and an executor set engine. These three modules work together to execute the key-value cache management method of this application.

[0093] The perception module is used to execute step 101. Specifically, without blocking the inference task, it periodically collects target metrics related to KV-Cache access behavior and system bottlenecks.

[0094] The policy engine is used to execute steps 102 and 103. Further, the policy engine may include an indicator shaper and a policy inferencer, both used to transform the indicators collected by the perception module into specific control actions. The types of control actions may include one or more of the following: key-value page migration, page locking, prefetching, DMA mode switching, hot / cold stratification, etc.

[0095] The strategy engine issues a command corresponding to the control action to the executor set engine, carrying the action type and action parameters. The executor set engine forwards the command to the corresponding executor based on the action type, and the corresponding executor executes the control action. The command can be a kernel-side configuration update command or a control instruction to notify the inference framework.

[0096] The indicator shaper is used to normalize the target indicator. Specifically, the indicator shaper uses a sliding window (e.g., 50–100ms) to smooth transient noise in the target indicator and performs normalization. The smoothing window (including overlap) used for sliding window indicator shaping is used to suppress transient noise. Based on the normalized target indicator, a congestion index can be calculated. The policy inference engine determines the target scheduling policy for the key-value cache based on the congestion index and the normalized target indicator, and translates it into control actions. Specifically, the policy engine follows... Figure 3 The state machine shown is in operation.

[0097] The actuator collection engine is used to perform step 104. For example, the actuator collection engine may include actuators such as: migration actuators, DMA tuning actuators, page lock controllers, hot / cold stratification and prefetch actuators.

[0098] The migration executor is used to execute control actions corresponding to the KV page migration strategy, such as migrating hot pages to the near-end storage medium or migrating cold pages to the far-end storage medium. The KV page migration strategy can adopt a batch processing mechanism, aggregating multiple pages to be migrated and submitting them at once, thereby reducing the frequency of lock acquisition and release, effectively alleviating lock contention, and improving the overall system throughput.

[0099] Furthermore, when the migration executor executes the control actions corresponding to the KV page migration strategy, it may include one or more of the following migration paths: 1) Cross-NUMA node migration. For example, to optimize CPU inference memory access latency, hot pages from remote NUMA nodes can be migrated to local NUMA nodes.

[0100] 2) Migration between GPU HBM and host DRAM. For example, when GPU memory is insufficient, cold pages with low access frequency are asynchronously migrated from GPU HBM to host DRAM.

[0101] 3) Point-to-point direct migration between GPUs. For example, using high-speed interconnects such as NVLink, key-value pages can be migrated between multiple GPUs on the same machine, bringing them closer to the GPU performing the computation.

[0102] 4) Migrate from host DRAM to extended storage. For example, migrate cold pages from host DRAM to CXL extended memory or NVMe SSD to free up host DRAM resources.

[0103] The DMA tuning executor is used to execute control actions corresponding to the DMA mapping strategy, such as switching the DMA mapping mode to large block aggregation mode or small block pipeline mode.

[0104] The bulk aggregation mode refers to aggregating multiple discrete, fine-grained I / O requests into a single, large data block, processing all requests through a single DMA mapping and a single data transfer. The goal of this mode is to reduce the total number of DMA mappings and improve throughput. For example, in scenarios with frequent DMA mappings and limited throughput, switching to bulk aggregation mode can reduce the total number of DMA mappings.

[0105] Small-block pipelined mode refers to splitting a large task or data block request into multiple independent small-block requests, and processing them in parallel using a "mapping-transfer-demapping" pipeline approach. The goal of this mode is to reduce PCIe transmission latency and improve response speed. For example, in scenarios where task requests are mainly in small batches and have low latency, switching to small-block pipelined mode can reduce the latency of a single transmission.

[0106] Furthermore, to achieve dynamic management of DMA mapping modes at the operating system level, this application deploys a lightweight kernel module or eBPF flow control program in the kernel. When the device driver calls standard interfaces such as dma_map_sg and dma_unmap_sg, this component can intercept its SG list (scatter-aggregate list) and rewrite the list in real time according to the target scheduling policy, thereby switching the DMA mapping mode. For example, multiple entries describing small, scattered physical memory blocks in the intercepted SG list can be merged and reorganized into entries describing contiguous large memory blocks, thus merging multiple scattered DMA operations into a single (or a few) large-block aggregation mode. Alternatively, entries describing large memory blocks in the intercepted SG list can be divided into multiple entries describing fixed-size small blocks according to a strategy, thus splitting a single large-block DMA operation into multiple small blocks for a small-block pipelined mode. This component operates in a kernel pass-through manner, intercepting and rewriting the SG list to achieve dynamic switching of DMA mapping modes without modifying the driver or the hardware itself.

[0107] The page lock controller is used to execute control actions corresponding to the KV page prefetching strategy, such as locking hot session pages and dynamically adjusting page lock quotas.

[0108] For example, when the hotset ratio increases or high-priority session latency fluctuates, the page lock controller dynamically increases the page lock quota for hot pages according to the target scheduling policy and performs a page lock (mlock) operation on the hot pages to ensure that critical pages remain resident in physical memory. When page lock quota pressure increases (e.g., page lock occupancy exceeds a preset value) or low-priority session activity decreases, the page lock controller releases (munlocks) non-core page locks according to the target scheduling policy and dynamically adjusts the page lock quotas at each level to reduce page lock quota pressure.

[0109] The hot and cold stratification and prefetching actuator is used to dynamically update the hot and cold tags of KV pages in the key-value cache, and to execute control actions corresponding to the page lock management strategy, such as pausing or resuming the prefetching function, expanding or shrinking the page lock quota of hot pages, etc.

[0110] Specifically, the hot-cold stratification executor is used to determine the popularity of KV pages, maintain hot and cold tags, and make stratification strategy decisions. The hot-cold stratification executor controls the popularity of KV pages by calling the memory management suggestion interface. The memory management suggestion interface includes: a first suggestion interface, such as madvise(addr, MADV_WILLNEED), used to mark the memory page (KV page) corresponding to a specified address (addr) as a hot page, increasing its memory resident priority; a second suggestion interface, such as madvise(addr, MADV_COLD), used to mark the memory page (KV page) corresponding to a specified address (addr) as a cold page, decreasing its memory resident priority; and a third suggestion interface, such as madvise(addr, MADV_PAGEOUT), used to suggest swapping the memory page (KV page) corresponding to a specified address (addr) to low-speed storage, decreasing its memory resident priority.

[0111] The hot-cold stratification actuator combines information such as physical page access bits and DMA access records to dynamically update the hot and cold tags of KV pages by maintaining a consistent version number.

[0112] The hot-cold tiering executor is used to trigger the migration of key-value pages between different tiers; the migration executor is used to execute the specific migration actions. That is, the hot-cold tiering executor determines which key-value pages need to be migrated, and the migration executor completes the physical migration of key-value pages between different NUMA nodes and different storage tiers by calling kernel interfaces such as move_pages and migrate_pages.

[0113] Furthermore, based on the hot / cold tiering mechanism, this application also provides a predictive prefetching strategy to further reduce the access latency of the KV-Cache. For example, based on the token window range of the current inference request, the KV pages to be accessed at the next level are preloaded; when the system predicts that the next token window will access a KV page at a new level, the predictive prefetching mechanism is automatically enabled to preload the target KV page (the predicted KV page to be accessed at the next level) into the near-end storage medium.

[0114] Furthermore, the key-value cache management method of this application has optimization and mitigation effects for bad case scenarios that may occur during operating system operation. The bad case refers to a severe state in which the congestion index I is greater than a first threshold, multiple target indicators exceed the limit simultaneously, and tail latency fluctuates drastically.

[0115] Normal congestion (congestion index I is greater than the first threshold) or abnormal congestion (a target indicator exceeds the limit) can be mitigated using conventional target scheduling strategies. For example, target scheduling strategies can employ gradual adjustments to action parameters. Bad cases, however, require stronger and more urgent target scheduling strategies, which may include: pausing prefetching, forcing page locking, fast migration, and forcibly switching DMA mapping modes.

[0116] When the congestion index I drops to between the second and first thresholds and remains stable, the system can be considered to have entered a recovery state, and emergency target scheduling strategies can be gradually withdrawn. When the congestion index I drops below the second threshold and remains stable, the system can be considered to have entered a normal state, and emergency target scheduling strategies can be suspended.

[0117] In one optional embodiment of this application, the method may further include: Once the target scenario is determined, target scenario metrics are collected periodically. Determining the target scheduling strategy for the key-value cache may include: Based on the congestion index and the target scenario indicator, a target scheduling strategy is determined for the key-value cache.

[0118] Furthermore, the target scenario may include a first scenario and / or a second scenario; when the target scenario is the first scenario, the target scenario index includes the first scenario index; when the target scenario is the second scenario, the target scenario index includes the second scenario index.

[0119] Multi-session inference scenarios refer to service scenarios in which multiple independent user requests (sessions) are processed concurrently in a single-instance or multi-instance deployed Large Language Model (LLM) service. Each session maintains its own independent inference context and KV-Cache state, while sharing underlying computing and memory resources. Its core characteristics are the concurrency and independence of sessions; each session executes in parallel, resources are allocated on demand, and the start, running, or termination of one session does not affect the normal execution of other sessions.

[0120] The first scenario includes a multi-session inference scenario that satisfies a first condition, also known as a multi-session burst load scenario, where the first condition indicates that a multi-session burst load occurs in the multi-session inference scenario.

[0121] In an example of a multi-session inference scenario, assume the large language model service is deployed with two instances (Instance A and Instance B), each running on an 8-GPU server. User requests arrive: 100 users simultaneously initiate requests. Assume 50 of these requests are sent to Instance A, and the other 50 to Instance B. Within Instance A, its 8 GPUs need to collaborate to process the 50 sessions assigned to it. These 50 sessions are distributed across the 8 GPUs in some way (e.g., tensor parallelism, pipelined parallelism), and each GPU core may process multiple sessions simultaneously in a time-slice round-robin manner. When a large number of these 50 sessions simultaneously enter the continuous generation phase, it triggers issues such as a surge in KV access frequency, HBM pressure, and PCIe load at the GPU hardware layer within Instance A.

[0122] When multiple instances and multiple GPUs provide parallel services, multiple sessions concurrently decode on the GPU of the same instance, which can cause the KV access frequency and HBM / PCIe pressure to rise simultaneously in a short period of time. The key-value cache management method of this application can be used to detect multi-session burst load scenarios and has adaptive scheduling capabilities in such scenarios, thereby suppressing congestion.

[0123] For example, the first condition may include one or more of the following: ① the number of concurrent sessions exceeds a preset number; ② the growth rate of the number of active sessions and KV hot sets within the same instance increases; ③ the request queue length or scheduling delay increases abnormally; ④ the upper-layer framework explicitly reports an increase in session concurrency.

[0124] When it is determined that the first condition is met, the current situation is identified as the first scenario, and the processing flow for the first scenario can be initiated. Specifically, in addition to periodically collecting target indicators and calculating the congestion index, this application also periodically collects indicators for the first scenario. When it is determined that the optimization conditions are met, a target scheduling strategy for the key-value cache is determined based on the congestion index and the indicators for the first scenario.

[0125] The first scenario metrics are specialized metrics collected for multi-session burst load scenarios. Compared to the target metrics, the first scenario metrics focus more on "change trends" and "session dimensions." For example, the first scenario metrics may include one or more of the following: memory locality metrics, link congestion metrics, mapping overhead metrics, session access concentration metrics, and page lock resource pressure metrics. The memory locality metrics include the changing trends of page faults and NUMA hint faults, used to characterize the degree of deterioration in memory access locality caused by a surge in cross-node memory access. The link congestion metrics include PCIe latency and bandwidth utilization, used to reflect the link congestion status in real time. The mapping overhead metrics include the call frequency and average latency of DM mapping / demapping operations, used to assess the overhead pressure caused by frequent address mapping translations. The session access concentration metrics include the proportion of hot pages and the distribution of access frequency within each session, used to characterize the intensity and concentration of access competition for KV Cache resources by different sessions. The page lock resource pressure metrics include the system- and session-level page lock quota occupancy rate, used to measure the resource pressure of pinned memory and the potential risk of forced reclamation.

[0126] In the first scenario, the perception module of this application can continuously collect the aforementioned first scenario indicators within a millisecond period. After normalization, these indicators can form the state vector of the input strategy engine.

[0127] The policy engine can determine the target scheduling policy applicable to the first scenario based on the received state vector and convert it into corresponding control actions. The target scheduling policy applicable to the first scenario includes, but is not limited to, one or more of the following: 1) Locking the key KV pages of high-priority sessions to ensure that they are not replaced; 2) Prioritizing the migration of high-activity KV pages to local NUMA to reduce remote access; 3) Adjusting the DMA mapping mode to reduce mapping overhead; 4) Suspending the prefetching behavior of some low-priority sessions to reduce instantaneous resource preemption; 5) Dynamically adjusting the page lock quota to free up more DRAM space for high-activity sessions.

[0128] For example, if the state vector indicates a surge in concurrent sessions and the rate of increase in hot key-value (KV) sets, along with an increase in the congestion index I, then the target scheduling strategy could include: prioritizing the locking of KV pages for high-priority sessions to prevent them from being replaced, and suspending the prefetching function for low-priority sessions to reduce instantaneous resource preemption. As another example, if the state vector indicates a high NUMA remote access rate but low DMA latency, then the target scheduling strategy could include: prioritizing NUMA migrations rather than switching DMA modes, such as prioritizing the migration of highly active KV pages to local NUMA nodes to reduce remote access. Furthermore, if the state vector indicates a significant increase in queue length and scheduling latency, then the target scheduling strategy could include: reducing batch migrations and increasing page locking limits to ensure the stability of hot sessions.

[0129] The executor executes the control actions according to the corresponding action parameters based on the control actions output by the policy engine for the first scenario. After execution, the system can usually observe the following effects: 1) The proportion of remote memory access tends to decrease; 2) DMA mapping efficiency tends to improve; 3) PCIe link pressure is relieved; 4) The token decoding process is more stable and jitter is reduced.

[0130] Long-context inference scenarios refer to service scenarios in large language model (LLM) services deployed in single or multiple instances where the length of the input context sequence or the size of the effective token window for a single inference reaches or exceeds a system-preset long-context threshold. Its core characteristics are the long sequence nature of the context and the dynamic growth of KV-Cache usage: the amount of KV page data generated during inference increases significantly and positively correlated with the increase in context length, and the access pattern shows that the hot set window continuously slides as new tokens are generated. The hot set window refers to the coverage of the KV hot set in the token sequence, which is usually wider than the active window and used for hot / cold stratification.

[0131] The second scenario includes long-context inference scenarios that satisfy the second condition. The second condition represents the load state caused by a surge in KV-Cache resource demand and a divergence in cross-level access popularity due to long sequence inputs. In the second scenario, the following problems may occur: a surge in local memory pressure, such as the host DRAM or GPU HBM utilization approaching saturation; an abnormally high proportion of remote access due to insufficient local capacity, such as an abnormally high hit rate for access to CXL extended memory or remote NUMA nodes, introducing higher latency; and frequent page swapping / out operations between HBM and DRAM or between DRAM and CXL due to the inability of hot KV page data to be fully accommodated in limited high-speed memory, resulting in additional I / O overhead.

[0132] The key-value cache management method of this application can be used to provide an optimization mechanism for the cold and hot layering capability and cross-level collaboration in long context reasoning scenarios, thereby effectively alleviating the above problems.

[0133] For example, the second condition may include one or more of the following: ① The context length of the session or the processed token window continuously exceeds a preset threshold; ② The total occupancy of the instance's KV-Cache shows a clear and continuous upward trend over time; ③ The session's hotset exhibits obvious sliding window characteristics as inference progresses; ④ The upper-layer inference framework explicitly reports that the session is a long context type.

[0134] When it is determined that the second condition is met, the current situation is identified as the second scenario, and the processing flow for the second scenario can be initiated. Specifically, in addition to periodically collecting target indicators and calculating the congestion index, this application also periodically collects second scenario indicators. When it is determined that the optimization conditions are met, a target scheduling strategy for the key-value cache is determined based on the congestion index and the second scenario indicators.

[0135] The second scenario metrics are specialized metrics collected for long-context inference scenarios. For example, the second scenario metrics may include one or more of the following: KV page popularity trend metric, KV page hierarchical distribution metric, Token window prediction metric, storage tier level metric, and prefetching performance metric. The KV page popularity trend metric includes the trend of KV page access frequency changes, reflecting the trend of KV pages changing from cold pages to hot pages or vice versa. The KV page hierarchical distribution metric includes the distribution of KV pages at different storage tiers (e.g., host DRAM / remote NUMA nodes / CXL extended memory), representing the proportion and position of KV pages corresponding to different Transformer layers of the model at different storage tiers. The Token window prediction metric includes the Token window prediction and context range of the current session, used to predict the context range (Token window) to be accessed in the near future. The storage tier level metric includes the memory tier level and capacity pressure, used to quantify the real-time utilization and bottleneck risk of each storage tier. The prefetching performance metrics include changes in prefetching behavior and access sequence, used to reflect whether the prefetch hit rate and order have changed, and to measure the degree of matching between prefetching behavior and actual access patterns.

[0136] In the second scenario, the perception module of this application can continuously collect the above-mentioned second scenario indicators within a millisecond period. After normalization, these indicators can form the state vector of the input strategy engine.

[0137] The policy engine determines the target scheduling policy applicable to the second scenario based on the state vector and translates it into corresponding control actions. The target scheduling policy applicable to the second scenario includes, but is not limited to, one or more of the following: 1) Migrating KV pages with a predicted high probability of future access from low-speed storage to high-speed storage in advance; 2) Implementing page locking or priority residency policies for KV pages associated with the currently active window; 3) Migrating KV pages no longer in the future decoding window to a larger remote storage layer; 4) Adjusting memory quotas and hot / cold boundaries for different sessions to avoid prolonged high levels of host DRAM; 5) Dynamically updating the hot / cold labels of KV pages based on changes in popularity to guide the kernel's KV page migration policy. Here, the currently active window refers to the token range corresponding to the set of frequently accessed KV pages at the current moment.

[0138] For example, if the state vector represents a continuously expanding token window and rising DRAM levels, determining the target scheduling strategy could include migrating cold pages to CXL extended memory or NVMe SSDs and limiting the prefetch step size. As another example, if the state vector represents a prediction window indicating an upcoming access to a remote storage layer and rising PCIe RTT, determining the target scheduling strategy could include prefetching high-probability KV pages to local DRAM. If the state vector represents a fast hot-set window sliding speed, determining the target scheduling strategy could include increasing the hot-page lockout limit and reducing the frequency of cross-layer migrations.

[0139] The executor executes the control actions according to the corresponding action parameters based on the control actions output by the strategy engine for the second scenario. After execution, the system can usually observe the following effects: 1) Hot key-value pages are more concentrated in low-latency storage media; 2) The access path in the decoding stage becomes shorter and more stable; 3) The high water level phenomenon of host DRAM is alleviated; 4) Tail latency jitter of long text inference is reduced; 5) Prefetching operations are more targeted, reducing unnecessary data movement.

[0140] Furthermore, the following alternative solutions can be provided for the execution process of this application: Regarding alternatives to the policy engine: In addition to rule-based models (such as RL models), the policy engine can also use any of the following alternatives: 1) Decision tree or gradient boosting tree (GBDT) models for predicting action types; 2) Finite state machine (FSM) models for defining key-value (KV) cold and hot state transitions; 3) Lightweight reinforcement learning models for adaptively adjusting actions in specific deployment environments.

[0141] Regarding alternatives for actuators: Different actuators can be used on different hardware platforms. For example: 1) Alternatives for migration actuators: Automatic NUMA balancing with new policy weights can be used, or page-granular CXL type switching components can be used; 2) Alternatives for DMA tuning actuators: IOMMU passthrough mode can be used to reduce mapping, or device-side DMA distributors can be used to schedule data paths; 3) Alternatives for prefetch actuators: GPU-driven Async Prefetch can be used, or cross-node prefetching based on CXL Fabric can be used.

[0142] Regarding alternatives to the hot / cold tiered actuator: various strategies can be combined, such as: 1) hardware-assisted statistics based on page access bits; 2) periodic scanning and LRU incremental updates; 3) offloading some hot / cold management to device-side caches (such as NIC Cache or SmartSSD Cache).

[0143] The above alternatives do not change the core process of this application: a KV-Cache scheduling closed-loop process that integrates real-time perception, policy decision-making, and multi-dimensional execution at the operating system level.

[0144] In summary, the key-value cache management method provided in this application addresses the KV-Cache management problem in the Large Language Model (LLM) inference process by managing it at the operating system level. Specifically, it periodically collects target metrics and calculates a congestion index based on these metrics. The system's congestion state can be determined based on the calculated congestion index and the collected target metrics. When the system's congestion state meets optimization conditions, a target scheduling strategy for the key-value cache is determined, and the corresponding target operation is executed. The target metrics are related to KV-Cache access behavior and system bottlenecks. Based on these metrics, a system performance degradation trend can be identified earlier, and corresponding target scheduling strategies can be implemented before the problem escalates. These strategies include deciding how to allocate, migrate, and schedule KV-Cache resources, providing a stable, reliable, and efficient resource guarantee mechanism for LLM inference, improving overall system performance, and ensuring the efficiency and stability of model inference.

[0145] Furthermore, this application also includes the following effects: (1) Achieve unified control capabilities across levels and components. This application forms a unified scheduling closed loop for KV-Cache at the operating system layer, incorporating KV page migration, DMA scheduling, page lock quota, hot / cold tiering, and prefetch / reclaim decisions into the same control plane. This enables the operating system not only to provide resources but also to decide how to allocate, migrate, and optimize resources, thereby significantly improving the overall controllability of the system.

[0146] (2) It has real-time sensing capabilities and can identify dynamic performance bottlenecks. This application periodically collects target indicators, including the access ratio of remote NUMA nodes, PCIe link status, DMA mapping trend, and KV heat changes, which can identify the trend of system performance decline earlier and optimize and adjust before the problem escalates.

[0147] (3) Provide an interpretable and reproducible strategy and threshold system. This application provides mechanisms such as congestion index I, decision thresholds (such as I_high and I_low), and action triggering conditions for target scheduling strategies. Adaptive regulation is achieved through rule models and heat prediction logic, avoiding the unverifiable nature of black-box strategies and making the system easier to debug, verify, and continuously optimize in engineering practice.

[0148] (4) Significantly improve resource utilization efficiency and multi-session service capabilities This application enables the execution of differentiated strategies at the operating system layer based on dimensions such as session activity, window prediction, key-value page popularity, and host NUMA pressure, thereby improving resource contention among different sessions. This allows the system to support more concurrent sessions and operate more stably in a multi-tenant environment.

[0149] (5) Effectively improves inference delay stability and reduces tail delay jitter risk. This application reduces latency fluctuations across NUMA nodes and PCIe paths by migrating hot key-value pages in real time, adjusting DMA mapping mode, maintaining page lock protection for high-priority sessions, and locality optimization, thus making model inference latency more stable.

[0150] (6) It has good universality and scalability. This application is independent of any specific inference framework, model architecture, or hardware type, and can run on various resource formats such as GPU, CPU, CXL, and NVMe, demonstrating cross-platform adaptability. This architecture can also be extended to future large model execution accelerators and novel storage architectures.

[0151] This application can improve the operating system's ability to regulate under complex load scenarios and provide a stable, reliable, and efficient resource guarantee mechanism for large model inference.

[0152] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0153] Reference Figure 4 This diagram illustrates a structural block diagram of an embodiment of a key-value cache management device according to this application, applied to a large language model inference system. The key-value cache of the large language model inference system is distributed in a multi-level heterogeneous storage system, which includes a near-end storage medium and a far-end storage medium. The device may include: The sensing module 401 is used to periodically collect target indicators, which include at least one of the following: a first indicator for representing bus transmission delay, a second indicator for representing the proportion of remote access, and a third indicator for representing the direct memory access mapping overhead. The strategy engine 402 is used to calculate the congestion index based on the target metric; and determine the target scheduling strategy for the key-value cache when the optimization conditions are met; the optimization conditions include: the congestion index is greater than a first threshold, and / or any target metric is greater than the corresponding sub-metric threshold. The executor 403 is used to execute the target operation corresponding to the target scheduling strategy.

[0154] Optionally, the policy engine includes: The indicator shaper is used to normalize the target indicator; obtain the weight parameter corresponding to each target indicator; the weight parameter is obtained by pre-training offline; and perform a weighted summation of each normalized target indicator and its corresponding weight parameter to obtain the congestion index. The strategy inference engine is used to input the normalized target index as a state vector into the rule model, output the control action corresponding to the target scheduling strategy through the rule model, and issue the command corresponding to the control action to the executor, carrying the action type and action parameters, so that the executor can execute the control action.

[0155] Optionally, the device further includes: The parameter adjustment module is used to record the quantitative data of the benefits generated by the target operation for each target indicator after the target operation is executed; and to adjust the weight parameters corresponding to each target indicator according to the quantitative data of the benefits.

[0156] Optionally, the policy engine 402 is specifically used for: Based on the remaining local memory, a key-value page migration strategy is determined; the key-value page migration strategy is used to migrate hot pages to the near-end storage medium, or to migrate cold pages to the far-end storage medium. The DMA mapping strategy is determined based on the average DMA mapping time; the DMA mapping strategy is used to switch the DMA mapping mode to either large block aggregation mode or small block pipeline mode. The KV page prefetch strategy is determined based on the prefetch hit rate and / or the average DMA mapping time; the KV page prefetch strategy is used to pause or resume the prefetch function, or to adjust the prefetch step size. Based on the proportion of hot pages, a page lock management strategy is determined; the page lock management strategy is used to increase or decrease the page lock quota for hot pages.

[0157] Optionally, the device further includes: The first recovery module is used to gradually reduce the execution intensity of the target scheduling strategy if the congestion index is detected to be between the second threshold and the first threshold and stabilized for a preset period. The second recovery module is used to cancel the target scheduling strategy and restore the default scheduling strategy if the congestion index is detected to be less than the second threshold and stabilized for a preset period.

[0158] Optionally, the device further includes: The scenario metric collection module is used to periodically collect target scenario metrics when it is determined that the current target scenario is being observed. The strategy engine is specifically used to: determine the target scheduling strategy for the key-value cache based on the congestion index and the target scenario indicator.

[0159] Optionally, the target scenario includes a first scenario and / or a second scenario; when the target scenario is the first scenario, the target scenario index includes a first scenario index; when the target scenario is the second scenario, the target scenario index includes a second scenario index; the first scenario includes a multi-session reasoning scenario that satisfies a first condition; the second scenario includes a long-context reasoning scenario that satisfies a second condition.

[0160] Optionally, the near-end storage medium includes at least one of the following: GPU high-bandwidth memory, local non-uniform memory architecture NUMA node; the far-end storage medium includes at least one of the following: far-end NUMA node, CXL extended memory, NVMe solid-state drive.

[0161] Optionally, the large language model inference system supports parallel deployment of multiple instances, with multiple independent instances sharing underlying hardware resources and each configured with its own dedicated resource quota; the dedicated resources include key-value caches and corresponding page lock quotas and memory quotas independently managed by each instance, and each independent instance independently executes the key-value cache management method based on its own corresponding target indicators.

[0162] The key-value cache management device provided in this application addresses the KV-Cache management problem in the Large Language Model (LLM) inference process by managing it at the operating system level. Specifically, it periodically collects target metrics and calculates a congestion index based on these metrics. The calculated congestion index and the collected target metrics determine the system's congestion state. When the system's congestion state meets optimization conditions, a target scheduling strategy for the key-value cache is determined, and the corresponding target operation is executed. The target metrics are related to KV-Cache access behavior and system bottlenecks. Based on these metrics, a system performance degradation trend can be identified earlier, and corresponding target scheduling strategies can be implemented before the problem escalates. These strategies include deciding how to allocate, migrate, and schedule KV-Cache resources, providing a stable, reliable, and efficient resource guarantee mechanism for LLM inference, improving overall system performance, and ensuring the efficiency and stability of model inference.

[0163] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0164] Reference Figure 5 This is a schematic diagram of the structure of the electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device includes: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other through the communication bus. The memory is used to store at least one executable instruction, which causes the processor to execute the steps of the key-value cache management method of the aforementioned embodiment.

[0165] This application provides a non-transitory computer-readable storage medium. When the instructions in the storage medium are executed by a terminal's program or processor, the terminal is able to perform the steps of the key-value cache management method described in the foregoing embodiments. The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably.

[0166] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0167] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0168] These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing terminal device to operate in a predictive manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0170] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0171] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A key-value cache management method, characterized in that, The method, applied to a large language model inference system, wherein the key-value cache of the large language model inference system is distributed in a multi-level heterogeneous storage system, the multi-level heterogeneous storage system including near-end storage media and far-end storage media, includes: Periodically collect target metrics, which include at least one of the following: a first metric for representing bus transmission latency, a second metric for representing the proportion of remote accesses, and a third metric for representing the overhead of direct memory access mapping. Calculate the congestion index based on the target indicators; When the optimization conditions are met, a target scheduling strategy for the key-value cache is determined; the optimization conditions include: the congestion index is greater than a first threshold, and / or any target indicator is greater than the corresponding sub-indicator threshold; Execute the target operation corresponding to the target scheduling strategy.

2. The method according to claim 1, characterized in that, The calculation of the congestion index based on the target indicator includes: The target indicators are normalized. Obtain the weight parameters corresponding to each target indicator; these weight parameters are obtained through pre-training offline. The congestion index is obtained by weighting and summing the normalized target indicators with their corresponding weight parameters.

3. The method according to claim 2, characterized in that, The method further includes: After executing the target operation, record the quantitative data of the benefit generated by the target operation for each target indicator; Adjust the weight parameters corresponding to each target indicator based on the aforementioned revenue quantification data.

4. The method according to claim 1, characterized in that, The execution of the target operation corresponding to the target scheduling policy includes: The target index is converted into a state vector of the large language model inference system; The state vector is input into a preset rule model, and the control action corresponding to the target scheduling strategy is output through the rule model. The actuator sends a command corresponding to the control action, carrying the action type and action parameters, and the actuator executes the control action.

5. The method according to claim 1, characterized in that, The determination of the target scheduling strategy for the key-value cache includes one or more of the following: Based on the remaining local memory, a key-value page migration strategy is determined; the key-value page migration strategy is used to migrate hot pages to the near-end storage medium, or to migrate cold pages to the far-end storage medium. Determine the DMA mapping strategy based on the average DMA mapping time; The DMA mapping strategy is used to switch the DMA mapping mode to either large block aggregation mode or small block pipeline mode. Determine the KV page prefetching strategy based on the prefetch hit rate and / or the average DMA mapping time; The KV page prefetching strategy is used to pause or resume the prefetching function, or to adjust the prefetching step size; Based on the proportion of hot pages, a page lock management strategy is determined; the page lock management strategy is used to increase or decrease the page lock quota for hot pages.

6. The method according to claim 1, characterized in that, The method further includes: If the congestion index is detected to be between the second threshold and the first threshold and remains stable for a preset period, the execution intensity of the target scheduling strategy will be gradually reduced. If the congestion index is detected to be less than the second threshold and remains stable for a preset period, the target scheduling strategy is revoked and the default scheduling strategy is restored.

7. The method according to claim 1, characterized in that, The method further includes: Once the target scenario is determined, target scenario metrics are collected periodically. Determining the target scheduling strategy for the key-value cache includes: Based on the congestion index and the target scenario indicator, a target scheduling strategy is determined for the key-value cache.

8. The method according to claim 7, characterized in that, The target scenario includes a first scenario and / or a second scenario; when the target scenario is the first scenario, the target scenario index includes a first scenario index; when the target scenario is the second scenario, the target scenario index includes a second scenario index; the first scenario includes a multi-session reasoning scenario that satisfies a first condition; the second scenario includes a long-context reasoning scenario that satisfies a second condition.

9. The method according to any one of claims 1 to 8, characterized in that, The near-end storage medium includes at least one of the following: GPU high-bandwidth memory, local non-uniform memory architecture NUMA node; the far-end storage medium includes at least one of the following: far-end NUMA node, CXL extended memory, NVMe solid-state drive.

10. The method according to any one of claims 1 to 8, characterized in that, The large language model inference system supports parallel deployment of multiple instances, with multiple independent instances sharing underlying hardware resources and each configured with its own dedicated resource quota. The dedicated resources include key-value caches and corresponding page lock quotas and memory quotas that are independently managed by each instance. Each instance independently executes the key-value cache management method based on its own target metrics.

11. A key-value cache management device, characterized in that, An apparatus for use in a large language model inference system, wherein the key-value cache of the large language model inference system is distributed in a multi-level heterogeneous storage system, the multi-level heterogeneous storage system including a near-end storage medium and a far-end storage medium, the apparatus comprising: A sensing module is used to periodically collect target indicators, which include at least one of the following: a first indicator for representing bus transmission delay, a second indicator for representing the proportion of remote accesses, and a third indicator for representing the overhead of direct memory access mapping. A strategy engine is used to calculate a congestion index based on the target metric; and to determine a target scheduling strategy for the key-value cache when optimization conditions are met; the optimization conditions include: the congestion index is greater than a first threshold, and / or any target metric is greater than the corresponding sub-metric threshold. An executor is used to execute the target operation corresponding to the target scheduling strategy.

12. An electronic device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the steps of the key-value cache management method as described in any one of claims 1 to 10.

13. A readable storage medium, characterized in that, The program or instructions are stored on the readable storage medium, and when the program or instructions are executed by a processor, they implement the steps of the key-value cache management method as described in any one of claims 1 to 10.

Images