A multi-request heterogeneous memory QoS management method and system for large language model reasoning

Abstract

The application provides a multi-request heterogeneous memory QoS management method and system for large language model reasoning, which divides three response speed commitment levels for reasoning requests, matches the differentiated initial memory quota of HBM, CXL extended memory and CPU DDR, realizes the cooperative scheduling of memory quota and prefetch priority by means of the global quota table, memory state table and SLA monitoring table maintained by the quota manager, combines the HBM quota dynamic recycling and release mechanism based on the delay threshold, the linkage strategy of step-by-step adjustment of the prefetch cache amount and the prefetch bandwidth in the normal, limited flow and emergency three types of running states, and realizes the shared access and elastic scheduling of the extended memory among multiple GPUs by relying on the memory pooling capability of the CXL Switch. The application significantly improves the SLA achievement rate of high-priority requests, and can realize the dynamic adaptation and efficient utilization of heterogeneous memory resources.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and in particular to a method and system for QoS management of multi-request heterogeneous memory for large language model inference. Background Technology

[0002] With the large-scale deployment of Large Language Models (LLMs) in scenarios such as dialogue interaction, content generation, and batch inference, inference servers typically need to handle a large number of concurrent inference requests. Furthermore, the Quality of Service (QoS) requirements for different types of requests vary significantly: real-time interactive dialogue requests have stringent requirements for first-word response latency, typically requiring a P99 latency of less than 500ms, and must ensure high-speed access to critical data such as key-value caches; offline batch inference requests prioritize throughput and can tolerate higher access latency. Meanwhile, to balance memory capacity and access performance, a three-tier heterogeneous memory architecture based on HBM, CXL extended memory, and CPU DDR has become the mainstream configuration for large model inference servers. How to efficiently and fairly allocate heterogeneous memory resources in multi-request concurrency scenarios has become a core technical issue restricting the overall performance of the inference service and the achievement rate of Service Level Agreements (SLAs).

[0003] Current mainstream large language model inference frameworks, such as vLLM systems based on the PagedAttention mechanism and SGLang systems for structured language model programs, generally adopt a uniform memory scheduling strategy to manage all inference requests indiscriminately. They improve the space utilization of key-value cache through paged memory management, but they do not design differentiated memory allocation rules for requests with different SLA levels. When the server simultaneously handles concurrent inference requests of different priorities, low-priority requests will occupy high-bandwidth memory resources, causing the key-value cache of high-priority requests to not reside fully in high-speed memory. Their inference latency is easily affected by low-priority requests, making it difficult to guarantee the quality of service in low-latency scenarios.

[0004] In terms of heterogeneous memory expansion, the CXL (Compute Express Link) protocol has become the mainstream technology for GPU memory expansion due to its hardware-level cache coherency, non-blocking transactions, and memory pooling features. However, in existing large language model inference systems, CXL extended memory is mostly used as a passive storage expansion device for a single GPU, used for offloading cold data. Each GPU needs to maintain its own key-value cache copy independently, resulting in redundant storage of the same context data across multiple GPUs. This not only wastes memory resources but also increases the overhead of cross-GPU data copying, making it difficult to further improve overall memory utilization.

[0005] At the resource dynamic scheduling level, existing inference systems mostly use statically configured memory quotas, which cannot dynamically adjust memory resource allocation based on the real-time latency performance of each request. When the load of high-priority requests surges and latency approaches the SLA limit, the system cannot automatically reclaim high-speed memory resources from low-priority requests; conversely, when the load of high-priority requests decreases, idle high-speed memory cannot be allocated to low-priority requests in a timely manner to improve overall throughput, resulting in insufficient flexibility in resource scheduling. Furthermore, in heterogeneous memory systems that support hardware prefetching, the prefetch engine typically uses a unified first-in-first-out scheduling strategy to process prefetch commands for all requests, without associating prefetch bandwidth priority with the QoS level of the request. When multiple requests initiate prefetch requests simultaneously, the prefetch operation of high-priority requests may be blocked by low-priority requests, leading to uncontrollable prefetch latency and further impacting the inference performance of high-priority requests.

[0006] It is evident that existing heterogeneous memory management methods for large language model inference suffer from several technical shortcomings, including undifferentiated scheduling of multiple requests, insufficient utilization of CXL memory pooling capabilities, inability to dynamically adapt memory quotas to load, and a disconnect between prefetch priority and QoS levels. These limitations make it difficult to simultaneously ensure service quality for high-priority requests and achieve efficient utilization of overall system memory resources in scenarios with concurrent multi-level SLA requests. A new solution is urgently needed to address these issues. Summary of the Invention

[0007] In view of this, embodiments of the present invention provide a method and system for QoS management of multi-request heterogeneous memory for large language model inference, in order to solve the problem that existing large language model inference processes lack dynamic, efficient and differentiated QoS memory scheduling capabilities for heterogeneous memory.

[0008] One aspect of the present invention provides a multi-request heterogeneous memory QoS management method for large language model inference. The method is applicable to a multi-GPU, multi-level heterogeneous memory architecture for large language model inference, comprising HBM, CXL extended memory, and CPU DDR. The CXL extended memory of each GPU utilizes memory pooling shared access and elastic scheduling via CXL Switch. The method is executed based on a linked scheduler of the inference engine and includes the following steps: Based on the QoS classification module pre-embedded in the request scheduling layer, the inference requests are marked with response speed commitment levels according to preset mapping rules. The response speed commitment levels are divided into first-level requests that meet latency priority, second-level requests that meet balance priority, and third-level requests that meet throughput priority, according to priority from high to low. Initial memory quotas are allocated to first-level requests, second-level requests, and third-level requests in the order of HBM, CXL extended memory, and CPU DDR. The system maintains a global quota table based on a preset quota manager to record the response speed commitment level and memory quota of each inference request; maintains a memory status table to record the total capacity, current usage, and available amount of the three-level heterogeneous memory; maintains an SLA monitoring table to record the actual P99 latency, SLA target latency, and latency deviation of each inference request; dynamically allocates three-level heterogeneous memory to each inference request according to a preset strategy based on the global quota table, the memory status table, and the SLA monitoring table, and provides the response speed commitment level of each inference request to the arbitrator of the hardware prefetch engine, so as to execute the prefetch of the first-level request, the second-level request, and the third-level request in sequence during prefetch scheduling, and adjusts inference requests with latency close to the SLA limit to higher priority; According to a set frequency, based on the quota manager, when the response delay of the first-level request is greater than a first threshold, a first set proportion of HBM memory is reclaimed from the second-level request to supply the first-level request; when the response delay of the first-level request is less than a second threshold, a second set proportion of HBM memory is released from the first-level request to supply the second-level and third-level requests; and, based on the response delay of the first-level request, three states are divided: normal, rate-limited, and emergency, and the prefetch cache amount and prefetch bandwidth of the first-level request are increased step by step in these three states.

[0009] In some embodiments, a preset mapping rule is used to mark the response speed commitment level of inference requests, including: The response speed commitment level is automatically assigned based on the source of the inference request, or the response speed commitment level of the inference request is specified based on the application layer; wherein, in real-time dialogue scenarios, inference requests with a P99 latency requirement of less than or equal to 500ms are marked as first-level requests; in online inference scenarios, inference requests with a P99 latency requirement of greater than 500ms and less than or equal to 2s are marked as second-level requests; and in offline batch inference scenarios, inference requests with a P99 latency requirement of greater than 2s and less than 10s are marked as third-level requests. The initial memory allocation includes: prioritizing the allocation of HBM memory for the first-level requests; after satisfying the memory requirements of the first-level requests, allocating the remaining memory to the second-level requests and the third-level requests; if the remaining HBM memory capacity does not meet the memory requirements of all first-level requests, downgrading the HBM memory for the second-level requests to CXL extended memory and CPU DDR, and then configuring the first-level requests in the order of HBM, CXL extended memory, and CPU DDR.

[0010] In some embodiments, based on the global quota table, the memory status table, and the SLA monitoring table, three levels of heterogeneous memory are dynamically allocated to each inference request according to a preset strategy, including: Based on historical access frequency, the key-value cache accessed by the first-level request, the second-level request, and the third-level request are divided into hot data, warm data, and cold data according to the access frequency from high to low according to preset standards. Based on the global quota table and the memory status table, hot data in the first-level request is preferentially matched with HBM memory, while warm and cold data in the first-level request are matched with CXL extended memory. After the allocation of the first-level request, hot data in the second-level request is preferentially matched with HBM memory, warm data is preferentially matched with CXL extended memory, and cold data is matched with CPU DDR. After the allocation of the second-level request, hot, warm, and cold data in the third-level request are preferentially matched with CXL extended memory and CPU DDR, wherein, if there is remaining HBM memory, it is preferentially matched with hot data in the third-level request.

[0011] In some embodiments, when the response delay of the first-level request exceeds a first threshold, the quota manager reclaims a first predetermined proportion of HBM memory from the second-level request to supply the first-level request at a set frequency, including: The actual P99 latency of the first-level request for each inference request is collected at a frequency of less than or equal to 100ms. The latency deviation is calculated with the SLA target latency recorded in the SLA monitoring table. If the latency deviation is greater than 10%, 5% to 20% of HBM memory is reclaimed from the second-level request. Among them, the second-level request with the highest HBM memory usage is preferentially selected as the reclamation target. A migration instruction is generated to migrate the key-value cache corresponding to the reclamation target from HBM memory to CXL extended memory. The migration instruction is written to the preset hardware prefetch engine migration queue through the MMIO protocol to perform data transfer. After the migration is completed, the global quota table is updated. When the response latency of the first-level request is less than the second threshold, a second predetermined proportion of HBM memory is released from the first-level request to supply the second-level and third-level requests, including: The actual P99 latency of the first-level request for each inference request is collected at a frequency of less than or equal to 100ms. The latency deviation is calculated with the SLA target latency recorded in the SLA monitoring table. If the latency deviation is less than -20%, 5% to 20% of HBM memory is released from the first-level request for allocation to the second-level request and the third-level request according to a third set ratio.

[0012] In some embodiments, the method further includes: multiple GPUs forming a unified shared memory pool from multiple CXL memory expansion devices via a CXL Switch, and each GPU directly accessing the key-value cache in the shared memory pool through non-blocking transactions and cache consistency primitives of the CXL.cache protocol; wherein, the first-level request preferentially uses the near-end region with lower latency in the shared memory pool, and the second-level request and the third-level request use the far-end region with higher latency; When the shared memory pool capacity is insufficient, cold data in the third-level requests and cold data in the second-level requests are eliminated in sequence according to the fourth preset ratio; wherein, the elimination operation is performed based on the hardware prefetch engine.

[0013] In some embodiments, the response latency of the first-level request is divided into three states: normal, rate-limited, and emergency. The prefetch cache size and prefetch bandwidth of the first-level request are increased progressively in each of these three states, including: The latency deviation of the first-level request is defined as less than 5% as a normal state, greater than 5% but less than 20% as a rate-limiting state, and greater than 20% as an emergency state. In the normal state, during prefetching scheduling, the prefetching of the first-level request, the second-level request, and the third-level request are executed sequentially, and a dual buffer is set up in HBM memory for ping-pong switching, which is used synchronously for current computing and background prefetching; in the rate-limiting state, the prefetching of the second-level request and the third-level request is downgraded to a single buffer, and the buffer space is released to prefetch the first-level request; in the emergency state, the prefetching of the second-level request and the third-level request is suspended.

[0014] In some embodiments, the response latency of the first-level request is divided into three states: normal, rate-limited, and emergency. The prefetch cache size and prefetch bandwidth of the first-level request are increased progressively in each of these three states, including: In the normal state, the prefetch bandwidth of the first-level request, the second-level request, and the third-level request is set to 5:3:2; in the rate-limited state, the prefetch bandwidth of the first-level request, the second-level request, and the third-level request is set to 8:2:0; in the emergency state, all prefetch bandwidth is allocated to the first-level request.

[0015] On the other hand, the present invention also provides a multi-request heterogeneous memory QoS management system for large language model inference. The system is a large language model inference architecture formed by multiple GPUs combined with HBM, CXL extended memory and CPU DDR three-level heterogeneous memory. The CXL extended memory of each GPU utilizes the memory pooling shared access and elastic scheduling of CXL Switch. The system is used to execute pre-loaded computer programs or instructions. When the computer program / instruction is executed, the system implements the steps of the above method.

[0016] On the other hand, the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0017] On the other hand, the present invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps of the above-described method. The multi-request heterogeneous memory QoS management method and system for large language model inference described in this invention divides inference requests into three levels of response speed commitment and matches them with differentiated initial memory quotas from HBM, CXL extended memory, and CPU DDR. It leverages a global quota table, memory status table, and SLA monitoring table maintained by a quota manager to achieve coordinated scheduling of memory quotas and prefetch priorities. This is combined with a dynamic HBM quota reclamation and release mechanism based on latency thresholds, and a linkage strategy that adjusts the prefetch cache size and prefetch bandwidth progressively under normal, rate-limited, and emergency operating states. Furthermore, it relies on CXL... Switch's memory pooling capability enables shared access and elastic scheduling of extended memory across multiple GPUs. This ensures low-latency inference performance for high-priority requests, preventing low-priority requests from crowding out high-speed memory resources and thus affecting the service quality of high-priority requests, significantly improving the SLA achievement rate of high-priority requests. It also enables dynamic adaptation and efficient utilization of heterogeneous memory resources, reducing redundant storage of key-value cache and cross-GPU copying overhead in multi-GPU scenarios. Furthermore, by linking prefetch priority with request level and runtime status, it ensures the prefetch efficiency of high-priority requests. Ultimately, while meeting differentiated service quality requirements, it effectively improves the overall concurrent inference throughput and heterogeneous memory resource utilization of the system.

[0018] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0019] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0020] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings: Figure 1 This is a flowchart illustrating a multi-request heterogeneous memory QoS management method for large language model inference according to an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of the dynamic quota adjustment and resource reclamation logic in the multi-request heterogeneous memory QoS management method for large language model inference according to another embodiment of the present invention.

[0022] Figure 3 This is the state machine and state transition diagram of the linkage scheduler in the multi-request heterogeneous memory QoS management method for large language model inference according to another embodiment of the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0024] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0025] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0026] As large language model inference services evolve towards multi-scenario concurrency, a three-tier heterogeneous memory architecture based on HBM, CXL extended memory, and CPUDDR has become the mainstream configuration for inference servers. Systems must simultaneously handle concurrent requests with varying SLA requirements, such as real-time dialogue, general online inference, and offline batch generation. Differentiated scheduling and quality of service (QoS) assurance of heterogeneous memory resources have become core bottlenecks restricting system performance. Specifically, mainstream inference frameworks generally adopt a unified memory scheduling strategy, failing to allocate memory resources differently based on the SLA level of requests. High-priority requests' key-value caches are easily overwhelmed by low-priority requests consuming high-speed memory space, leading to excessive latency. QoS assurance in low-latency scenarios is lacking. CXL extended memory is often used merely as a passive storage extension for a single GPU, failing to fully utilize the switching capabilities of the CXL switch to achieve multi-GPU memory pooling. Each GPU independently maintains a copy of the key-value cache, resulting in redundant storage, high cross-GPU data copying overhead, and low overall memory utilization. Memory quotas are mostly statically configured, unable to dynamically reclaim and release resources based on real-time latency performance, resulting in insufficient flexibility in resource adaptation during load fluctuations. The hardware prefetch engine uses a unified priority scheduling mechanism that is not linked to the QoS level of the request. Prefetching operations for high-priority requests are easily blocked, further exacerbating latency fluctuations.

[0027] In view of this, the present invention provides a multi-request heterogeneous memory QoS management method for large language model inference. This method is applicable to a multi-GPU, multi-level heterogeneous memory architecture for large language model inference, consisting of HBM, CXL extended memory, and CPU DDR. The CXL extended memory of each GPU utilizes the memory pooling shared access and elastic scheduling of the CXL Switch. The method is executed based on the linkage scheduler of the inference engine, such as... Figure 1 As shown, the method includes the following steps S101~S103: Step S101: Based on the QoS classification module pre-embedded in the request scheduling layer, the inference requests are marked with response speed commitment levels according to preset mapping rules. The response speed commitment levels are divided into first-level requests that prioritize latency, second-level requests that prioritize load balancing, and third-level requests that prioritize throughput, according to priority from high to low. Initial memory quotas are allocated to first-level requests, second-level requests, and third-level requests in the order of HBM, CXL extended memory, and CPU DDR.

[0028] Step S102: Based on the preset quota manager, maintain a global quota table to record the response speed commitment level and memory quota of each inference request; maintain a memory status table to record the total capacity, current usage, and available amount of the three-level heterogeneous memory; maintain an SLA monitoring table to record the actual P99 latency, SLA target latency, and latency deviation of each inference request; dynamically allocate three-level heterogeneous memory to each inference request according to a preset strategy based on the global quota table, memory status table, and SLA monitoring table, and provide the response speed commitment level of each inference request to the arbitrator of the hardware prefetch engine, so as to execute the prefetch of the first-level request, the second-level request, and the third-level request in sequence during prefetch scheduling, and adjust the inference requests with latency close to the SLA limit to higher priority.

[0029] Step S103: Based on the quota manager, when the response delay of the first-level request is greater than the first threshold, reclaim a first set proportion of HBM memory from the second-level request to supply the first-level request; when the response delay of the first-level request is less than the second threshold, release a second set proportion of HBM memory from the first-level request to supply the second-level and third-level requests; and, based on the response delay of the first-level request, divide it into three states: normal, rate-limited, and emergency, and increase the prefetch cache amount and prefetch bandwidth of the first-level request in each of these three states.

[0030] This invention relates to a server-grade three-tier heterogeneous memory hardware architecture optimized for large language model inference. Using a server motherboard as the carrier, the core hardware includes multiple GPU processors, a CPU main processor, a CXL switch chip, multiple independent CXL memory expansion devices, and CPU-side DDR main memory. Multiple GPUs are interconnected with the CXL switch via PCIe / CXL physical links. The CXL switch connects to multiple CXL memory expansion devices, and the CPU is also connected to the switching system via CXL / PCIe links, forming a tree-like topology containing compute nodes, switching nodes, and memory pool nodes.

[0031] Each GPU integrates its own dedicated high-bandwidth memory (HBM), which is the storage medium with the highest access bandwidth and lowest latency in the entire architecture. It is mainly used to store model weights and key-value cache hot data for high-priority inference requests. Each GPU's memory controller integrates a key-value cache access heat awareness unit and a hardware prefetch engine (HPC), which can collect the access heat of each attention head with zero software overhead and independently perform data prefetching and memory migration operations without consuming GPU computing core resources.

[0032] Multiple CXL memory expansion devices are interconnected via CXL Switches to form a globally unified addressed shared memory pool, which is the core expansion layer of the architecture. This layer has a capacity much larger than GPU HBM, and its access latency is between HBM and DDR. It mainly stores key-value cached warm data for low-priority requests, as well as non-hot data for high-priority requests.

[0033] The CPU-side DDR main memory provides the system with maximum storage capacity as a backup, primarily used for archiving infrequently accessed key-value cache cold data, as well as carrying runtime data for system processes and the inference framework. This level has the highest access latency and is only used for long-term cold data storage when the HBM and CXL memory pools are insufficient. Data access must be completed in coordination with the system bus and memory scheduling.

[0034] The CXL Switch is the core hardware for implementing memory pooling and elastic scheduling. It performs unified global address space orchestration for all downstream CXL memory devices, with address mapping managed uniformly by the inference engine software. Routing rules can be dynamically configured through the MMIO interface. Based on the CXL.cache protocol, it achieves cache consistency when multiple GPUs access shared memory, and ensures data correctness when multiple GPUs concurrently read the same key-value cache data through corresponding protocol primitives. It supports dynamically adjusting the memory pool capacity quota according to the real-time load of each GPU, realizing on-demand capacity allocation. It can divide the memory pool into a near-end low-latency area and a far-end high-capacity area, with high-priority requests being allocated to the near-end area first, supporting differentiated QoS services.

[0035] In step S101, the QoS classification module marks the newly arrived inference requests with a classification level according to preset mapping rules. The mapping rules can be automatically matched based on the API access endpoint of the request, the request header identifier, the business scenario type, or the user service level, or can be explicitly specified by the upper layer application through the interface. The requests are divided into three priority levels from high to low: the first level corresponds to the latency priority scenario, serving low-latency demand services such as real-time dialogue and interactive programming; the second level corresponds to the balanced scenario, serving general online inference services; and the third level corresponds to the throughput priority scenario, serving offline batch inference, batch content generation, and other services.

[0036] In some embodiments, a preset mapping rule is used to mark the response speed commitment level of inference requests, including: automatically assigning the response speed commitment level based on the source of the inference request, or specifying the response speed commitment level of the inference request based on the application layer; wherein, inference requests with a response speed requirement of P99 latency less than or equal to 500ms in real-time dialogue scenarios are marked as Level 1 requests; inference requests with a response speed requirement of P99 latency greater than 500ms and less than or equal to 2s in online inference scenarios are marked as Level 2 requests; and inference requests with a response speed requirement of P99 latency greater than 2s and less than 10s in offline batch inference scenarios are marked as Level 3 requests.

[0037] After completing the tier labeling, the tiering module, considering the available spare capacity of the current system's three-tier heterogeneous memory, pre-allocates initial memory quotas for each request, following the principle of prioritizing high-speed memory towards higher-priority requests: Tier 1 requests are prioritized for HBM resources, with their KV Cache hot data residing in HBM by default; Tier 2 requests are evenly allocated between HBM and CXL extended memory; Tier 3 requests prioritize CXL extended memory and CPU DDR, allocating a small quota only when HBM space is sufficient. The initial quota uses a best-effort allocation strategy: Tier 1 requests are prioritized for HBM memory; after satisfying the memory requirements of Tier 1 requests, the remaining memory is allocated to Tier 2 and Tier 3 requests; if the remaining HBM memory capacity is insufficient to meet the memory requirements of all Tier 1 requests, the HBM memory for Tier 2 requests is downgraded to CXL extended memory and CPU DDR, and then Tier 1 requests are configured sequentially in the order of HBM, CXL extended memory, and CPU DDR. Specifically, priority is given to ensuring the HBM quota requirements of all Level 1 requests. If the current available HBM space is insufficient to meet the quota standards for all Level 1 requests, the HBM quotas for Level 2 and Level 3 requests are directly reduced, and their KV Caches are preferentially scheduled to CXL extended memory or DDR main memory to free up high-speed memory space for Level 1 requests. Simultaneously, the response speed commitment level for each request is bound to its KV Cache data throughout its entire lifecycle, remaining effective throughout subsequent data migration, memory scheduling, and prefetch scheduling stages to ensure the consistency of the QoS policy.

[0038] In step S102, the quota manager continuously maintains three types of core data tables to support scheduling decisions: the first is a global quota table, which records the response speed commitment level of each online inference request, the actual usage and target quota value of each level of memory (HBM, CXL extended memory, and CPU DDR), serving as the basis for resource allocation; the second is a memory status table, which calculates the total capacity, usage, and available capacity of the three levels of storage media in real time to understand the global resource level; and the third is an SLA monitoring table, which continuously collects the actual P99 inference latency of each request, the corresponding SLA target latency, and calculates the latency deviation value to evaluate the SLA achievement in real time. Based on these three types of data tables, the quota manager dynamically adjusts the memory resource distribution of each request according to a tiered allocation strategy: hot data for first-level requests always resides in HBM, while warm data is scheduled to CXL extended memory; hot data for second-level requests is retained in HBM, warm data is stored in CXL extended memory, and cold data is archived to DDR main memory; most data for third-level requests is stored in CXL extended memory and DDR main memory, with only a small amount of high-frequency access data temporarily residing in HBM. Meanwhile, the quota manager synchronizes the response speed commitment level of each request to the priority arbiter of the hardware prefetch engine in real time. When scheduling the prefetch command queue, the arbiter strictly executes the prefetch operation in the order of first priority, second priority, and third priority last. When the SLA monitoring table detects that the latency of a certain request is close to the SLA limit, the quota manager sends a temporary priority escalation instruction to the arbiter to raise the prefetch priority of the request, so as to prevent the latency from being further deteriorated due to prefetch queuing.

[0039] For example, based on the global quota table, memory status table, and SLA monitoring table, three levels of heterogeneous memory are dynamically allocated to each inference request according to a preset strategy, including steps S1021 and S1022: Step S1021: Based on historical access popularity, the key-value cache accessed by first-level requests, second-level requests, and third-level requests are divided into hot data, warm data, and cold data according to the access popularity from high to low according to preset standards.

[0040] Step S1022: Based on the global quota table and memory status table, hot data in the first-level request is preferentially matched with HBM memory, and warm and cold data in the first-level request are matched with CXL extended memory; after the allocation of the first-level request is completed, hot data in the second-level request is preferentially matched with HBM memory, warm data is preferentially matched with CXL extended memory, and cold data is matched with CPUDDR; after the allocation of the second-level request is completed, hot data, warm data, and cold data in the third-level request are preferentially matched with CXL extended memory and CPU DDR, wherein, if there is remaining HBM memory, it is preferentially matched with hot data in the third-level request.

[0041] Step S103 comprises two actions. The first part involves the dynamic reclamation and release of HBM quotas: The quota manager collects the actual P99 latency of first-level requests at a set frequency and compares it with preset dual thresholds. When the latency of a first-level request exceeds the first threshold, a quota reclamation process is triggered, prioritizing the second-level request with the highest HBM usage as the reclamation target. Its HBM quota is reclaimed according to a set ratio. The hardware prefetch engine asynchronously moves the KV Cache data of the second-level request from HBM to CXL extended memory. This migration process does not consume GPU computing resources. The released HBM space is then allocated to the first-level request. When the latency of a first-level request is less than the second threshold, a quota release process is triggered, reducing the HBM quota of the first-level request. The released high-speed memory space is allocated proportionally to the second and third-level requests, improving the overall system concurrency throughput. To avoid performance fluctuations caused by frequent adjustments, a minimum time interval is set for quota adjustments, and the maximum data migration amount in a single adjustment does not exceed a set ratio of the target request's HBM usage.

[0042] For example, based on a set frequency and the quota manager, when the response latency of a first-level request exceeds a first threshold, a first set proportion of HBM memory is reclaimed from the second-level requests to supply the first-level requests. This includes: collecting the actual P99 latency of each inference request for the first-level requests at a frequency of less than or equal to 100ms, calculating the latency deviation with the SLA target latency recorded in the SLA monitoring table, and if the latency deviation is greater than 10%, then 5% to 20% of HBM memory is reclaimed from the second-level requests. Among these, the second-level requests with the highest HBM memory usage are preferentially selected as the reclamation targets, and migration instructions are generated to migrate the key-value cache corresponding to the reclamation target from HBM memory to CXL extended memory. The migration instructions are written to a preset hardware prefetch engine migration queue via the MMIO protocol to perform data transfer, and the global quota table is updated after the migration is completed.

[0043] When the response latency of a Level 1 request is less than a second threshold, a second set proportion of HBM memory is released from the Level 1 request to supply the Level 2 and Level 3 requests. This includes: collecting the actual P99 latency of each Level 1 inference request at a frequency of less than or equal to 100ms, calculating the latency deviation with the SLA target latency recorded in the SLA monitoring table, and if the latency deviation is less than -20%, then 5% to 20% of HBM memory is released from the Level 1 request for allocation to the Level 2 and Level 3 requests according to a third set proportion.

[0044] In some embodiments, the method further includes: multiple GPUs forming a unified shared memory pool from multiple CXL memory expansion devices via a CXL Switch; each GPU directly accessing the key-value cache in the shared memory pool through non-blocking transactions and cache consistency primitives of the CXL.cache protocol; wherein, first-level requests preferentially use the near-end region of the shared memory pool with lower latency, while second-level and third-level requests use the far-end region with higher latency. When the shared memory pool capacity is insufficient, cold data in third-level requests and cold data in second-level requests are evicted in sequence according to a fourth predetermined ratio; wherein the evicting operation is performed based on a hardware prefetch engine.

[0045] In some embodiments, the response latency of the first-level request is divided into three states: normal, rate-limited, and emergency. The prefetch cache size and prefetch bandwidth of the first-level request are increased progressively under these three states, including steps S1031 to S1033: Step S1031: Define the latency deviation of the first-level request as a normal state if it is less than 5%, a rate-limiting state if it is greater than 5% but less than 20%, and an emergency state if it is greater than 20%.

[0046] Step S1032: Under normal conditions, during prefetch scheduling, the prefetching of Level 1, Level 2, and Level 3 requests is executed sequentially, and a double buffer is set up in HBM memory for ping-pong switching, which is used synchronously for current computation and background prefetching; under rate limiting conditions, the prefetching of Level 2 and Level 3 requests is downgraded to single buffering, and the buffer space is released to prefetch Level 1 requests; under emergency conditions, the prefetching of Level 2 and Level 3 requests is suspended.

[0047] Step S1033: Under normal conditions, the prefetch bandwidth for the first-level request, the second-level request, and the third-level request is set to 5:3:2; under rate-limited conditions, the prefetch bandwidth for the first-level request, the second-level request, and the third-level request is set to 8:2:0; under emergency conditions, all prefetch bandwidth is allocated to the first-level request.

[0048] On the other hand, the present invention also provides a multi-request heterogeneous memory QoS management system for large language model inference. The system is a large language model inference architecture formed by multiple GPUs combined with HBM, CXL extended memory and CPU DDR three-level heterogeneous memory. The CXL extended memory of each GPU utilizes the memory pooling shared access and elastic scheduling of CXL Switch. The system is used to execute pre-loaded computer programs or instructions. When the computer program / instruction is executed, the system implements the steps of the above method.

[0049] On the other hand, the present invention also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0050] On the other hand, the present invention also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the steps of the above-described method. The present invention will now be described with reference to a specific embodiment: This embodiment provides a multi-request heterogeneous memory QoS management method for large language model inference. It constructs a five-layer linkage multi-request QoS management system with request hierarchy, quota allocation, dynamic adjustment, memory pooling, and coordinated scheduling, upgrading heterogeneous memory resources from unified management and first-come-first-served to differentiated services and elastic scheduling.

[0051] Specifically, the method in this embodiment includes the following five steps and is applied to a server system that includes multiple GPUs, a CXL Switch shared memory pool, and CPU main memory: Step 1: Request Hierarchical and SLA Mapping A QoS classification module is introduced into the request scheduling layer of the inference engine to mark the SLA level and pre-allocate memory quotas for each inference request entering the system.

[0052] 1.1 QoS Level Definition: P0 (Latency Priority): Designed for low-latency scenarios such as real-time dialogues, with an SLA target of P99 latency <500ms. Hot data in the KV Cache is prioritized for permanent residency in HBM, with prefetching having the highest priority. Preferably, the proportion of P0-level KV Cache in HBM quota, CXL quota, and DDR quota is 70%, 20%, and 10%, respectively.

[0053] P1 (Balanced): Designed for general online inference scenarios, with an SLA target of P99 latency <2s, the KV Cache is evenly distributed between HBM and CXL, and the prefetch priority is medium. Preferably, the proportion of P1-level KV Cache in HBM quota, CXL quota, and DDR quota is 40%, 40%, and 20%, respectively.

[0054] P2 (Throughput Priority): Designed for offline batch inference scenarios, this approach prioritizes best-effort KV Cache, using CXL and DDR, allocating only a small quota when HBM is plentiful. Preferably, the P2-level KV Cache allocates 20% of its HBM quota, 40% of its CXL quota, and 40% of its DDR quota.

[0055] 1.2 SLA Mapping Mechanism: When a request arrives, the inference engine automatically assigns a QoS level based on the request source, such as the API endpoint, request header tags, and user level. Alternatively, the application layer can explicitly specify the QoS level via API. The QoS level for each request is bound to its KV Cache data and remains unchanged throughout the entire lifecycle of data migration and memory allocation.

[0056] 1.3 Initial Quota Pre-allocation: Based on the QoS level and current system memory usage, pre-allocate HBM / CXL / DDR quotas for each request. Pre-allocation employs a best-effort strategy: HBM quotas for P0 requests are prioritized, with remaining space allocated to P1 / P2. If HBM space is insufficient to meet the quota requirements of all P0 requests, the HBM quota for P2 requests is downgraded, and its KV Cache is preferentially allocated to CXL / DDR to free up HBM space for P0 requests.

[0057] Step 2: QoS Differentiated Memory Quota Scheduling In multi-request concurrent scenarios, heterogeneous memory quotas and prefetch priorities are allocated based on the QoS levels of each inference request: 2.1 Quota Manager: Implement a quota manager within the inference engine, maintaining the following data structures: A Global Quota Table, recording the QoS level, current HBM / CXL / DDR usage, and target quota for each request; a Tier Status Table, recording the total capacity, used capacity, and available capacity for each HBM / CXL / DDR tier; and an SLA Monitor Table, recording the actual P99 latency, SLA target latency, and latency deviation for each request.

[0058] 2.2 Quota Allocation Strategy: P0 request: Prioritize allocating HBM space, hot KV Cache data resides in HBM, and warm data resides in CXL.

[0059] P1 request: Balanced allocation between HBM and CXL, hot data resides in HBM, warm data resides in CXL, and cold data resides in DDR.

[0060] P2 Request: Prioritize CXL and DDR, and allocate a small amount of HBM quota only when HBM space is plentiful.

[0061] 2.3 Prefetch priority linkage: The quota manager synchronizes the QoS level of each request to the arbitrator of the hardware prefetch engine in real time. When scheduling based on the migration command queue, the prefetch arbitrator prioritizes the prefetch command of P0 request, followed by P1, and lastly P2. When the latency of a P0 request approaches the SLA limit, its prefetch priority is temporarily raised to urgent to preempt the prefetch bandwidth of other requests.

[0062] Step 3: Dynamic Quota Adjustment and Resource Recovery like Figure 2 As shown, the quota manager continuously monitors the SLA achievement of each request and automatically performs quota adjustments when a delay deviation is detected. 3.1 Perform monitoring and detection every 100ms to collect the actual P99 latency of each request, compare it with the SLA target latency, and calculate the latency deviation: dev = (actual P99 latency - SLA target latency) / SLA target latency.

[0063] 3.2 When the latency of a P0 request exceeds the limit, quota reclamation is triggered. The triggering condition is that the P0 request dev > 0.1, i.e., the latency exceeds the limit by 10%. The reclamation strategy is to reclaim HBM quota from P2 requests, reclaiming 5% each time.

[0064] The reclamation process includes: selecting the P2 request with the highest HBM usage as the reclamation target; generating migration instructions to migrate a portion of the target request's KV Cache from HBM to CXL extended memory; writing the migration instructions to the Hardware Prefetch Engine (HPC) migration instruction queue via MMIO, with the HPC hardware performing the actual data transfer; updating the global quota table after migration and releasing HBM space for P0 requests; and performing the migration operations through the Hardware Prefetch Engine (HPC) without blocking GPU computation for P0 requests.

[0065] 3.3 When the latency of a P0 request is significantly lower than the target, quota release is triggered. The triggering condition is that the P0 request's dev < -0.2, meaning the latency is significantly lower than the target. The release strategy is to release a portion of the HBM quota to P1 / P2 requests to improve overall throughput. The quota for P1 / P2 requests is allocated proportionally based on the remaining HBM space.

[0066] 3.4 Frequency adjustment control includes: a minimum interval of 100ms for quota adjustments to prevent frequent adjustments from causing data migration storms. The maximum migration amount for a single adjustment shall not exceed 20% of the target request HBM usage to avoid migration delays affecting normal inference. Adjustment commands are written to the HPC migration command queue in batches via MMIO to reduce the number of MMIO interactions.

[0067] Step 4: Introduce CXL memory pooling and multi-GPU shared KV Cache Leveraging the memory pooling capabilities of the CXL Switch, shared access and elastic scheduling of the KV cache are achieved in multi-GPU servers: A shared memory pool architecture is constructed, in which multiple CXL memory expansion devices are combined into a unified shared memory pool via a CXL Switch. Multiple GPUs directly access the KV Cache data in this memory pool through the CXL.cache protocol, without needing to go through CPU DDR. The address space of the shared memory pool is managed uniformly by software, and address mapping is configured through the MMIO interface.

[0068] Cross-GPU KV Cache reuse is implemented. When inference requests on multiple GPUs need to access the same KV Cache data, such as the shared context in a multi-turn dialogue, only one copy of the data in the shared memory pool is required. Each GPU reads concurrently through the cache coherency mechanism of the CXL.cache protocol, reducing the overhead of cross-GPU KV Cache copying; the expected reduction in copying overhead is over 60%.

[0069] Elastic capacity allocation is implemented, and the capacity of the shared memory pool can be dynamically allocated according to the actual needs of each GPU. When the number of P0 requests on a GPU increases, resources are automatically allocated from the memory pool space released by other GPUs. Capacity allocation is tied to QoS levels: P0 requests prioritize using the near-end region of the memory pool with the lowest latency, while P1 / P2 requests use the far-end region.

[0070] The CXL.cache protocol utilizes non-blocking transactions and cache consistency primitives, such as CrdUp and WrInv, to achieve efficient data transfer between the GPU and the shared memory pool. CXL.cache requests issued by the prefetch engine are non-blocking operations, returning immediately after being sent, with data transfer completed asynchronously in the background. When multiple GPUs concurrently read data from the same KV cache, the CXL.cache protocol guarantees cache consistency.

[0071] When the memory pool capacity is insufficient, an eviction policy is executed, prioritizing the eviction of cold data requested by P2 (i.e., the bottom 20% of data in terms of popularity). Next, cold data requested by P1 is evictioned. Data requested by P0 is not evictioned unless the request completes or times out. The eviction operation is performed via HPC hardware and does not block normal inference.

[0072] Step 5: Adaptive scheduling linking QoS and prefetch priority To achieve coordinated optimization of QoS policies and prefetching systems, a linked adaptive scheduling mechanism is introduced, such as... Figure 3 As shown, the details are as follows: 5.1 A Joint Scheduler is introduced, implemented within the inference engine, to coordinate the work of the quota manager and the prefetch engine. Inputs include: SLA achievement rate, HBM utilization, prefetch hit rate, and memory pool utilization for each request. Outputs include: quota adjustment instructions, prefetch priority adjustment instructions, and pipeline depth adjustment instructions.

[0073] 5.2 Execute a global SLA monitoring closed loop. The coordinated scheduler collects the global status every T_ms period, with a default interval of 100ms. It comprehensively evaluates the SLA achievement of each request and the system resource utilization, and outputs adjustment instructions based on the evaluation results, forming a closed loop that includes monitoring, evaluation, adjustment, and execution.

[0074] 5.3 is the design state machine for the linkage dispatcher, with states including: Normal, Throttled, and Emergency.

[0075] In the NORMAL state, all P0 request SLAs satisfy dev < 0, prefetch bandwidth is allocated according to the default priority, and the double-buffered pipeline operates normally. In the THROTTLED state, some P0 request SLAs are close to their limits (dev > 0.05), the P0 prefetch priority is increased to urgent, and P2 requests are downgraded to single-buffered. In the EMERGENCY state, P0 request SLAs are severely exceeded (dev > 0.2), P2 request prefetching is suspended, all prefetch bandwidth is allocated to P0, and HBM data from P2 requests is migrated in batches to CXL. Rollback conditions: When a P0 request dev < -0.1, rollback from EMERGENCY to THROTTLED; when dev < -0.2, rollback to NORMAL.

[0076] The upgrade condition is based on any P0 request and is triggered based on the worst-case scenario. The rollback condition is based on all P0 requests and rollback is only performed after global safety is ensured. This asymmetric design prevents premature rollback from causing the SLA to deteriorate again.

[0077] 5.4 Dynamic Prefetch Bandwidth Allocation: In NORMAL state, prefetch bandwidth is allocated in a ratio of P0:P1:P2 = 5:3:2; in THROTTLED state, the ratio is adjusted to P0:P1:P2 = 8:2:0; in EMERGENCY state, all prefetch bandwidth is allocated to P0. Bandwidth allocation is achieved by adjusting the priority weights of the prefetch arbiter, which is updated in real time through the MMIO configuration register.

[0078] 5.5 CXL Memory Pool Capacity and QoS Linkage: The capacity allocation strategy of the shared memory pool is bound to the QoS level: P0 requests prioritize the use of the near-end area with the lowest latency in the memory pool; when the overall capacity of the memory pool is insufficient, cold data of P2 requests are prioritized to be evicted to make room for P0 / P1 requests; the capacity adjustment command is written to the address routing configuration register of the CXL Switch via MMIO.

[0079] Corresponding to the above method, the present invention also provides an apparatus / system including a computer device, the computer device including a processor and a memory, the memory storing computer instructions, the processor executing the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the apparatus / system performs the steps of the method as described above.

[0080] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0081] In summary, the multi-request heterogeneous memory QoS management method and system for large language model inference described in this invention, through the synergistic effect of multiple technical mechanisms such as three-level response speed commitment levels and differentiated memory quota matching, dynamic reclamation and release of HBM quotas, CXLSwitch memory pooling and sharing, and linkage between prefetch priority and runtime status, constructs multi-layer protection for high-priority inference requests from three dimensions: initial resource allocation, dynamic runtime expansion, and prefetch bandwidth guarantee. This can improve the SLA achievement rate of high-priority requests to over 99%, effectively solving the problem of low-priority requests crowding out high-speed memory and dragging down the latency of high-priority requests under unified scheduling mode. On the other hand, through a two-dimensional hierarchical storage strategy of SLA level and heat and dynamic quota scheduling, it achieves precise reuse of three-level heterogeneous memory of HBM, CXLS extended memory, and CPU DDR, improving HBM utilization efficiency to 80%~90%, while relying on the shared memory pool to eliminate key-value pairs between multiple GPUs. Redundant cache copies reduce cross-GPU data copying overhead by more than 60%, and the dynamically adjustable prefetch bandwidth arbitration mechanism ensures the prefetch efficiency of high-priority requests. Ultimately, while fully meeting differentiated service quality requirements, the overall concurrent inference throughput of the system can be increased by 40% to 60%, significantly improving the utilization efficiency and scheduling flexibility of heterogeneous memory resources.

[0082] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0083] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0084] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0085] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-request heterogeneous memory QoS management method for large language model inference, characterized in that, The method is applicable to a multi-GPU, multi-level heterogeneous memory architecture for large language model inference using HBM, CXL extended memory, and CPU DDR. The CXL extended memory of each GPU utilizes memory pooling shared access and elastic scheduling via the CXL Switch. The method is executed based on the inference engine's linked scheduler and includes the following steps: Based on the QoS classification module pre-embedded in the request scheduling layer, the inference requests are marked with response speed commitment levels according to preset mapping rules. The response speed commitment levels are divided into first-level requests that meet latency priority, second-level requests that meet balance priority, and third-level requests that meet throughput priority, according to priority from high to low. Initial memory quotas are allocated to first-level requests, second-level requests, and third-level requests in the order of HBM, CXL extended memory, and CPU DDR. The system maintains a global quota table based on a preset quota manager to record the response speed commitment level and memory quota of each inference request; maintains a memory status table to record the total capacity, current usage, and available amount of the three-level heterogeneous memory; maintains an SLA monitoring table to record the actual P99 latency, SLA target latency, and latency deviation of each inference request; dynamically allocates three-level heterogeneous memory to each inference request according to a preset strategy based on the global quota table, the memory status table, and the SLA monitoring table, and provides the response speed commitment level of each inference request to the arbitrator of the hardware prefetch engine, so as to execute the prefetch of the first-level request, the second-level request, and the third-level request in sequence during prefetch scheduling, and adjusts inference requests with latency close to the SLA limit to higher priority; According to a set frequency, based on the quota manager, when the response delay of the first-level request is greater than a first threshold, a first set proportion of HBM memory is reclaimed from the second-level request to supply the first-level request; when the response delay of the first-level request is less than a second threshold, a second set proportion of HBM memory is released from the first-level request to supply the second-level and third-level requests; and, based on the response delay of the first-level request, three states are divided: normal, rate-limited, and emergency, and the prefetch cache amount and prefetch bandwidth of the first-level request are increased step by step in these three states.

2. The multi-request heterogeneous memory QoS management method for large language model inference according to claim 1, characterized in that, The response speed commitment level of inference requests is marked using preset mapping rules, including: The response speed commitment level is automatically assigned based on the source of the inference request, or the response speed commitment level of the inference request is specified based on the application layer; wherein, in real-time dialogue scenarios, inference requests with a P99 latency requirement of less than or equal to 500ms are marked as first-level requests; in online inference scenarios, inference requests with a P99 latency requirement of greater than 500ms and less than or equal to 2s are marked as second-level requests; and in offline batch inference scenarios, inference requests with a P99 latency requirement of greater than 2s and less than 10s are marked as third-level requests. The initial memory allocation includes: prioritizing the allocation of HBM memory for the first-level requests; after satisfying the memory requirements of the first-level requests, allocating the remaining memory to the second-level requests and the third-level requests; if the remaining HBM memory capacity does not meet the memory requirements of all first-level requests, downgrading the HBM memory for the second-level requests to CXL extended memory and CPU DDR, and then configuring the first-level requests in the order of HBM, CXL extended memory, and CPU DDR.

3. The multi-request heterogeneous memory QoS management method for large language model inference according to claim 2, characterized in that, Based on the global quota table, the memory status table, and the SLA monitoring table, three levels of heterogeneous memory are dynamically allocated to each inference request according to a preset strategy, including: Based on historical access frequency, the key-value cache accessed by the first-level request, the second-level request, and the third-level request are divided into hot data, warm data, and cold data according to the access frequency from high to low according to preset standards. Based on the global quota table and the memory status table, hot data in the first-level request is preferentially matched with HBM memory, while warm and cold data in the first-level request are matched with CXL extended memory. After the allocation of the first-level request, hot data in the second-level request is preferentially matched with HBM memory, warm data is preferentially matched with CXL extended memory, and cold data is matched with CPU DDR. After the allocation of the second-level request, hot, warm, and cold data in the third-level request are preferentially matched with CXL extended memory and CPU DDR, wherein, if there is remaining HBM memory, it is preferentially matched with hot data in the third-level request.

4. The multi-request heterogeneous memory QoS management method for large language model inference according to claim 3, characterized in that, According to a set frequency, based on the quota manager, when the response delay of the first-level request exceeds a first threshold, a first set proportion of HBM memory is reclaimed from the second-level requests to supply the first-level requests, including: The actual P99 latency of the first-level request for each inference request is collected at a frequency of less than or equal to 100ms. The latency deviation is calculated with the SLA target latency recorded in the SLA monitoring table. If the latency deviation is greater than 10%, 5% to 20% of HBM memory is reclaimed from the second-level request. Among them, the second-level request with the highest HBM memory usage is preferentially selected as the reclamation target. A migration instruction is generated to migrate the key-value cache corresponding to the reclamation target from HBM memory to CXL extended memory. The migration instruction is written to the preset hardware prefetch engine migration queue through the MMIO protocol to perform data transfer. After the migration is completed, the global quota table is updated. When the response latency of the first-level request is less than the second threshold, a second predetermined proportion of HBM memory is released from the first-level request to supply the second-level and third-level requests, including: The actual P99 latency of the first-level request for each inference request is collected at a frequency of less than or equal to 100ms. The latency deviation is calculated with the SLA target latency recorded in the SLA monitoring table. If the latency deviation is less than -20%, 5% to 20% of HBM memory is released from the first-level request for allocation to the second-level request and the third-level request according to a third set ratio.

5. The multi-request heterogeneous memory QoS management method for large language model inference according to claim 4, characterized in that, The method further includes: multiple GPUs forming a unified shared memory pool by connecting multiple CXL memory expansion devices through a CXL Switch; each GPU directly accessing the key-value cache in the shared memory pool through non-blocking transactions and cache consistency primitives of the CXL.cache protocol; wherein, the first-level request preferentially uses the near-end region with lower latency in the shared memory pool, while the second-level request and the third-level request use the far-end region with higher latency. When the shared memory pool capacity is insufficient, cold data in the third-level requests and cold data in the second-level requests are eliminated in sequence according to the fourth preset ratio; wherein, the elimination operation is performed based on the hardware prefetch engine.

6. The multi-request heterogeneous memory QoS management method for large language model inference according to claim 5, characterized in that, Based on the response latency of the first-level request, three states are categorized: normal, rate-limited, and emergency. The prefetch cache size and prefetch bandwidth for the first-level request are increased progressively across these three states, including: The latency deviation of the first-level request is defined as less than 5% as a normal state, greater than 5% but less than 20% as a rate-limiting state, and greater than 20% as an emergency state. In the normal state, during prefetching scheduling, the prefetching of the first-level request, the second-level request, and the third-level request are executed sequentially, and a dual buffer is set up in HBM memory for ping-pong switching, which is used synchronously for current computing and background prefetching; in the rate-limiting state, the prefetching of the second-level request and the third-level request is downgraded to a single buffer, and the buffer space is released to prefetch the first-level request; in the emergency state, the prefetching of the second-level request and the third-level request is suspended.

7. The multi-request heterogeneous memory QoS management method for large language model inference according to claim 6, characterized in that, Based on the response latency of the first-level request, three states are categorized: normal, rate-limited, and emergency. The prefetch cache size and prefetch bandwidth for the first-level request are increased progressively across these three states, including: In the normal state, the prefetch bandwidth of the first-level request, the second-level request, and the third-level request is set to 5:3:2; in the rate-limited state, the prefetch bandwidth of the first-level request, the second-level request, and the third-level request is set to 8:2:0; in the emergency state, all prefetch bandwidth is allocated to the first-level request.

8. A multi-request heterogeneous memory QoS management system for large language model inference, characterized in that, The system comprises a large language model inference architecture consisting of multiple GPUs, HBM, CXL extended memory, and CPU DDR three-level heterogeneous memory. The CXL extended memory of each GPU utilizes the memory pooling shared access and elastic scheduling of the CXL Switch. The system is used to execute pre-loaded computer programs or instructions. When the computer program / instruction is executed, the system implements the steps of the method as described in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 7.

Images