A kv cache anti-jitter asynchronous offloading scheduling method and system for large language model reasoning

Abstract

The application belongs to the technical field of computer memory management and heterogeneous computing scheduling, and discloses a KV cache anti-jitter asynchronous offloading scheduling method and system for large language model inference, which performs semantic isolation and real attention score on the key-value cache on the basis of the block type KV Cache management structure, selects recoverable redundant physical blocks based on the score results, acquires a basic pressure signal, establishes an I / O jitter quantization model based on a sliding time window, and calculates the recent average bus jitter rate of the system in real time; when the safe threshold is broken and the crisis state is entered, the candidate pool with decreasing priority is extracted to perform asynchronous offloading, and the maximum active concurrency limit is reduced; when the danger is removed, the number of concurrent is smoothly increased. Through the logical mapping update and physical block logout mechanism, the application reduces the underlying data handling in the redundant block recycling process, realizes block-level memory recycling, and guarantees the stability of the throughput; through high-precision hot and cold isolation, the offloading loss is minimized, and the OOM robustness is improved.

Description

Technical Field

[0001] This invention belongs to the field of computer memory management and heterogeneous computing scheduling technology, specifically relating to a KV cache anti-jitter asynchronous offloading scheduling method and system for large language model inference. Background Technology

[0002] In the autoregressive inference process of generative large language models, the model needs to generate subsequent content word by word based on the existing context. To avoid repeated attention calculations on the historical sequence during each forward propagation, the inference engine typically allocates and maintains a key-value cache (KV Cache) for each request to store intermediate tensor representations of historical tokens. However, as application scenarios continue to evolve towards long texts (such as document analysis and ultra-long-turn dialogues) and high concurrency, the memory usage of KV Cache grows linearly or even superlinearly. On resource-constrained computing nodes, the explosive growth of KV Cache can lead to the rapid exhaustion of available memory (OOM), forming a core memory wall that limits the throughput and concurrency ceiling of large model services.

[0003] To alleviate the aforementioned memory crisis, the industry has proposed a series of memory optimization solutions, but all of them have revealed significant technical flaws in actual high-pressure production environments. Existing block-based or paging-based KV cache management methods can alleviate the pressure of contiguous memory allocation, but their focus is usually on cache address management or memory allocation, and they do not fully solve the I / O jitter problem caused by frequent unloading and prefetching under critical memory conditions. The static compression strategy of discarding tokens at a fixed ratio cannot detect the dynamic load of the system, which not only destroys the physical continuity of memory and generates a large number of fragments, but may also lose critical context information in long sequences; while blindly relying on system virtual memory swapping when memory is insufficient, due to the lack of anti-jitter mechanism and memory life cycle awareness, frequent invalid data movement will cause bus congestion, leading to a sharp deterioration in tail latency, and ultimately losing online service capabilities.

[0004] As disclosed in relevant prior art, patent application CN121412032A presents a method, program product, computing node, and storage medium for distributed parallel training of large models. While addressing the memory overhead and fault tolerance issues in distributed parallel training of large models, this approach is only applicable to the model training phase and completely fails to solve the KV cache memory crisis in scenarios with extremely long text concurrency during the inference phase. Specifically, this solution offloads model gradients and optimizer states, rather than the KV (key-value) cache that consumes a large amount of memory during inference, and lacks semantic awareness of long text features. Furthermore, in autoregressive inference in a single-card-constrained environment, the coarse-grained memory transfer of this solution leads to severe computational flow blockage, failing to achieve just-in-time memory reloading ("pull back before computation, kick out after computation"). If forcibly applied to the decoding phase, the lack of a physical chunking mechanism will cause severe I / O latency and memory fragmentation, making it impossible to handle high-concurrency requests for long texts.

[0005] Patent application CN121542395A discloses an adaptive key-value (KV) cache compression method and system for large language models. It proposes a KV cache logical compression method for large language model inference. This scheme is a typical example of pure logical-layer semantic compression, and its most fatal flaw is that it completely destroys the physical continuity of GPU memory. This scheme achieves compression by randomly discarding individual tokens. When combined with the underlying PagedAttention in a single-card environment, it generates severe "internal fragmentation of GPU memory" within a preset physical block, further exacerbating the GPU memory shortage problem. Furthermore, this scheme lacks a fallback mechanism for heterogeneous storage. When the compressed KV cache still reaches the physical limits of GPU memory under high-concurrency scenarios, the system will directly crash due to OutOfMemoryError (OOM). Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a KV cache anti-jitter asynchronous offloading scheduling method and system for large language model inference. This method prevents OOM (Out of Memory) under extreme concurrency when running large language models in a limited computing environment, and solves the problems of bus congestion storms, cold and hot data isolation failure, and false eviction.

[0007] The present invention discloses a KV cache anti-jitter asynchronous offloading scheduling method and a system for large-scale language model inference, comprising the following steps: Step 1: Receive large model inference requests, schedule requests in a queue according to the sequence lifecycle state machine, and maintain the block state information corresponding to the request sequence based on the preset block-based KV Cache storage structure. Step 2: In the autoregressive decoding stage of the sequence, semantic isolation and real attention scoring are performed on the KV Cache partitioned in Step 1. Based on the scoring results, triple pooling block alignment is performed on the logical blocks of non-critical features. Through logical mapping update and physical block release, the underlying data movement in the redundant block reclamation process is reduced, and the risk of fragmentation is reduced. Step 3: During the recycling process in the autoregressive decoding stage of Step 2, the dual water level lines are monitored in real time, and the heterogeneous memory exhaustion signal of the external underlying physical device is collected simultaneously. The basic pressure signal reflecting the memory capacity reserve is output comprehensively. Step 4: Establish an I / O jitter quantization model based on a sliding time window to calculate the recent average bus jitter rate of the system in real time; if the basic pressure signal does not fall below the safety threshold, the current system memory capacity is determined to be safe, and the process returns to step 2 to maintain the normalized autoregressive decoding and memory block reclamation cycle; if the basic pressure signal falls below the safety threshold, the system internal scheduling preconditions are combined to trigger and switch to a global takeover crisis state, and step 5 is executed next. Step 5: In the crisis state triggered in step 4, based on the real-time scheduling status and idle cooling period of the sequences in the same active decoding set, multiple offloading candidate pools with decreasing priority are dynamically extracted; within the same candidate pool, multi-dimensional composite sorting is performed by combining the temporal locality characteristics of the sequence and the spatial release benefits, and the physical blocks of the selected sequence are strictly and asynchronously offloaded to the host memory in the order of the candidate pool. Step 6: During the crisis, if the memory capacity gap after unloading in Step 5 is not filled, a non-linear backpressure reduction is performed on the maximum active concurrency limit allowed by the system; at the same time, the system maintains strict hysteresis recovery latch; when the basic pressure signal indicates that both the internal memory and external devices have recovered to safety, and the recent average bus jitter rate has been continuously lower than the tolerance threshold within a consecutive preset time window, the system determines that the danger has been eliminated and exits the crisis state, and then performs a linear smooth recovery on the maximum active concurrency limit.

[0008] Furthermore, step 1 specifically includes: 1) Total video memory capacity of the target computing node for dynamic detection and the static weight usage of large models After deducting the preset safety reserve space Then, based on the set video memory allocation ratio coefficient... Calculate the total byte budget allocated to the key-value cache; 2) Divide the budget into fixed-size physical blocks, assuming the total number of layers in the large model is... The number of tokens contained in a single physical block is The number of attention heads is The dimension of a single-head hidden layer is The number of bytes occupied by the data type is The maximum number of physical blocks the system can accommodate Determined as: (1) For any input request sequence, the generated KV feature vector is segmented at the token granularity and distributed into fixed-size physical blocks; the global absolute position index in the given sequence Its corresponding logical block index The calculation formula is: (2) The system maintains a dynamic mapping table independently for each inference request, recording the mapping relationship between the logical block index and the actual physical block address in the memory pool, thereby achieving physically non-contiguous but logically continuous addressing.

[0009] Further, in step 1, block pre-filling calculation is performed on the input request sequence, specifically as follows: When the received length is When dealing with a long input sequence, it is divided into multiple consecutive sub-sequence blocks according to a preset block capacity C; Perform gradient-free forward propagation on each subsequence block in turn; When calculating the current sub-block, the accumulated KV Cache state generated by the previous sub-block is passed in as the historical context, and the updated key-value state is extracted after calculation and passed to the next sub-block.

[0010] Furthermore, in step 2, a true attention score is performed on the KV Cache, specifically as follows: For a total length of Based on the causal features generated by the autoregressive model, the complete input sequence is strictly divided into three logically independent isolation regions along the time step dimension; among them, the Sink region, which is the truncated sequence with a length of [missing information], is the initial grammar protection region. Continuous tokens; the tail local observation region, i.e., the Obs region, is the truncated sequence with a length of [missing value]. A continuous set of tokens; the middle compressible area, or Mid area, is located between the Sink area and the Obs area, with a length of [missing information]. ; (5) Perform scaling dot product attention operation, calculate For compressible bond tensors Global attention weight distribution matrix A: (6) in, The hidden dimension of the model's attention head; The query tensor is the actual projection of the Obs region at the end of the sequence into the current forward computation; the generated weight matrix A has the following shape. , For batch size, This represents the number of long positions.

[0011] Furthermore, in step 2, based on the scoring results, triple-pooling block alignment is performed on the logical blocks of non-critical features. Through logical mapping updates and physical block releases, the underlying data movement during redundant block reclamation is reduced, and the risk of fragmentation is mitigated. Specifically: The global attention weight distribution matrix A is subjected to mean pooling along the window dimension and max pooling along the head dimension in sequence to eliminate score drift caused by different observation window sizes, thus obtaining a one-dimensional word-level score. ,Will Mapping back to physical block dimension form Perform a summation operation within the block to obtain the initial block score. Where N is the number of physical blocks; We introduce a one-dimensional sliding max pooling method with kernel size k, stride s, and padding p to generate the smoothed final block importance: (7) Under the constraint of retention ratio r, the target retention index is obtained based on the final block importance score. And by updating the logical mapping table, the reserved blocks are located between the Sink area and the Obs area in the logical address space; unselected physical blocks are directly released and reset to the free state; For unselected underlying physical blocks, their corresponding physical memory pages are directly handed over to the underlying lifecycle state machine to be reset to an idle state.

[0012] Furthermore, in step 3, the double waterline is calculated as follows: (3) (4) in, and The preset capacity monitoring ratio coefficient strictly meets the requirements. Interval distribution; The threshold for the low water level line; The high water level threshold; This represents the maximum number of physical blocks that the physical storage pool can hold.

[0013] Furthermore, in step 3, the dual water level lines are monitored in real time simultaneously, and the heterogeneous memory depletion signal of the external underlying physical devices is collected simultaneously. The basic pressure signal reflecting the remaining memory capacity is output comprehensively, specifically as follows: Based on monitoring of the dual water level lines, the number of absolutely free physical blocks in the system memory mapping table is calculated. With dynamic low water level Judgment; when At that time, internal logic memory pressure Set to the boolean value True; For the total available physical video memory Preset safety warning threshold If and only if there is currently an active decoding computation task and At that time, the actual memory consumption state of the underlying hardware device Set to True; In instantaneous determination mode, the system executes Boolean logic to generate a fused basic pressure signal: (8).

[0014] Furthermore, in step 4, a crisis state of global takeover is triggered and initiated, specifically as follows: Let the total number of successful physical block unloadings and the total number of successful prefetches in the system at time step t be respectively... and Define the instantaneous page increment as: (9) The system obtains the total number of currently active request sequences participating in the computation. ;when At that time, calculate the normalized instantaneous page rate. ; This represents the total number of request sequences participating in the computation at the current time step t. Create a length of For a circular buffer sliding window, at any time step T, the macroscopic jitter resistance quantification index of the system is defined as the mathematical expectation within this window: (10) Input-based and The system determines whether the available video memory pool needs to enter a high-pressure crisis state of global takeover. Uninstallation begins; the system currently meets any of the following internal scheduling preconditions: uninstallation has already been attempted. There exists a candidate sequence for expulsion, i.e. Or the system is already in a managed takeover mode. Its triggering condition is defined as: (11).

[0015] Furthermore, in step 6, the system determines that the danger has been eliminated and exits the crisis state. Define the comprehensive recovery signal: (12) This means that three conditions must be met simultaneously: sufficient logic block level, secure external underlying physical devices, and a restored I / O bus; among which... This is the preset low-frequency jitter tolerance threshold; Only when The signal is in continuous The state machine truly relinquishes control and exits the intervention state only when the condition remains true for an extended period of time. (13) (14).

[0016] This invention also provides a KV cache anti-jitter asynchronous offloading scheduling method and a system for large model inference scheduling, for implementing the above method. The system includes: The physical paging memory capacity allocation and mapping module is responsible for the computational resource planning during system initialization, calculating the total byte budget of KVCache and dividing it into fixed-size physical blocks, establishing the mapping relationship between logical blocks and physical blocks, and realizing physically non-contiguous but logically contiguous addressing. The physical block lifecycle and dual water level monitoring module maintains the lifecycle residence state for each inference sequence, defines six mutually exclusive residence states, and ensures the integrity of forward computation and the atomicity of asynchronous transfer. The anti-jitter dual water level monitoring and hysteresis alarm module calculates the high water level and low water level based on the maximum total number of physical blocks that the physical storage pool can hold; it polls the number of idle physical blocks in real time and sends an unloading trigger signal or releases the pool pressure warning based on their relative position to the dual water level. The long sequence block pre-filling module uses block computing technology to process ultra-long input sequences, transforming global attention calculation into serial calculation and limiting the instantaneous peak memory usage during the pre-filling stage. The attention scoring module performs semantic isolation on the KV Cache, calculates the attention weight distribution matrix based on the real query tensor and key tensor, and quantifies the importance of long sequence historical features; The triple pooling block alignment and reclamation module takes the attention scoring results and aligns the feature scores to the physical block memory boundary through triple pooling operations, thereby achieving zero-fragmentation reclamation of redundant physical blocks. The dual memory pressure sensing module simultaneously detects the logical memory view inside the framework and the physical view of the underlying device, and outputs a high-confidence basic pressure signal to solve the false negative problem of "logical idle but physical exhausted". The anti-jitter window and state machine module calculates the average bus jitter rate based on a sliding time window, and drives the state machine to trigger or exit a crisis state to avoid bus storms and memory pressure oscillations. The three-layer candidate dynamic unloading scheduling and cold / hot isolation expulsion module constructs a three-layer priority candidate pool in crisis situations, executes composite sorting logic, and unloads physical blocks in sequence asynchronously according to order to avoid mistakenly killing high-frequency active sequences. The dynamic concurrency backpressure and load adaptive module performs non-linear reduction of the concurrency limit in a crisis state and linear smooth recovery after the crisis ends, ensuring the minimum service capacity of the system and avoiding a second rebound of memory pressure.

[0017] The beneficial effects of this invention are as follows: 1) This invention reduces low-level data movement during redundant block reclamation through logical mapping updates and physical block deregistration mechanisms, and achieves block-level memory reclamation, greatly expanding the logical capacity of limited memory. Existing memory defragmentation technologies typically require expensive in-memory or cross-bus data copying. This invention introduces an attention-based semantic isolation and triple pooling alignment mechanism, which, after identifying redundant feature blocks, performs virtual merging and direct deregistration only at the logical mapping layer. This design achieves fragmentation-free memory reclamation while reducing low-level data movement during redundant block reclamation. 2) This invention solves the PCIe bus storm problem near the critical point of video memory, ensuring the stability of the maximum throughput. Existing single waterline scheduling is prone to falling into a blind "swap-in-swap-out" dead loop when video memory is fully loaded. This invention innovatively decouples the triggering and recovery conditions of the crisis: it uses only absolute capacity pressure as the sole trigger for the crisis, and introduces the recent average bus jitter rate based on the sliding time window as a strict hysteresis recovery latch to solve the problem that the video memory seems to be idle in the asynchronous inference of large models, but the bus is already filled with data being moved back and forth, thus preventing a sudden drop in system throughput; 3) High-precision hot and cold data isolation and cascading eviction are achieved, minimizing the performance loss of a single unloading. Addressing the pain point of existing systems blindly eviction of active requests leading to high-frequency cache misses, this invention abandons static queue partitioning and instead dynamically slices the same active decoding set into an unloading candidate pool with decreasing priority. 4) A dynamic counter-pressure and smooth recovery defense line under extreme high pressure is constructed, improving the robustness against OOM. Faced with concurrent surges far exceeding the physical carrying capacity limit, this invention breaks through the passive defense strategy of relying solely on memory movement and introduces a system-level active counter-pressure mechanism. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the system structure; Figure 2 This is a flowchart of triple pooling block-level compression; Figure 3 It is an anti-jitter state machine and crisis triggering flowchart; Figure 4 This is a three-level cascading unloading flowchart; Figure 5 This is a flowchart of dynamic back pressure and smooth recovery control; Figure 6 It is a flowchart of the method. Detailed Implementation

[0019] To make the content of this invention easier to understand, the invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0020] like Figure 1 As shown, the KV cache anti-jitter asynchronous offloading scheduling method and system for large language model inference described in this invention mainly involves the collaboration of several modules, including a physical paging memory capacity allocation and mapping module, a physical block lifecycle and dual watermark monitoring module, an anti-jitter dual watermark monitoring and hysteresis alarm module, a long sequence block pre-filling module, an attention scoring module, a triple pooling block alignment and reclamation module, a dual memory pressure sensing module, an anti-jitter window and state machine module, a three-layer candidate dynamic offloading scheduling and cold / hot isolation eviction module, and a dynamic concurrency backpressure and load adaptation module. The specific contents of each module are as follows:

[0021] (1) Physical paging video memory capacity allocation and mapping module;

[0022] This module is primarily responsible for computational resource planning during system initialization, as well as the virtual paging mechanism that breaks the contiguous video memory allocation limitation, providing the underlying physical storage structure for inference of large language models. During the initialization phase, the system dynamically detects the total video memory capacity of the target computing nodes. and the static weight usage of the model After deducting the preset safety reserve space... Then, based on the set video memory allocation ratio coefficient... Calculate the total byte budget allocated to the KV Cache.

[0023] The system divides the budget into fixed-size physical blocks. Assume the total number of layers in the model is... The number of tokens contained in a single physical block is The number of attention heads is The dimension of a single-head hidden layer is The number of bytes occupied by the data type is The maximum number of physical blocks the system can accommodate. Determined as: (1) For a long input text sequence, the generated KV feature vector is segmented at the token level and distributed into the aforementioned discrete physical blocks. The global absolute position index in the given sequence... Its corresponding logical block index The calculation formula is: (2) The system maintains an independent dynamic mapping table (Block Table) for each inference request, recording the mapping relationship between logical block indexes and actual physical block addresses in the memory pool, thereby achieving physically non-contiguous but logically continuous addressing.

[0024] (2) Physical block lifecycle and dual water level monitoring module;

[0025] This module maintains the lifetime residency state for each sequence participating in inference. Although the KV cache of a sequence is split and distributed across various network layers of the model in the underlying memory mapping, to ensure the integrity of forward attention computation and the atomicity of asynchronous transfers, the system defines six strictly mutually exclusive discrete residency states in macro-scheduling: ON_GPU (Memory Resident): This sequence has all KV Cache physical blocks in all network layers residing in the GPU memory of the computing device, and can directly and completely participate in the forward computation of the model; ON_CPU (Main Memory Resident): This sequence has been completely asynchronously unloaded to CPU main memory in all network layer KV Caches, and all physical blocks originally occupied in video memory have been released and returned to the global free pool. PINNED (Page Locked): The complete data of this sequence is currently in non-pageable main memory, preparing for efficient DMA transfer; OFFLOAD_INFLIGHT (Unloading in progress): The system has initiated an asynchronous bus copy event for this sequence from video memory to main memory. During this period, all underlying physical blocks associated with this sequence are strictly locked until the full layer data transfer is completed. PREFETCH_INFLIGHT: The system has initiated an asynchronous readback event from main memory to video memory for this sequence, waiting for its full layer of data to be reallocated and ready in video memory; MIXED (Mixed Resident Transitional State): Due to partial unloading of the sequence, some physical blocks are on the GPU and some are on the CPU.

[0026] (3) Anti-vibration dual water level monitoring and hysteresis alarm module;

[0027] During system initialization or when the total video memory budget changes, this module is based on the maximum total number of physical blocks that the physical memory pool can hold. The system dynamically calculates two core safety thresholds: a low watermark and a high watermark. To ensure the robustness of system performance under extremely limited memory conditions, the watermark calculation strictly follows mathematical logic that includes lower limit protection. (3) (4) in, and The system's preset capacity monitoring ratio strictly meets the requirements. Interval distribution.

[0028] At each time step of the system's forward inference, this module polls the underlying storage pool in real time to count the total number of physical blocks that are currently in an absolutely idle state. The system is based on The state machine logic is executed based on the relative position of the two water level lines.

[0029] When the number of available physical blocks is detected to have fallen below the low water level defense line, i.e. Upon this, this module immediately changes the system's global scheduling state, determines that the available video memory pool is under high pressure, and immediately sends an unloading trigger signal to the external dynamic unloading scheduling module; after the unloading is triggered, as some underlying physical blocks are asynchronously reclaimed, It will begin to recover. When the system is in... When the system is within the buffer zone, it maintains a high-pressure status indicator, allowing unloading operations to continue without changing the alarm status. The system will only allow unloading operations to continue if and only if a complete sequence of unloading or inference requests ends, effectively exceeding the high-water mark in the global number of free physical blocks. Only then will this module officially lift the pool pressure warning and send a suspension signal to the scheduling engine, actively pausing new unloading behavior.

[0030] (4) Long sequence block pre-filling module;

[0031] This module employs block-based computation technology to process ultra-long input contexts, aiming to transform global attention computation into serial computation, thereby limiting the instantaneous peak memory usage during the pre-filling stage and providing basic data preparation and assurance for subsequent cache compression and asynchronous scheduling.

[0032] When the received length is When dealing with long input sequences, the system divides them into multiple consecutive sub-sequence blocks according to a preset block size C. The system then performs gradient-free forward propagation on each sub-sequence block sequentially. When computing the current sub-block, the system passes the accumulated KV Cache state generated by the previous sub-block as the historical context, and extracts the updated key-value state after computation to pass to the next sub-block. This conventional partitioning mechanism reduces the peak memory complexity of a single pre-filling operation from... Successfully reduced to a manageable level scope.

[0033] (5) Attention scoring module;

[0034] This module, configured after the pre-filling stage, aims to address the redundancy of historical context in long text reasoning and accurately quantify the importance of each historical feature in a long sequence, providing high-confidence data support for subsequent compression scheduling. The specific implementation steps are as follows: For a total length of Given the complete input sequence, the system first strictly divides the complete KV Cache into three logically independent isolation regions along the time step dimension based on the causal features generated by the autoregressive model. The Sink region is the region that extracts the very beginning of the sequence with a length of [missing information]. Continuous tokens; the tail local observation region (Obs region) is the truncated sequence with a length of [missing value]. A continuous set of tokens; the middle compressible area (Mid area) is located between the Sink area and the Obs area, with a length of [missing information]. This region typically contains a large amount of redundant background information and is marked as the target area for dynamic evaluation and compression. Whether it's the Sink region, Obs region, or Mid region, their size is... Multiples of integers. The specific division method is as follows: (5)

[0035] To accurately assess the retention value of massive historical features within the Mid region, this module abandons the traditional estimation method of using approximate key vectors instead of query vectors. Instead, it strictly adopts the query tensor that is actually projected from the end of the sequence's Obs region in the current forward computation. As an observation benchmark, the system performs scaled dot product attention operations to calculate... For compressible bond tensors Global attention weight distribution matrix A: (6) in, Let A be the hidden dimension of the model's attention head. The generated weight matrix A has the following shape. ( For batch size, (This represents the number of heads). This matrix mathematically maps precisely the distribution of the attention dependency of the latest local context on historical long text segments, serving as the core metric for subsequent feature selection.

[0036] (6) Triple pooling block alignment and recycling module;

[0037] This module builds upon the attention scoring matrix from Module 5, such as... Figure 2 The diagram shown illustrates the sub-flowchart of the triple-pooling block-level compression method described in this invention. Its core technological breakthrough lies in completely abandoning traditional token-level discrete pruning. Instead, it proposes a multi-dimensional pooling smoothing algorithm that strictly aligns fine-grained feature scores to the memory boundaries of the underlying physical blocks. This achieves an extremely high compression ratio while ensuring zero-fragmentation reclamation of physical memory. Directly removing low-scoring features at the token level would generate a large amount of internal memory fragmentation within the fixed-capacity physical blocks at the bottom layer, preventing the physical memory from being substantially reclaimed by the system.

[0038] The probability matrix A is subjected to mean pooling along the window dimension and max pooling along the head dimension in sequence to eliminate score drift caused by different observation window sizes, thus obtaining a one-dimensional word-level score. ,Will Mapping back to physical block dimension form Perform a summation operation within the block to obtain the initial block score. Where N is the number of physical blocks. Finally, to prevent critical context breakage caused by physical segmentation, a one-dimensional sliding max pooling with kernel size k, stride s, and padding p is introduced to generate the smoothed final block importance: (7)

[0039] Under the constraint of retention ratio r, the target retention index is obtained based on the final block importance score. By updating the logical mapping table, the selected reserved blocks are located between the Sink area and the Obs area in the logical address space; for unselected physical blocks, their physical pages are directly released and reset to an idle state. This architecture ensures that each compaction operation releases complete physical block resources.

[0040] (7) Dual memory pressure sensing module;

[0041] This module is configured in the front-end perception layer of the global scheduling engine and aims to solve the false negative problem of "logical idle but physical exhausted" caused by memory fragmentation or memory allocator caching mechanisms in deep learning frameworks. By simultaneously probing the logical memory view inside the framework and the real physical view of the underlying device within discrete time steps, this module can output a high-confidence basic stress signal, which is divided into internal logical stress and external device physical stress.

[0042] The internal logic pressure is derived from the monitoring metrics of Module 3, based on the number of absolutely free physical blocks in the system memory mapping table. With dynamic low water level Judgment; when hour, Set to a boolean value of True. External device physical load is determined by the system directly obtaining the total available physical memory by calling the underlying computing device interface. The system sets a safety warning threshold. If and only if there is currently an active decoding computation task and hour, Set to True; In instantaneous determination mode, the system executes Boolean logic to generate a fused basic pressure signal: (8)

[0043] (8) Anti-jitter window and state machine module;

[0044] This module receives the basic pressure signal from module seven. This aims to solve the problem of bus storms caused by blindly issuing unloading and prefetching instructions near the memory critical point. For example... Figure 3 The diagram shown illustrates the anti-jitter state machine and crisis triggering flowchart described in this invention. By introducing a time-window-based I / O jitter quantization model, this module drives a state machine to mitigate bus jitter characteristics, controlling the triggering and smooth exit of system intervention states.

[0045] Let the total number of successful physical block unloadings and the total number of successful prefetches in the system at time step t be respectively... and Define the instantaneous page increment as: (9) The system obtains the total number of currently active request sequences participating in the computation. ;when At that time, calculate the normalized instantaneous page rate. To eliminate sudden noise in single-step measurements, this module establishes a length of... A circular buffer sliding window. At any time step T, the system's macroscopic jitter resistance quantification index is defined as the mathematical expectation within this window: (10) Based on the input of module seven and The system determines whether the available video memory pool needs to enter a high-pressure crisis state of global takeover. Uninstallation begins. Its trigger condition is defined as: (11) The underlying memory pressure signal must be satisfied. (True), and the system currently meets any of the following internal scheduling preconditions: Unloading has been attempted ( There are candidate sequences available for expulsion. ), or the system is already in a managed takeover mode ( ).

[0046] To completely prevent the oscillating cycle of intervention being lifted immediately after the danger has passed, followed by rapid relapse due to high concurrency, this module has developed an extremely stringent comprehensive recovery signal: (12) This means that three conditions must be met simultaneously: sufficient logic block level, security of external underlying physical devices, and a return to normal I / O bus activity. This is the preset low-frequency jitter tolerance threshold.

[0047] jitter rate Its core function is to prevent the system from prematurely exiting. Even if the video memory margin seems to have recovered to a safe level, as long as the system detects that frequent block-level swapping is causing the bus load to exceed the tolerance threshold, the system will still maintain a crisis intervention state to prevent a precipitous drop in throughput. Only when... The signal is in continuous The state machine truly relinquishes control and exits the intervention state only when the condition remains true for an extended period of time. (13) (14)

[0048] (9) Three-layer candidate dynamic unloading scheduling and cold and hot isolation expulsion module.

[0049] This module is configured in the execution layer of the global scheduling engine and receives the crisis takeover signal from module eight. To prevent the blind unloading of high-frequency active sequences under extreme memory pressure from causing them to be mistakenly killed, this module strictly constructs a precise three-layer degradation defense line (L1 / L2 / L3) and composite sorting logic based on the underlying queue state, cooldown characteristics, and residency threshold.

[0050] like Figure 4 The diagram shown illustrates the three-tiered cascading unloading process described in this invention. When the system is in a crisis takeover state, this module scans the sequence in priority order and constructs three independent candidate pools sequentially. L1 is the ready queue that has been pre-filled but not yet executed; L2 is the currently active decoding queue, which is selected only from the remaining sequences of "active but not scheduled in this step"; L3 serves as a fallback mechanism for extreme states. As a fallback mechanism for extreme states, it is not necessarily executed in every scheduling, but is controlled by a four-dimensional composite triggering condition: (15) If and only if both conservative L1 and L2 fail to provide valid candidates, and the current ready queue does not experience severe overcrowding (length...), then... Only when the system's logical or physical memory pressure remains, can an unauthorized trigger be established. Once triggered, the system will broaden the search scope to the entire active decoding set, forcibly mining deep redundant physical blocks.

[0051] For L1 candidates, an ascending sort is performed, with absolute priority given to unloading the sequence with the most currently freeable physical blocks, maximizing the benefit of a single unloading. For L2 and L3, when sorting decoded-state candidates, the system prioritizes evicting the cold sequence with the longest idle cooling time (i.e., the longest unused strategy). Only when multiple sequences have the same idle time is the sequence with the largest freeable volume prioritized for unloading. When issuing unloading commands, the system strictly follows... The insurmountable priority. That is, L1 candidates are cleared first; if L1 is exhausted and the gap is filled. If the conditions are still not met, then L2 candidates can be explored; L3 is only activated when both L1 and L2 are exhausted and the L3 limit trigger condition is met. The absolute recovery gap formula is as follows: (16)

[0052] (10) Dynamic Concurrent Backpressure and Load Adaptive Module

[0053] This module is configured in the global request scheduler, serving as the highest-level control instruction for preventing extreme memory overflows. For example... Figure 5 The diagram shown is a flowchart of the dynamic backpressure and smooth recovery control described in this invention. When the system is in a crisis takeover state... When conventional physical block unloading methods are insufficient to quickly curb the trend of memory degradation, the system dynamically adjusts the maximum concurrency limit allowed to participate in decoding, forcing a switch to a degraded operating mode.

[0054] The system is initiating and maintaining [its operation]. During this state, it was determined that simple memory movement could no longer alleviate the current computational pressure, and a concurrent anti-pressure mechanism was activated. The system reads the currently effective active decoding concurrency limit. And according to the preset attenuation coefficient (in Nonlinear reduction is applied to it. The formula for calculating single-step reduction is: (17) This logic ensures that the system concurrency will decrease rapidly and exponentially to free up memory space, guaranteeing that the system can still maintain a minimum processing capacity of at least 2 concurrent requests even when memory is low, thus preventing the inference service from completely shutting down.

[0055] Module 8 determines whether the cumulative score during the recovery window meets the conditions (i.e.) The system officially terminates the takeover and exits crisis mode. After exiting crisis mode, the system does not immediately restore the initial high-concurrency configuration, but instead executes a linear smooth recovery strategy. In each safe scheduling step, the system proceeds in fixed step sizes. Gradually increase the concurrency limit: (18) in This represents the maximum physical concurrent load capacity measured during system initialization. This asymmetric adjustment mechanism, characterized by rapid decrease and slow increase, effectively filters out secondary rebounds in memory pressure and bus oscillations caused by the instantaneous maximization of concurrency.

[0056] like Figure 6 As shown, the present invention provides a KV cache anti-jitter asynchronous offloading scheduling method and a system for large-scale language model inference, which is based on the above-mentioned system and includes the following steps: Step 1: Receive large model inference requests, queue up the requests according to the sequence lifecycle state machine, and maintain the block state information corresponding to the request sequence based on the preset block-type KV Cache storage structure to provide scheduling objects for subsequent compression, recycling and asynchronous unloading. Step 2: In the autoregressive decoding stage of the sequence, semantic isolation and real attention scoring are performed on the key-value cache divided in Step 1. Based on the scoring results, triple pooling block alignment is performed on the logical blocks of non-key features. Through logical mapping update and physical block release, the underlying data transfer in the redundant block reclamation process is reduced, and the risk of fragmentation is reduced. Step 3: During the recycling process in the autoregressive decoding stage of Step 2, the dual water level lines of the available logic blocks inside the system are monitored in real time, and the heterogeneous memory exhaustion signal of the external underlying physical device is collected simultaneously. The basic pressure signal reflecting the memory capacity reserve is output in a comprehensive manner. Step 4: Establish an I / O jitter quantization model based on a sliding time window to calculate the recent average bus jitter rate of the system in real time; if the basic pressure signal does not fall below the safety threshold, the current system memory capacity is determined to be safe, and the process returns to step 2 to maintain the normalized autoregressive decoding and memory block reclamation cycle; if the basic pressure signal falls below the safety threshold, the system internal scheduling preconditions are combined to trigger and switch to a global takeover crisis state, and step 5 is executed next. Step 5: In the crisis state triggered in step 4, based on the real-time scheduling status and idle cooling period of the sequences in the same active decoding set, multiple offloading candidate pools with decreasing priority are dynamically extracted; within the same candidate pool, multi-dimensional composite sorting is performed by combining the temporal locality characteristics of the sequence and the spatial release benefits, and the physical blocks of the selected sequence are strictly and asynchronously offloaded to the host memory in the order of the candidate pool. Step 6: During the crisis, if the memory capacity gap after unloading in Step 5 is not filled, a non-linear backpressure reduction is performed on the maximum active concurrency limit allowed by the system. At the same time, the system maintains strict hysteresis recovery latch. When the basic pressure signal mentioned in Step 3 indicates that both the internal memory and external devices have recovered safely, and the recent average bus jitter rate calculated in Step 4 is continuously lower than the tolerance threshold within a consecutive preset time window, the system determines that the danger has been eliminated and exits the crisis state. Subsequently, a linear smooth recovery is performed on the maximum active concurrency limit.

[0057] To further illustrate the collaborative working mechanism of the technical solution of this invention, the following uses an edge inference node configured with limited physical video memory as an example to describe in detail the complete response and scheduling recovery process of the system when encountering excessive concurrency peaks.

[0058] The initial system parameters are set as follows: maximum physical concurrent capacity. Low water level Block, high water level Block, single-step bus offload budget Block, recovery window period At this step, the system is currently handling 28 concurrent long-sequence dialogue requests.

[0059] During normal autoregressive decoding, the system calculates attention scores in real time, virtually merges and pools key-value cache blocks corresponding to a large number of historically useless tokens, and directly cancels physical blocks at the logical level, effectively slowing down the exhaustion of GPU memory.

[0060] As the dialogue rounds deepened and the sequence length increased dramatically, the monitoring module detected internal idle physical blocks within the system. It plummeted to 85 (below) Generate basic pressure signal The state machine immediately detects the signal and forces the system's global state into crisis takeover mode. (This pauses regular scheduling and prepares to force unload or suspend some low-priority sequences.)

[0061] The system needs to unload the KV cache from the CPU to salvage video memory. The system first scans the ready queue, extracting two sequences that have completed pre-filling but have not yet begun decoding as L1 candidates. After sorting, the sequence with the largest number of blocks (largest volume) is asynchronously unloaded, reclaiming 60 physical blocks. After L1 unloading, It rose to 145, but still has not reached a high level. The system then extended the search scope to L2. After filtering out the hottest sequences currently being executed and strictly eliminating sequences with an idle cooling step count Age < 1, the system locked three Least Recently Used (LRU) cold sequences that met the criteria and issued transfer commands, consuming 50 units of the single-step unload budget.

[0062] Despite physical video memory The number of chips has gradually recovered to 310 (temporarily surpassing the high water level), but the state machine, through a sliding time window calculation, found that due to the recent intensive unloading, the recent average bus jitter rate has increased. The concurrency level remains as high as 0.60, far exceeding the tolerance threshold. The system determines that the bus is still under pressure and refuses to exit the crisis state. Given the continued crisis state and severe bus congestion, the system triggers a dynamic backpressure mechanism, exponentially reducing the current concurrency limit from 32. The system forcibly suspended and evicted the three most active sequences with the lowest priority, thus forcibly reducing the rate of system heat generation and memory consumption.

[0063] As the concurrency limit is lowered, the system's memory handling burden decreases. Bus jitter. It quickly dropped below 0.2. At this point, the system's overall recovery signal... The light comes on, and the anti-vibration counter begins to increment. After 16 consecutive scheduling steps ( Within that timeframe, the signal remained uninterrupted. The system officially determined the danger had been averted and cleared the area. The system initiates a recovery mechanism after the crisis is resolved, and in subsequent safety scheduling steps, the concurrency limit is reduced in increments of 1. The step size gradually and linearly increases from 25 to 32, and the system eventually smoothly transitions back to the limit throughput state.

[0064] The above description is merely a preferred embodiment of the present invention and is not intended to further limit the present invention. All equivalent changes made based on the description and drawings of the present invention are within the protection scope of the present invention.

Claims

1. A KV cache anti-jitter asynchronous offloading scheduling method for large language model inference, characterized in that, Includes the following steps: Step 1: Receive large model inference requests, schedule requests in a queue according to the sequence lifecycle state machine, and maintain the block state information corresponding to the request sequence based on the preset block-based KV Cache storage structure. Step 2: In the autoregressive decoding stage of the sequence, semantic isolation and real attention scoring are performed on the KV Cache partitioned in Step 1. Based on the scoring results, the logical blocks with non-critical features are aligned by triple pooling. By updating the logical mapping and releasing the physical blocks, the underlying data transfer during the redundant block reclamation process is reduced, and the risk of fragmentation is lowered. Step 3: During the recycling process in the autoregressive decoding stage of Step 2, the dual water level lines are monitored in real time, and the heterogeneous memory exhaustion signal of the external underlying physical device is collected simultaneously. The basic pressure signal reflecting the memory capacity reserve is output comprehensively. Step 4: Establish an I / O jitter quantization model based on a sliding time window and calculate the recent average bus jitter rate of the system in real time; if the basic pressure signal does not fall below the safety threshold, it is determined that the current system memory capacity is safe and return to step 2; if the basic pressure signal falls below the safety threshold, it is combined with the preconditions of the system's internal scheduling to trigger and switch to a global takeover crisis state, and then proceed to step 5. Step 5: In the crisis state triggered in step 4, based on the real-time scheduling status and idle cooling period of the sequences in the same active decoding set, multiple offloading candidate pools with decreasing priority are dynamically extracted; within the same candidate pool, multi-dimensional composite sorting is performed by combining the temporal locality characteristics of the sequence and the spatial release benefits, and the physical blocks of the selected sequence are strictly and asynchronously offloaded to the host memory in the order of the candidate pool. Step 6: During a crisis, if the memory capacity gap after unloading in Step 5 is not filled, a non-linear backpressure reduction is performed on the maximum allowed active concurrency limit of the system. Meanwhile, the system maintains strict hysteresis recovery latch; when the basic pressure signal indicates that both the internal video memory and external devices have recovered to safety, and the recent average bus jitter rate has been continuously lower than the tolerance threshold within a preset time window, the system determines that the danger has been eliminated and exits the crisis state, and then performs a linear smooth recovery on the maximum active concurrency limit.

2. The KV cache anti-jitter asynchronous offloading scheduling method for large language model inference according to claim 1, characterized in that, Step 1 is as follows: 1) Total video memory capacity of the target computing node for dynamic detection and the static weight usage of large models ; After deducting the preset safety reserve space Then, based on the set video memory allocation ratio coefficient... Calculate the total byte budget allocated to the key-value cache; 2) Divide the budget into physical blocks of fixed size, assuming the total number of layers in the large model is... The number of tokens contained in a single physical block is The number of attention heads is The dimension of a single-head hidden layer is The number of bytes occupied by the data type is The maximum number of physical blocks the system can accommodate Determined as: ， For any input request sequence, the generated KV feature vector is segmented at the token level and distributed into fixed-size physical blocks; the global absolute position index in the given sequence Its corresponding logical block index The calculation formula is: 。 3. The KV cache anti-jitter asynchronous offloading scheduling method for large language model inference according to claim 1, characterized in that, In step 1, block pre-filling computation is performed on the input request sequence, including: When the received length is When dealing with a long input sequence, it is divided into multiple consecutive sub-sequence blocks according to a preset block capacity C; Perform gradient-free forward propagation on each subsequence block in turn; When calculating the current sub-block, the accumulated KV Cache state generated by the previous sub-block is passed in as the historical context, and the updated key-value state is extracted after calculation and passed to the next sub-block.

4. The KV cache anti-jitter asynchronous offloading scheduling method for large language model inference according to claim 1, characterized in that, In step 2, the KV Cache is given a true attention score, specifically: For a total length of Based on the causal features generated by the autoregressive model, the complete input sequence is strictly divided into three logically independent isolation regions along the time step dimension of the KVCache; among them, the starting syntax protection region, i.e., the Sink region, is the truncated sequence with a length of [missing information]. Continuous tokens; the tail local observation region, i.e., the Obs region, is the truncated sequence with a length of [missing value]. A continuous set of tokens; the middle compressible area, or Mid area, is located between the Sink area and the Obs area, with a length of [missing information]. ; , Perform scaling dot product attention operation, calculate For compressible bond tensors Global attention weight distribution matrix A: ， in, The hidden dimension of the model's attention head; The query tensor is the actual projection of the Obs region at the end of the sequence into the current forward computation; the generated weight matrix A has the following shape. , Batch size, This represents the number of long positions.

5. The KV cache anti-jitter asynchronous offloading scheduling method for large language model inference according to claim 4, characterized in that, In step 2, based on the scoring results, triple-pooling block alignment is performed on logical blocks with non-critical features. Through logical mapping updates and physical block releases, the underlying data movement during redundant block reclamation is reduced, and the risk of fragmentation is mitigated. Specifically: The global attention weight distribution matrix A is subjected to mean pooling along the window dimension and max pooling along the head dimension in sequence to obtain a one-dimensional word-level score. ,Will Mapping back to physical block dimension form Perform a summation operation within the block to obtain the initial block score. Where N is the number of physical blocks; We introduce a one-dimensional sliding max pooling method with kernel size k, stride s, and padding p to generate the smoothed final block importance: ， Under the constraint of retention ratio r, the target retention index is obtained based on the final block importance score. And by updating the logical mapping table, the reserved blocks are located between the Sink area and the Obs area in the logical address space; unselected physical blocks are directly released and reset to the free state; For unselected underlying physical blocks, their corresponding physical memory pages are directly handed over to the underlying lifecycle state machine to be reset to an idle state.

6. The KV cache anti-jitter asynchronous offloading scheduling method for large language model inference according to claim 1, characterized in that, In step 3, the double water level is calculated as follows: ， in, and The preset capacity monitoring ratio strictly meets the requirements. Interval distribution; The threshold for the low water level line; The high water level threshold; This represents the maximum number of physical blocks that the physical storage pool can hold.

7. The KV cache anti-jitter asynchronous offloading scheduling method for large language model inference according to claim 6, characterized in that, In step 3, the dual water level lines of the internal underlying paging physical video memory are monitored simultaneously, and the heterogeneous video memory depletion signal of the external underlying physical devices is collected simultaneously. The overall output is a basic pressure signal reflecting the remaining video memory capacity, specifically: Based on monitoring of the dual water level lines, the number of absolutely free physical blocks in the system memory mapping table is calculated. With dynamic low water level Judgment; when At that time, internal logic memory pressure Set to the boolean value True; For the total available physical video memory Preset safety warning threshold If and only if there is currently an active decoding computation task and At that time, the actual memory consumption state of the underlying hardware device Set to True; In instantaneous determination mode, the system executes Boolean logic to generate a fused basic pressure signal: 。 8. The KV cache anti-jitter asynchronous offloading scheduling method for large language model inference according to claim 7, characterized in that, In step 4, a global takeover crisis state is triggered and initiated, specifically as follows: Let the total number of successful physical block unloadings and the total number of successful prefetches in the history of the system at time step t be respectively... and Define the instantaneous page increment as: ， The system obtains the total number of currently active request sequences participating in the computation. ;when At that time, calculate the normalized instantaneous page rate. ; This represents the total number of request sequences participating in the computation at the current time step t. Create a length of For a circular buffer sliding window, at any time step T, the macroscopic jitter resistance quantification index of the system is defined as the mathematical expectation within this window: ， Input-based and The system determines whether the available video memory pool needs to enter a high-pressure crisis state of global takeover. Start uninstalling; Its triggering condition is defined as: 。 9. A KV cache anti-jitter asynchronous offloading scheduling method for large language model inference according to claim 8, characterized in that, In step 6, the system determines that the danger has been eliminated and exits the crisis state, specifically as follows: Satisfy the underlying memory pressure signal If true, and the system currently meets any of the following internal scheduling preconditions: Unloading has been attempted... There exists a candidate sequence for expulsion. Or the system is already in a managed takeover mode. ; Define the comprehensive recovery signal: ， This means that three conditions must be met simultaneously: sufficient logic block level, secure external underlying physical devices, and a restored I / O bus; among which... This is the preset low-frequency jitter tolerance threshold; Only when The signal is in continuous The state machine truly relinquishes control and exits the intervention state only when the condition remains true within the specified time window. ， 。 10. A KV cache anti-jitter asynchronous offloading scheduling system for large language model inference, characterized in that... The system for implementing the method of any one of claims 1-9 comprises: The physical paging memory capacity allocation and mapping module is responsible for the computational resource planning during system initialization, calculating the total byte budget of KVCache and dividing it into fixed-size physical blocks, establishing the mapping relationship between logical blocks and physical blocks, and realizing physically non-contiguous but logically contiguous addressing. The physical block lifecycle and dual water level monitoring module maintains the lifecycle residence state for each inference sequence, defines six mutually exclusive residence states, and ensures the integrity of forward computation and the atomicity of asynchronous transfer. The anti-jitter dual water level monitoring and hysteresis alarm module calculates the high water level and low water level based on the maximum total number of physical blocks that the physical storage pool can hold; it polls the number of idle physical blocks in real time and sends an unloading trigger signal or releases the pool pressure warning based on their relative position to the dual water level. The long sequence block pre-filling module uses block computing technology to process ultra-long input sequences, transforming global attention calculation into serial calculation and limiting the instantaneous peak memory usage during the pre-filling stage. The attention scoring module performs semantic isolation on the KV Cache, calculates the attention weight distribution matrix based on the real query tensor and key tensor, and quantifies the importance of long sequence historical features; The triple pooling block alignment and reclamation module takes the attention scoring results and aligns the feature scores to the physical block memory boundary through triple pooling operations, thereby achieving zero-fragmentation reclamation of redundant physical blocks. The dual memory pressure sensing module simultaneously detects the logical memory view inside the framework and the physical view of the underlying device, and outputs a high-confidence basic pressure signal. The anti-jitter window and state machine module calculates the average bus jitter rate based on a sliding time window, which drives the state machine to trigger or exit a crisis state. The three-layer candidate dynamic unloading scheduling and cold / hot isolation eviction module constructs a three-layer priority candidate pool in crisis situations, executes composite sorting logic, and unloads physical blocks in sequence asynchronously according to order. The dynamic concurrency backpressure and load adaptive module performs non-linear reduction of the concurrency limit during a crisis and performs linear smooth recovery after the crisis ends.

Images