Key value cache scheduling method and system for language model reasoning

Abstract

The invention provides a key value cache scheduling method and system for language model inference, and is applied to the technical field of video memory management and key value cache multiplexing of a large-scale language model inference engine. The method comprises the following steps: acquiring an input sequence, extracting a candidate multiplexing fragment, calculating a fragment hash value of the candidate multiplexing fragment, querying a corresponding historical logic block hash list in a fragment index table, and generating a current logic block hash list for the fragment in a pre-filling process, mapping the Hash of the current logic block to the physical address of the historical physical key value cache block through a prefix Hash table to realize logic Hash alias, and scheduling key value cache data in an underlying video memory to participate in autoregressive decoding according to the logic Hash alias; by means of the scheme, prefix tree matching limitation can be broken through, zero-copy cross-context multiplexing of semantic fragments at any position is achieved, pre-filling calculation redundancy and first word delay are remarkably reduced, and GPU video memory utilization efficiency and reasoning throughput under a multi-agent concurrent scene are improved.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and computer architecture, and in particular to a key-value cache scheduling method and system for language model inference. Background Technology

[0002] In large-scale language model inference, key-value caches are used to store the key and value tensors already computed in the attention mechanism, forming short-term working memory for the agent to perform tasks. Existing technologies generally employ a prefix caching strategy, reusing the corresponding key-value cache only when the prefix of a newly requested input sequence is completely identical to the prefix of a historically computed sequence. This strategy relies on a prefix tree structure for cache matching, which has significant limitations in multi-agent collaborative scenarios. The system prompt templates, role settings, and context organization methods used by each agent are highly heterogeneous, meaning that even if the input sequence contains identical semantic fragments, cache reuse cannot be triggered simply because of prefix differences. This leads to a large amount of repetitive pre-padding computation, significantly increasing first-word latency and exacerbating GPU memory usage. Furthermore, rotational position encoding deeply couples key-value tensors with absolute positions; directly reusing physical cache blocks across contexts will cause positional information misalignment, affecting model output stability. Simultaneously, when multiple agents concurrently access the same historical key-value cache block, the lack of a collaborative management mechanism for the lifecycle of physical memory blocks easily leads to system-level errors such as dirty writes or post-use release.

[0003] Therefore, there is an urgent need for a key-value cache scheduling method that can break through prefix constraints, support the reuse of semantic fragments at arbitrary positions, be compatible with positional encoding calibration, and ensure concurrency safety.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a key-value cache scheduling method and system for language model inference, aiming to solve the technical problems in the prior art caused by strict prefix matching constraints, such as coarse granularity of key-value cache reuse, inability to reuse cross-context semantic segments, inability to share working memory among multiple agents, degradation of generation quality caused by positional encoding misalignment, and insecurity of physical memory management under concurrent access. By constructing a decoupled architecture of semantic segment index and logical hash pre-aliasing, performing low-level pointer overwriting in the prefix hash table to achieve zero-copy cross-context reuse, combining the positional encoding offset calibration mechanism of first and last local recalculation and tensor concatenation, and introducing reference counting based on atomic operations to drive memory lifecycle management, the present invention supports efficient, safe, and high-quality KV cache reuse of repeated semantic segments at arbitrary positions, significantly reducing first-word latency (TTFT), reducing peak GPU memory usage, and ensuring the system robustness of multi-agent concurrent inference.

[0006] This invention provides a key-value cache scheduling method for language model inference, applied to a computer device, comprising:

[0007] Obtain the input sequence of the current inference request and extract candidate reuse fragments from the input sequence;

[0008] Calculate the fragment hash value of the candidate reuse fragment, query the fragment hash value in the fragment index table, and obtain the historical logical block hash list corresponding to the fragment hash value; wherein, the fragment index table records the mapping relationship between fragment hash values ​​with independent semantics and the corresponding logical block hash list;

[0009] During the pre-filling computation of the language model, a corresponding current logical block hash list is generated for the candidate reuse fragments based on the current context of the input sequence;

[0010] In the prefix hash table, the physical address pointer corresponding to the current logical block hash list is updated to the physical address of the historical physical key-value cache block pointed to by the historical logical block hash list, so as to establish a logical hash alias mapping;

[0011] Based on the updated physical address pointer in the prefix hash table, the key-value cache data corresponding to the historical physical key-value cache block in the underlying video memory is scheduled so that the key-value cache data can participate in the autoregressive decoding calculation of the language model.

[0012] In some alternative embodiments, extracting candidate multiplexed segments from the input sequence includes: extracting text substrings by parsing explicit boundary labels carried in the input sequence and using the text substrings as candidate multiplexed segments; or, identifying upstream output text sequences referenced in the input sequence according to workflow default rules and using the upstream output text sequences as candidate multiplexed segments.

[0013] In some optional embodiments, before calculating the fragment hash value of the candidate reuse fragment, the method includes: performing text normalization processing on the candidate reuse fragment, wherein the text normalization processing includes at least one of whitespace removal, punctuation unification, and case conversion; or, converting the candidate reuse fragment into a word sequence using a word segmenter of a language model, and calculating the fragment hash value based on the word sequence.

[0014] In some optional embodiments, the physical address pointer corresponding to the current logical block hash list is updated to the physical address of the historical physical key-value cache block pointed to by the historical logical block hash list, including: traversing the current logical block hash in the current logical block hash list; replacing the value of the key-value pair indexed by the current logical block hash in the prefix hash table with the offset mapped in the underlying video memory of the historical logical block hash at the corresponding sequential position in the historical logical block hash list.

[0015] In some optional embodiments, after establishing the logical hash alias mapping, the following steps are included: comparing the current position encoding information of the candidate multiplexed fragment in the current context with the historical position encoding information in the historical context; and triggering a position encoding offset calibration mechanism if the current position encoding information and the historical position encoding information are inconsistent.

[0016] In some optional embodiments, a positional encoding offset calibration mechanism is triggered, including: extracting the first and last lexical subsequences of the candidate multiplexing segment; and calling the language model to perform partial pre-padding calculations on the first and last lexical subsequences to obtain first and last tensor data carrying the updated rotated positional encoding.

[0017] In some optional embodiments, the triggering position encoding offset calibration mechanism further includes: determining a recalculation ratio threshold for partial pre-padding calculation based on the total number of tokens contained in the candidate reused segment and the total token length of the current context; adjusting the truncation lengths of the first and last token subsequences according to the recalculation ratio threshold; and concatenating the first tensor data, the last tensor data, and the intermediate segment tensor data read from the historical physical key-value cache block so that the concatenated tensor data participates in the autoregressive decoding calculation.

[0018] In some optional embodiments, when updating the physical address pointer corresponding to the current logical block hash list to the physical address of the historical physical key-value cache block pointed to by the historical logical block hash list, the following is included: In the underlying memory management pool of the language model, an atomic increment operation is performed on the reference counter corresponding to the historical physical key-value cache block to intercept the asynchronous release operation and dirty write operation performed by concurrent inference requests on the historical physical key-value cache block.

[0019] In some optional embodiments, after the language model completes the autoregressive decoding computation for the input sequence, the process includes: performing a cleanup operation on the virtual physical address associated with the current logical block hash list, and performing an atomic decrement operation on the reference counter of the historical physical key-value cache block.

[0020] In some optional embodiments, the method further includes: monitoring the physical memory occupancy rate of the underlying memory management pool; when the physical memory occupancy rate exceeds a safety threshold, locating an idle physical key-value cache block with a reference counter of zero in the underlying memory management pool according to the least recently used strategy, performing physical memory reclamation operation on the idle physical key-value cache block, and clearing the hash mapping relationship corresponding to the idle physical key-value cache block in the prefix hash table in parallel.

[0021] In some optional embodiments, after completing the autoregressive decoding calculation and generating the output sequence, the following steps are included: segmenting the output sequence into target product segments based on one of the following indicators: agent role identifier, tool call instruction boundary, and maximum lexical length threshold.

[0022] In some optional embodiments, the method further includes: calculating a new fragment hash value for the target product fragment; obtaining the new physical key-value cache block allocated during the generation process of the target product fragment and the corresponding new logical block hash list from the underlying memory management pool of the language model; and writing the mapping relationship between the new fragment hash value and the new logical block hash list into the fragment index table to update the collaborative working memory.

[0023] In some optional embodiments, the mapping in the fragment index table is bound to at least one of the following permission identifiers: task domain isolation identifier and tenant permission identifier.

[0024] In some optional embodiments, querying the fragment hash value in the fragment index table includes: extracting the target permission identifier carried by the current inference request; comparing the target permission identifier with the permission identifier recorded in the fragment index table; and if the target permission identifier matches the permission identifier, obtaining a list of historical logical block hashes corresponding to the fragment hash value.

[0025] In some optional embodiments, calculating the fragment hash value of the candidate multiplexed fragment and querying the fragment hash value in the fragment index table includes: calculating the locality-sensitive hash value of the candidate multiplexed fragment; performing an approximate matching query in the fragment index table based on the locality-sensitive hash value to obtain the target mapping item; if the similarity of the target mapping item exceeds a preset threshold, extracting the complete sequence features of the candidate multiplexed fragment and performing a secondary precise hash anti-collision comparison operation; if the secondary precise hash anti-collision comparison operation passes the verification, obtaining the historical logical block hash list corresponding to the fragment hash value.

[0026] This invention provides a key-value cache scheduling system for language model inference, comprising:

[0027] The input parsing module is configured to obtain the input sequence of the current inference request and extract candidate reuse fragments from the input sequence;

[0028] The index addressing module is configured to calculate the fragment hash value of the candidate reuse fragment, query the fragment hash value in the fragment index table, and obtain the historical logical block hash list corresponding to the fragment hash value; wherein, the fragment index table records the mapping relationship between fragment hash values ​​with independent semantics and the corresponding logical block hash list;

[0029] The context allocation module is configured to generate a corresponding current logical block hash list for candidate reuse fragments based on the current context of the input sequence during the pre-filling computation of the language model.

[0030] The address redirection module is configured to update the physical address pointer corresponding to the current logical block hash list in the prefix hash table to the physical address of the historical physical key-value cache block pointed to by the historical logical block hash list, so as to establish a logical hash alias mapping.

[0031] The decoding execution module is configured to schedule key-value cache data corresponding to historical physical key-value cache blocks in the underlying video memory based on the updated physical address pointer in the prefix hash table, so that the key-value cache data can participate in the autoregressive decoding calculation of the language model.

[0032] In some optional embodiments, the system further includes a tensor alignment module; the tensor alignment module is configured to detect the rotation position encoding difference of the candidate reuse fragment in the current context and the historical context after the address redirection module establishes the logical hash alias mapping; in the case of rotation position encoding difference, the tensor alignment module performs local pre-filling calculation on the boundary word sequence of the candidate reuse fragment according to the recalculation ratio to obtain local tensor features, and performs a concatenation operation on the local tensor features and the intermediate tensor features extracted from the historical physical key-value cache block, so that the decoding execution module can call it to participate in the autoregressive decoding calculation.

[0033] In some optional embodiments, the system further includes a concurrent memory locking module; the concurrent memory locking module is configured to send a reference count increment instruction for the historical physical key-value cache block to the underlying memory in parallel when the address redirection module updates the physical address pointer to the physical address of the historical physical key-value cache block, so as to intercept asynchronous memory erase operations and rewrite operations; and after the decoding execution module completes the autoregressive decoding calculation, it sends a release instruction and performs a free physical key-value cache block reclamation operation based on the reference state according to the underlying memory remaining monitoring results.

[0034] In some optional embodiments, the system further includes a working memory sedimentation module; the working memory sedimentation module is configured to capture the output sequence finally generated by the current inference request, divide the output sequence into target product fragments according to truncation rules, obtain the new logical block hash list and the corresponding new fragment hash value allocated in the underlying video memory for the target product fragment, and write the association between the new fragment hash value and the new logical block hash list into the fragment index table to achieve persistent sharing of the underlying tensor state.

[0035] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention.

[0036] The key-value cache scheduling method and system for language model inference of the present invention have the following beneficial effects:

[0037] This invention overcomes the strict dependence of prefix caching on the start position of the sequence by using semantic fragment-level logical hash alias mapping, enabling zero-copy reuse of key-value caches for repeated text fragments at any position. Without moving physical GPU memory data, it significantly reduces pre-filling computation overhead and shortens the first-word latency (TTFT). It supports working memory sharing across contexts and prompt templates among multiple agents, improving inference throughput and GPU memory utilization efficiency. At the same time, relying on atomic reference counting and permission identifier binding mechanisms, it ensures GPU memory security and tenant data isolation in high-concurrency scenarios. Attached Figure Description

[0038] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings.

[0039] Figure 1 This is a flowchart of a key-value cache scheduling method for language model inference according to an embodiment of the present invention;

[0040] Figure 2 This is a schematic diagram of the structure of a key-value cache scheduling system for language model inference according to an embodiment of the present invention. Detailed Implementation

[0041] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the invention will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0042] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0043] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined. Therefore, the actual execution order may change depending on the specific circumstances.

[0044] In large-scale language model inference, the key-value cache (KV Cache), as a data structure storing historical context states in the attention mechanism, has a decoupled mapping relationship between its physical memory layout and logical addressing. The prefix hash table (PHT) is essentially an address translation table that maps logical block hash identifiers to physical memory block offsets. This mapping relationship does not depend on the absolute position or context order of the input sequence, but only on the current inference engine's memory management strategy for KV blocks. When semantically identical text segments appear in different input sequences, although their starting positions, prefix lengths, and position encoding parameters differ in their respective contexts, as long as the segment is divided into KV cache blocks of the same number and size, a mapping relationship exists between its corresponding historical logical block hash list and the current logical block hash list. The one-to-one index relationship allows new requests to reuse key-value tensor data already existing in physical memory without triggering pre-filling computation by modifying the physical address pointed to by the target logical block hash in the PHT. This address redirection operation bypasses the mandatory constraint of sequence start consistency imposed by traditional prefix matching, refining the granularity of cache reuse from "complete prefix sequence" to "semantically independent fragments". This enables KV state sharing across contexts, requests, and arbitrary insertion positions in multi-agent heterogeneous prompting scenarios. As a result, while keeping the original attention computation logic of the model unchanged, the matrix multiplication load in the pre-filling stage is significantly reduced, the first word latency is shortened, and the bandwidth and capacity overhead caused by repeated allocation and writing to physical memory blocks is reduced.

[0045] like Figure 1 As shown, this invention provides a key-value cache scheduling method for language model inference, which includes the following steps:

[0046] Step S100: Obtain the input sequence of the current inference request and extract one or more candidate reuse fragments from the input sequence; wherein, the candidate reuse fragment is a continuous text substring with relatively independent semantics in the multi-agent collaborative task flow, and its boundary is determined by predefined structured tags, message role division, tool call context or semantic integrity of output content.

[0047] Step S200: Perform a hash calculation on each candidate reuse fragment to generate a unique fragment hash value; the hash calculation is based on the text content itself or its equivalent representation after normalization to ensure that semantically consistent fragments can still generate the same hash value under different prompt templates; use the fragment hash value as a key to perform a lookup operation in the fragment index table that resides in memory. If a match is found, obtain the historical logical block hash list associated with the fragment hash value; the fragment index table stores the mapping relationship between fragment-level hashes and the corresponding logical block hash list, does not save the key-value cache data body, and only maintains index information.

[0048] Step S300: During the pre-filling calculation stage of the language model, a new set of current logical block hash lists is dynamically generated for candidate reuse fragments based on the overall context structure of the current input sequence and the allocated logical address space. The current logical block hash list reflects the sequence of logical block positions that the fragment should occupy in this request, and its composition is affected by the current prefix length, the number of occupied blocks, and the block partitioning strategy of the inference engine.

[0049] Step S400: Access the prefix hash table, which is a dictionary-style data structure that supports fast random read and write. Its key is the logical block hash, and its value is the offset pointing to the physical key-value cache block in the underlying video memory. For each item in the current logical block hash list, use it as the key to locate the corresponding entry in the prefix hash table, and update the value of the entry to the physical offset mapped by the historical logical block hash that is in the same order in the historical logical block hash list. This establishes an alias mapping relationship within the logical namespace, so that multiple different logical block hashes can point to the same physical key-value cache block.

[0050] Step S500: In the subsequent autoregressive decoding process, the attention mechanism initiates a video memory read request based on the current logical block hash list. Since the pointer overwriting was completed in step S400, after the request is parsed through the prefix hash table, the actual access is to the video memory address where the historical physical key-value cache block is located. The key tensors and value tensors stored in the historical physical key-value cache block are directly loaded into the computation pipeline to participate in the attention weight calculation and context vector generation in the current context, thereby skipping the repeated pre-filled matrix multiplication operation for this segment.

[0051] The above steps form a tightly coordinated technical loop: Step S100 identifies semantic units with reuse potential, providing targets for subsequent scheduling; Step S200 uses a hash mechanism to determine cross-context content consistency and completes the mapping from semantics to logical addresses using a fragment index table; Step S300 establishes the local logical address requirements of the fragment to be reused in the current context, constituting a prerequisite for aliasing operations; Step S400 performs pointer redirection at the underlying memory management level, which is an action to achieve zero-copy reuse, essentially constructing a many-to-one mapping topology within the logical address space; Step S500 makes this mapping relationship effective at runtime, ensuring that the hardware computing path naturally inherits the cached state. This four-level linkage mechanism of "semantic recognition—index lookup—logical aliasing—physical reuse" jointly solves the prefix mismatch problem caused by heterogeneous prompt words in multi-agent environments, breaks through the rigid dependence of traditional prefix caching on the sequence start position, realizes cross-context key-value cache reuse of semantic fragments at any position, and fundamentally reduces the redundant computing load and memory bandwidth pressure in the pre-filling stage.

[0052] Through the above scheme, this embodiment enables different agents to reuse the key-value cache state corresponding to the same historical generated content in their independently constructed prompt contexts without transferring physical tensor data. This significantly shortens the first-word latency (TTFT), reduces the peak GPU memory usage, and supports the continuous accumulation and collaborative evolution of working memory among multiple agents. In other optional implementations, hash calculation can employ one-way encrypted hashing, locality-sensitive hashing, or combined multi-level hashing. The fragment index table can be deployed in local process memory, a single-machine multi-process shared memory area, or in a large-scale inference cluster across computing nodes, as a centralized memory database or a sharded distributed key-value index service. In cross-node sharing mode, each computing node synchronizes the logical hash mapping state through a remote direct memory access (RDMA) network or high-speed Ethernet, thereby supporting multi-agent collaborative working memory sharing at the global multi-computing node level.

[0053] In one specific implementation, the extraction of candidate reuse fragments is achieved through two mutually exclusive and independently activatable mechanisms: one is an explicit label parsing mechanism, and the other is a workflow implicit identification mechanism.

[0054] First, for the explicit tag parsing mechanism, the input parsing module is configured to recognize predefined structured boundary markers. In one implementation, these markers are paired XML-style tags, e.g., ; in another, they are semantically specific tags, e.g., or , whose semantic type is bound to the agent role. When such tags are present in the input sequence, the module performs regular expression matching and extracts the complete text substring enclosed by them; subsequently, a boundary stripping operation is performed on this substring, removing the tags themselves and adjacent newlines and indentation whitespace, resulting in clean candidate reusable fragments. This mechanism supports recursive unpacking and flattening of nested tags, as well as parallel extraction of multiple non-overlapping tag segments from the same input.

[0055] Specifically, for the implicit workflow recognition mechanism, the input parsing module does not rely on any manually inserted tags, but instead performs context-aware recognition based on the workflow protocol preset by the inference service layer. This protocol defines the reference relationship pattern between the upstream agent's output and the downstream agent's input. In one implementation, this pattern manifests as an explicit reference to the upstream output hash value contained in the downstream prompt, for example, embedded in the form of [REF:sha256_abc123]. After capturing this reference, the module initiates a reverse query to the Memory Table to locate the corresponding historical logical block hash list and backtracks to restore the original upstream output text as a candidate reuse fragment. In another implementation, this pattern is based on token-level sequence matching: the module divides the current input sequence into several sliding windows, each window length equal to a preset minimum reusable fragment length threshold (e.g., 64 tokens); calculates the MinHash signature for each window and performs an approximate search with the locally cached recent upstream output fragment signature set; if the Jaccard similarity exceeds 0.92, a precise SHA-256 comparison is triggered, and after confirming the match, the original text corresponding to that window is truncated as a candidate reuse fragment. The mechanism also supports reference resolution across message rounds, allowing downstream requests to reference upstream outputs from one or more previous rounds, with the time span controlled by session lifecycle parameters in the workflow state machine.

[0056] In some other alternative implementations, explicit labels can be replaced with fragment metadata carried by HTTP header fields, gRPC metadata, or the $comment field in a JSON Schema; workflow implicit identification can also be replaced with fragment boundary prediction based on LLM self-labeling—at the upstream agent output stage, the model is prompted to generate a structured suffix, such as {"segment_boundary": "end", "semantic_type": "code_patch"}, which the downstream parsing module uses to locate the start and end positions of the fragment.

[0057] Through the above scheme, this embodiment can achieve dual compatibility of candidate reuse fragment extraction methods: it supports business parties to actively annotate high-value semantic units in the prompt to ensure reuse determinism, and it also supports the system to automatically discover duplicate content in non-intrusive deployment scenarios, thereby completing the transparent injection of KV cache sharing capabilities without changing the upper-layer application logic; the two mechanisms can be dynamically switched according to the request granularity, or they can be mixed in the same request - for example, explicit parsing is used for tagged parts, and implicit matching is used for untagged but frequently occurring document reference segments.

[0058] In one specific implementation, text normalization is performed on the candidate reuse fragment to eliminate hash inconsistency caused by differences in the agent's output format. First, the system calls the underlying string preprocessing unit to remove leading, trailing, and consecutive whitespace characters from the fragment. This includes replacing tabs, newlines, carriage returns, and multiple consecutive spaces with a single space, and further trimming the whitespace at both ends. Specifically, this operation is based on matching and replacing whitespace character categories (\p{Zs}, \p{Zl}, \p{Zp}) in the Unicode standard to ensure consistent processing of cross-language text. Then, the system performs a unified punctuation operation: mapping full-width Chinese punctuation (such as ",", ".", ";") to corresponding half-width ASCII punctuation (", ".", ";"), and standardizing quotation marks ("'", """) to right-angle single quotation marks (') and right-angle double quotation marks ("). Simultaneously, it normalizes dashes (——, ―) and ellipses (...) to two consecutive hyphens (--) and three consecutive periods (...), respectively. Next, the system decides whether to perform case conversion based on the configuration strategy: in code-oriented or structured text reuse scenarios, lowercase conversion is enabled, mapping all ASCII letters and common Latin extended characters (U+00C0–U+017F) to their lowercase forms; in scenarios that preserve the semantics of proper nouns, this step is skipped. After completing at least one of the above normalization operations, the system calculates the SHA-256 hash value of the normalized text, which serves as the fragment hash value for the candidate reuse fragment.

[0059] In some alternative implementations, instead of directly normalizing the original text, the system inputs candidate reuse fragments into the tokenizer bound to the current language model to generate corresponding word sequences. This tokenizer is a pre-trained tokenizer instance loaded from the HuggingFace Tokenizers library, supporting BPE or WordPiece algorithms, and strictly enabling the add_prefix_space=False and trim_offsets=True parameters to ensure cross-agent tokenization consistency. Before performing MinHash calculation, the system first uses a sliding window algorithm to transform the discrete word sequences into an N-gram (e.g., 3-gram) feature set composed of multiple continuous subsequences. Subsequently, multiple independent hash functions are used to perform dimensionality reduction mapping on the feature set, and the minimum value of each group is used to construct the MinHash. The signature matrix generates a fingerprint vector of length 128, which is then hashed using SHA3-256 to obtain the final fragment hash value. This method avoids the text-level sensitivity to punctuation / spaces / case sensitivity, and instead constructs hashes based on semantic units actually perceived by the model, thus improving the reusability hit rate in scenarios where the word segmenter is highly robust.

[0060] Through the above scheme, this embodiment can effectively suppress fragmented hash collisions caused by heterogeneous output of multiple agents, so that the same semantic content can still be stably mapped to the same fragment index item under different prompt templates, different encoding habits, and different terminal rendering formats, thereby significantly improving the query accuracy and cross-context reuse coverage of the fragment index table. Compared with the prior art, this embodiment incorporates semantically equivalent but surface-formatted text fragments into the same cache reuse path without increasing the storage overhead of the video memory entity, avoiding invalid pre-fill calculations caused by minor format deviations, and providing a high-confidence matching basis for the subsequent logical hash aliasing mechanism.

[0061] In one specific implementation, based on the above embodiments, the values ​​of key-value pairs indexed by the current logical block hash in the prefix hash table are replaced with the physical offsets mapped by the historical logical block hashes at the corresponding positions. Specifically, this includes the following operations: First, the system obtains the current logical block hash list. With historical logical block hash list Both are of the same length and are strictly aligned according to the spatial order of the blocks in the sequence; then, for each index position The system queries the underlying video memory management module to... For each key's Prefix Hash Table (PHT) entry, its value field is extracted—a 64-bit unsigned integer representing the starting byte offset (Physical BlockOffset) of the physical key-value cache block mapped to that logical block in the GPU's global memory address space. Then, the system performs an atomic write operation, directly overwriting this offset into the PHT. This is the value field corresponding to the key; the overwrite process does not trigger any memory allocation, data copying, or tensor computation, but only modifies the address mapping stored in the PHT. In some other optional implementations, the reading and overwriting of offsets can be done in batch mode, that is, constructing a batch containing... right The compact structure allows for batch updates via a single CUDA Unified Memorymemcpy or host-to-device pinned memory write; alternatively, the offset can be further encapsulated into a metadata structure with a verification field, which includes, in addition to the physical offset, a block size identifier, device ID, and version stamp to support logical alias mapping across GPU devices.

[0062] Through the above scheme, this embodiment can ensure that the hash of each logical block in the current request is accurately redirected to the original storage location of the historical physical KV block during the address resolution stage, thereby achieving zero-copy cross-context reuse without changing the existing PagedAttention operator call path. Compared with the prior art, this embodiment avoids redundant memory allocation and pre-filling calculation caused by the rigid binding of logical hash and physical address, while maintaining the inference engine's full compatibility with block-level address mapping, making the scheduling behavior completely transparent to the upper-layer attention operator.

[0063] Because current large language models widely employ Rotated Positional Encoding (RoPE), the key-value (KV) tensor state of any word is deeply bound to its absolute and relative position in the context. Therefore, if the prefix of the input sequence changes even slightly, the positional encodings of all subsequent words will shift, causing the historical key-value cache data to become completely invalid. Existing architectures typically can only adopt extremely conservative, strict prefix tree matching strategies, abandoning the potential for reusing the same intermediate text segments following different context prefixes.

[0064] In one specific implementation of this embodiment, after establishing the logical hash alias mapping, the system first obtains the starting absolute position index of the candidate reuse fragment in the current context. The starting absolute position index corresponding to this fragment in the historical context Then calculate the difference between the two. and judge Is it zero; if If the current location coding information is inconsistent with the historical location coding information, the location coding offset calibration mechanism will be triggered.

[0065] Specifically, the system is based on the total word length of the candidate reuse fragments. Overall sequence length in the current context Calculate the recalculation ratio threshold Then, the head truncation length is determined based on this threshold. Length of tail cut-off Then, extract the first word subsequence from the candidate reuse fragment. and suffix word sequence .

[0066] Subsequently, the system for Allocate a new temporary logical block hash list And call the attention operator of the language model, in order to... Perform local pre-padding calculations at the RoPE rotation angle of the starting position to generate a key tensor carrying the correct position encoding. AND-value tensor Similarly, for In position offset Perform the same operation at the location to generate Middle segment word subsequence Corresponding key-value tensor and It then reads directly from the historical physical key-value cache block that has been overridden by pointers.

[0067] Specifically, the local pre-padding computation injection rotation position encoding is performed, and its underlying mathematical mechanism is: extracting the absolute position index of the current context. For the first or last word sequence, the first... Each terminology generates a query vector. With key vector Multiply by absolute position Constructed rotation matrix The mathematical expression is:

[0068]

[0069] as well as

[0070]

[0071] This explicitly injects the latest positional bias into the tensor data generated by the locally pre-filled computation.

[0072] Next, the system constructs the target key tensor and value tensor along the sequence dimension in the GPU memory. It should be noted that the tensor concatenation is not a forced data copy performed in physical memory (such as a full concatenation operation in torch.cat that causes memory reallocation), but rather a concatenation based on a virtual logical view.

[0073] At the underlying hardware implementation level, the system calls the virtual block table concatenation primitive to sequentially fill the new physical block number of the first tensor, the reused physical block number of the historical intermediate segment, and the new physical block number of the last tensor into the slots of the virtual block table of the current inference request, according to the logical sequence. During the decoding execution module's operation, it uses this virtual block table to achieve logically continuous reading across non-contiguous physical memory blocks, thereby strictly ensuring zero data movement (zero-copy) of the intermediate segment tensor. The target tensor, after logical concatenation, is then injected into the KV cache structure of the current inference request to participate in the attention calculation of the subsequent autoregressive decoding stage.

[0074] In some other alternative implementations, the recalculation ratio threshold can be set based on a piecewise function: when At the beginning, take 16 tokens from the first and last 16 tokens; when Time-based linear interpolation ;when When a dynamic sliding window strategy is enabled, only the continuous token intervals affected by the offset of the coverage position are recalculated; or, the concatenation operation can be replaced by memory view-based concatenation to avoid explicit data copying and directly generate a composite tensor descriptor pointing to three consecutive video memory regions.

[0075] Through the above scheme, this embodiment can accurately calibrate the RoPE misalignment problem caused by positional offset while maintaining cross-context KV cache reuse. This ensures that the concatenated tensor meets the continuity constraint of rotational positional encoding in the sequence dimension, thereby guaranteeing the semantic consistency and instruction compliance of the model output. Compared with the prior art, this embodiment only introduces controllable local recalculation overhead at the beginning and end while reusing the intermediate long text KV cache, avoiding the computational redundancy caused by full prefill. Moreover, it does not require modification of the model architecture or training process and is fully compatible with the positional encoding implementation of mainstream LLM inference frameworks. In addition, the positional encoding offset calibration mechanism and tensor concatenation logic provided in this embodiment successfully overcome the technical bias that the intermediate segment cache cannot be reused due to the strong binding of rotational positional encoding (RoPE). This invention does not require full recalculation of the entire offset intermediate segment. It only needs to absorb the boundary attention mutation caused by positional deviation by truncating and recalculating the local word subsequences at the beginning and end, and seamlessly stitch the newly generated correct tensors at the beginning and end with the directly reused historical intermediate segment tensors in the logical sequence dimension.

[0076] In one specific implementation, based on the above embodiments, after the language model completes the autoregressive decoding calculation for the input sequence, the system first identifies the virtual address mapping entries associated with each logical block hash in the current logical block hash list; specifically, the context allocation module traverses the current logical block hash list. For each Locate the corresponding virtual address entry in the Prefix Hash Table (PHT) and perform an explicit cleanup operation: remove the entry from the PHT or set its value to a null pointer / invalid offset to unbind the logical hash from any physical memory block; then, the system calls the atomic operation interface atomicSub provided by the CUDA runtime to send an instruction to the underlying memory management pool to perform an atomic decrement operation on the reference counter maintained in the metadata structure of the historical physical key-value cache block; this operation is performed directly on the GPU device side to ensure the indivisibility and visibility of the counter update in a multi-threaded or multi-stream concurrent environment; next, the system synchronously updates the status field in the metadata of the physical block and marks it as "pending reclamation check"; this status field does not trigger immediate release, but only serves as a basis for subsequent judgment by the background monitoring thread.

[0077] In some other alternative implementations, the virtual address cleanup operation can adopt a delayed lazy cleanup strategy: instead of immediately deleting entries from the PHT, the corresponding logical hash entries are marked as "invalid" and cleaned up uniformly in the hash collision handling process of subsequent new requests; the atomic decrement operation can also be replaced by the atomic_fetch_sub instruction based on memory order semantics to be compatible with the memory consistency model requirements of different GPU architectures; the storage location of the reference counter can be located in the reserved metadata area at the head of the physical KV block, or it can reside independently in the global hash index structure of the video memory management pool, as long as its update and read are synchronized through the same atomic instruction set.

[0078] Through the above scheme, this embodiment can ensure that the logical hash alias mapping relationship is completely removed after the end of a single inference lifecycle, while making the reference state of the historical physical key-value cache block accurately reflect the actual usage, avoiding memory leakage or erroneous reclamation caused by the counter not being decremented in time. Compared with the prior art, this embodiment maintains the determinism and timing security of physical memory block lifecycle management in high-concurrency multi-agent scenarios, making the sharing and reuse of the same historical KV block by multiple inference requests both scalable and predictable, fundamentally eliminating system-level anomalies such as Use-After-Free or Double-Free caused by inaccurate reference counting.

[0079] In one specific implementation, the system is configured with a video memory usage monitoring module. This module periodically polls the hardware registers or driver interfaces exposed by the underlying video memory management pool to obtain the number of allocated bytes and the total capacity bytes of the current physical video memory (VRAM), and calculates the real-time physical video memory usage rate accordingly. When the usage rate continuously exceeds a preset safety threshold, a video memory reclamation process is triggered: First, the system traverses the physical key-value cache block metadata linked list maintained by the underlying video memory management pool, filtering out all physical blocks with a reference counter value of zero; then, for the set of filtered free physical blocks, a least recently used (LRU) sort is performed based on the most recently accessed timestamp recorded in their metadata; then, from the sorted set of free physical blocks, the least recently used (LRU) sort is performed; Starting from the head of the queue, the system sequentially calls the CUDA memory management API (such as cudaFree) or the BlockAllocator::free_block interface of the vLLM underlying layer to perform video memory release operations on the selected physical key-value cache blocks. At the same time, in parallel, the system accesses the prefix hash table (PHT) data structure, locates all key-value pairs with the offset of the reclaimed physical block as the value, and clears the mapping entries corresponding to the key (i.e., logical block hash). This clearing operation is completed through the hash table's native erase interface or a lock-free deletion mechanism based on atomic compare-and-swap (CAS), ensuring the consistency of the mapping relationship in multi-threaded concurrent reading of the PHT scenario.

[0080] In some other alternative implementations, the security threshold can be configured as a fixed percentage (e.g., 85%) or dynamically adjusted to segmented thresholds based on the GPU model (e.g., 82% for A100 and 87% for H100); the LRU policy can be replaced with the LFU (Least Frequently Used) policy, sorted according to the access frequency counter maintained in the metadata; the method for collecting physical memory usage can be replaced by querying GPU memory usage snapshots through the NVIDIA Management Library (NVML) or reading the memory status through the Linux kernel / sys / class / drm / card / device / mem_info node; the clearing operation of the mapping entries in the prefix hash table can also adopt a delayed cleanup mechanism: only the entries to be cleared are marked as "stale", and the background asynchronous cleanup thread performs the actual deletion in batches during low-load periods.

[0081] In addition, the physical memory reclamation mechanism can also adopt an expiration elimination strategy based on time-to-live (TTL). The system allocates a dynamic lifecycle countdown for each historical physical key-value cache block, and automatically strips its logical alias mapping and releases the physical memory after the timeout.

[0082] Through the above scheme, this embodiment can achieve proactive and state-aware reclamation of physical memory resources, ensuring the stability of high-concurrency inference services while avoiding invalid address references or memory leaks caused by mapping residues. Compared with the prior art, this embodiment superimposes memory level-driven LRU eviction logic on the basis of the reference counting mechanism, so that the life cycle management of physical KV blocks can simultaneously meet the requirements of security (zero reference count is a prerequisite for reclamation) and efficiency (prioritizing the release of cold data), thereby maintaining a dynamic balance of memory occupancy in high-frequency reuse scenarios of multiple agents and significantly reducing the request failure rate caused by memory overflow (OOM).

[0083] In one specific implementation, based on the above embodiments, after completing the autoregressive decoding calculation and generating the output sequence, the system performs a segmentation operation on the output sequence according to the agent role identifier: First, it identifies the agent role type associated with the current inference request. This role type is indicated by predefined role tags carried in the request metadata, including but not limited to "code reviewer", "test generator", "document digester" or "vulnerability analyst". Then, for each role type, a segmentation strategy matching its semantic output pattern is pre-configured. For example, for "code reviewer", the system extracts substrings marked with as boundaries in the output sequence as independent target product fragments. For "test generator", it performs syntax-aware segmentation based on the test_case: prefix and its subsequent indented block structure. Specifically, the segmentation process calls a lightweight rule parser, which does not rely on the construction of a complete syntax tree, but achieves low-latency boundary recognition based on a regular expression engine and a context-sensitive state machine.

[0084] In some other alternative implementations, the segmentation operation is performed based on the boundaries of the tool call instruction: the system detects string patterns in the output sequence that conform to a preset tool call protocol format. This pattern contains structured fields such as tool_name, arguments, and output, and is enclosed in standardized delimiters (such as --- TOOL CALL ---). When such a pattern is detected, it is treated as a single target product segment, and the boundary of this segment is determined by the position of the leading whitespace character before the starting delimiter and the position of the trailing newline character after the ending delimiter. Alternatively, the segmentation operation is performed based on a maximum word length threshold: the system inputs the output sequence into a word segmenter consistent with the language model to obtain a continuous word sequence. Then, using a preset maximum word length threshold as the sliding window size, non-overlapping segmentation is performed along the sequence direction. When the number of remaining words is less than the threshold, the remaining part is merged into the previous window or used to form a separate end segment. The threshold is configured to any integer value among 256, 512, or 1024 and supports dynamic loading at runtime. In other equivalent implementations, the boundaries of the truncation rule can also be based on: the natural language syntax dimension, such as using punctuation marks like newlines and periods as boundaries; the structured document dimension, such as using the various levels of heading tags in Markdown format as boundaries; and the abstract syntax tree (AST) dimension, where, in code generation tasks, the scope terminator of function definition closures or class definitions is used as the truncation boundary.

[0085] Through the above scheme, this embodiment can achieve adaptive segmentation of heterogeneous output forms of multiple agents. Without introducing additional model inference overhead, it can accurately separate semantically coherent and functionally complete sub-units into target product fragments, thereby ensuring the semantic granularity consistency and reuse effectiveness of the fragment index table written by the subsequent working memory sedimentation module.

[0086] In one specific implementation, based on the above embodiments, after the language model completes the autoregressive decoding calculation for the current inference request and generates an output sequence, the working memory accumulation module first segments the output sequence according to a preset truncation rule to determine the target product fragment. The truncation rule includes at least one of the following: identifying semantic boundaries based on agent role identifiers, dividing functional units based on start and end markers of tool call instructions, or performing equal-length segmentation of continuous text based on the maximum word length threshold. Specifically, the system calls the metadata interface of the underlying memory management pool to obtain the address information of the new physical key-value cache block allocated to the target product fragment during the inference process and its corresponding new logical block hash list, wherein the new logical block hash list is aligned by the inference engine according to a fixed block size during the pre-filling stage. The system then generates and records the initial logical index of each block in the prefix hash table. Next, based on the original text content of the target product fragment or its normalized lexical sequence, the system calculates a new fragment hash value. The new fragment hash value is generated using the SHA-256 algorithm, and optionally performs whitespace removal, unified punctuation, and conversion to lowercase normalization before calculation. Then, the new fragment hash value is used as the key and the new logical block hash list is used as the value, and written into the fragment index table to establish a new mapping relationship. In some other optional implementations, the new fragment hash value can be replaced with a multi-level hash combination, including the concatenation of fragment content hash and task domain identifier hash. The new logical block hash list can also be expanded into an enhanced index structure containing block version numbers or timestamps to support subsequent version-based backtracking or conflict detection.

[0087] Through the above scheme, this embodiment can persist the effective semantic output generated by the agent in a single inference as an index item that can be reused across contexts and requests, so that the collaborative working memory continues to grow with the interaction of multiple agents; at the same time, because it only stores hashes and logical block indexes and does not copy physical KV tensors, this mechanism realizes the ability to share the underlying tensor state across agents without significantly increasing memory overhead.

[0088] In one specific implementation, based on the above embodiments, each mapping record in the fragment index table, in addition to containing the fragment hash value and the corresponding historical logical block hash list, is also bound to at least one permission identifier field. This permission identifier field is configured to store either a task domain isolation identifier or a tenant permission identifier, where the task domain isolation identifier is a unique task instance ID generated by the workflow scheduler, and the tenant permission identifier is a hexadecimal tenant code assigned by the multi-tenant SaaS platform. During the write operation, the system first extracts the target permission identifier from the context metadata of the current inference request, and then serializes the target permission identifier, along with the fragment hash value and the historical logical block hash list, into a structured key-value pair, which is then written to the persistent storage area of ​​the fragment index table. This structured key-value pair uses a nested dictionary format, with the top-level key being the fragment hash value, and subkeys including logical_block_hashes, tenant_id, and task_id. The tenant_id and task_id fields are set to empty strings when the corresponding isolation strategy is not enabled and do not participate in subsequent matching decisions.

[0089] Specifically, during the query phase, after the index addressing module receives the fragment hash value of the candidate reuse fragment, it first performs a hash lookup to obtain the original record item. Then, the system parses the permission identifier field bound to the record item and extracts the target permission identifier carried by the current inference request. Then, it performs a comparison according to the preset permission matching strategy: if tenant-level isolation is enabled, the corresponding historical logical block hash list is only allowed to be returned if the tenant_id field in the record item is not empty and is exactly equal to the target tenant code; if task domain isolation is enabled, the subsequent address redirection process is only triggered if the task_id field in the record item is not empty and is strictly consistent with the current task instance ID; if neither is enabled, the permission field is ignored and the default no-isolation mode is used. All comparison operations are completed in memory in constant time, without introducing additional I / O latency.

[0090] In other alternative implementations, the permission identifier field can be extended to support multi-level nested identifiers, such as a composite permission tag (tenant_id:task_id:agent_role); or an attribute-based access control (ABAC) model can be adopted, replacing the permission identifier with a set of Boolean policy expressions, such as "is_internal_tenant has_read_permission"; or in a distributed deployment scenario, the permission identifier can be jointly validated with a node affinity tag (node_affinity_tag) to achieve fine-grained cache visibility control across physical nodes.

[0091] In scenarios involving multi-user collaborative dialogue or group agent interaction, the permission identifier field can also be configured as a session group ID, allowing candidate reuse fragments to be shared only to a limited extent among agent requests belonging to the same virtual session space.

[0092] Through the above scheme, this embodiment can achieve precise access control of semantic fragment-level KV cache reuse behavior under the shared fragment index table architecture, ensuring that inference requests from different task domains or different tenants are logically invisible to each other, eliminating the risk of cross-domain cache leakage from the underlying data structure level, while maintaining low overhead and high concurrency security of the index query path.

[0093] In one specific implementation, each mapping record stored in the fragment index table is associated with at least one permission identifier, which is a structured string containing at least one of a task domain isolation identifier or a tenant permission identifier. When the system performs a query operation, it first extracts the target permission identifier from the metadata of the current inference request. This metadata is injected by the inference service framework during the request initialization phase and passed to the index addressing module via the context propagation mechanism. Then, the index addressing module compares the extracted target permission identifier with the permission identifier bound to the corresponding hash key in the fragment index table field by field. Only if the two are completely consistent is the historical logical block hash list associated with the hash key be allowed to be returned. If the comparison is inconsistent, it is directly determined as a cache miss, and subsequent address redirection operations are not triggered. Specifically, the permission identifier uses a fixed format encoding, such as "tenant_id:abc123;task_id:plan_v2", with semicolons as field separators and colons separating field names and values. The indexing and addressing module strictly verifies the existence, order, and literal equality of field values ​​during the comparison process, without performing fuzzy matching or wildcard expansion. In other optional implementations, the permission identifier can be encoded using a binary bitmask, where different bits represent access control dimensions such as tenant affiliation, task lifecycle stage, and agent role type; alternatively, the permission identifier can be issued with a digital signature, and the indexing and addressing module verifies the signature validity before decoding and verifying the permission content.

[0094] Through the above scheme, this embodiment can achieve fine-grained access control of fragment-level KV cache reuse paths. While sharing physical memory resources, it ensures that semantic fragment indexes between different tenants or different task domains are invisible and cannot be cross-referenced, thereby achieving dual protection of logical isolation and data security at the underlying memory scheduling level. Compared with the prior art, this embodiment avoids the risk of cross-domain data leakage caused by multiple tenants sharing the same fragment index table, and can complete permission adjudication without additional copying of KV cache entities, maintaining the design goal of low-overhead index management.

[0095] In one specific implementation, based on the above embodiments, the process of calculating the fragment hash value of the candidate reuse fragment and querying the fragment hash value in the fragment index table includes the following operations: First, perform Local Sensitive Hash (LSH) calculation on the extracted candidate reuse fragment, specifically using the MinHash algorithm, and then normalize and segment the fragment to obtain the word sequence. The process involves mapping a fixed-length signature vector. Then, based on this MinHash signature vector, a hash bucket index lookup is performed in the memory-resident fragment index table to locate the set of candidate mapping items within the corresponding hash bucket. Next, for each candidate mapping item, the Jaccard similarity between its MinHash signature and the current fragment's MinHash signature is calculated. If the similarity of any candidate mapping item exceeds a preset threshold of 0.95, the complete word sequence of the historical fragment associated with that candidate mapping item is extracted and compared element-by-element with the complete word sequence of the current candidate reused fragment. Subsequently, SHA-256 hash values ​​are calculated for both, and if the hash values ​​are completely identical, the match is confirmed, thus obtaining the hash list of the historical logical blocks bound to that mapping item.

[0096] In other alternative implementations, Locality Sensitive Hash (LSH) can be replaced with SimHash or Top-K N-gram Hash; the similarity threshold can be dynamically adjusted according to the task scenario, with a range of 0.85 to 0.98; the hash algorithm used for secondary precise comparison can be replaced with BLAKE3 or MD5, provided that its output length and collision probability meet the deterministic and security requirements of the memory index service; provided that the fragment index table supports a multi-level key structure, LSH can be combined with tenant identifier and task domain identifier to form a composite hash key to reduce the size of the candidate set for approximate matching.

[0097] Through the above solution, this embodiment can significantly reduce the cache miss rate caused by differences in text format, such as the addition or deletion of spaces, changes in line breaks, and variations in punctuation, while maintaining millisecond-level query latency, thus balancing high-throughput retrieval efficiency with strong consistency semantic matching capabilities. Compared with existing technologies that rely on linear traversal or full string comparison using a single precise hash, this embodiment reduces the time complexity of fragment matching from... Optimize to average Furthermore, it maintains a precision rate of over 99% even with an index size of tens of millions of fragments.

[0098] The key-value cache scheduling method of this invention is not a purely mathematical algorithm, but a system-level improvement that deeply impacts the computer's video memory architecture. By performing pointer tampering from logical address to physical address in the underlying prefix hash table, it effectively blocks the massive physical electrical signal read / write operations of high-dimensional tensor data between computing units such as SRAM and high-bandwidth video memory HBM, greatly reducing the system's data bus occupancy and dynamic hardware power consumption.

[0099] like Figure 2 As shown, this embodiment of the invention also provides a key-value cache scheduling method and system for language model inference, used to implement the above method. The system includes:

[0100] The input parsing module M100 is configured to receive the input sequence corresponding to the current inference request and extract one or more candidate reuse segments from the input sequence according to preset boundary recognition rules. The boundary recognition rules include matching based on explicit semantic tags, segmentation based on message role identifiers, truncation based on tool call command boundaries, or segmentation based on a fixed-length sliding window. In one embodiment, the input parsing module M100 scans the input sequence using a regular expression engine. <code>or <plan>Pairs of closing tags are used to identify the text substrings wrapped by the tags as candidate reusable segments; in some other optional embodiments, the input parsing module M100 directly locates and extracts the start and end offsets of the annotated semantic segments in response to the structured metadata fields injected by the middleware. In an application scenario oriented to Retrieval-Augmented Generation (RAG), the boundary recognition rule can also be configured to intercept according to the chunk boundaries of the retrieved documents. The input parsing module can directly identify each independently recalled reference text paragraph as a candidate reusable segment based on the document chunk metadata returned by the external knowledge base retrieval, so as to achieve zero-copy sharing of the external knowledge key-value cache when querying the same factual document in multiple rounds or by multiple agents.

[0101] The index addressing module M200 is configured to perform a hash calculation on each candidate reusable segment and, based on the calculated segment hash value, perform a query operation in the segment index table to obtain the historical logical block hash list associated with the segment hash value; wherein, the segment index table is implemented as a hash map structure resident in the host memory or the GPU unified virtual address space, with the normalized segment hash value as the key and an ordered list composed of one or more logical block hashes as the value; in one embodiment, the index addressing module M200 first performs text normalization processing on the candidate reusable segment, including removing leading and trailing and consecutive whitespace characters, unifying the forms of Chinese and English punctuation, converting to lowercase, and converting it into a sequence of tokens by calling the tokenizer配套 with the language model, and then generates the segment hash value using the SHA-256 algorithm; in some other optional embodiments, the index addressing module M200 uses the locality-sensitive hashing algorithm to generate approximate hash values, first performs a fast bucket positioning, and then performs a secondary exact hash comparison on the candidate set to complete the anti-collision verification.

[0102] The context allocation module M300 is configured to dynamically generate a set of current logical block hashes for each candidate reusable segment during the prefill calculation stage of the language model, according to the overall context structure of the current input sequence, the allocated prefix length, and the KV cache block size constraint; wherein, the length of the current logical block hash list is jointly determined by the number of tokens covered by the candidate reusable segment and the KV block capacity parameter adopted by the inference engine; in one embodiment, the context allocation module M300 inherits from the BlockManagerV1 of the vLLM framework and allocates sequentially numbered logical block hashes for the segment according to the total sequence length of the current request and the number of occupied blocks; in some other optional embodiments, the context allocation module M300 supports a sparse block allocation strategy, allowing non-consecutive logical block hash indexes to point to the same physical cache block to adapt to irregular semantic segment distributions.

[0103] The address redirection module M400 is configured to access a prefix hash table and, for each logical block hash in the current logical block hash list, update the physical address pointer of its corresponding entry in the prefix hash table to the physical address pointed to by the logical block hash of the same position in the historical logical block hash list. The prefix hash table is implemented as a one-dimensional array or hash table structure directly addressable by the GPU, with each entry storing a 64-bit offset pointing to the starting address of the physical key-value cache block. In one embodiment, the address redirection module M400 traverses the index pairs between the current logical block hash list and the historical logical block hash list. And perform an atomic write operation: PHT[g_i] = PHT[h_i], thereby establishing a many-to-one alias mapping relationship in the logical namespace; in some other optional implementations, the address redirection module M400 updates the PHT entries in batches before the pre-filled Kernel starts by injecting CUDA Graph nodes, so as to reduce host-device synchronization overhead.

[0104] The decoding execution module M500 is configured to, when calling the attention operator during the autoregressive decoding stage, read the corresponding historical physical key-value cache block data from the underlying video memory based on the updated physical address pointer in the prefix hash table, and use this data as the Key and Value input for the current attention calculation to participate in the Softmax weighted summation operation. The decoding execution module M500 is compatible with the native attention instruction sets of PagedAttention, FlashAttention, or Hopper architectures. In one embodiment, the decoding execution module M500 is implemented based on vLLM's AttentionWrapper, transparently using the redirected PHT entry in the BlockTable lookup path without modifying the Attention Kernel source code. In other optional implementations, the decoding execution module M500 directly loads the physical address mapped by the PHT through LDG.64 instructions, bypassing the traditional page table traversal path and further reducing address resolution latency.

[0105] The aforementioned modules work together to form a closed-loop key-value cache scheduling mechanism: the input parsing module M100 identifies text units with independent semantics, the index addressing module M200 locates the logical index entry of the existing computation state, the context allocation module M300 constructs an adapted logical address view for it in the new context, the address redirection module M400 binds this view to historical physical resources through underlying pointer overwriting, and finally the decoding execution module M500 naturally consumes this shared state in the standard attention process. This cooperative approach breaks the rigid dependence of traditional prefix caching on the start position of the sequence, allowing repeated semantic fragments appearing at any position to trigger zero-copy reuse across contexts; at the same time, since all operations occur at the logical address mapping layer rather than the physical data transport layer, it avoids memory bandwidth bottlenecks and redundant GEMM computations, thereby substantially alleviating the problem of repeated pre-filling caused by prefix divergence in multi-agent heterogeneous prompting scenarios, and solving the technical defects of existing technologies such as coarse reuse granularity, serious memory waste, and lack of concurrency security.

[0106] Through the above scheme, this embodiment can achieve fine-grained, cross-context, and low-overhead scheduling of key-value cache resources during language model inference. Without changing the model structure and training state, it significantly reduces first-word latency (TTFT), compresses peak physical memory usage, and supports the sustainable growth of working memory sharing among multiple agents. Compared with the prior art, the beneficial effects of this embodiment are: while maintaining compatibility with mainstream inference frameworks such as vLLM and TGI, it moves the cache reuse unit down from "complete prefix sequence" to "semantic fragment", and uses a logical hash alias mechanism to advance the reuse decision to the pre-filling stage, enabling the system to achieve synergistic optimization of computational efficiency and memory efficiency without sacrificing generation quality and operational robustness.

[0107] In one specific implementation, the tensor alignment module is configured as a separate hardware-accelerated coprocessor submodule, integrated between the CUDA kernel scheduling layer of the inference engine and the attention operator execution path. This module includes a positional encoding difference detection unit, a boundary truncation control unit, a local pre-padding computation unit, and a tensor splicing scheduling unit. The positional encoding difference detection unit receives a completion signal from the address redirection module M400 and synchronously reads the current context's starting absolute position index. relative to the absolute position index of the historical context Calculate the difference between the two. ;when When a rotational position encoding (RoPE) offset is detected, the subsequent calibration process is triggered; the boundary truncation control unit performs a recalculation based on a preset recalculation ratio threshold function. Determine the start and end cut-off lengths: For lengths of... The candidate reuse fragment, its head truncation length Length of tail section The local pre-filled computation unit calls the model's native Attention forward operator, shifting at the current context position. Below, independent KV cache block allocation and RoPE injection-style pre-filling are performed only on the first and last word subsequences to generate sequences carrying the correct positional encoding. and The tensor splicing scheduling unit extracts all KV tensor slices corresponding to the intermediate segments from the historical physical key-value cache blocks in block order. It then calls the underlying tensor concatenation instruction along the sequence dimension (dim=1) in the GPU memory to construct the complete target KV tensor. and The concatenated result is then registered as an effective KV cache source for the next stage of attention calculation by the decoding execution module M500.

[0108] In some alternative implementations, the tensor alignment module can be implemented as a lightweight kernel function running in the GPU stream (CUDAStream), dynamically loaded and embedded into the vLLM or Triton custom Attention operator; the scaling threshold function is recalculated. It can be replaced with a piecewise linear function, for example, when hour ,when hour ,when hour The first and last truncation can also be replaced by a single-end recalculation mode, that is, only recalculating the first part. Each word element, or only the tail Local pre-filling is performed on individual lexical units, while the tail and middle segments are read uniformly from the historical cache; tensor concatenation operations can also be completed by the cross-block view mechanism directly supported by PagedAttention's Block Manager, without the need for explicit memory copying or temporary buffer allocation.

[0109] It's important to note that tensor concatenation is not a forced data copy in physical memory, such as the full concatenation operation of torch.cat that triggers memory reallocation. Instead, it's concatenation based on a virtual logical view. At the underlying hardware implementation level, the system updates the virtual block table for the current inference request, sequentially filling the slots in the virtual block table with the new physical block number of the first tensor, the reused physical block number of the historical intermediate segment, and the new physical block number of the last tensor, in logical order. During computation, the decoding execution module uses this virtual block table to achieve logically continuous reading across non-contiguous physical memory blocks, thus strictly guaranteeing zero data movement (zero-copy) of the intermediate tensors.

[0110] Through the above scheme, this embodiment can achieve RoPE position alignment of cross-context semantic fragment KV cache with controllable local recalculation overhead without disrupting the existing attention calculation process; while maintaining the stability of model output quality, it significantly reduces the risk of generation deviation caused by position encoding misalignment, and makes zero-copy reuse of intermediate long text segments a feasible system-level capability in engineering.

[0111] The logical hash alias mapping mechanism and the Rotated Position Encoding (RoPE) dynamic calibration mechanism in this embodiment are not simply an overlay of existing technologies, but rather a highly interconnected and organic combination. While simple physical address alias mapping can achieve memory reuse, it can lead to fatal positional encoding semantic errors in large autoregressive models. On the other hand, simple local positional calibration, if separated from the redirection of underlying physical pointers, requires high costs for copying and allocating video memory. This invention, through underlying pointer hijacking combined with local boundary term recalculation and tensor concatenation, forms a closed loop at the computer system level. This synergistic effect produces an unexpected technical effect that achieves both zero-copy video memory and ensures strict alignment of positional semantics in the attention mechanism of large models, fundamentally overcoming the dual bottlenecks of computation and storage when reusing across contexts.

[0112] In one specific implementation, the memory concurrency locking module is implemented as an independent structural unit that works in conjunction with the address redirection module M400 and the decoding execution module M500. Its physical deployment location is in the intermediate abstraction layer between the GPU driver layer and the inference framework memory manager. This module includes a reference counting management submodule, an atomic instruction scheduling submodule, and a garbage collection strategy engine submodule. The reference counting management submodule maintains a metadata structure corresponding one-to-one with each historical physical key-value cache block. This metadata structure resides in a read-only resident page in the GPU's unified virtual address space and contains a 64-bit unsigned integer field as a reference. The atomic instruction scheduling submodule submits a cudaAtomicAdd instruction to the CUDA runtime environment simultaneously with the address redirection module M400 completing the physical address overwrite operation for any current logical block hash in the prefix hash table. The target address of this instruction is the reference counter field in the metadata structure of the corresponding historical physical key-value cache block, and the operand is 1. This instruction ensures that concurrent access to the metadata of the same cache block satisfies the sequential consistency constraint at the GPU hardware level. After the decoding execution module M500 completes the autoregressive decoding calculation and triggers the request context destruction event, the atomic instruction scheduling submodule synchronously submits a cudaAtomicAdd instruction. The `udaAtomicSub` instruction decrements the same reference counter by 1. The reclamation strategy engine submodule continuously monitors the memory usage event stream exposed by the underlying memory management pool. When it receives a high-water mark alarm signal indicating that the usage exceeds 85%, it starts a scanning task, traverses all physical key-value cache block metadata, filters out entries with a reference counter value of 0, selects target blocks based on the Least Recently Used (LRU) timestamp field, then calls `cudaFreeAsync` to asynchronously release the associated memory pages, and sends a mapping cleanup notification to the address redirection module M400 to synchronously clear the corresponding historical logic in the prefix hash table. Block hash index; in some other optional implementations, the reference counter field can adopt a CAS (Compare-and-Swap) structure with version numbers to support finer-grained concurrency control; the atomic instruction scheduling submodule can be replaced with a cross-GPU device-level reference counting broadcast mechanism based on NVLink interconnect for distributed multi-GPU inference scenarios; the triggering condition of the reclamation policy engine submodule can be replaced with an active eviction policy based on workflow semantic awareness, that is, when it is detected that the semantic fragment carried by a certain historical physical key-value cache block has been marked as "expired task product", the reference count status is ignored and it is directly included in the reclamation candidate set.

[0113] Through the above scheme, this embodiment can ensure that when multiple inference requests share the same historical physical key-value cache block, their lifecycle is strictly controlled by the atomic increment and decrement operations of the reference counter, fundamentally eliminating Use-After-Free or Dirty Write errors caused by asynchronous erasure or concurrent writing of the background garbage collection thread; at the same time, the coupled design of the garbage collection strategy engine and the memory usage monitoring makes the release of physical memory resources both meet the premise of security and have dynamic response capability, avoiding the problem of decreased memory utilization caused by static pre-allocation or conservative delayed garbage collection.

[0114] In one specific implementation, based on the above embodiments, the working memory deposition module is configured to, in response to the decoding execution module M500 completing the autoregressive decoding calculation of the current inference request and outputting the final sequence, capture the complete content of the output sequence; the module further performs segmentation operation on the output sequence according to preset truncation rules to generate one or more target product fragments; the truncation rules include at least one of semantic boundary recognition based on agent role identification, structured segmentation based on tool call instruction tags, and forced truncation based on the maximum word length threshold; after segmentation, the working memory deposition module calls the underlying video memory management interface to query the KV cache covered by each target product fragment. A new logical block hash list corresponding to the region is generated. This list is allocated and registered to the prefix hash table by the inference engine during the pre-filling phase. Simultaneously, the module calls the hash calculation unit to perform deterministic hashing on the text content of each target product fragment or its corresponding lexical sequence, generating a unique new fragment hash value. Subsequently, the module persistently stores the new fragment hash value as the key and the corresponding new logical block hash list as the value in the fragment index table. This write operation establishes a new mapping entry in the fragment index table, enabling the target product fragment to be retrieved by other contexts in subsequent inference requests and triggering logical hash alias mapping, thereby achieving continuous accumulation and reuse of the underlying tensor state across requests and agents. In other optional implementations, the truncation rule can be replaced by automatically dividing fragment boundaries based on message role tags such as "assistant" and "tool_response"; or by using a dynamic sliding window combined with a semantic coherence scoring model to select the optimal segmentation point; or by preloading differentiated segmentation strategy templates according to task type and performing strategy matching at runtime based on the task domain identifier of the current request.

[0115] Through the above scheme, this embodiment can automatically precipitate the effective semantic output generated by each agent in the multi-agent system into indexable and reusable underlying tensor states at the fragment granularity, forming a collaborative working memory that expands continuously as the task evolves. This mechanism does not rely on explicit manual annotation, is compatible with the KV block lifecycle management logic of mainstream inference frameworks, and supports cross-context and cross-agent key-value cache sharing without increasing redundant copying of physical memory, thereby improving the overall system throughput efficiency and state consistency.

[0116] To verify the technical effectiveness of this invention, a reuse test was conducted on an intermediate shared segment of 8,192 tokens in a performance benchmark test scenario for multi-agent collaborative reasoning. Experimental data shows that the traditional baseline scheme based on strict prefix matching requires full recalculation due to prefix mismatches, resulting in a first-word generation latency (TTFT, Time-To-First-Token) of approximately 818.89 milliseconds. However, by employing the segmented key-value caching scheduling method of this embodiment, the pre-filled matrix multiplication calculation of these 8,000 tokens is successfully skipped through a logical hash alias mechanism, significantly reducing the first-word generation latency to approximately 114.76 milliseconds. These experimental results objectively demonstrate that this system can substantially eliminate the bottleneck of redundant computation in multi-agent long-context interaction scenarios.

[0117] The technical solution disclosed in this invention utilizes the local isomorphism of semantic segments in natural language sequences and directly applies this pattern to the memory scheduling hardware in computer architecture. Specifically, the key-value cache scheduling algorithm alters the addressing path of the graphics processing unit (GPU) or tensor processing unit (TPU) memory controller by forcibly overwriting the physical address offset in the prefix hash table during the pre-filling phase. This algorithm-level reconstruction effectively blocks redundant physical electrical signal transmission between computing units (such as SRAM) and high-bandwidth memory (HBM) for high-dimensional tensor data, significantly reducing the dynamic power consumption and data bus occupancy of the underlying hardware. This demonstrates the system-level technical architecture improvement brought about by the close integration of the algorithm and the underlying hardware.< / plan> < / code>

Claims

1. A key-value cache scheduling method for language model inference, applied to computer devices, characterized in that, include: Obtain the input sequence of the current inference request, and extract candidate reuse fragments from the input sequence; Calculate the fragment hash value of the candidate reuse fragment, query the fragment hash value in the fragment index table, and obtain the historical logical block hash list corresponding to the fragment hash value; wherein, the fragment index table records the mapping relationship between fragment hash values ​​with independent semantics and the corresponding logical block hash list; During the pre-filling computation of the language model, a corresponding current logical block hash list is generated for the candidate reuse fragments in the current context of the input sequence; In the prefix hash table, the physical address pointer corresponding to the current logical block hash list is updated to the physical address of the historical physical key-value cache block pointed to by the historical logical block hash list, so as to establish a logical hash alias mapping; Based on the updated physical address pointer in the prefix hash table, key-value cache data corresponding to the historical physical key-value cache block in the underlying video memory is scheduled so that the key-value cache data participates in the autoregressive decoding calculation of the language model.

2. The method according to claim 1, characterized in that, The step of extracting candidate multiplexed segments from the input sequence includes: Text substrings are extracted by parsing the explicit boundary labels carried in the input sequence, and the text substrings are used as candidate reuse fragments; Alternatively, the upstream output text sequence referenced in the input sequence can be identified according to the workflow default rules, and the upstream output text sequence can be used as the candidate reuse fragment.

3. The method according to claim 1, characterized in that, Before calculating the fragment hash value of the candidate multiplexed fragment, the process includes: The candidate reuse fragments are subjected to text normalization processing, wherein the text normalization processing includes at least one of the following operations: removing whitespace characters, unifying punctuation marks, and converting capitalization. Alternatively, the candidate reuse fragment can be converted into a word sequence by the word segmenter of the language model, and the hash value of the fragment can be calculated based on the word sequence.

4. The method according to claim 1, characterized in that, The step of updating the physical address pointer corresponding to the current logical block hash list to the physical address of the historical physical key-value cache block pointed to by the historical logical block hash list includes: Iterate through the current logical block hashes in the current logical block hash list; Replace the key-value pair indexed by the current logical block hash in the prefix hash table with the offset of the historical logical block hash in the underlying video memory at the corresponding sequential position in the historical logical block hash list.

5. The method according to claim 1, characterized in that, After establishing the logical hash alias mapping, the following is included: Compare the current position encoding information of the candidate multiplexed segment in the current context with the historical position encoding information in the historical context; If the current location encoding information is inconsistent with the historical location encoding information, a location encoding offset calibration mechanism is triggered.

6. The method according to claim 1, characterized in that, When updating the physical address pointer corresponding to the current logical block hash list to the physical address of the historical physical key-value cache block pointed to by the historical logical block hash list, the following is included: In the underlying memory management pool of the language model, an atomic increment operation is performed on the reference counter corresponding to the historical physical key-value cache block to intercept the asynchronous release operation and dirty write operation performed by concurrent inference requests on the historical physical key-value cache block.

7. The method according to claim 1, characterized in that, After completing the autoregressive decoding calculation and generating the output sequence, the process includes: The output sequence is segmented into target product fragments based on one of the following indicators: agent role identifier, tool call instruction boundary, and maximum word length threshold.

8. The method according to claim 1, characterized in that, The mapping relationship in the fragment index table is bound to at least one of the following permission identifiers: task domain isolation identifier and tenant permission identifier.

9. The method according to claim 1, characterized in that, The step of calculating the fragment hash value of the candidate reuse fragment and querying the fragment hash value in the fragment index table includes: Calculate the local sensitivity hash value of the candidate reuse segment; perform an approximate matching query in the segment index table based on the local sensitivity hash value to obtain the target mapping item; If the similarity of the target mapping items exceeds a preset threshold, extract the complete sequence features of the candidate reuse fragment and perform a second precise hash anti-collision comparison operation; If the second precise hash anti-collision comparison operation passes the verification, the historical logical block hash list corresponding to the fragment hash value is obtained.

10. A key-value cache scheduling system for language model inference, characterized in that, include: The input parsing module is configured to obtain the input sequence of the current inference request and extract candidate reuse fragments from the input sequence; The index addressing module is configured to calculate the fragment hash value of the candidate reuse fragment, query the fragment hash value in the fragment index table, and obtain the historical logical block hash list corresponding to the fragment hash value; wherein, the fragment index table records the mapping relationship between fragment hash values ​​with independent semantics and the corresponding logical block hash list; The context allocation module is configured to generate a corresponding current logical block hash list for the candidate reuse fragments based on the current context of the input sequence during the pre-filling calculation of the language model. The address redirection module is configured to update the physical address pointer corresponding to the current logical block hash list in the prefix hash table to the physical address of the historical physical key-value cache block pointed to by the historical logical block hash list, so as to establish a logical hash alias mapping. The decoding execution module is configured to schedule key-value cache data corresponding to the historical physical key-value cache block in the underlying video memory based on the updated physical address pointer in the prefix hash table, so that the key-value cache data participates in the autoregressive decoding calculation of the language model.

Images