Shared context-based large language model inference cache apparatus and method

Abstract

The application discloses a large language model inference cache device and method based on shared context. It includes: a cache packaging module encapsulates shared fragments into cache units carrying quality records; a multiplexing and quality verification module first determines lossless multiplexing or approximate multiplexing candidates with a prefix state summary, and then measures the relative integrity of the multiplexing results at the sampling probe position before multiplexing, and calculates the top-k sorting consistency rate and KL divergence. If it does not meet the standard, it will fall back to complete forward calculation and write the measured divergence back to the quality record; a quality-aware management module calculates the retention priority according to the access heat and the quality risk derived from the quality record, and eliminates it accordingly, and feeds back the fallback and divergence to the update of the quality risk. The application guarantees the output quality while accelerating the multiplexing, manages the cache based on quality, and maintains consistency, thereby improving the stability and resource utilization efficiency of inference acceleration.

Description

Technical Field

[0001] This invention relates to the field of large language model inference acceleration and key-value caching management technology, specifically to a large language model inference acceleration caching device and method based on shared context. Background Technology

[0002] Large language models typically use an autoregressive approach to generate output. During inference, the model needs to calculate the key and value tensors in the attention mechanism for the tokens that have already been processed, and save the calculation results as a key-value cache for use when generating subsequent tokens, thereby avoiding redundant calculations for existing tokens.

[0003] In applications that include system prompts, search documents, few-shot examples, and multi-turn dialogue history, there is often a large amount of repetitive or shareable context between different inference requests. As the context length increases, the computational load required in the model pre-filling stage increases accordingly, and the GPU memory space occupied by the key-value cache also increases. For long context requests or high-concurrency inference scenarios, repeatedly performing pre-filling calculations increases the latency of the first token generation and consumes more computational and GPU memory resources.

[0004] To reduce redundant calculations, existing technologies typically cache already calculated key-value pairs for use in subsequent requests. Some solutions organize the cache according to the token prefix of the request, reusing the corresponding key-value cache when the current request and a historical request have the same prefix; some solutions encapsulate the key-value cache as a cache object that can be stored, retrieved, migrated, or unloaded to support the use of key-value cache across requests or inference instances; some solutions also attempt to reuse key-value caches that do not form a complete identical prefix through fragment matching, position correction, partial recalculation, or hierarchical recalculation.

[0005] Cache reuse based on the same prefix can guarantee that the reused result is consistent with the original calculation result, but its applicable conditions are relatively strict. When the same shared segment is located after different causal contexts, its position in the input sequence changes, or only part of the context is the same between requests, the existing cache usually cannot be directly reused, and the range of speedup that can be obtained is limited.

[0006] Using fragment reuse, position correction, or approximate reuse can expand the applicability of caching. However, the key-value tensor is not only related to the token sequence of the fragment itself, but also affected by factors such as the context preceding the fragment, token position, model parameters, word segmentation method, and caching precision. Even if the token sequence of the fragment to be reused is the same, its actual computation state in different requests may differ. Directly using an existing cache may cause the attention calculation result to deviate from the complete forward calculation result, and further affect the token distribution and output content generated by the model.

[0007] Existing solutions typically determine whether a cache can be reused based on token matching, feature similarity, positional differences, or preset recalculation rules. These judgments primarily reflect the similarity between input or intermediate features and cannot directly indicate whether a reuse will substantially affect the model output of the current request. For the same cache unit, the error generated by reuse may differ across different causal contexts, model layers, or generation tasks. Existing technologies lack a mechanism to assess the reliability of reuse results for the current request, and it is difficult to switch to full computation in a timely manner when the reuse result is unreliable, thus making it difficult to control the output quality risks caused by approximate reuse.

[0008] The quality differences in cache reuse also affect cache management. Existing cache eviction policies typically determine the retention order of cache units based on access frequency, recent access time, cache length, or expected hit probability. These metrics reflect the likelihood of a cache unit being accessed again, but they do not reflect the impact of the cache unit's reuse on the model's output quality. Cache units that are frequently accessed but prone to significant computational bias may occupy cache space for a long time, while cache units that are accessed less frequently but have stable reuse results may be evicted prematurely. This results in limited cache space not being prioritized for reliable cached data with actual acceleration value.

[0009] The effectiveness of key-value caching also depends on the input content and model runtime environment at the time the cache was generated. When historical input is edited, system suggestions, search results, or tool returns change, or model weights, tokenizers, and related inference parameters are updated, the original cache may no longer match the current request. In a multi-replica deployment environment, the cache state stored by different inference instances may also differ. If cache changes or expiration states are not promptly propagated, inference instances may continue to use expired or inconsistent cache versions, resulting in output errors or inconsistencies in inference results between different replicas.

[0010] Existing caching systems typically allocate cache space using fixed quotas, fixed proportions, or cache length-based allocation methods. The pre-filling computation saved, storage space occupied, actual hit rates, and reuse reliability vary depending on the cache length. Fixed allocation methods fail to reflect the actual contribution of various caches to the overall inference performance of the system, potentially leading to underutilization of some cache space or a large amount of space being occupied by low-yield caches.

[0011] When the caching system is running for the first time, a model version is being deployed for the first time, or historical statistics are insufficient, the cache hit rate and reuse reliability are not yet clear. At this time, directly performing cache reuse and resource allocation based on fixed thresholds and initial quotas may result in unverified caches being used for inference, or the judgment conditions may be too conservative, failing to utilize reusable caches, thus affecting the output stability and inference acceleration effect during the system's cold start phase.

[0012] Therefore, existing shared context key-value cache reuse technologies still struggle to simultaneously meet the requirements of expanding cache reuse scope, controlling model output quality, managing cache space effectively, and maintaining the effectiveness of multi-instance caches. Summary of the Invention

[0013] To address the aforementioned problems, the present invention aims to provide a cache device and method for accelerating large language model inference based on shared context. This method accelerates inference, reduces latency and memory usage, and performs online quality verification on reused results while rolling back in case of unreliability. This enables cache management to be aware of the quality risks of reuse and maintains cache consistency under historical changes, model upgrades, and multi-copy deployments. As a result, the stability of inference acceleration and resource utilization efficiency are improved while ensuring output quality.

[0014] To overcome the shortcomings of existing technologies, this invention proposes a large language model inference acceleration caching device based on shared context, including a cache encapsulation module, a reuse and quality verification module, and a quality-aware management module, and the three form a feedback loop with the online measurement results of reuse quality. The cache encapsulation module is used to encapsulate shared fragments into cache units. Each cache unit carries a quality record that migrates with it. The quality record includes the quantization error of the cache unit, the logits divergence of historical reuse, the top-k sorting inconsistency rate, and the number of backoffs. The reuse and quality verification module is used to: first compare the causal preceding computation state represented by the candidate cache unit with the prefix state summary of the current request; when the two are consistent, perform lossless reuse of the segment by skipping the preceding computation; when the two are inconsistent but the token sequence is the same, only use the segment as a gated approximate reuse candidate. Before reuse, the top-klogits sorting consistency rate and KL divergence of the reuse result relative to the complete calculation result are calculated at the sampling probe position. When the consistency rate is lower than the first threshold or the KL divergence is higher than the second threshold, reuse is determined to be unusable and the process is rolled back to the complete forward calculation. At the same time, the measured divergence is written back to the quality record of the corresponding cache unit. The quality awareness management module is used to calculate the retention priority of cache units based on access popularity and quality risk derived from the quality records. When the cache space is insufficient, cache units with low access popularity and high quality risk are prioritized for elimination. The module also feeds back the rollback and the measured divergence to the update of the quality risk, so that the cache space prioritizes the retention of cache units that have been verified by measurement to be safe for reuse.

[0015] Furthermore, it also includes an adaptive quota unit and a consistency control unit; The adaptive quota unit is used to allocate quotas to the cache space according to the token length range, and the quota for the length range k in the next statistical period is allocated according to... Update: In the formula, t is the sequence number of the statistical period, and k is the number of the token length interval. Let k be the cache quota allocated to the length interval k in the t-th statistical period. in The effective hit revenue percentage of this interval after passing the quality verification in the previous statistical period is determined by the computation time saved by recalculation, the number of bytes of reused key tensors and value tensors, the rollback rate and the quality risk, with β ranging from 0.05 to 0.3, so that the quota is dynamically adjusted with the net acceleration revenue and quality risk rather than allocated according to a fixed ratio. The consistency control unit is used to maintain a monotonically increasing version number and content checksum for each cache unit. When historical input, system prompts, or tool return results change, or when model weights are upgraded, cascading failure is performed based on the position of the modified token and its prefix dependencies. Cache units that are on the same causal chain after the modified token are marked as pending verification or invalid. The changes are propagated among multiple model replicas with an incremental message containing the cache unit identifier, version number, range of changed tokens, range of affected layers, and checksum. When verification fails or the version is outdated, the complete cache unit is requested or the local full computation is rolled back. The cache namespace is isolated according to the model weight checksum to support version rollback.

[0016] Furthermore, the prefix state summary is a summary of the hidden state or attention key value corresponding to the causal preceding text of the shared segment, or it is the hash value of the summary; The cache encapsulation module records the model weight checksum, word segmenter version, and prefix status summary in the metadata layer of the cache unit. The reuse and quality verification module only allows lossless reuse when the model weight checksum is consistent, the word segmenter is consistent, and the prefix state summary is consistent, thereby excluding fragments with the same token sequence but different causal preceding text from the lossless reuse path.

[0017] Furthermore, the probe location is selected from the boundary of the shared segment, the absolute position offset point, the layer with high quantization error, and the key position before generation; The first threshold ranges from 95% to 99.9%, and the second threshold ranges from 0.001 to 0.05. When the same cache unit is rolled back N times consecutively, the access frequency of that cache unit is reduced or it is removed from the near reuse candidate. The value of N ranges from 3 to 10. For cache cells quantized with INT8 or lower precision, the quality verification is performed using a higher probe location sampling density than for unquantized cache cells.

[0018] Furthermore, the quality risk is the combined value of the quantization error, probe position logits divergence, top-k sorting inconsistency rate, and historical backoff count of the cache unit after normalization. The retention priority is based on The calculation is performed, where u is the cache unit and t is the current time. in The normalized value of the number of visits, where This represents the cumulative number of historical accesses to cache unit u. This represents the maximum cumulative number of accesses to all cache units in the current cache. Let be the time decay value, where τ is the time decay constant; The time interval since the most recent access, where The time when cache unit u was last accessed. To prevent the minimum time interval constant from being zero; S(u) is the normalized value of space occupancy, and E(u) is the normalized value of the quality risk. The value ranges from 0.3 to 0.8. and It ranges from 0 to 0.5.

[0019] Furthermore, it also includes a cold start and parameter initialization module, which is used to establish an empty cache namespace and read the model weight checksum, word segmenter version and rotation position encoding base frequency parameters to initialize each threshold and quota when the cache is empty, historical request statistics are insufficient or the model version is launched for the first time; When the cache is empty, the first batch of requests are directly passed through to the underlying inference engine to complete the full forward computation. At the same time, the token sequence, prefix state summary and quality baseline are recorded in the bypass and the cache unit is built asynchronously after the request is completed. Adaptive quota and threshold updates will only be initiated when the cumulative number of requests is not less than the set threshold, or when the number of verifiable cache units accumulated in each length interval is not less than the set number and the statistical period is not less than the set duration. In length intervals with insufficient statistics, the default parameters will continue to be used.

[0020] Furthermore, it also includes a context-aware module, which generates a similarity hash fingerprint for the normalized token sequence using a sliding window, recalls candidate cache units in historical fragments using the Hamming distance of the fingerprint, and then performs a precise comparison of the token sequence and a rolling hash comparison on the recalled candidates. When generating similarity hash fingerprints, each token vector within the window is processed according to... After weighted summation, the sign of each dimension is taken, where This is the sequence number of the token within the window. For the first in the window A token vector of tokens. Let the weight be the weight of the token vector, where the weight is... , This is the inverse document frequency normalized value. The position decay weight increases as the token approaches the generation boundary. The value ranges from 0.2 to 0.8.

[0021] Furthermore, the multiplexing includes position-correctable multiplexing, used to, when the prefix state digest is consistent but the absolute position has an offset δ, adjust the key tensor encoded by rotation position by angularly adjusting the dimension pair (2j, 2j+1) at an angle. Re-rotation, Where j is the index of the key tensor dimension pair, base is the rotation position encoding fundamental frequency, and d is the key tensor dimension; The gated approximate multiplexing candidate passes through the bottleneck residual adapter. Generate candidate features, where h is the hidden state or key-value digest of the cache unit, and b is the boundary state of the current request's private context. and This is the low-rank projection matrix of the bottleneck residual adapter. It is a non-linear activation function; Furthermore, the reuse is recorded in layers, and only the key tensors and value tensors of the layers that have passed the quality verification are reused, while the remaining layers undergo complete forward computation.

[0022] A method for accelerating caching of large language model inference based on shared context, used in the aforementioned apparatus, includes: The shared fragments are encapsulated as cache units carrying quality records, which include quantization error, logits divergence of historical reuse, top-k sorting inconsistency rate, and number of backoffs; For candidate cache units, first compare the causal preceding computation state they represent with the prefix state digest of the current request. If they match, perform lossless reuse by skipping the forward computation. If they do not match but the token sequence is the same, they are only used as gated approximate reuse candidates. Before reuse, calculate the top-klogits ranking consistency rate and KL divergence of the reuse result relative to the complete computation result at the sampling probe position. If the threshold is not met, backtrack to the complete forward computation and write the measured divergence back to the quality record. The retention priority is calculated based on both access frequency and quality risk derived from quality records. When cache space is insufficient, cache units with low access frequency and high quality risk are prioritized for elimination, and the rollback and measured divergence are fed back to the update of the quality risk.

[0023] Furthermore, it also includes: Cache space is allocated according to token length ranges, with the quota for each length range determined by... Update: In the formula, t is the sequence number of the statistical period, and k is the number of the token length interval. Let k be the cache quota allocated to the length interval k in the t-th statistical period, where The percentage of effective hits after passing the quality verification in this interval is determined by the computation time saved by recalculation, the number of bytes of the reused key tensors and value tensors, the rollback rate, and the quality risk. In addition, a monotonically increasing version number and content checksum are maintained for each cache unit. When historical input, system prompts, or tool return results change, or model weights are upgraded, cascading failures are performed based on the position of the modified token and its prefix dependencies. Cache units on the same causal chain after the modified token are marked as pending verification or invalidated, and the changes are propagated among multiple model replicas with an incremental message containing the cache unit identifier, version number, range of changed tokens, range of affected layers, and checksum.

[0024] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention provides a large language model inference acceleration caching device and method based on shared context. By performing online quality measurement on the reuse results before reuse and rolling back to full calculation when the results are not up to standard, unreliable reuse is kept out of the output, making the output quality of similar reuse, approximate reuse and cross-position reuse controllable, and avoiding damage to the generation quality due to the pursuit of hit rate.

[0025] 2. This invention provides a cache device and method for accelerating large language model inference based on shared context. It uses online measurement of reuse quality for both reuse gating and cache eviction, forming a feedback loop. This allows the cache to be retained and evicted based on quality risk rather than just access frequency, reducing cache units that degrade output due to long-term retention. This improves the effective hit rate of safe reuse within a given memory budget.

[0026] 3. This invention provides a cache device and method for accelerating large language model inference based on shared context. By using version number, checksum and causal cascading failure by token position, as well as incremental synchronization between multiple replicas, it avoids reusing expired caches during historical editing, model upgrades and multi-replica deployment, thus ensuring output correctness.

[0027] 4. This invention provides a cache device and method for accelerating large language model inference based on shared context. By dynamically allocating length interval quotas according to the net acceleration benefit after quality verification, the utilization efficiency of cache space and the overall speedup ratio are improved.

[0028] 5. This invention provides a cache device and method for accelerating large language model inference based on shared context. Through pass-through and bypass construction in the cold start phase, it remains robust when statistics are insufficient, avoiding misuse or insufficient reuse. Attached Figure Description

[0029] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0030] Figure 1 This is a block diagram of the overall architecture of the device of the present invention; Figure 2 This is a schematic diagram of the structure of the cache unit of the present invention; Figure 3 This is the main flowchart for accelerating inference in this invention; Figure 4 This is a flowchart of the quality gate control and backtracking determination process of the present invention; Figure 5 This is a schematic diagram of the quality perception elimination and feedback closed loop of the present invention; Figure 6 This is a schematic diagram illustrating the consistency control and cascading failure of the present invention; Figure 7 This is a schematic diagram of the cold start process of the present invention. Detailed Implementation

[0031] The technical solution of the present invention will be more clearly and completely explained below with reference to the accompanying drawings and through the description of preferred embodiments of the present invention.

[0032] To make the objectives, technical solutions, and advantages of the present invention clearer, the specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments are only some embodiments of the present invention, and not all embodiments; based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] like Figure 1 and Figure 3As shown, the caching device in this embodiment is deployed before or beside the underlying inference engine, including a cache encapsulation module, a reuse and quality verification module, and a quality-aware management module. It may further include a request normalization module, a context-aware module, an adaptive quota unit, a consistency control unit, and a cold start and parameter initialization module. The shared context includes, but is not limited to, system prompts, repeatedly referenced search documents in search enhancement generation, few-shot examples, and the history of multi-turn dialogues. This content is heavily repeated across different requests and is the primary target for reuse acceleration in this embodiment. The cache encapsulation module encapsulates reusable fragments of the shared context into cache units and stores them in a cache pool. The reuse and quality verification module receives inference requests, recalls candidate cache units from the cache pool, determines and performs reuse, or, if unreliable, falls back to the complete forward computation of the underlying inference engine. The quality-aware management module manages the cache pool for retention, eviction, quota, and consistency. The three modules form a feedback loop based on the online measurement results of reuse quality: the reuse and quality verification module feeds back the quality measurement results of each reuse to the quality awareness management module, which updates the quality risk of the cache unit accordingly and sends this quality risk back as the gating basis for subsequent reuse, thereby enabling the system to continuously prioritize the retention and reuse of cache units that have been verified by measurement to be safe for reuse. This embodiment can be used in conjunction with paging-based video memory management, precise prefix reuse, and the underlying framework that objectifies key-value caches and stores, retrieves, migrates, and unloads them across multiple instances.

[0034] like Figure 3 As shown, the main process for processing a reasoning request includes: the request normalization module normalizes the input text to be reasoned by whitespace and control characters, and normalizes it by Unicode. It then encodes the input text into a token sequence using a tokenizer consistent with the model to be reasoned, inserts session boundary markers, and records the tokenizer's identifier, vocabulary version, and special token configuration. Based on this, a prefix state summary of the current request is generated. The context-aware module recalls candidate cache units. It determines whether the causal context calculation state represented by the candidate cache unit is consistent with the current prefix state summary. If they are consistent, they are used for lossless reuse, and the forward calculation of the corresponding segment is skipped. If they are inconsistent but the token sequence is the same, they are used as gated approximate reuse candidates. Online quality gating is performed before using the above reuse. If the quality meets the standard, the reuse result is adopted; if it does not meet the standard, the process reverts to the complete forward calculation. Finally, the result is output and the quality record and access popularity of the relevant cache units are updated.

[0035] like Figure 2As shown, the caching unit in this embodiment adopts a hierarchical structure. The metadata layer records the model weight checksum, tokenizer version, rotation position encoding fundamental frequency parameters, the start and end positions of the token for the segment, the token sequence digest, and the prefix state digest; the feature layer stores the key tensor, value tensor, and rotation position encoding state of the key tensor corresponding to the segment, or stores the digest of the above tensors; the index layer stores the similarity hash fingerprint used for similarity recall and the rolling hash and prefix state digest used for precise comparison; the quality record layer records the quantization error, logits divergence of historical reuse, top-k sorting inconsistency rate, and number of backoffs for the caching unit. The quality record is stored, retrieved, and migrated together with the caching unit, so that quality gating and quality-aware eviction can still be performed after the caching unit is migrated across instances.

[0036] The prefix state summary is a summary of the hidden state or attention key corresponding to the causal preceding text of the shared segment, or a hash value of the summary, which represents the influence of all preceding contexts on the computational state of the segment. The reuse and quality verification module only allows lossless reuse of the segment by skipping the forward computation when the model weight checksum, tokenizer, and prefix state summary are consistent. This excludes segments with the same token sequence but different causal preceding text from the lossless reuse path, avoiding erroneous reuse due to ignoring preceding text differences.

[0037] As an optional implementation, the context-aware module discovers and confirms candidate shared fragments in two stages. In the first stage, a similarity hash fingerprint is generated for the normalized token sequence using a sliding window. The window length is 32 to 2048 tokens, and the step size is 8 to 256 tokens. The window length and step size can be determined by the 50th to 90th percentile of the shared fragment length distribution in historical requests. For each token within the window, its embedding vector in the model input embedding matrix is ​​read, or, if the embedding is inaccessible, a random projection vector determined by the token identifier and model version is used. The vector dimension is 128 to 1024. The similarity hash fingerprint of the window is obtained by weighted summing of the token vectors within the window using the following formula and taking the sign of each dimension of the result: ; The weights are calculated according to Sure, This is the inverse document frequency normalized value based on historical request statistics. The position decay weight increases as the token approaches the generation boundary. The threshold is set to 0.2 to 0.8. In the second stage, candidate cache units are recalled from historical fragments using the Hamming distance between fingerprints. The Hamming distance threshold is set to 2% to 8% of the number of fingerprint bits. Then, the recalled candidates are subjected to precise comparison of token sequences and rolling hash comparison to confirm the reusable range. The context-aware module can also maintain a multi-level index including a model type index, a length range index, a similarity hash fingerprint bucket index, a token rolling hash index, and a prefix state summary index. Candidate cache units are retrieved efficiently through joint filtering of multi-level indexes. It should be noted that similarity hash recall and its weighting are only an optional implementation method to improve the candidate recall rate. The reuse security of this invention is guaranteed by subsequent causal comparison and online quality gating, and does not depend on this recall method.

[0038] like Figure 4 As shown, the reuse and quality verification module performs online quality gating before using any reuse: at the sampling probe position, it calculates the top-k logits ranking consistency rate and KL divergence of the reuse result relative to the complete calculation result. When the consistency rate is lower than a first threshold or the KL divergence is higher than a second threshold, the reuse is deemed unusable and a rollback to the complete forward calculation is performed; otherwise, the reuse is deemed usable. Regardless of whether a rollback is performed, the measured divergence is written back to the quality record of the corresponding cache unit. In one example, the first threshold is 95% to 99.9%, the second threshold is 0.001 to 0.05, and an additional requirement can be added that the quantization error of the cache unit is not higher than 0.001 to 0.02 for reuse to be allowed. The probe position is selected at the boundary of the shared segment, the absolute position offset point, the layer with high quantization error, and the key position before generation. When the same cache unit is rolled back N times consecutively, the access frequency of the cache unit is reduced or it is removed from the approximate reuse candidate, where N is 3 to 10. For cache units quantized with INT8 or lower precision, a higher probe location sampling density than that of non-quantized cache units is used for quality verification to more strictly control the quality risks brought about by quantization.

[0039] The reuse and quality verification module selects a reuse method from three options—precise reuse, position-correctable reuse, and approximate boundary adaptation—based on the matching between the current request and the cache unit. Precise reuse is performed when the model, token segmenter, token sequence, and prefix state digest are all consistent. For cases where the prefix state digest is consistent but the absolute position has an offset of δ, position-correctable reuse is used. This involves re-rotating the key tensor using rotational position encoding along the dimension pair (2j, 2j+1), where the rotation angle for each dimension pair is... ;

[0040] `base` is the fundamental frequency encoded by the rotation position, and `d` is the dimension of the key tensor; the corresponding correction matrix is...

[0041] The corrected key vector is Value tensors without rotation position encoding remain unchanged; when the cache unit stores an unrotated key tensor, it is directly rotated to generate a key tensor according to the target position.

[0042] For segments with identical token sequences but not completely identical prefixes, and gated as approximate reuse candidates, candidate features are generated through a bottleneck residual adapter: ; Where h is the hidden state or key-value digest of the cache unit, and b is the boundary state of the current request's private context. , It is a low-rank projection matrix with a bottleneck rank ranging from 8 to 64. The activation is nonlinear; the bottleneck residual adapter can adopt a low-rank residual module of LoRA form and be trained separately for each model version. During training, the base model is frozen, and the complete computation results of the teacher model are used as supervision. The loss function is the weighted sum of the mean square error of the hidden state and the KL divergence of the output logits. In addition, the effective range of reuse is recorded layer by layer. Only the key tensors and value tensors of the layers that pass the quality verification are reused, and the remaining layers perform complete forward computation, i.e., layered reuse. In scenarios with extremely high quality requirements, the device can also turn off approximate reuse and only retain lossless reuse and position-correctable reuse.

[0043] like Figure 5 As shown, the quality-aware management module derives the quality risk of cache units from the quality records. This risk is calculated by synthesizing a comprehensive value after normalizing the quantization error, probe position logits divergence, top-k sorting inconsistency rate, and historical backoff count for each cache unit. The module then calculates retention priority using a heat function, prioritizing the eviction of cache units with low access frequency and high quality risk when cache space is insufficient. Specifically, the access frequency and time decay terms in the heat function give higher retention priority to frequently accessed and recently accessed cache units, the space occupancy term suppresses excessively large cache units, and the quality risk term prioritizes the eviction of cache units prone to deterioration. In one example... Take a value between 0.3 and 0.8. and The value is between 0 and 0.5, representing the minimum time interval in the time decay term. Take 1 second to 60 seconds, time constant The timeframe ranges from 10 minutes to 7 days. The quality awareness management module feeds back the rollback and measured divergence of each reuse to the corresponding cache unit quality risk update. The updated quality risk participates in the above-mentioned elimination decision on the one hand, and serves as the gating basis for subsequent reuse, which is then sent back to the reuse and quality verification module. This forms a closed loop, enabling the system to continuously converge to a state of prioritizing the reuse of cache units that can be safely reused as it runs.

[0044] The adaptive quota unit allocates cache space quotas according to token length ranges, and the quota for each length range in the next statistical period is calculated using the aforementioned exponentially weighted moving average formula. Updated, where the smoothing coefficient β ranges from 0.05 to 0.3. The percentage of effective hits for a given length interval after passing quality verification in the previous statistical period is determined by the computation time saved by recalculation, the number of bytes of reused key and value tensors, the backoff rate, and the aforementioned quality risk. This allows the quota to be dynamically adjusted based on net acceleration gains and quality risk, rather than being allocated at a fixed ratio. In one example, the quota for short sequence intervals is 20% to 60%, for medium sequence intervals it is 20% to 50%, and for long sequence intervals it is 10% to 40%. More cache space is allocated to length intervals with higher net gains proven through quality verification, while less cache space is allocated to length intervals with high backoff rates or high quality risks.

[0045] like Figure 6 As shown, the consistency control unit maintains a monotonically increasing version number and content checksum for each cache unit. When historical input, system prompts, or tool results change, or model weights are upgraded, cascading invalidation is performed based on the position of the modified token and its prefix dependencies. Cache units on the same causal chain after the modified token are marked as pending verification or invalidated, thus avoiding the reuse of caches that are no longer valid due to previous changes. Changes are propagated among multiple model replicas via incremental messages containing the cache unit identifier, old version number, new version number, range of changed tokens, range of affected layers, and checksum. Upon receiving the message, the replica first compares the version number and then verifies the content checksum. If the verification fails or the version is outdated, it requests the complete cache unit or rolls back to local full computation. The device also isolates the cache namespace by model weight checksum, ensuring that caches of different model versions do not interfere with each other, and supports rolling back the cache to a historically valid state by version number.

[0046] like Figure 7As shown, when the cache is empty, historical request statistics are insufficient, or the model version is launched for the first time, the cold start and parameter initialization module establishes an empty cache namespace and reads the model weight checksum, tokenizer version, and rotation position encoding base frequency parameters to initialize various thresholds and quotas. When the cache is empty, the first batch of requests is directly passed through to the underlying inference engine to complete the full forward computation. At the same time, the token sequence, prefix state digest, window fingerprint, key-value tensor digest, and quality baseline are recorded in the bypass, and cache units are asynchronously built after the request is completed. The adaptive quota and threshold update is only initiated when the cumulative number of requests is not less than the set threshold, or when the number of verifiable cache units accumulated in each length interval is not less than the set number and the statistical period is not less than the set duration. In length intervals with insufficient statistics, the default parameters continue to be used, thereby avoiding misuse or insufficient reuse when statistics are unreliable.

[0047] This invention is applicable to various Transformer-based large language model inference systems employing key-value caching, including but not limited to models using rotational position encoding and key-value caching using INT8 or lower precision quantization. The specific values ​​given above, such as window length, stride, vector dimension, Hamming distance threshold, various quality thresholds, bottleneck rank, weight coefficient, time constant, and quota upper and lower limits, are merely examples. Those skilled in the art can adjust them according to model size, hardware conditions, and quality requirements. The technical features in the above embodiments can be combined with each other without contradiction, and all such combinations fall within the scope of protection of this invention.

[0048] The above-described specific embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Various modifications, substitutions, and improvements made by those skilled in the art to the technical solutions of the present invention based on the provided textual description and drawings, without departing from the design concept and spirit of the present invention, should all fall within the scope of protection of the present invention. The scope of protection of the present invention is determined by the claims.

Claims

1. A large language model inference acceleration caching device based on shared context, characterized in that, It includes a cache encapsulation module, a reuse and quality verification module, and a quality awareness management module, and the three form a feedback loop based on the online measurement results of reuse quality; The cache encapsulation module is used to encapsulate shared fragments into cache units. Each cache unit carries a quality record that migrates with it. The quality record includes the quantization error of the cache unit, the logits divergence of historical reuse, the top-k sorting inconsistency rate, and the number of backoffs. The reuse and quality verification module is used to: first compare the causal preceding computation state represented by the candidate cache unit with the prefix state summary of the current request; when the two are consistent, perform lossless reuse of the segment by skipping the preceding computation; when the two are inconsistent but the token sequence is the same, only use the segment as a gated approximate reuse candidate. Before reuse, the top-klogits sorting consistency rate and KL divergence of the reuse result relative to the complete calculation result are calculated at the sampling probe position. When the consistency rate is lower than the first threshold or the KL divergence is higher than the second threshold, reuse is determined to be unusable and the process is rolled back to the complete forward calculation. At the same time, the measured divergence is written back to the quality record of the corresponding cache unit. The quality awareness management module is used to calculate the retention priority of cache units based on access popularity and quality risk derived from the quality records. When the cache space is insufficient, cache units with low access popularity and high quality risk are prioritized for elimination. The module also feeds back the rollback and the measured divergence to the quality risk update, so that the cache prioritizes the retention of cache units that have been verified by measurement to be safe for reuse.

2. The large language model inference acceleration caching device based on shared context according to claim 1, characterized in that, It also includes an adaptive quota unit and a consistency control unit; The adaptive quota unit is used to allocate quotas to the cache space according to the token length range, and the quota for the length range k in the next statistical period is allocated according to... renew, in The percentage of valid hits that passed the quality verification in the previous statistical period is the percentage of hits within that interval. This percentage is determined by the computation time saved during recalculation, the number of bytes in the reused key and value tensors, the rollback rate, and the quality risk. β ranges from 0.05 to 0.

3. Let $k$ be the cache quota allocated to length interval $k$ in the $t$-th statistical period. The consistency control unit is used to maintain a monotonically increasing version number and content checksum for each cache unit. When historical input, system prompts, or tool return results change, or when model weights are upgraded, cascading failure is performed based on the position of the modified token and its prefix dependencies. Cache units that are on the same causal chain after the modified token are marked as pending verification or invalid. The changes are propagated among multiple model replicas with an incremental message containing the cache unit identifier, version number, range of changed tokens, range of affected layers, and checksum. When verification fails or the version is outdated, the complete cache unit is requested or the local full computation is rolled back. The cache namespace is isolated according to the model weight checksum to support version rollback.

3. The large language model inference acceleration caching device based on shared context according to claim 1, characterized in that, The prefix state summary is a summary of the hidden state or attention key value corresponding to the causal preceding text of the shared segment, or it is the hash value of the summary; The cache encapsulation module records the model weight checksum, word segmenter version, and prefix status summary in the metadata layer of the cache unit. The reuse and quality verification module only allows lossless reuse when the model weight checksum is consistent, the word segmenter is consistent, and the prefix state summary is consistent, thereby excluding fragments with the same token sequence but different causal preceding text from the lossless reuse path.

4. The large language model inference acceleration caching device based on shared context according to claim 1, characterized in that, The probe locations are selected from the boundaries of shared segments, absolute position offset points, layers with high quantization errors, and key locations before generation; The first threshold ranges from 95% to 99.9%, and the second threshold ranges from 0.001 to 0.

05. When the same cache unit is rolled back N times consecutively, the access frequency of that cache unit is reduced or it is removed from the near reuse candidate. The value of N ranges from 3 to 10. For cache cells quantized with INT8 or lower precision, the quality verification is performed using a higher probe location sampling density than for unquantized cache cells.

5. The large language model inference acceleration caching device based on shared context according to claim 1, characterized in that, The quality risk is the combined value of the quantization error, probe position logits divergence, top-k sorting inconsistency rate, and the normalized number of historical backoffs for the cache unit. The retention priority is based on calculate, in The normalized value of the number of visits, where This represents the cumulative number of historical accesses to cache unit u. This represents the maximum cumulative number of accesses to all cache units in the current cache. This is the time decay value; The time interval since the most recent access, where The time when cache unit u was last accessed. To prevent the minimum time interval constant from being zero; S(u) is the normalized value of space occupancy, and E(u) is the normalized value of the quality risk. The value ranges from 0.3 to 0.

8. and It ranges from 0 to 0.

5.

6. The large language model inference acceleration caching device based on shared context according to claim 1, characterized in that, It also includes a cold start and parameter initialization module, which is used to create an empty cache namespace and read the model weight checksum, word segmenter version and rotation position encoding base frequency parameters to initialize each threshold and quota when the cache is empty, historical request statistics are insufficient or the model version is launched for the first time. When the cache is empty, the first batch of requests are directly passed through to the underlying inference engine to complete the full forward computation. At the same time, the token sequence, prefix state summary and quality baseline are recorded in the bypass and the cache unit is built asynchronously after the request is completed. Adaptive quota and threshold updates will only be initiated when the cumulative number of requests is not less than the set threshold, or when the number of verifiable cache units accumulated in each length interval is not less than the set number and the statistical period is not less than the set duration. In length intervals with insufficient statistics, the default parameters will continue to be used.

7. The large language model inference acceleration caching device based on shared context according to claim 1, characterized in that, It also includes a context-aware module, which generates a similarity hash fingerprint of the normalized token sequence by sliding window, recalls candidate cache units in historical fragments by Hamming distance of the fingerprint, and then performs a precise comparison of the token sequence and a rolling hash comparison on the recalled candidates; When generating similarity hash fingerprints, each token vector within the window is processed according to... After weighted summation, the sign of each dimension is taken, where in the formula This is the sequence number of the token within the window. For the first in the window A token vector of tokens. Let the weight be the weight of the token vector, where the weight is... , This is the inverse document frequency normalized value. The position decay weight increases as the token approaches the generation boundary. The value ranges from 0.2 to 0.

8.

8. The large language model inference acceleration caching device based on shared context according to claim 1, characterized in that, The multiplexing includes position-correctable multiplexing, used when the prefix state digest is consistent but the absolute position is offset. At that time, for the key tensor using rotational position encoding, the angle is used to represent the dimension pair (2j, 2j+1). Re-rotation, Where j is the index of the key tensor dimension pair, base is the rotation position encoding fundamental frequency, and d is the key tensor dimension; The gated approximate multiplexing candidate passes through the bottleneck residual adapter. Generate candidate features, where h is the hidden state or key-value digest of the cache unit, and b is the boundary state of the current request's private context. and This is the low-rank projection matrix of the bottleneck residual adapter. It is a non-linear activation function; Furthermore, the reuse is recorded in layers, and only the key tensors and value tensors of the layers that have passed the quality verification are reused, while the remaining layers undergo complete forward computation.

9. A method for accelerating large language model inference based on shared context, used to implement the device for accelerating large language model inference based on shared context as described in any one of claims 1-8, characterized in that, include: The shared fragments are encapsulated as cache units carrying quality records, which include quantization error, logits divergence of historical reuse, top-k sorting inconsistency rate, and number of backoffs; For candidate cache units, first compare the causal preceding computation state they represent with the prefix state digest of the current request. If they match, perform lossless reuse by skipping the forward computation. If they do not match but the token sequence is the same, they are only used as gated approximate reuse candidates. Before reuse, calculate the top-klogits ranking consistency rate and KL divergence of the reuse result relative to the complete computation result at the sampling probe position. If the threshold is not met, backtrack to the complete forward computation and write the measured divergence back to the quality record. The retention priority is calculated based on both access frequency and quality risk derived from quality records. When cache space is insufficient, cache units with low access frequency and high quality risk are prioritized for elimination, and the rollback and measured divergence are fed back to the update of the quality risk.

10. The method for accelerating caching of large language model inference based on shared context according to claim 9, characterized in that, Also includes: Cache space is allocated according to token length ranges, with the quota for each length range determined by... Update: In the formula, t is the sequence number of the statistical period, and k is the number of the token length interval. Let k be the cache quota allocated to the length interval k in the t-th statistical period, where The percentage of effective hits after passing the quality verification in this interval is determined by the computation time saved by recalculation, the number of bytes of the reused key tensors and value tensors, the rollback rate, and the quality risk. In addition, a monotonically increasing version number and content checksum are maintained for each cache unit. When historical input, system prompts, or tool return results change, or model weights are upgraded, cascading failures are performed based on the position of the modified token and its prefix dependencies. Cache units on the same causal chain after the modified token are marked as pending verification or invalidated, and the changes are propagated among multiple model replicas with an incremental message containing the cache unit identifier, version number, range of changed tokens, range of affected layers, and checksum.

Images