Compression method, device and equipment of key value cache, storage medium and program product

Abstract

The application relates to a compression method and device of a key-value cache, equipment, a storage medium and a program product. The method comprises the following steps: acquiring token sequences and token cache quantities corresponding to each network layer in a large language model; for any network layer, the token sequence of the network layer is split into key sink tokens, task instruction tokens and context semantic tokens according to the positions of the tokens in the token sequence of the network layer; according to the token cache quantity, target semantic tokens matched with the task instruction tokens of the network layer are selected from the context semantic tokens; the key-value information corresponding to the key sink tokens and the target semantic tokens in the network layer is compressed to obtain compressed key-value reasoning information of the network layer; and the compressed key-value reasoning information of all the network layers is cached. The method can improve the accuracy of key-value information and the reasoning accuracy of the large language model.

Description

Technical Field

[0001] This application relates to the field of caching technology, and in particular to a compression method, apparatus, device, storage medium, and program product for key-value caching. Background Technology

[0002] The key and value intermediate calculation results generated during the reasoning and generation process of large language models have reusable characteristics, and such intermediate calculation results can be uniformly stored and managed through a caching mechanism.

[0003] In related technologies, in order to save the cache space occupied by intermediate calculation results, compression strategies such as selective compression and quantization compression are usually adopted to compress and cache intermediate calculation results, so as to provide reusable key-value state support for subsequent self-attention calculation of tokens.

[0004] However, the key-value information cached by the caching compression method in related technologies has low accuracy, resulting in low inference accuracy of large language models. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, device, storage medium, and program product for compressing key-value cache to address the above-mentioned technical problems, thereby improving the accuracy of key-value information and thus enhancing the inference accuracy of large language models.

[0006] Firstly, this application provides a compression method for key-value caching, the method comprising:

[0007] Obtain the lexical sequences and lexical cache counts corresponding to each network layer in a large language model;

[0008] For any network layer, the word sequence of the network layer is divided into key sink words, task instruction words, and context semantic words according to the position of each word in the word sequence. Key sink words include base words that represent the overall semantics of the word sequence. Task instruction words include instruction words that clarify the execution goal and behavior of the large language model. Context semantic words include words in the word sequence other than key sink words and task instruction words.

[0009] Based on the number of lexical caches, target semantic lexical units that match the task instruction lexical units of the network layer are selected from the context semantic lexical units.

[0010] The key-value information corresponding to the key sink words and target semantic words in the network layer is compressed to obtain the compressed key-value inference information of the network layer.

[0011] Compressed key-value inference information for all network layers is cached.

[0012] The key-value caching compression method provided in this application involves obtaining the lexical sequence and the number of lexical caches corresponding to each network layer in a large language model; for any network layer, the lexical sequence of the network layer is split into key sink lexical units, task instruction lexical units, and context semantic lexical units according to the position of each lexical unit in the lexical sequence of the network layer; based on the number of lexical caches, target semantic lexical units that match the task instruction lexical units of the network layer are selected from the context semantic lexical units; the key-value information corresponding to the key sink lexical units and target semantic lexical units in the network layer is compressed to obtain the compressed key-value inference information of the network layer; and the compressed key-value inference information of all network layers is cached. Specifically, key sink lexical units include base point lexical units that represent the overall semantics of the lexical sequence; task instruction lexical units include instruction lexical units that clarify the execution target and behavior direction of the large language model; and context semantic lexical units include lexical units in the lexical sequence other than key sink lexical units and task instruction lexical units. In this method, for each network layer, after classifying the lexical sequence to identify key sink lexicals, task instruction lexicals, and contextual semantic lexicals, the key-value information corresponding to the key sink lexicals is forcibly retained to ensure the relative stability of the model's attention weight distribution. Simultaneously, based on the number of lexical cached units, target semantic lexicals associated with the task instruction lexicals are selected, and their key-value information is compressed and stored. In this way, both key-value information of key sink lexicals and target semantic lexicals in each network layer are accurately retained. The former ensures the large language model can think normally and avoids computational crashes, while the latter aims to ensure the large language model thinks correctly and relevantly, injecting the information required for the task. This ensures the comprehensiveness and accuracy of the lexicals corresponding to the key-value information, facilitating subsequent inference by the model and improving the accuracy of the inference results.

[0013] In one embodiment, obtaining the number of lexical caches corresponding to each network layer in the large language model includes:

[0014] Obtain the budget cache of the large language model, the number of network layers in the large language model, and the key-value cache size of a single word;

[0015] The allocated cache for each network layer is determined based on the budget cache and the number of network layers;

[0016] The number of lexical caches corresponding to each network layer is determined based on the allocated cache size of each network layer and the key-value cache size of a single lexical.

[0017] In the key-value caching compression method provided in this application embodiment, the allocated cache for each network layer is first determined based on the budget cache of the large language model and the number of network layers of the large language model. Then, based on the allocated cache of each network layer and the key-value cache size of a single word, the number of word caches corresponding to each network layer is determined to avoid idle and wasted cache resources or local cache overflow.

[0018] In one embodiment, the large language model includes a top-level network layer, a third-level network layer, and at least one intermediate network layer; the allocation cache for each network layer is determined based on the budget cache and the number of network layers, including:

[0019] Based on the budget cache and the number of network layers, determine the allocation cache for the top-level network layer and the allocation cache for the bottom-level network layer;

[0020] Calculate the cache difference between the allocated cache of the bottom network layer and the allocated cache of the top network layer, and determine the cache difference between adjacent layers as the ratio of the cache difference to the number of network layers;

[0021] The allocation cache of each intermediate network layer is determined based on the difference between the allocation cache of the bottom network layer and the cache of the adjacent layer.

[0022] In the key-value caching compression method provided in this application embodiment, the allocation cache of the top-level network layer and the allocation cache of the bottom-level network layer are determined according to the budget cache and the number of network layers; then, the difference between the two and the number of network layers are combined to obtain the cache difference between adjacent layers, and then the allocation cache of each intermediate network layer is derived in an arithmetic progression; under the premise that the sum of the allocation caches of all network layers is equal to the budget cache, the bottom-level network layer cache quota is greater than the adjacent upper layer in a hierarchical incremental allocation, which fits the stronger dependence of the deep network of the bottom layer of the large language model on contextual information, and improves the accuracy and stability of the model inference operation.

[0023] In one embodiment, determining the allocation cache for the top-level network layer and the allocation cache for the bottom-level network layer based on the budget cache and the number of network layers includes:

[0024] Calculate the ratio between the budget buffer and the number of network layers;

[0025] The ratio and preset hyperparameters are fused to obtain the allocation cache of the top-level network layer;

[0026] The allocation cache of the lower network layer is determined based on the difference between the ratio and the allocation cache of the top network layer.

[0027] In the key-value caching compression method provided in this application embodiment, the ratio between the budgeted cache and the number of network layers is used as the basis. The ratio and preset hyperparameters are fused to obtain the allocation cache of the top-level network layer. The allocation cache of the bottom-level network layer is determined based on the difference between the ratio and the allocation cache of the top-level network layer. This is equivalent to allocating more cache to the bottom-level network layer with more dispersed information and less cache to the top-level network layer with less sparse information, in order to give full play to the role of the allocation cache.

[0028] In one embodiment, the lexical sequence of the network layer is split into key sink lexical units, task instruction lexical units, and contextual semantic lexical units, including:

[0029] The word in the word sequence of the network layer at the first preset position is identified as the key sink word; the word in the word sequence of the network layer at the second preset position is identified as the task instruction word.

[0030] Contextual semantic lexical units are determined based on the lexical units in the lexical sequence, excluding key sink lexical units and task instruction lexical units.

[0031] In the key-value caching compression method provided in this application embodiment, key sink words and task instruction words in the word sequence are quickly defined by fixed positions, and the remaining parts are uniformly classified as context semantic words. The three types of words can be efficiently and quickly split without complex semantic recognition. This provides a regular and orderly data foundation for subsequent selection and matching of semantic words based on the number of caches, key-value information compression and hierarchical cache optimization, and improves the execution efficiency and running stability of the overall cache optimization process.

[0032] In one embodiment, target semantic lexical units that match the task instruction lexical units of the network layer are selected from the context semantic lexical units based on the number of lexical units cached, including:

[0033] Calculate the attention score between each context semantic word and each word in the task instruction word to obtain the semantic quantization value of each context semantic word;

[0034] The semantic lexical units of each context are sorted according to their semantic quantization values.

[0035] Target semantic words that match the number of word caches are determined from the sorting results of semantic words in each context.

[0036] In the key-value caching compression method provided in this application embodiment, the attention score between each context semantic word and each word in the task instruction word is calculated to obtain the semantic quantization value of each context semantic word, so as to quantify the correlation between each context semantic word and the task instruction word. Then, according to the size of the semantic quantization value, each context semantic word is sorted to determine the target semantic word that matches the word cache quantity, so as to remove context semantic words with low correlation and retain high-quality context semantic words.

[0037] In one embodiment, the attention score between each context semantic unit and each unit in the task instruction unit is calculated to obtain the semantic quantization value of each context semantic unit, including:

[0038] For any contextual semantic word, through each attention head of the network layer, the attention score between the contextual semantic word and each word in the task instruction word is calculated, and the head-level attention score of the contextual semantic word in each attention head is obtained;

[0039] The semantic quantization value of the context semantic lexical is obtained by fusing the head attention scores of different attention heads.

[0040] In the key-value caching compression method provided in this application embodiment, for any context semantic lexical, the attention score between the context semantic lexical and each lexical in the task instruction lexical is first calculated through each attention head of the network layer to obtain the head-level attention score of the context semantic lexical at each attention head. Then, the head-level attention scores of the context semantic lexical at different attention heads are fused to obtain the semantic quantization value of the context semantic lexical, eliminating the influence of abnormal activation in individual heads, and accurately quantifying the overall contribution of the lexical to understanding the current task generation instruction. This provides a more accurate ranking basis for subsequent filtering of highly relevant target semantic lexicals based on the number of lexical caches.

[0041] In one embodiment, the target semantic word element matching the word element cache quantity is determined from the sorting results of each context semantic word element, including:

[0042] The number of target semantic words is obtained by subtracting the number of words in the cache from the number of words in the key sink words and the number of words in the task instruction words.

[0043] Identify target semantic words that match the number of target semantic words from the sorting results.

[0044] In the key-value caching compression method provided in this application embodiment, the number of lexical caches is subtracted from the number of key sink lexicals and the number of task instruction lexicals in sequence to obtain the number of target semantic lexicals. Then, the target semantic lexicals that match the number of target semantic lexicals are determined from the sorting results. This is equivalent to prioritizing the storage and use needs of key sink lexicals and task instruction lexicals on the basis of caching allocation in the adaptation network layer, forcing the model to make the most effective selection of semantic information within limited resources, thereby avoiding the pruning of key information and ensuring the core semantic logic of the reasoning task.

[0045] Secondly, this application also provides a compression device for key-value caching, comprising:

[0046] The sequence acquisition module is used to obtain the word sequence and word cache number corresponding to each network layer in the large language model;

[0047] The lexical segmentation module is used to split the lexical sequence of any network layer into key sink lexical units, task instruction lexical units, and context semantic lexical units according to the position of each lexical unit in the lexical sequence of the network layer. Key sink lexical units include base point lexical units that represent the overall semantics of the lexical sequence. Task instruction lexical units include instruction lexical units that clarify the execution goal and behavior of the large language model. Context semantic lexical units include lexical units in the lexical sequence other than key sink lexical units and task instruction lexical units.

[0048] The lexical filtering module is used to filter target semantic lexical units that match the task instruction lexical units of the network layer from the context semantic lexical units based on the number of lexical units cached.

[0049] The lexical compression module is used to compress the key-value information corresponding to the key sink lexical and target semantic lexical in the network layer to obtain the compressed key-value inference information of the network layer.

[0050] The key-value caching module is used to cache compressed key-value inference information for all network layers.

[0051] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method in any of the embodiments of the first aspect described above.

[0052] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method in any of the embodiments of the first aspect described above.

[0053] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method in any of the embodiments of the first aspect described above.

[0054] The aforementioned key-value caching compression method, apparatus, device, storage medium, and program product obtain the lexical sequence and lexical cache quantity corresponding to each network layer in the large language model. For any network layer, according to the position of each lexical in the network layer's lexical sequence, the lexical sequence of the network layer is split into key sink lexicals, task instruction lexicals, and context semantic lexicals. Based on the lexical cache quantity, target semantic lexicals matching the task instruction lexicals of the network layer are selected from the context semantic lexicals. The key-value information corresponding to the key sink lexicals and target semantic lexicals in the network layer is compressed to obtain the compressed key-value inference information of the network layer. The compressed key-value inference information of all network layers is cached. Among them, key sink lexicals include base point lexicals that represent the overall semantics of the lexical sequence; task instruction lexicals include instruction lexicals that clarify the execution target and behavior direction of the large language model; and context semantic lexicals include lexicals in the lexical sequence other than key sink lexicals and task instruction lexicals. In this method, for each network layer, after classifying the lexical sequence to identify key sink lexical units, task instruction lexical units, and contextual semantic lexical units, the key-value information corresponding to the key sink lexical units is forcibly retained to ensure the relative stability of the model's attention weight distribution. Simultaneously, based on the number of lexical units cached, target semantic lexical units associated with task instruction lexical units are selected, and their key-value information is compressed and stored. In this way, both key-value information of key sink lexical units and target semantic lexical units in each network layer are accurately retained. The former ensures the large language model can think normally and avoids computational crashes, while the latter aims to ensure the large language model thinks correctly and relevantly, injecting the information required for the task. This approach ensures that the accuracy of the inference output is not affected, while adapting to various business tasks and low-computing-power deployment scenarios, achieving efficient, lightweight, and highly adaptable operation of the large language model's inference process. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is an internal structural diagram of a computer device in one embodiment;

[0057] Figure 2 This is a flowchart illustrating a compression method for key-value caching in one embodiment;

[0058] Figure 3 This is a schematic diagram of word segmentation in one embodiment of a word sequence;

[0059] Figure 4 This is a flowchart illustrating the steps for determining the number of lexical caches in one embodiment;

[0060] Figure 5 This is a flowchart illustrating the step of determining the number of lexical caches in another embodiment;

[0061] Figure 6 This is a flowchart illustrating the target semantic unit determination steps in one embodiment;

[0062] Figure 7 This is a flowchart illustrating the steps for determining the semantic quantization value of a lexical unit in one embodiment;

[0063] Figure 8 This is a flowchart illustrating the target semantic unit determination step in another embodiment;

[0064] Figure 9 This is a schematic diagram of compressed key-value inference information for each network layer of a large language model in one embodiment.

[0065] Figure 10 This is a flowchart illustrating the reasoning method of a large language model in one embodiment;

[0066] Figure 11 This is a structural block diagram of a compression device for key-value caching in one embodiment. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0069] First, the technical context involved in the embodiments of this application will be explained as follows.

[0070] With the rapid development of deep learning technology, Large Language Models (LLMs) based on the Transformer architecture have moved from the laboratory to widespread application, becoming a crucial engine driving the generalization of artificial intelligence. Leveraging their powerful modeling capabilities for long-distance dependencies through their self-attention mechanism, LLMs have not only achieved record-breaking performance in traditional natural language tasks such as text generation, question-answering, and machine translation, but also demonstrated enormous potential for cross-modal understanding and collaborative generation through deep integration with visual and audio modalities. With the exponential growth of parameter scale and training data, these models are gradually acquiring human-like knowledge integration, contextual reasoning, and semantic generalization capabilities, accelerating the evolution of artificial intelligence (AI) systems from "task-specific" to "scenario-general."

[0071] However, when LLMs shift from static, finite-length offline applications to dynamic, continuously interactive streaming deployment environments, such as real-time dialogues, long document processing, or intelligent agent systems, the full-sequence key-value cache (KV Cache) mechanism upon which their underlying architecture relies exposes severe efficiency and stability bottlenecks. This not only restricts the practical deployment of models on resource-constrained devices but also fundamentally limits their reliable operation in infinite context streams.

[0072] Specifically, during autoregressive inference, the model needs to cache the key and value states corresponding to all historical tokens to support attention calculations for subsequent tokens. This full caching strategy has two fundamental drawbacks: First, memory consumption increases linearly with the length of the context, which will quickly exhaust the device's GPU memory and cause uncontrollable inference latency in long-duration interaction scenarios. Second, mainstream large language models are usually pre-trained on sequences of limited length (such as 4K or 8K tokens). When the input context significantly exceeds this range, even with sufficient hardware resources, the language generation quality of the model will still degrade sharply, manifesting as logical breaks, factual illusions, or semantic drift.

[0073] Further research indicates that the aforementioned performance degradation is not simply caused by information loss, but rather stems from the model's internal attention mechanism's high sensitivity to specific contextual structures. Without effective guidance, arbitrary pruning or compression of the historical cache can easily disrupt the key signals maintaining a stable attention distribution, thereby triggering a systematic failure in the inference process.

[0074] Therefore, a new KV cache management paradigm is urgently needed. This paradigm must not only efficiently compress redundant historical information under strictly limited memory budgets, but also possess the ability to structurally identify and permanently retain core contextual units that play a decisive role in the long-term inference stability of the model. Only in this way can we achieve high robustness and continuous availability of large language models under infinitely long-flowing input while ensuring low resource overhead.

[0075] Among related technologies, the methods for compressing key-value (KV) caches in large language models mainly fall into the following three categories:

[0076] The first category of methods is selective compression strategies. The core of these methods lies in dynamically identifying and retaining key sink terms that are crucial for model inference, while discarding or compressing less important term data. For example, they leverage hierarchical attention uncertainty to dynamically allocate cache budgets, achieve adaptive compression through global-local importance scoring, and share similar KV caches between layers to improve memory efficiency. These methods generally achieve significant compression ratios and perform well in long text tasks such as Needle-in-a-Haystack. They typically maintain model performance well but rely on accurate importance assessment mechanisms and are sensitive to computational overhead.

[0077] The second category of methods is quantization compression techniques. The core of these methods is to reduce memory usage and computational load by lowering the numerical precision of key-value vectors in the KV cache. This involves achieving high throughput with low precision through a custom quantization kernel with perceptual activation, proposing a 2-bit KV cache quantization scheme that requires no tuning, or employing adaptive quantization with aligned precision to reduce memory access overhead. These methods demonstrate outstanding performance in compression ratio (typically 4-10 times) and throughput improvement, making them particularly suitable for large-scale deployments and long-context inference scenarios. However, quantization may introduce precision loss, requiring a trade-off between compression ratio and model accuracy, and its generalization ability to different models and tasks still needs further validation.

[0078] The third category of methods is attention compression mechanisms. These methods start by optimizing the attention computation structure itself, improving efficiency by refining the organization and access of key-value caches. For example, cache eviction is modeled as a sub-module optimization problem, dynamically managing key sink lexical units; hierarchical context compression is implemented based on redundancy assumptions; and efficient compression is achieved through inter-layer cosine similarity analysis. These methods can typically significantly reduce memory usage and improve inference speed while maintaining semantic accuracy, making them particularly suitable for scenarios with strict requirements on generation quality. However, their design is often closely related to specific model architectures and attention patterns, and challenges remain in their portability across different models and their synergistic integration with other techniques such as quantization.

[0079] In summary, the compression schemes in related technologies have the following shortcomings:

[0080] (1) It has weak generalization ability and poor scene adaptability.

[0081] Methods based on importance scoring (such as attention scores and gradient magnitudes) or fixed heuristics (such as preserving first and last lexical terms and using sliding windows) perform well on specific models or datasets, but their performance degrades significantly when transferred to different architectures (such as from LLaMA to Qwen), different tasks (such as dialogue vs. long document summarization), or different contextual distributions. For example, some lexical terms may be unimportant in question answering tasks but are key syntactic anchors in code generation, and static scoring cannot capture such semantic differences.

[0082] (2) Stability and efficiency are difficult to balance.

[0083] While heuristic methods can ensure the stability of long sequence reasoning, they do not effectively compress intermediate redundant information, resulting in limited memory savings. On the other hand, aggressive dynamic elimination strategies (such as high-proportion sparsification) can significantly reduce GPU memory usage, but may accidentally remove implicit key sink words (which may not be initial words) that "anchor" the attention distribution, leading to sudden changes in output quality or even a crash.

[0084] (3) It introduces additional overhead, which goes against the original intention of lightweight design.

[0085] While learning-driven or automatic tuning methods (such as Bayesian optimization of search cache budget and training compensation modules) are theoretically superior, they require online evaluation, gradient calculation, or additional parameters, increasing inference latency and implementation complexity. This contradicts the fundamental goal of KV cache compression to "reduce inference overhead," making it particularly difficult to implement in edge devices or high-concurrency service scenarios.

[0086] (4) Lack of system-level coordination.

[0087] While quantization reduces precision, it may weaken the semantic separability of key-value vectors, rendering subsequent attention-based selection mechanisms ineffective. Dynamic cache eviction, on the other hand, often disrupts memory continuity, contradicting the efficient memory access requirements of parallel computing frameworks. Ultimately, existing methods are mostly independently designed "single-point optimizations," lacking a systematic collaborative design across quantization, filtering, memory management, and hardware adaptation, making it difficult to achieve optimal overall performance.

[0088] Considering the above factors, this application provides a key-value caching compression method to efficiently save GPU memory while ensuring model inference accuracy, improve the accuracy of cached key values, and ensure that the model can handle infinitely long text input sequences without fine-tuning.

[0089] The key-value caching compression method provided in this application can be applied to computer devices. The computer device can be a terminal or a server. Taking a server as an example... Figure 1 This is a diagram of the internal structure of a computer device. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data used for implementing a key-value caching compression method. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a key-value caching compression method.

[0090] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0091] In one exemplary embodiment, such as Figure 2 As shown, a compression method for key-value caching is provided, including the following steps:

[0092] S201, obtain the lexical sequence and lexical cache number corresponding to each network layer in the large language model.

[0093] Large language models refer to pre-trained large-scale Transformer architecture language models with natural language understanding, generation, and semantic reasoning capabilities, such as text generation models and question answering models. The core of a large language model consists of multiple internally stacked network layers, namely encoding and decoding layers, which are the basic hierarchical units for model feature extraction and semantic operations.

[0094] A lexical sequence refers to the ordered array of lexical units arranged sequentially according to word order after word segmentation of the input text in a large language model. This array forms the basic language data that the model can recognize. In this embodiment, the lexical sequences of all network layers in the large language model originate from the word segmentation sequence of the input text. However, the lexical sequences corresponding to different network layers are different. For the first network layer of the large language model, the word segmentation sequence of the input text is the lexical sequence. For subsequent network layers, the output lexical sequence of the previous layer serves as the input lexical sequence of the next layer. In the large language model, shallow network layers tend to focus on the original literal lexical sequence of the text, while deep network layers tend to focus on the abstract lexical sequence after incorporating contextual semantics. The deeper the layer, the greater the difference in semantic features between the sequences.

[0095] When obtaining the word sequence corresponding to each network layer in the large language model, the input text of the large language model is segmented to obtain the word sequence corresponding to the first network layer. After the first network layer performs attention calculation on the input word sequence, the processed word sequence is output, which is the word sequence corresponding to the second network layer. After the second network layer performs attention calculation on the input word sequence, the processed word sequence is output, which is the word sequence corresponding to the third network layer, and so on, to obtain the word sequence corresponding to each network layer.

[0096] The term cache size refers to the maximum cache quota of terms that can be stored and retained for each network layer, i.e., the upper limit of the term data capacity allowed to be stored by that network layer. In the embodiments of this application, considering that dense attention can be observed at the bottom layer and sparse attention can be observed at the higher layers, a "pyramid pattern" aggregation phenomenon is presented from the bottom layer to the top layer. The cache capacity allocated to each network layer is different, and the term cache size corresponding to each network layer is different. Moreover, for two adjacent network layers, the term cache size of the bottom network layer is greater than that of the top network layer.

[0097] Optionally, cache capacity can be allocated to each network layer from bottom to top according to the number of network layers, thereby determining the number of lexical caches corresponding to each network layer.

[0098] S202, for any network layer, according to the position of each word in the word sequence of the network layer, the word sequence of the network layer is split into key sink word words, task instruction word words, and context semantic word words.

[0099] Among them, key sink lexical units include base point lexical units that represent the overall semantics of the lexical unit sequence; task instruction lexical units include instruction lexical units that clarify the execution goals and behavioral directions of the large language model; and context semantic lexical units include lexical units in the lexical unit sequence other than key sink lexical units and task instruction lexical units.

[0100] For any network layer, based on the position of each word in the word sequence, the words in the word sequence are divided into key sink words, task instruction words, and contextual semantic words. Key sink words include those that connect the overall semantics, determine the main logic and core theme of the statement, and serve as the semantic support basis for the entire content; they are usually located at the beginning of the word sequence. Task instruction words are used to clarify the model's execution goals, operational requirements, and behavioral directions; they are usually located at the end of the word sequence. Contextual semantic words refer to auxiliary words used to supplement scene information, background content, and auxiliary explanations; they enrich the context and improve the dialogue content, and are usually located in the middle of the word sequence.

[0101] For example, the lexical units in the lexical unit sequence can be divided into three subsequences from left to right according to a preset ratio. The lexical units in the first subsequence are identified as key sink lexical units, the lexical units in the second subsequence are identified as context semantic lexical units, and the lexical units in the third subsequence are identified as task instruction lexical units.

[0102] In an exemplary embodiment, the word sequence of the network layer is split into key sink words, task instruction words, and contextual semantic words, including:

[0103] The word in the word sequence of the network layer at the first preset position is identified as the key sink word; the word in the word sequence of the network layer at the second preset position is identified as the task instruction word; and the context semantic word is identified based on the word in the word sequence other than the key sink word and the task instruction word.

[0104] The first N lexical units in the first preset position of the lexical unit sequence are identified as key sink lexical units; the last M lexical units in the second preset position of the lexical unit sequence are identified as task instruction lexical units; and the lexical units in the lexical unit sequence other than the key sink lexical units in the first preset position and the task instruction lexical units in the second preset position are identified as context semantic lexical units. Here, M and N are positive numbers greater than 1, and M and N can be equal or unequal.

[0105] Taking a lexical sequence containing 14 lexical units as an example, Figure 3 This is a diagram illustrating the segmentation of a word sequence. Figure 3 The lexical sequence is divided from left to right: the first 4 lexical units are identified as key sink lexical units, the fifth 10 lexical units are identified as contextual semantic lexical units, and the eleventh 14 lexical units are identified as task instruction lexical units.

[0106] In this embodiment, key sink words and task instruction words in the word sequence are quickly defined by fixed positions, and the remaining parts are uniformly classified as context semantic words. The three types of words can be efficiently and quickly split without complex semantic recognition. This provides a regular and orderly data foundation for subsequent selection and matching of semantic words based on cache quantity, key-value information compression and hierarchical cache optimization, and improves the execution efficiency and running stability of the overall cache optimization process.

[0107] S203, based on the number of lexical caches, select target semantic lexicals from the context semantic lexicals that match the task instruction lexicals of the network layer.

[0108] For each context semantic unit, the semantic correlation between that context semantic unit and each task instruction unit is calculated to obtain the reference weight of that context semantic unit. This is used to obtain the reference weights of all context semantic units. Then, in descending order of reference weight, the context semantic units corresponding to the number of units cached are obtained sequentially, i.e., the target semantic units.

[0109] S204. The key-value information corresponding to the key sink terms and target semantic terms in the network layer is compressed to obtain the compressed key-value inference information of the network layer.

[0110] The key-value information corresponding to the key sink words in the network layer is compressed, as is the key-value information corresponding to the target semantic words in the network layer. Then, according to the order of the key sink words and the target semantic words in the corresponding word sequence, the compressed key-value information corresponding to the key sink words and the compressed key-value information corresponding to the target semantic words are summarized to obtain the compressed key-value inference information of the network layer.

[0111] S205 caches compressed key-value inference information for all network layers.

[0112] Based on the association relationship of a set of compressed key-value inference information corresponding to each layer, the compressed key-value inference information of each network layer is cached in a preset cache space.

[0113] In this embodiment, for each network layer, after classifying the lexical sequence to identify key sink lexicals, task instruction lexicals, and contextual semantic lexicals, the key-value information corresponding to the key sink lexicals is forcibly retained to ensure the relative stability of the model's attention weight distribution. Simultaneously, based on the number of lexical cached lexicals, target semantic lexicals associated with task instruction lexicals are selected, and their key-value information is compressed and stored. In this way, both key-value information of key sink lexicals and key-value information of target semantic lexicals in each network layer are accurately retained. The former ensures the large language model can think normally and avoids computational crashes, while the latter aims to ensure the large language model thinks correctly and relevantly, injecting the information required for the task to guarantee the comprehensiveness and accuracy of the lexicals corresponding to the key-value information, facilitating subsequent inference by the model and improving the accuracy of the inference results.

[0114] In the key-value caching compression method, the number of lexical caches for each network layer is first determined, and then the target semantic lexical units are further filtered to achieve key-value information compression and caching of the target semantic lexical units. Based on this, the feasible methods for determining the number of lexical caches for each network layer in a large language model are explained below.

[0115] In one exemplary embodiment, such as Figure 4 As shown, the number of lexical caches corresponding to each network layer in the large language model is obtained, including:

[0116] S401, obtain the budget cache of the large language model, the number of network layers in the large language model, and the key-value cache size of a single lexical.

[0117] Among them, the budget cache of the large language model refers to the upper limit of the cache pre-allocated for inference of the large language model, the number of network layers of the large language model refers to the total number of stacked transformer layers inside the large language model, and the key-value cache size of a single word refers to the storage memory space occupied by the key vector and value vector data corresponding to a single word.

[0118] In one scenario, the budget cache size of the large language model and the key-value cache size of a single word can be manually defined and stored in a preset register. Thus, the budget cache size of the large language model and the key-value cache size of a single word can be obtained by reading the contents of the register.

[0119] In another scenario, a model parsing tool can be invoked to perform a structured decomposition of a large language model and determine the number of network layers in the large language model based on the decomposition results.

[0120] S402 determines the allocated cache for each network layer based on the budget cache and the number of network layers.

[0121] Based on the principle that the sum of the allocated caches of all network layers equals the budget cache, and that the allocated cache of the lower network layer is greater than the allocated cache of the upper network layer in any two adjacent network layers, the budget cache is allocated to each network layer to obtain the allocated cache of each network layer.

[0122] For example, a baseline value for the incremental difference of the cache for each layer is set, and an allocation cache expression for each network layer is constructed. At the same time, an expression is constructed to make the sum of the allocation caches of multiple network layers equal to the budget cache. Combining the above expressions, the allocation cache for each network is calculated.

[0123] For example, the allocation cache of the underlying network layer is set to 1 / 2 of the preset cache, and then the allocation cache of each network layer on the underlying network layer is set layer by layer according to the arithmetic sequence.

[0124] S403 determines the number of lexical caches for each network layer based on the allocated cache size for each network layer and the key-value cache size for a single lexical.

[0125] It should be noted that the key-value cache size of a single word is fixed. Based on this, the ratio between the allocated cache of each network layer and the key-value cache size of a single word can be calculated, and the ratio can be used to determine the word cache size of the corresponding network layer.

[0126] In this embodiment, the allocated cache for each network layer is first determined based on the budget cache of the large language model and the number of network layers of the large language model. Then, based on the allocated cache of each network layer and the key-value cache size of a single word, the number of word caches corresponding to each network layer is determined to avoid idle and wasted cache resources or local cache overflow.

[0127] Next, the implementation method for determining the allocation of caches for each network layer in the foregoing embodiments will be further explained. In an exemplary embodiment, such as Figure 5 As shown, the large language model includes a top-level network layer, a third-level network layer, and at least one intermediate network layer; based on the budget cache and the number of network layers, the allocated cache for each network layer is determined, including:

[0128] S501 determines the allocation cache for the top-level network layer and the allocation cache for the bottom-level network layer based on the budget cache and the number of network layers.

[0129] In an exemplary embodiment, determining the allocation cache for the top-level network layer and the allocation cache for the bottom-level network layer based on the budget cache and the number of network layers includes:

[0130] Calculate the ratio between the budgeted cache and the number of network layers; fuse the ratio with preset hyperparameters to obtain the allocated cache of the top-level network layer; determine the allocated cache of the bottom-level network layer based on the difference between the ratio and the allocated cache of the top-level network layer.

[0131] The preset hyperparameter is a value greater than 1. The allocation cache of the top-level network layer is determined by first calculating the ratio between the budgeted cache and the number of network layers, and then multiplying the ratio by the reciprocal of the preset hyperparameter. The product is the allocation cache of the top-level network layer.

[0132] The allocation cache of the lower network layer is calculated and determined after the allocation cache of the top network layer is determined. Specifically, the product of the ratio and 2 is calculated first, and then the difference between the product and the allocation cache of the top network layer is determined. The difference is the allocation cache of the lower network layer.

[0133] Budget caching For example, if the number of network layers is m, the allocation cache of the top network layer ( The expression for ) is as follows:

[0134]

[0135] The underlying network layer allocation cache ( The expression for ) is as follows:

[0136]

[0137] Where β is a preset hyperparameter, which is a value greater than 1.

[0138] In this embodiment, the ratio between the budget cache and the number of network layers is used as the basis. The ratio and preset hyperparameters are fused to obtain the allocation cache of the top network layer. The allocation cache of the bottom network layer is determined based on the difference between the ratio and the allocation cache of the top network layer. This is equivalent to allocating more cache to the bottom network layer with more dispersed information and less cache to the higher network layer with less sparse information, in order to give full play to the role of the allocation cache.

[0139] S502, calculate the cache difference between the allocated cache of the bottom network layer and the allocated cache of the top network layer, and determine the ratio of the cache difference to the number of network layers as the cache difference between adjacent layers.

[0140] Based on the allocation cache of the top-level network layer The underlying network layer allocates cache as For example, a large language model has m layers starting from layer 0. The expression for the cache difference between adjacent layers is: .

[0141] S503 determines the allocation cache of each intermediate network layer based on the allocation cache of the bottom network layer and the difference between the caches of adjacent layers.

[0142] Continue using the allocation cache of the top-level network layer as The underlying network layer allocates cache as This illustrates the allocation cache of the l-th network layer ( The expression for ) is as follows:

[0143]

[0144] In this embodiment, the allocation cache of the top-level network layer and the allocation cache of the bottom-level network layer are determined based on the budget cache and the number of network layers. Then, the difference between the two and the number of network layers is used to obtain the cache difference between adjacent layers, and the allocation cache of each intermediate network layer is derived in an arithmetic progression. Under the premise that the sum of the allocation caches of all network layers is equal to the budget cache, the cache quota of the bottom-level network layer is greater than that of the adjacent upper layer in a hierarchical incremental allocation. This is in line with the requirement that the deep network of the bottom layer of the large language model has a stronger dependence on contextual information, and improves the accuracy and stability of the model's inference operation.

[0145] Given the number of lexical caches for each network layer, we can further use this as a basis to filter target semantic lexical units from multiple contextual semantic lexical units in each network layer, and persistently store the key-value information of the target semantic lexical units for subsequent reuse.

[0146] In one exemplary embodiment, such as Figure 6 As shown, based on the number of lexical caches, target semantic lexical units that match the task instruction lexical units of the network layer are selected from the context semantic lexical units, including:

[0147] S601, calculate the attention score between each context semantic word and each word in the task instruction word to obtain the semantic quantization value of each context semantic word.

[0148] In practical applications, a task instruction lexicon consists of multiple lexicons, and these multiple task instruction lexicons together constitute a task instruction window.

[0149] For each context semantic lexical, the attention score between the context semantic lexical and each lexical in the task instruction window is calculated to obtain multiple attention scores for a context semantic lexical. Then, the multiple attention scores are fused to obtain the semantic quantization value of the context semantic lexical.

[0150] The semantic quantization value of a context semantic unit is positively correlated with the probability that the context semantic unit is the target semantic unit. The larger the semantic quantization value, the closer the connection between the context semantic unit and the task instruction unit as a whole, and the higher the probability that the context semantic unit will be retained as the target semantic unit; conversely, the smaller the semantic quantization value, the sparser the connection between the context semantic unit and the task instruction unit as a whole, and the lower the probability that the context semantic unit will be retained as the target semantic unit.

[0151] S602, sort the semantic lexical units of each context according to the size of the semantic quantization value.

[0152] The context semantic units are sorted in descending order of their semantic quantization values ​​to obtain the sorting results. The earlier a context semantic unit appears in the list, the higher its probability of being retained as a target semantic unit; the later a context semantic unit appears in the list, the lower its probability of being retained as a target semantic unit.

[0153] S603, determine the target semantic lexical that matches the number of lexical caches from the sorting results of each context semantic lexical.

[0154] From the sorting results of each context semantic lexical, extract the context semantic lexical that matches the number of lexical caches (that is, several context semantic lexical ...

[0155] In this embodiment, the attention score between each context semantic word and each word in the task instruction word is calculated to obtain the semantic quantification value of each context semantic word, so as to quantify the correlation between each context semantic word and the task instruction word. Then, according to the size of the semantic quantification value, each context semantic word is sorted to determine the target semantic word that matches the word cache quantity, so as to remove context semantic words with low correlation and retain high-quality context semantic words.

[0156] In an exemplary embodiment, one possible implementation of the aforementioned step S601, "calculating the attention score between each context semantic lexical and each lexical in the task instruction lexical to obtain the semantic quantization value of each context semantic lexical," will be described, such as... Figure 7 As shown, it includes:

[0157] S701, for any contextual semantic word, calculate the attention score between the contextual semantic word and each word in the task instruction word through each attention head of the network layer, and obtain the head-level attention score of the contextual semantic word in each attention head.

[0158] For any contextual semantic lexical, the attention score between the contextual semantic lexical and each lexical in the task instruction lexical is calculated through an attention head of the network layer, and multiple attention scores are fused, for example, superimposed, to obtain the head-level attention score of the contextual semantic lexical calculated by the attention head.

[0159] Taking a task instruction lexicon consisting of four lexicons as an example, the calculation principle of a single attention head is explained: the semantic correlation between the context semantic lexicon and the four task instruction lexicons is calculated separately to obtain the attention score between the context semantic lexicon and the four task instruction lexicons. Then, the four attention scores are superimposed to obtain a head-level attention score for the context semantic lexicon.

[0160] It should be noted that each network layer includes more than one attention head, and the parameters of different attention heads are also different. Therefore, for the same context semantic lexical, different attention heads will calculate different head attention branches.

[0161] S702, fuse the head attention scores of context semantic units from different attention heads to obtain the semantic quantization value of the context semantic units.

[0162] Optionally, the head attention scores of context semantic units from different attention heads can be superimposed, and the superposition result can be used as the semantic quantification value of the context semantic units.

[0163] Optionally, weights are assigned to each attention head, and weighted fusion is performed in combination with the corresponding head attention score to obtain a weighted fusion result, which is used as the semantic quantification value of the context semantic units.

[0164] In this embodiment, for any contextual semantic lexical, the attention score between the contextual semantic lexical and each lexical in the task instruction lexical is first calculated through each attention head of the network layer to obtain the head-level attention score of the contextual semantic lexical at each attention head. Then, the head-level attention scores of the contextual semantic lexical at different attention heads are fused to obtain the semantic quantification value of the contextual semantic lexical. This eliminates the influence of abnormal activation in individual heads and accurately quantifies the overall contribution of the contextual semantic lexical to understanding the current task's generated instruction. This provides a more accurate ranking basis for subsequent filtering of highly relevant target semantic lexicals based on the number of lexical caches.

[0165] In an exemplary embodiment, one possible implementation of the aforementioned step S603, "determining the target semantic word that matches the word cache quantity from the sorting results of each context semantic word," is described, such as... Figure 8 As shown, it includes:

[0166] S801, the number of cached lexical units is subtracted from the number of key sink lexical units and the number of task instruction lexical units in turn to obtain the number of target semantic lexical units.

[0167] The lexical cache count refers to the total number of lexical units that the network layer needs to cache. Since the key-value information corresponding to the key sink lexical units and task instruction lexical units of each network layer in this embodiment needs to be forcibly retained, this will occupy a certain number of units. Therefore, when determining the target semantic lexical unit count, the key sink lexical units and task instruction lexical units of each network layer need to be removed from the lexical cache count.

[0168] First, the number of cached lexical units is subtracted from the number of key sink lexical units to obtain the first subtraction result, which is the total number of lexical units of the task instruction lexical units and the target semantic lexical units. Then, the first subtraction result is subtracted from the number of lexical units of the task instruction lexical units to obtain the second subtraction result, which is the number of target semantic lexical units.

[0169] S802, determine the target semantic words that match the number of target semantic words from the sorting results.

[0170] From the sorting results of each context semantic word, extract the context semantic words that match the number of target semantic words (that is, several context semantic words with larger semantic quantization values) as the target semantic words.

[0171] In this embodiment, the number of cached lexical units is subtracted from the number of key sink lexical units and the number of task instruction lexical units in turn to obtain the number of target semantic lexical units. Then, the target semantic lexical units that match the number of target semantic lexical units are determined from the sorting results. This is equivalent to prioritizing the storage and usage needs of key sink lexical units and task instruction lexical units on the basis of caching in the adaptation network layer. This forces the model to make the most effective selection of semantic information within limited resources, thereby avoiding the pruning of key information and ensuring the core semantic logic of the reasoning task.

[0172] In one exemplary embodiment, a compression method for key-value caching is provided, comprising the following steps:

[0173] (1) Obtain the word sequence corresponding to each network layer in the large language model.

[0174] (2) Obtain the budget cache of the large language model, the number of network layers in the large language model, and the key-value cache size of a single word.

[0175] (3) Calculate the ratio between the budget cache and the number of network layers; combine the ratio with the preset hyperparameters to obtain the allocation cache of the top network layer.

[0176] (4) Determine the allocation cache of the bottom network layer based on the difference between the ratio and the allocation cache of the top network layer.

[0177] (5) Calculate the cache difference between the allocation cache of the bottom network layer and the allocation cache of the top network layer, and determine the ratio of the cache difference to the number of network layers as the cache difference between adjacent layers.

[0178] (6) Determine the allocation cache of each intermediate network layer based on the difference between the allocation cache of the bottom network layer and the cache of the adjacent layer.

[0179] (7) Determine the number of word caches corresponding to each network layer based on the allocated cache of each network layer and the key value cache size of a single word.

[0180] (8) For any network layer, the word in the word sequence of the network layer at the first preset position is determined as the key sink word; the word in the word sequence of the network layer at the second preset position is determined as the task instruction word; and the context semantic word is determined based on the word in the word sequence other than the key sink word and the task instruction word.

[0181] (9) For any context semantic word, calculate the attention score between the context semantic word and each word in the task instruction word through each attention head of the network layer to obtain the head-level attention score of the context semantic word in each attention head; fuse the head-level attention scores of the context semantic word in different attention heads to obtain the semantic quantization value of the context semantic word.

[0182] (10) Sort the semantic lexical units of each context according to the size of the semantic quantization value.

[0183] (11) Subtract the number of word caches from the number of word caches for key sink words and the number of word caches for task instruction words to obtain the number of target semantic words.

[0184] (12) Identify the target semantic words that match the number of target semantic words from the sorting results.

[0185] (13) Compress the key-value information corresponding to the key sink words and target semantic words in the network layer to obtain the compressed key-value inference information of the network layer.

[0186] (14) Cache the compressed key-value inference information of all network layers.

[0187] Please see Figure 9 , Figure 9 This is a schematic diagram of compressed key-value inference information for each network layer of a large language model. Figure 9 In this model, the key sink terms and task instruction terms corresponding to the key value information retained by different network layers are fixed, while the target semantic terms corresponding to the key value information retained by different network layers are different.

[0188] In this embodiment, for each network layer, after classifying the lexical sequence to identify key sink lexical units, task instruction lexical units, and contextual semantic lexical units, the key-value information corresponding to the key sink lexical units is forcibly retained to ensure the relative stability of the model's attention weight distribution. Furthermore, based on the number of lexical units cached, target semantic lexical units associated with task instruction lexical units are selected, and their key-value information is compressed and stored. In this way, both key-value information of key sink lexical units and key-value information of target semantic lexical units in each network layer are accurately retained. The former ensures that the large language model can think normally and avoids computational crashes, while the latter aims to ensure that the large language model thinks correctly and relevantly, injecting the information required for the task. This ensures that the accuracy of the inference output is not affected, while adapting to various business tasks and low-computing-power deployment scenarios, achieving efficient, lightweight, and highly adaptable operation of the large language model's inference process.

[0189] In one embodiment, a reasoning method for a large language model is provided, such as... Figure 10 As shown, it includes:

[0190] S1001, Begin.

[0191] S1002, Obtain the input text for the large language model.

[0192] S1003, convert the input text into a text sequence.

[0193] S1004 performs hierarchical caching management for each network layer of the large language model.

[0194] S1005 allocates a cache for each network layer.

[0195] S1006 forces the retention of key sink words and task instruction words in the word sequence of each network layer, and selects target semantic words that match the allocation cache from the context semantic words.

[0196] To address the vulnerability of attention distribution caused by the constraints of the Softmax normalization layer in large language models, this study analyzes the attention patterns of different layers and heads of the model to identify lexical positions that consistently receive abnormally high attention weights regardless of the input content. Experiments show that in most autoregressive language models, the initial 1-4 lexical units in a sequence are assigned an "abnormally high" number of attention scores. Verification through "lexical replacement experiments" (e.g., replacing initial lexical units with meaningless newlines) confirms that even when the semantics of these lexical units are stripped away, the model still assigns them high attention, and removing them leads to a significant increase in perplexity, thus confirming their role as structural sinks rather than semantic keypoints.

[0197] Furthermore, based on structural stability, the density and utility of semantic information in the cache space are maximized. When identifying semantically important terms, the "instruction window" or "recent context" of the current decoding step is used as the query benchmark. This window typically contains the last few terms (e.g., 8), representing the most direct task instructions and immediate context. For each historical term (except for sinks preserved by the structural sink preservation track), the cumulative sum of its attention scores for each query term within the "instruction window" across all attention heads is calculated. This cumulative score quantifies the overall contribution of the historical term to understanding the currently generated instruction. To eliminate the influence of "abnormal activations" in individual heads, a cross-head pooling operation can be introduced in the score calculation to obtain more robust evaluation results.

[0198] Within each layer, the system selects the Top-K lexics with the highest semantic importance scores from all historical lexics, based on the dynamic cache budget allocated to that layer (determined by an independent inter-layer budget allocation scheme). The selection process is competitive, with the retention or rejection of a lexicographical term entirely dependent on its real-time semantic value relative to the current generation task. Ultimately, the KV cache retained at that layer consists of both "forced-retained sink lexics" and "winners of the competition for semantically important lexics."

[0199] Thus, the design process fully considers that the "dual tracks" are not two independent processes, but a highly collaborative and orderly unified system. In the process of lexical identification and retention, it is explicitly stipulated that the retention of structural sinks has a higher priority than semantic importance filtering. The cache budget used for semantic importance filtering is the total layer budget minus the fixed overhead reserved for storing key sink lexical terms. This forces the model to make the most effective selection of semantic information within limited resources. In this way, through the collaborative work of the "dual tracks," the first track ensures the model can think normally and avoids computational crashes; the second track focuses on ensuring the model thinks correctly and relevantly, injecting the information required for the task—both are indispensable.

[0200] S1007, cache target semantic terms.

[0201] S1008, Output result.

[0202] S1009, Expel non-target semantic words from context semantic words.

[0203] S1010, release cached resources.

[0204] S1011 caches the key-value information of key sink terms, target semantic terms, and task instruction terms in each network layer.

[0205] S1012, based on the cached information of each network layer, performs text inference to obtain the inference result corresponding to the input text.

[0206] In this embodiment, different hierarchical key-value compression processes are applied to different Transformer layers of the large model. This not only significantly reduces the amount of key-value memory occupied but also preserves the long-context understanding capability of the large model, significantly improving the inference efficiency of the large model and thus fully utilizing the computing power of domestically produced accelerator cards. The method in this embodiment can be used as a general optimization module, adapted to mainstream Transformer architectures, and has good scalability and engineering implementation potential. Its advantages include, but are not limited to, the following:

[0207] (1) Stability guarantee of attention mechanism.

[0208] By using structural sink anchoring, the attention distribution drift problem caused by abrupt changes in the denominator of Softmax is completely resolved, ensuring the model maintains mathematical stability even under extreme cache compression. Even when the starting lexical unit is replaced with a meaningless symbol, the model performance recovers, proving that this method addresses positional bias dependencies rather than simple semantic dependencies. This enables large models to stably handle infinitely long text streams, resolving the fundamental problem of performance collapse in traditional window attention methods after exceeding the cache limit.

[0209] (2) A leapfrog improvement in memory efficiency.

[0210] Experimental evaluation using the LongBench benchmark shows that, at the same generation quality, it matches the performance of a model with a full KV cache while retaining only 12% of the KV cache, thus significantly reducing memory usage. In scenarios where memory efficiency is critical, it outperforms other KV cache compression techniques while maintaining only 0.7% of the KV cache.

[0211] (3) Supports plug-and-play, broad compatibility, and efficient deployment.

[0212] As a key-value cache management layer that requires no modification to the Transformer architecture and weights, it can be seamlessly integrated into mainstream inference frameworks through standardized interfaces and has been validated for effectiveness on various autoregressive models using different encodings. The system maintains positional consistency through in-cache position remapping, with computational overhead accounting for only about 0.01%–0.05% of inference time. While maintaining support for multiple precisions such as FP16 / INT8, it is fully compatible with paged attention memory management, enabling efficient and flexible deployment across all scenarios from edge to cloud.

[0213] (4) The model has better performance.

[0214] The performance of the Mistral7B-Instruct model was compared on a domestic accelerator card using the LongBench long test dataset. The proposed method outperformed the original Pyramid KV cache compression method when the KV size was set to 64. The experimental data are shown in Table 1. Table 1 includes the multiple task types involved in the comparative experiment, the dataset for each task type, the number of samples in each dataset, the evaluation scores of the basic large model (Mistral-7B-Instruct, KV cache budget limit = 64 words) on each dataset for the corresponding task, and the evaluation scores of the compression method of this application on each dataset for the corresponding task. The higher the overall evaluation score, the less cache the model occupies and the higher the model inference accuracy.

[0215] Table 1

[0216]

[0217] As shown in Table 1, when performing corresponding tasks based on multiple datasets, the overall test score of the model inference method provided in this application embodiment is higher than that of the caching and compression methods in related technologies.

[0218] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0219] Based on the same inventive concept, this application also provides a key-value cache compression apparatus for implementing the key-value cache compression method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more key-value cache compression apparatus embodiments provided below can be found in the limitations of the key-value cache compression method described above, and will not be repeated here.

[0220] In one exemplary embodiment, such as Figure 11As shown, a key-value caching compression device is provided, comprising: a sequence acquisition module 1101, a lexical segmentation module 1102, a lexical filtering module 1103, a lexical compression module 1104, and a key-value caching module 1105, wherein:

[0221] Sequence acquisition module 1101 is used to acquire the lexical sequences and the number of lexical caches corresponding to each network layer in the large language model;

[0222] The lexical segmentation module 1102 is used to split the lexical sequence of any network layer into key sink lexical units, task instruction lexical units, and context semantic lexical units according to the position of each lexical unit in the lexical sequence of the network layer. Key sink lexical units include base point lexical units that represent the overall semantics of the lexical sequence. Task instruction lexical units include instruction lexical units that clarify the execution goal and behavior of the large language model. Context semantic lexical units include lexical units in the lexical sequence other than key sink lexical units and task instruction lexical units.

[0223] The lexical filtering module 1103 is used to filter target semantic lexical units that match the task instruction lexical units of the network layer from the context semantic lexical units based on the number of lexical units cached.

[0224] The lexical compression module 1104 is used to compress the key-value information corresponding to the key sink lexical and target semantic lexical in the network layer to obtain the compressed key-value inference information of the network layer.

[0225] The key-value caching module 1105 is used to cache the compressed key-value inference information of all network layers.

[0226] In an exemplary embodiment, the sequence acquisition module 1101 includes: a cache acquisition unit, a cache allocation unit, and a quantity determination unit, wherein:

[0227] The cache retrieval unit is used to retrieve the budget cache of the large language model, the number of network layers in the large language model, and the key-value cache size of a single word.

[0228] The cache allocation unit is used to determine the allocated cache for each network layer based on the budget cache and the number of network layers;

[0229] The quantity determination unit is used to determine the number of lexical caches corresponding to each network layer based on the allocation cache of each network layer and the key-value cache size of a single lexical.

[0230] In an exemplary embodiment, the large language model includes a top-level network layer, a bottom-level network layer, and at least one intermediate network layer; the cache allocation unit is further configured to determine the allocation cache of the top-level network layer and the allocation cache of the bottom-level network layer based on the budget cache and the number of network layers; calculate the cache difference between the allocation cache of the bottom-level network layer and the allocation cache of the top-level network layer, and determine the ratio of the cache difference to the number of network layers as the adjacent layer cache difference; and determine the allocation cache of each intermediate network layer based on the allocation cache of the bottom-level network layer and the adjacent layer cache difference.

[0231] In an exemplary embodiment, the cache allocation unit is further configured to calculate the ratio between the budgeted cache and the number of network layers; fuse the ratio with preset hyperparameters to obtain the allocation cache of the top-level network layer; and determine the allocation cache of the bottom-level network layer based on the difference between the ratio and the allocation cache of the top-level network layer.

[0232] In an exemplary embodiment, the word segmentation module 1102 includes: a first segmentation unit and a second segmentation unit, wherein:

[0233] The first segmentation unit is used to identify the word in the word sequence of the network layer at the first preset position as the key sink word; and to identify the word in the word sequence of the network layer at the second preset position as the task instruction word.

[0234] The second segmentation unit is used to determine the context semantic lexical units based on the lexical units in the lexical sequence, excluding key sink lexical units and task instruction lexical units.

[0235] In an exemplary embodiment, the lexical filtering module 1103 includes: a quantization value sorting unit, a lexical sorting unit, and a lexical matching unit, wherein:

[0236] The quantization value sorting unit is used to calculate the attention score between each context semantic word and each word in the task instruction word, and obtain the semantic quantization value of each context semantic word.

[0237] The lexical sorting unit is used to sort the semantic lexical units of each context according to the size of their semantic quantization values.

[0238] The lexical matching unit is used to determine the target semantic lexical that matches the number of lexical caches from the sorting results of the semantic lexical ...

[0239] In an exemplary embodiment, the quantization value sorting unit is further configured to, for any context semantic lexical, calculate the attention score between the context semantic lexical and each lexical in the task instruction lexical through each attention head of the network layer, to obtain the head-level attention score of the context semantic lexical at each attention head; and fuse the head-level attention scores of the context semantic lexical at different attention heads to obtain the semantic quantization value of the context semantic lexical.

[0240] In an exemplary embodiment, the quantization value sorting unit is further configured to subtract the number of lexical caches from the number of lexicals of key sink lexicals and the number of lexicals of task instruction lexicals in turn to obtain the number of target semantic lexicals; and to determine the target semantic lexicals that match the number of target semantic lexicals from the sorting results.

[0241] Each module in the aforementioned key-value caching compression device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0242] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0243] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0244] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0245] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0246] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0247] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0248] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A compression method for key-value caching, characterized in that, The method includes: Obtain the lexical sequences and lexical cache counts corresponding to each network layer in a large language model; For any network layer, the word sequence of the network layer is split into key sink words, task instruction words, and context semantic words according to the position of each word in the word sequence. The key sink words include base words that represent the overall semantics of the word sequence. The task instruction words include instruction words that clarify the execution goal and behavior of the large language model. The context semantic words include words in the word sequence other than the key sink words and the task instruction words. Based on the number of word caches, target semantic words that match the task instruction words of the network layer are selected from the context semantic words, including: calculating the attention score between each context semantic word and each word in the task instruction words to obtain the semantic quantization value of each context semantic word; sorting each context semantic word according to the size of the semantic quantization value; and determining the target semantic words that match the number of word caches from the sorting results of each context semantic word. The key-value information corresponding to the key sink words and target semantic words in the network layer is compressed to obtain the compressed key-value inference information of the network layer. Compressed key-value inference information for all network layers is cached.

2. The method according to claim 1, characterized in that, The process of obtaining the number of lexical caches corresponding to each network layer in the large language model includes: Obtain the budget cache of the large language model, the number of network layers in the large language model, and the key-value cache size of a single word; Based on the budget cache and the number of network layers, determine the allocated cache for each network layer; The number of lexical caches corresponding to each network layer is determined based on the allocated cache size of each network layer and the key-value cache size of a single lexical.

3. The method according to claim 2, characterized in that, The large language model includes a top-level network layer, a third-level network layer, and at least one intermediate network layer; determining the allocation cache for each network layer based on the budget cache and the number of network layers includes: Based on the budget cache and the number of network layers, determine the allocation cache for the top-level network layer and the allocation cache for the bottom-level network layer; Calculate the cache difference between the allocated cache of the bottom network layer and the allocated cache of the top network layer, and determine the ratio of the cache difference to the number of network layers as the cache difference between adjacent layers; The allocation cache of each intermediate network layer is determined based on the difference between the allocation cache of the bottom network layer and the cache of the adjacent layer.

4. The method according to claim 3, characterized in that, The step of determining the allocation cache for the top-level network layer and the allocation cache for the bottom-level network layer based on the budget cache and the number of network layers includes: Calculate the ratio between the budget cache and the number of network layers; The ratio and preset hyperparameters are fused to obtain the allocation cache of the top-level network layer; The allocation cache of the bottom network layer is determined based on the difference between the ratio and the allocation cache of the top network layer.

5. The method according to any one of claims 1-4, characterized in that, The step of splitting the lexical sequence of the network layer into key sink lexical units, task instruction lexical units, and contextual semantic lexical units includes: The word in the word sequence of the network layer at the first preset position is determined as the key sink word; the word in the word sequence of the network layer at the second preset position is determined as the task instruction word. The context semantic lexical units are determined based on the lexical units in the lexical sequence excluding the key sink lexical units and the task instruction lexical units.

6. The method according to claim 1, characterized in that, The step of calculating the attention score between each context semantic lexical and each lexical in the task instruction lexical to obtain the semantic quantization value of each context semantic lexical includes: For any contextual semantic word, through each attention head of the network layer, the attention score between the contextual semantic word and each word in the task instruction word is calculated to obtain the head-level attention score of the contextual semantic word at each attention head; The semantic quantization value of the context semantic lexical is obtained by fusing the head-level attention scores of different attention heads.

7. The method according to claim 1, characterized in that, Determining the target semantic word that matches the word cache quantity from the sorting results of each of the context semantic words includes: The number of target semantic words is obtained by subtracting the number of words in the cached word from the number of words in the key sink word and the number of words in the task instruction word. The target semantic words that match the number of target semantic words are determined from the sorting results.

8. A compression device for key-value caching, characterized in that, The device includes: The sequence acquisition module is used to obtain the word sequence and word cache number corresponding to each network layer in the large language model; The lexical segmentation module is used to, for any network layer, divide the lexical sequence of the network layer into key sink lexical units, task instruction lexical units, and contextual semantic lexical units according to the position of each lexical unit in the lexical sequence of the network layer; the key sink lexical units include base point lexical units that represent the overall semantics of the lexical sequence; the task instruction lexical units include instruction lexical units that clarify the execution goal and behavior direction of the large language model; the contextual semantic lexical units include lexical units in the lexical sequence other than the key sink lexical units and the task instruction lexical units; The lexical filtering module is used to filter target semantic lexical units that match the task instruction lexical units of the network layer from the context semantic lexical units according to the number of lexical units cached. This includes: calculating the attention score between each context semantic lexical unit and each lexical unit in the task instruction lexical unit to obtain the semantic quantization value of each context semantic lexical unit; sorting each context semantic lexical unit according to the magnitude of the semantic quantization value; and determining the target semantic lexical units that match the number of lexical units cached from the sorting results of each context semantic lexical unit. The lexical compression module is used to compress the key-value information corresponding to the key sink lexical and target semantic lexical in the network layer to obtain the compressed key-value inference information of the network layer. The key-value caching module is used to cache compressed key-value inference information for all network layers.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images