Storage optimization method and device, electronic device and storage medium

Abstract

The present application discloses a storage optimization method and device, electronic device and storage medium, and relates to the field of data processing technology. The storage optimization method and device, electronic device and storage medium of the present application compress the space occupied by the cache by adopting the strategy of retention and selection. When allocating cache to each network layer, the cache is allocated in a gradually decreasing manner from the low layer to the high layer key value cache, so as to avoid missing key words at the low layer and avoiding saving unimportant words at the high layer. Then, the attention focus degree of the input word and the cache word, that is, the target attention value, is used to select the most influential key words in the cache word for the input word and cache it. Then, representative representative words are selected from the remaining words for storage, and only words that have an impact on the model processing data can be stored. The technical effect of compressing the space occupied by the KV cache is achieved, thereby providing the processing capability for more requests.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a storage optimization method and device, an electronic device, and a storage medium. Background Art

[0002] Large Language Models (LLMs) have achieved significant breakthroughs in the field of natural language processing, demonstrating strong generalization capabilities and practical effects in many tasks, including text generation, language understanding, code completion, question-answering systems, and translation.

[0003] To improve efficiency during LLM reasoning, a key-value cache (KV) mechanism is used to help the model avoid recalculating previous tokens (i.e., words, which refer to the smallest units with independent meaning into which text is segmented during natural language processing) when processing data, thereby accelerating the reasoning process. When using KV caching, the model caches the calculated key values. When processing the next round of data, the model can directly use the cached key values ​​without recalculating all inputs. However, the KV cache will continue to grow as the reasoning process progresses, resulting in a sharp decrease in video memory space, which in turn reduces the LLM's data processing ability. Summary of the Invention

[0004] The present application provides a storage optimization method and device, an electronic device, and a storage medium to at least solve the problem in the related art of how to effectively reduce the storage space occupied by the KV cache without affecting the model accuracy and reasoning quality.

[0005] This application provides a storage optimization method, including:

[0006] Performing key-value cache allocation processing in a preset model to obtain first key-value caches corresponding to multiple network layers in the preset model, wherein the higher the level of the multiple network layers, the smaller the first key-value cache;

[0007] Calculating target attention values ​​of multiple cached word-grams corresponding to multiple input word-grams and multiple cached key values ​​in a preset model, and performing word-gram selection processing from the multiple cached word-grams based on the first key value cache and the target attention values ​​to obtain a keyword word-gram;

[0008] Constructing word-meta vectors corresponding to each of the multiple other word-meta based on the target attention value, and performing word-meta selection processing from the multiple other word-meta based on the first key-value cache and the word-meta vector to obtain a representative word-meta, wherein the multiple other word-meta are cached word-meta in the multiple cached word-meta except the keyword word-meta;

[0009] The key value corresponding to the keyword word, the input key values ​​corresponding to each of the plurality of input word units, and the representative key value corresponding to the representative word unit are stored and processed.

[0010] The present application also provides a storage optimization device, comprising:

[0011] an allocation unit, configured to perform key-value cache allocation processing in a preset model to obtain first key-value caches corresponding to respective multiple network layers in the preset model, wherein a higher level of the multiple network layers corresponds to a smaller first key-value cache;

[0012] A calculation unit, configured to calculate target attention values ​​of a plurality of cached word elements corresponding to a plurality of input word elements and a plurality of cached key values ​​in a preset model;

[0013] A first selection unit is configured to select a word from a plurality of cached word elements according to the first key value cache and the target attention value to obtain a keyword word;

[0014] A construction unit, used to construct word element vectors corresponding to multiple other word elements according to the target attention value;

[0015] a second selection unit, configured to select a word from a plurality of other word-grams according to the first key-value cache and the word-gram vector to obtain a representative word-gram, wherein the plurality of other word-grams are cached word-grams other than the keyword word-gram in the plurality of cached word-grams;

[0016] The storage unit is used to store and process the key key value corresponding to the key word, the input key values ​​corresponding to each of the multiple input word units, and the representative key value corresponding to the representative word unit.

[0017] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned storage optimization methods when executing the computer program.

[0018] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned storage optimization methods are implemented.

[0019] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above storage optimization methods when executed by a processor.

[0020] The storage optimization method and device, electronic device and storage medium of the present application compress the space occupied by the cache by adopting the strategy of retention and selection. When allocating cache for each network layer, the cache is allocated in a gradually decreasing manner from the low layer to the high layer, so as to avoid missing key words at the low layer and saving unimportant words at the high layer. Then, the attention focus degree between the input word and the cache word, that is, the target attention value, is used to select the most influential key words in the cache word for the input word and cache them. Then, representative representative words are selected from the remaining words for storage. Only words that have an impact on the model processing data can be stored, without storing all cached words. Therefore, the technical problem of how to effectively reduce the storage space occupied by the KV cache without affecting the model accuracy and reasoning quality can be solved, so as to achieve the technical effect of compressing the space occupied by the KV cache and providing the processing capability for more requests. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 A flowchart of a storage optimization method provided in an embodiment of the present application;

[0023] Figure 2 A flowchart of a storage optimization process provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of the architecture of a storage optimization method provided in an embodiment of the present application;

[0025] Figure 4 A storage optimization device provided in an embodiment of the present application;

[0026] Figure 5 Another storage optimization device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0028] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0029] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0030] Figure 1 A flowchart of a storage optimization method provided in an embodiment of the present application is provided, and the method is described in detail in conjunction with the execution process of the storage optimization method.

[0031] like Figure 1 As shown, the storage optimization method includes:

[0032] Step 101 : performing key-value cache allocation processing in a preset model to obtain first key-value caches corresponding to multiple network layers in the preset model, wherein the higher the level of the multiple network layers, the smaller the corresponding first key-value cache.

[0033] In an embodiment of the present application, the preset model is a custom-selected language model, namely an LLM, such as the Generative Pre-trained Transformer (GPT) and the Large Language Model Meta AI (LLaMA). The key-value cache refers to the upper limit of the memory resources allowed for storing the keys and values ​​of historical tokens during inference of the preset model, usually measured by the number of cacheable tokens. That is, the first key-value cache is the upper limit of the memory resources allowed for storing the keys and values ​​of historical tokens at each network layer in the preset model.

[0034] A network layer refers to the multi-layer decoder structure that makes up the pre-set model, arranged from lowest to highest (from the bottom layers closest to the input to the top layers closest to the output). A key characteristic of the allocation strategy is that lower-level networks receive larger first-key-value caches, while higher-level networks receive smaller first-key-value caches. This allocation strategy stems from the inherent characteristics of the model: the low-level attention mechanism tends to focus broadly on all input information, requiring a larger cache space to retain potentially useful information; while the high-level attention mechanism is highly focused on a small number of key words that embody core semantics, so a smaller cache space can be allocated.

[0035] Specifically, the principle behind allocating key-value caches in the pre-set model is that the model's attention distribution at low levels is roughly uniform, tending to broadly aggregate information from all input content without paying particular attention to any particular input fragment. At mid-levels, attention gradually converges on local tokens within each document, gradually summarizing local information. Ultimately, at high levels, the attention mechanism significantly focuses on a few key tokens, with these tokens receiving extremely high attention scores, indicating that the pre-set model has effectively aggregated key information onto these tokens. Therefore, a gradually decreasing approach is used when allocating key-value caches to each layer to avoid missing key tokens at low levels and saving unimportant tokens at high levels.

[0036] Step 102 calculates target attention values ​​of multiple cached words corresponding to multiple input words and multiple cache key values ​​in a preset model, and performs word selection processing from the multiple cached words based on the first key value cache and the target attention value to obtain keyword words.

[0037] In the embodiments of this application, a word-gram is a basic semantic unit of the input text (request information) of a preset model after word segmentation. Multiple input words refer to words in a new input sequence that is currently being processed, and multiple cached words refer to historical words corresponding to key values ​​that have been generated or processed and are currently cached in video memory.

[0038] The target attention value quantifies the degree of attention or dependence of each input word on each historical cache word. It is usually calculated through the softmax in the attention mechanism and reflects the importance of historical information in the current input context.

[0039] Based on the calculated target attention value and the first key-value cache allocated for the current layer (which includes a sub-cache space specifically allocated for keyword tokens), a token selection process is performed to filter out keyword tokens from multiple cached tokens. Keyword tokens are the cached tokens that are most important for processing the current input sequence and have the greatest impact on maintaining the coherence and accuracy of the generated content. The selection process is usually based on the ranking of the target attention values ​​(for example, selecting the cached tokens with the highest attention scores) and ensuring that the number of selected keyword tokens does not exceed the first key-value cache quota of each network layer.

[0040] Step 103: construct word element vectors corresponding to multiple other word elements according to the target attention value, and perform word element selection processing from multiple other word elements according to the first key value cache and the word element vector to obtain a representative word element, wherein the multiple other word elements are cached word elements in the multiple cached word elements except the keyword word element.

[0041] In the embodiment of the present application, for the remaining cached word-grams (i.e., multiple other word-grams) that were not selected as key words, corresponding word-gram vectors are constructed based on the target attention value. The word-gram vector is an abstract representation whose dimension is related to the number of attention heads in the preset model.

[0042] The construction method includes but is not limited to: comparing the normalized attention value of each other word on each attention head with a preset conversion threshold to generate a vector consisting of binary bits (0 or 1). Each binary bit represents the decision of the corresponding attention head to retain (1) or discard (0) the other word. Therefore, the word vector compactly encodes the semantic importance distribution characteristics of the other word under different attention modes of the preset model.

[0043] Also based on the first key-value cache of the network layer (which also includes a sub-cache space specifically allocated for representative terms), the term vector is used to perform further term selection processing to obtain a representative term. The selection strategy includes at least a clustering algorithm, such as calculating the mean anchor vector of all term vectors, calculating the Hamming distance (i.e., the number of binary bit differences) between each term vector and the anchor vector, sorting the terms according to the Hamming distance and grouping them (buckets), and finally randomly selecting a term from each term group as the representative of that term group, i.e., the representative term. The representative term can approximate the overall semantic information of the remaining term group with a smaller number of terms.

[0044] In step 104 , the key value corresponding to the key word, the input key values ​​corresponding to the plurality of input word units, and the representative key value corresponding to the representative word unit are stored.

[0045] In the embodiment of the present application, the filtered keyword units and the multiple input word units currently input (referring to the keyword units at the end of the target word unit sequence obtained after the input text (request information) is processed by the preset model, such as: the keyword units at the end of the target word unit sequence The KVs of multiple input tokens also need to be cached), along with the key values ​​of the selected representative tokens, are stored in the KV cache area of ​​the graphics memory. The retained token keys and values ​​are reused in subsequent attention calculations, allowing the preset model to continue generating the next token.

[0046] By allocating key-value caches in descending order based on network levels, the approach aligns with the inherent principle of LLM, which aggregates broad information at the bottom layer and focuses on core information at the top layer, effectively avoiding the omission of key information at the bottom layer and the storage of redundant information at the top layer. By accurately screening key words based on target attention values, the historical context most relevant to the current reasoning task is prioritized, maintaining the accuracy and contextual consistency of the model output. By converting the remaining words into attention pattern vectors and efficiently clustering and selecting representative words, the semantic representation capabilities of the compressed word group are maintained to the greatest extent possible while significantly compressing the cache space. Overall, this method can significantly reduce the memory usage of the KV cache, enabling the LLM reasoning service to support larger batch processing scales (increasing throughput) and process longer input sequences (enhancing long-context task capabilities) under the same hardware resources, while maintaining model generation quality. This improves resource utilization efficiency and expands the application scope of large models.

[0047] The storage optimization method of the present application compresses the space occupied by the cache by adopting a retention and selection strategy. When allocating cache space for each network layer, it allocates it in a gradually decreasing manner from the low layer to the high layer key-value cache, so as to avoid missing key words at the low layer and avoiding saving unimportant words at the high layer. Then, the attention focus degree between the input word and the cache word, that is, the target attention value, is used to select the most influential key words in the cache word for the input word and cache them. Then, representative representative words are selected from the remaining words for storage. Only words that have an impact on the model processing data can be stored, without storing all cached words. Therefore, the technical problem of how to effectively reduce the storage space occupied by the KV cache without affecting the model accuracy and reasoning quality can be solved, so as to achieve the technical effect of compressing the space occupied by the KV cache and providing the ability to process more requests.

[0048] At the same time, in order to facilitate understanding of the implementation process of the embodiment of the present application, the embodiment of the present application provides a flowchart of a storage optimization process, such as Figure 2 As shown, it includes KV cache management module, key token (word element) retention module, and representative token selection module. The function of the KV cache management module is to perform key-value cache allocation processing, the function of the key token (word element) retention module is to select key words, and the function of the representative token selection module is mainly to select representative words. Figure 2 The budget in is the size of the key value cache, such as: the key budget is the key key value cache, the reserved budget is the reserved total cache, and the specific description of the subsequent embodiments can be referred to. Figure 2 For further understanding.

[0049] In one implementation of the embodiment of the present application, when performing key-value cache allocation processing in a preset model, it can also be implemented in the following manner but is not limited to: determining a preset total key-value cache, wherein the preset total key-value cache includes a first total cache of multiple network layers and a reserved total cache of multiple network layers; allocating the reserved total cache according to the number of levels of the multiple network layers, the number of attention heads corresponding to each of the multiple network layers, and the preset number of inputs to obtain a first reserved cache corresponding to each of the multiple network layers, wherein the preset number of inputs is the number of pre-configured input words, and each network layer includes multiple attention heads; allocating the first total cache according to the number of levels, the levels corresponding to each of the multiple network layers, and the preset layer cache ratio to obtain an initial key-value cache corresponding to each of the multiple network layers, wherein the preset layer cache ratio is the ratio between the initial key-value cache corresponding to the lowest level network layer and the initial key-value cache corresponding to the highest level network layer in the pre-configured multiple network layers; and adding the first reserved cache and the initial key-value cache corresponding to each of the multiple network layers to obtain a first key-value cache corresponding to each of the multiple network layers.

[0050] In an embodiment of the present application, first, a preset total key-value cache is determined (i.e., the maximum cache resource limit allocated in the preset model, usually measured in terms of the number of storable words). The preset total key-value cache includes at least two parts: the first total cache is a key-value cache for caching words allocated to multiple network layers; the reserved total cache is a key-value cache dedicated to caching multiple input words. Among them, the preset input number refers to the pre-configured number of input words to be retained (such as the last 5 words of the target word sequence), the number of attention heads is the number of parallel computing units in the attention mechanism of each network layer (i.e., the number of attention heads included in each network layer), and the number of layers is the total number of multiple network layers.

[0051] The first reserved cache is the cache amount of each network layer dedicated to caching input words; the preset layer cache ratio refers to the ratio of the initial key-value cache between the bottom and top layers of multiple network layers, which is used to control the cache reduction rate; the initial key-value cache is the key-value cache of the basic cache words of each network layer that does not include the reserved cache for input words.

[0052] When allocating the reserved total buffer, the reserved total buffer is allocated to each network layer according to the number of network layers, the number of attention heads in each layer (usually the same for each layer), and the preset number of inputs. The first reserved buffer obtained by each network layer is , that is, each network layer needs to reserve for each attention head word space, used to store input word, where Enter the quantity for the preset, is the number of attention heads. At this point, the total reserved buffer can be determined to be , is the number of levels.

[0053] Subsequently, the first total cache is hierarchically allocated. The allocation method of the first total cache can be expressed by, but not limited to, formula (1):

[0054] Formula (1)

[0055] in, The preset layer cache ratio is defined as the ratio of the lowest network layer (layer 1) to the highest network layer (layer 2). Initial key-value cache ratio of the layer); is the number of levels, is the first total cache, The level represents the first of multiple network layers. Layer Network Layer, For the Initial key-value cache for the network layer.

[0056] Finally, the two types of caches at each network layer are merged, that is, the first reserved cache allocated to each network layer Added to the initial key-value cache, the final first key-value cache of the network layer is obtained (e.g., The first key-value cache of the network layer is ).

[0057] By separating the first total cache (historical word-meta cache) from the reserved total cache (end-word-meta cache), task focus information (such as the user's latest command) is forcibly retained. By presetting the cache ratio at each level, the cache decreases as the network layer increases. This precisely matches the LLM's characteristic of broad aggregation at the bottom layer and core focus at the top layer, preventing key information from being missed at the lower layers or redundant storage at the higher layers. Finally, through cache merging, each layer's cache space covers both historical keyword metata and current input focus, significantly improving video memory utilization. This allocation mechanism lays the resource foundation for subsequent keyword metata screening and representative word metata compression, optimizing storage efficiency from the source.

[0058] In one possible implementation method of the embodiment of the present application, after obtaining the first key-value cache, it can also be implemented in but not limited to the following methods: the initial key-value caches corresponding to multiple network layers are divided and processed according to a preset key cache ratio to obtain key key-value caches and representative key value caches corresponding to multiple network layers; in multiple network layers, the key key-value caches are evenly distributed according to the number of attention heads to obtain attention key-value caches corresponding to multiple attention heads.

[0059] In an embodiment of the present application, the preset key cache ratio is a custom-set ratio, which is a preset parameter used to determine the key key cache ratio when dividing the initial key value cache, for example, 20%. The key key cache refers to the sub-key cache quota in each network layer that is specifically used to filter and retain key words. The representative key cache refers to the sub-key cache quota in each network layer that is specifically used for clustering, selecting, and retaining representative words. The attention key cache is a cache amount independently obtained by each attention head for filtering its corresponding key word.

[0060] The initial key-value cache corresponding to each network layer is divided according to a preset key-cache ratio, splitting the initial key-value cache into two specialized sub-caches: a key-value cache and a representative key-value cache. The key-value cache is used to subsequently retain the core historical tokens (keyword tokens) that have the greatest impact on the current task; the representative key-value cache supports clustering and selecting representative tokens from the remaining tokens, ensuring a balanced allocation of cache resources at each layer between retaining critical information and compressing redundant information.

[0061] Specifically, the key value cache and the representative key value cache can be obtained by, but not limited to, formula (2):

[0062] ,

[0063] Formula (2)

[0064] in, Indicates the Key-value cache of the network layer, Indicates the The network layer represents the key-value cache, The preset key cache ratio.

[0065] After completing the hierarchical cache division, the key-value cache needs to be adapted to a finer-grained computing unit, namely the attention head. Within each network layer, the key-value cache is evenly distributed according to the number of attention heads contained in this network layer (that is, the number of independent parallel computing units in the multi-head attention mechanism). Each attention head is allocated an equal amount of attention key-value cache, and the value of the attention key-value cache is equal to the key-value cache of this network layer divided by the number of attention heads. This ensures that each attention head has an independent resource quota for subsequent screening of key words from its own perspective, fully respecting the characteristics that different attention heads may focus on different semantic patterns (such as local dependencies, grammatical structures, or specific symbols).

[0066] Specifically, the key-value cache can be obtained by, but not limited to, formula (3):

[0067] Formula (3)

[0068] in, For the The attention key-value cache of each attention head in the layer network, is the number of attention heads.

[0069] By presetting the key cache ratio, the resource allocation for retaining key information and compressing representative information is clearly defined within each network layer, making cache allocation more closely aligned with model processing requirements. By evenly dividing the key-value cache among each attention head to form an attention key-value cache, not only is independent resource quota management of the computing unit achieved, but it also ensures that each submodule in the multi-head attention mechanism can autonomously filter important historical information based on its specific pattern. This fine-grained cache control mechanism lays a quantitative foundation for the subsequent accurate retention of key context and efficient compression of redundant data, further improving the utilization efficiency of video memory resources and the fidelity of semantic information.

[0070] In one implementable method of an embodiment of the present application, when allocating and processing the first total cache, it can also be implemented in but not limited to the following method: obtaining a set cache difference between adjacent network layers in multiple network layers, a first set initial key value cache corresponding to the lowest-level network layer, and a second set initial key value cache corresponding to the highest-level network layer, wherein the set cache difference, the first set initial key value cache, and the second set initial key value cache are set hypothetical values; performing key value cache inference on multiple network layers based on the number of layers, the layers, the preset layer cache ratio, the first total cache, the set cache difference, the first set initial key value cache, and the second set initial key value cache to obtain the initial key value cache corresponding to each of the multiple network layers.

[0071] In the embodiment of the present application, setting the cache difference refers to the difference between the initial key value caches of adjacent layers preset for deriving the calculation method corresponding to the initial key value cache. The first setting of the initial key value cache refers to the initial key value cache of the lowest network layer (layer 1) assumed for deriving the calculation method corresponding to the initial key value cache. The second setting of the initial key value cache refers to the highest network layer (layer 2) assumed for deriving the calculation method corresponding to the initial key value cache. The initial key-value cache for a network layer (e.g., layer 1, layer 2, etc.) is constructed using a mathematical model. Assumptions represent temporary, non-actual parameters used to construct mathematical relationships. Key-value cache inference refers to the process of mathematically deducing the true initial key-value cache based on the assumed parameters and overall constraints. Layers refer to the position of a network layer in the pre-set model (e.g., layer 1, layer 2, etc.), with increasing numbers indicating higher layers.

[0072] Specifically, the key-value cache reasoning of multiple network layers can be achieved through but not limited to the following methods: Set the initial key-value cache size for the lowest level (first level), Set the initial key-value cache size for the top (highest) level, i.e. the second level. The cache difference between two adjacent layers is set as the cache difference, so , and then each layer is calculated in a decreasing manner as ,

[0073] So ;

[0074] because ;

[0075] so ;

[0076] Then you can get the The initial key-value cache calculation method of the network layer is:

[0077] .

[0078] By establishing hypothetical values ​​to construct a mathematical model for cache decrement, key-value cache reasoning is used to convert the preset layer cache ratio and total cache constraints into actual allocation values ​​at each layer. This ensures strict arithmetic decrement of cache allocation and accurately adapts to the preset model's attention mechanism of broad aggregation at the bottom layer and core focus at the top layer. Furthermore, by mathematically eliminating intermediate variables, the final allocation result is completely determined by the configurable preset layer cache ratio and the first total cache, enhancing the transparency and reproducibility of the solution.

[0079] In one implementable method of an embodiment of the present application, when allocating the reserved total cache, it can also be implemented by, but not limited to, the following method: multiplying the preset input quantity by the number of attention heads corresponding to each of the multiple network layers to obtain the first reserved cache corresponding to each of the multiple network layers, wherein the reserved total cache is the product of the first reserved cache corresponding to each of the multiple network layers and the number of levels.

[0080] In the embodiments of the present application, each network layer reserves a preset number of tokens for each of its attention heads, which are used to store the preset number of input tokens at the end of the target token sequence. Therefore, the size of the first reserved buffer for a single layer is determined by the number of attention heads and the preset number of tokens at the end of the target token sequence.

[0081] Through simple product processing, the hardware resource requirements for the end-of-layer token cache are precisely guaranteed. Each attention head independently retains a preset number of input tokens, ensuring that the user's latest requested instructions or task focus information (typically located at the end of the input sequence) are fully preserved across all network layers, preventing them from being mistakenly eliminated due to resource competition. Furthermore, the explicit mathematical relationship between the total cache and the layered cache enhances the predictability and configurability of system resource allocation.

[0082] In one implementable method of an embodiment of the present application, when calculating the target attention values ​​of multiple input words and multiple cache words in a preset model, the following method can also be used but is not limited to: in the target network layer, the attention values ​​of the multiple cache words are calculated for each of the multiple input words based on the target attention head to obtain multiple first attention values ​​corresponding to each of the multiple cache words, wherein the target network layer is any network layer among the multiple network layers, and the target attention head is any attention head among the multiple attention heads included in the target network layer; the multiple first attention values ​​corresponding to each of the multiple cache words are summed up respectively to obtain the second attention value corresponding to each of the multiple cache words calculated based on the target attention head; until the attention value calculation based on all attention heads in the preset model is completed, the target attention value is obtained, wherein the target attention value includes multiple attention values ​​calculated based on all attention heads, and all attention heads correspond to multiple attention values.

[0083] In an embodiment of the present application, the first attention value refers to the attention weight of a single input word and a single cached word on a target attention head. The summation process refers to the operation of adding the first attention values ​​of the same cached word to all input words. The second attention value refers to the sum of the attention weights of a single cached word to all input words on the target attention head, which represents its comprehensive importance. All attention heads refer to the set of all attention calculation units in multiple network layers in a preset model. The target attention value includes the attention values ​​of all cached words on all attention heads, and the target attention value can be in the form of a three-dimensional evaluation matrix consisting of all attention heads, all cached words and their corresponding attention values.

[0084] The target network layer is any network layer currently being processed, for example: At the same time, in order to facilitate the understanding of the implementation process in the embodiment of the present application, the subsequent target network layer is based on the first The network layer is used as an example for explanation.

[0085] For the target attention head in the target network layer (i.e., any attention calculation unit in the target network layer), the following operations can be performed: multiple cached tokens in the current video memory are combined with multiple input tokens of the current input (usually the tokens at the end of the sequence, such as the last token). ) perform attention interaction calculation to obtain the first attention value.

[0086] Specifically, the calculation of the first attention value can be performed using, but not limited to, formula (4):

[0087] Formula (4)

[0088] in, Indicates that any cached word among multiple cached words is a cached word , Any input word among multiple input words is the input word , represents the target attention head, Cache word and input word The first attention value on the target attention head, Customize the activation function selected when performing attention calculation for the preset model. Input tokens generated for the preset model The query vector, Cache tokens generated for the preset model The key vector of A custom scaling factor for attention calculations of the preset model.

[0089] When summing all the first attention values ​​of multiple cached word-units, traverse all input word-units related to the cached word-unit, add the corresponding first attention values, and obtain the second attention value of the cached word-unit on the current target attention head. The calculation of the second attention value can be performed using, but is not limited to, formula (5):

[0090] Formula (5)

[0091] in, Cache word The second attention value on the target attention head, Comprehensively reflects the cache word The overall importance of multiple input tokens.

[0092] Finally, traverse all attention heads (that is, all The above two steps are repeated: each attention head independently outputs the attention value of all cached words under its perspective; the final target attention value can be a three-dimensional data structure:

[0093] The first dimension: all attention heads (total ); the second dimension: all cached words (such as: The third dimension is the attention value corresponding to each attention head-cache word combination. That is, the target attention value can be expressed as a set .

[0094] By performing independent calculations on each attention head, the algorithm fully preserves the differential evaluation of cached tokens by different attention modes (such as grammatical focus and semantic focus). By summing across input tokens, the dependency of the final context on historical information is aggregated into a single importance metric. The resulting target attention value structure fully records the semantic contribution of each cached token under various attention perspectives. This fine-grained evaluation mechanism provides a quantitative basis for subsequent, precise keyword selection, ensuring that the retained historical information is highly relevant to the current task while avoiding the omission of important information due to differences in attention modes.

[0095] In one implementable method of an embodiment of the present application, when performing word element selection processing from multiple cached word elements, it can also be implemented in but not limited to the following methods: according to the attention key value cache, among multiple second attention values, attention values ​​are selected in order from large to small to obtain multiple selected second attention values ​​corresponding to the target attention head; the cached word elements corresponding to each of the multiple selected second attention values ​​are determined as multiple target keyword elements corresponding to the target attention head; until the multiple first keyword elements corresponding to all attention heads are determined, the keyword elements are obtained, wherein the keyword elements include the multiple first keyword elements corresponding to all attention heads.

[0096] In an embodiment of the present application, a keyword element is a set of globally important historical word elements formed by merging the first keyword elements of all attention heads. The first keyword element represents the keyword element corresponding to all attention heads. Therefore, the first keyword element includes the target keyword element.

[0097] First, for the target attention head (the single attention calculation unit currently being processed), the following operations can be performed: Get the attention key value cache corresponding to the target attention head (for example: if the target network layer is layer network layer, the attention key cache corresponding to the target attention head is ), which is the upper limit of the number of keywords allowed to be retained.

[0098] Extract the second attention value calculated by the target attention head for all cached tokens ( ), that is, the importance score of each cached word to the target attention head (based on multiple second attention values ​​calculated by the target attention head), sort the second attention values ​​in descending order, and select the top ranked The extracted second attention value is called the selected second attention value, and its number is strictly equal to the attention key value cache quota.

[0099] Each cached word corresponding to the second attention value is then marked as a target keyword word exclusive to the target attention head, i.e., multiple target keyword words corresponding to the target attention head. The multiple target keyword words corresponding to the target attention head represent the most important historical information for the current task from the perspective of the target attention head.

[0100] Finally, traverse all attention heads and repeat the above two steps: each attention head independently outputs the first set of keyword meta-elements it has filtered out; the results of all attention heads are merged to form the final set of keyword meta-elements.

[0101] By independently screening the attention heads, the differences in different attention patterns (such as local dependency, global correlation, etc.) are fully respected to ensure that all kinds of semantically important information can be captured; by sorting by score and caching constraints, the number of word units retained in each head is precisely controlled to avoid resource waste; the final merged keyword unit set comprehensively covers the core historical context under multi-perspective evaluation, providing high-value information for subsequent generation tasks, while significantly reducing redundant storage.

[0102] In one implementable method of an embodiment of the present application, when selecting multiple selected second attention values ​​corresponding to a target attention head, it can also be implemented by, but not limited to, the following method: sorting the multiple second attention values ​​in descending order to obtain the sorted second attention value corresponding to the target attention head; in the sorted second attention values, the multiple second attention values ​​whose number is the attention key value cache are determined as the multiple selected second attention values ​​corresponding to the target attention head in descending order.

[0103] In an embodiment of the present application, the second attention values ​​calculated by the target attention head for all cached tokens are sorted in descending order. This generates an ordered sequence of second attention values. The first rank corresponds to the cached token with the highest second attention value, and the last rank corresponds to the cached token with the lowest second attention value. This clearly demonstrates the relative importance of each token from the perspective of the target attention head.

[0104] After the sorting is completed, according to the attention key value cache assigned to the target attention head, starting from the starting end of the sorted second attention value (that is, the position with the highest second attention value), the previous attention key value cache values ​​are selected in sequence. The selected values ​​are the selected second attention values ​​corresponding to the target attention head, and their number is strictly equal to the attention key value cache quota, and they are all the values ​​with the highest importance ranking under the evaluation of the target attention head. That is, according to Sort from large to small and select the target attention head cached words.

[0105] By enforcing descending sorting, we ensure that the filtering process always prioritizes the most important terms, avoiding subjective bias. By implementing hard constraints on the cache size, we explicitly control the number of terms retained by each head, ensuring that resource consumption strictly matches the pre-allocated quota. This combination ensures objectivity, predictability, and resource efficiency in keyword filtering, providing high-value historical context for subsequent generation tasks.

[0106] In one implementable method of an embodiment of the present application, when constructing word element vectors corresponding to multiple other word elements, the following method can also be used but is not limited to: determining multiple other attention values ​​corresponding to the target other word elements in the target network layer, wherein the target other word element is any word element among the multiple other word elements, and the multiple other attention values ​​are multiple attention values ​​calculated for the target other word elements based on the multiple attention heads included in the target network layer; performing vector conversion processing on the multiple other attention values ​​to obtain target word element vectors corresponding to the target other word elements, until the first word element vectors corresponding to the multiple other word elements in the multiple network layers are respectively determined to obtain word element vectors, wherein the word element vectors include the first word element vectors corresponding to the multiple other word elements in the multiple network layers.

[0107] In an embodiment of the present application, for a target other word (any one of the remaining cached words in the target network layer) in the target network layer (any network layer currently being processed), multiple other attention values ​​associated with it are extracted. The multiple other attention values ​​are derived from the evaluation results of the target other word on all attention heads in the target network layer, that is, the attention value calculated by each attention head for the target other word, indicating that if the target network layer contains attention heads, then the other target words correspond to Other attention values.

[0108] Perform vector conversion processing on multiple other attention values ​​of other target words, The other attention values ​​of the target word are directly converted into the target word vector of the target word, without changing the data dimension and content. It is only encapsulated in form to provide a unified data structure for subsequent clustering.

[0109] Finally, traverse all remaining cached words (i.e., multiple other words) in the target network layer and repeat the above two steps to generate the first word element vector corresponding to each other word element in the target network layer. Further expand to all network layers and complete the same operation layer by layer to finally form a complete word element vector set: word element vector = {word element vector set of the first network layer, word element vector set of the second network layer, ..., Layer network layer word element vector set}, where the first The word element vector set of the network layer = { The first word element vector corresponding to each of the multiple other words in the layer network layer}.

[0110] By extracting the multi-head attention score of each remaining word to construct a vector, the importance distribution characteristics of the word under different semantic focus modes of the model are fully preserved; through layered independent processing, each network layer is ensured to generate word representation according to its own characteristics; the final word vector set provides structured input for subsequent representative word clustering, so that the compression process can accurately capture the group semantic pattern and significantly improve the efficiency and accuracy of redundant information filtering.

[0111] In one implementable method of an embodiment of the present application, when performing vector conversion processing on multiple other attention values, it can also be implemented by but not limited to the following method: compare the multiple other attention values ​​with the preset conversion threshold respectively to obtain a first comparison result; when determining the multiple other attention values ​​based on the first comparison result, if there are other attention values ​​that are less than the preset conversion threshold, the other attention values ​​that are less than the preset conversion threshold are set to the first value; when determining the multiple other attention values ​​based on the first comparison result, if there are other attention values ​​that are not less than the preset conversion threshold, the other attention values ​​that are not less than the preset conversion threshold are set to the second value; perform vector construction processing based on the first value and the second value to obtain the target word element vector corresponding to the target other word element.

[0112] In an embodiment of the present application, the preset conversion threshold is a custom-set attention value critical point (such as 0.5) for binary decision-making. Comparison processing refers to the operation of comparing the size of each other attention value with the preset conversion threshold. The first comparison result refers to the Boolean judgment set generated after the comparison (for example, each value corresponds to a less than or not less than state). The first numerical value represents a discrete state code in which other attention values ​​are lower than the preset conversion threshold, indicating that it is recommended to discard the target other word element, for example: 0. The second numerical value represents a discrete state code in which other attention values ​​are not lower than the preset conversion threshold, indicating that it is recommended to retain the target other word element. Vector construction processing refers to the process of arranging discrete state codes into binary vectors in order. The first word element vector is a fixed-length binary vector (dimension = 0) composed of the first numerical value and the second numerical value. ), which represents the semantic preservation pattern of other target words on each attention head of the target network layer.

[0113] The multiple other attention values ​​of the target other word (that is, the attention values ​​of the target other word on each attention head in the target network layer) are compared one by one with the preset conversion threshold (such as 0.5) to determine the necessity of semantic retention of the attention value. The first comparison result is divided into two categories: if a certain other attention value is less than the preset conversion threshold, the other attention value is set to the first value (usually 0), indicating that the corresponding attention head believes that the target other word can be discarded; if a certain attention value is not less than (that is, greater than or equal to) the preset conversion threshold, the other attention value is set to the second value (usually 1), indicating that the corresponding attention head recommends retaining the target other word.

[0114] The above steps can be summarized as follows: each target word In the attention head Score on Normalize, where Other target words The attention value on the target attention head is the other attention value, and then a threshold (usually 0.5) is used to determine whether the target other word is retained. The judgment is saved in binary form, that is, it is retained as 1 and discarded as 0. Finally, a bit is formed for each other word. The binary vector of , that is, the first word element vector corresponding to other target words, such as: }.

[0115] The core idea behind binary substitution of tokens is that for any attention head, the binary decision of whether to retain or discard a token inherently reflects the token's semantic importance. Using binary substitution to represent token vectors also reduces computational overhead for subsequent clustering algorithms, enabling efficient computation through masking and bitwise comparisons.

[0116] This operation discretizes each other attention value into a binary state bit (0 or 1). For example, the scores of a target other word on 5 attention heads are [0.3, 0.6, 0.4, 0.8, 0.5], which are converted to a vector [0, 1, 0, 1, 1] after comparison with a threshold of 0.5.

[0117] Finally, vector construction is performed based on all set values: the first and second values ​​generated are arranged in sequence according to the order of attention heads; a fixed length of (number of attention heads), that is, the target word element vector corresponding to the other target words.

[0118] By presetting the conversion threshold, the importance decision is binarized, and the complex attention score is compressed into a compact binary signature. The comparison operation only requires a single numerical comparison, significantly reducing the complexity of subsequent clustering calculations. Each binary bit encodes the independent judgment of the attention head on whether to retain or discard a word, preserving the differences in multi-head evaluations. The word vector is converted from a floating-point array (high video memory usage) to a bit sequence (extremely low video memory usage), significantly reducing the size of the intermediate representation.

[0119] In one implementable method of an embodiment of the present application, in the process of performing word unit selection processing from multiple other word units, it can also be implemented in but not limited to the following methods: in the target network layer, multiple first word unit vectors are matrix constructed to obtain a first vector matrix corresponding to multiple other word units in the target network layer; word unit selection processing is performed from multiple other word units based on the first key-value cache and the first vector matrix to obtain multiple target representative word units corresponding to the target network layer; until the multiple first representative word units corresponding to each of the multiple network layers are determined respectively, the representative word units are obtained, wherein the representative word units include multiple first representative word units corresponding to each of the multiple network layers.

[0120] In the embodiment of the present application, in the target network layer (the network layer currently being processed), the first word element vectors of all other words in the target network layer (each word element corresponds to a length of Binary vectors) are processed by matrix construction:

[0121] The first word element vector is stacked in the order of other words to form a dimension of The first vector matrix where is the number of other words, and the first word vector matrix is ​​as follows:

[0122]

[0123] The rows of the first word-unit vector matrix correspond to multiple other words, and the columns correspond to multiple attention heads. The element values ​​are the first value (0) or the second value (1), which fully encodes the attention retention mode of all words. Based on the first key-value cache and the first vector matrix allocated by the target network layer, word-unit selection processing is performed. The word-unit selection processing at least includes: calculating anchor vectors, Hamming distance sorting, uniform bucketing and representative selection, and obtaining multiple target representative words corresponding to the target network layer. Finally, all network layers are traversed, and the above operations are repeated layer by layer. The multiple first representative words output by each layer are merged to form the final set of representative words. The first representative word represents the representative word corresponding to each of the multiple network layers. Therefore, the first representative word includes the target representative word.

[0124] Matrix construction structures word-meta semantic patterns, providing efficient input for clustering. Hamming distance sorting and a uniform bucketing strategy ensure that representative words cover diverse attention patterns. A random selection mechanism avoids clustering bias and enhances robustness. Finally, layered independent compression fully adapts to the information processing characteristics of different network layers. Overall, this significantly reduces redundant storage while maintaining the collective semantic representation of eliminated words.

[0125] In one implementation of the embodiment of the present application, when performing word-unit selection processing from multiple other word-units, the following method may also be used, but is not limited to it: performing column-wise mean calculation processing on the first vector matrix to obtain a first anchor vector corresponding to the target network layer, wherein the number of columns of the first vector matrix is ​​the number of multiple other word-units, and the number of rows of the first vector matrix is ​​the number of attention heads corresponding to the target network layer; respectively calculating the Hamming distance between the first anchor vector and multiple first row vectors in the first vector matrix, wherein the multiple rows of the first vector matrix are multiple first row vectors, and the multiple first row vectors are the first word-unit vectors corresponding to multiple other word-units in the target network layer; sorting the other word-units corresponding to each of the multiple first row vectors according to the Hamming distance in a preset sorting method to obtain sorted other word-units corresponding to the target network layer, wherein the preset sorting method at least includes sorting from largest to smallest or sorting from smallest to largest; allocating the sorted other word-units based on the representative key-value cache to obtain multiple word-unit categories, and performing word-unit extraction processing on each of the multiple word-unit categories to obtain multiple target representative word-units corresponding to the target network layer.

[0126] In the embodiments of the present application, column-wise mean calculation refers to averaging the elements in each column of the first vector matrix. The first anchor vector is a vector composed of the column means of the first vector matrix, representing the typical binary pattern of the other word group. The Hamming distance, which measures the difference in the number of bits between two binary vectors at the same position, is used to measure the degree of difference. The first row vector is the vector corresponding to a single row in the first vector matrix (i.e., the binary representation of a single other word).

[0127] Convert the first vector into a matrix (dimensions: , is the number of other words, is the number of attention heads) and the mean is calculated by column: the arithmetic mean is calculated for each column of the first vector matrix (that is, the binary results of all other words on the same attention head); Dimensional continuous vector, get the first anchor vector , each element of which is 0 or 1, indicating the average tendency of the attention head to retain suggestions for multiple other word groups.

[0128] The Hamming distance between the first anchor vector and each row of the first vector matrix (i.e., the first row vector, corresponding to the binary vector of a single other word) is calculated separately. The Hamming distance is defined as the number of bits that differ between two binary strings. The first anchor vector is also a binary vector (e.g., 1 if it is greater than or equal to 0.5, and 0 otherwise). It is directly compared bit by bit with each first row vector. The distance value between each first row vector and the first anchor vector is finally obtained, which quantifies the degree of deviation of multiple other words from the group average pattern.

[0129] Sort the other words in a preset order (such as from large to small or from small to large) according to the Hamming distance to generate a sequence of other words; divide the ordered sequence into word categories representing the key-value cache, each category contains other lemmas (the remainder is randomly assigned); randomly select a lemma from each lemma category as a representative, and obtain The target represents a word.

[0130] The typical attention patterns of word groups are automatically captured through the first anchor vector, providing an objective benchmark for similarity calculation; efficient difference measurement of binary vectors is achieved through Hamming distance, significantly reducing computational overhead; a preset sorting method ensures the orderliness of grouping; uniform bucketing ensures that all types of patterns are covered; and finally, random representatives are extracted to avoid clustering bias, maintaining the semantic diversity of the compressed words to the greatest extent possible while strictly adhering to cache constraints.

[0131] In one implementable method of an embodiment of the present application, when determining the first anchor vector corresponding to the target network layer, it can also be implemented by but not limited to the following method: performing mean calculation processing on the first vector matrix by column to obtain an initial anchor vector, and comparing the multiple anchor point values ​​in the initial anchor point vector with the preset conversion threshold to obtain a second comparison result; in the case where there is an anchor point value less than the preset conversion threshold among the multiple anchor point values ​​determined according to the second comparison result, the anchor point value less than the preset conversion threshold is set as the first value; in the case where there is an anchor point value not less than the preset conversion threshold among the multiple anchor point values ​​determined according to the second comparison result, the anchor point value not less than the preset conversion threshold is set as the second value; and vector construction processing is performed based on the first value and the second value to obtain the first anchor vector.

[0132] In the embodiment of the present application, the first vector matrix is ​​processed by column mean calculation: the arithmetic mean of all first word element vectors on the attention head dimension is calculated column by column; The initial anchor vector of dimension φ represents the average tendency strength of each attention head to retain suggestions for the word group.

[0133] Each element of the initial anchor vector is compared with a preset conversion threshold (e.g., 0.5). The size relationship is determined element by element, and a Boolean result set (the second comparison result) is generated. If an element is smaller than the preset conversion threshold, the element is set to the first value (e.g., 0). If an element is smaller than the preset conversion threshold, the element is set to the second value (e.g., 1).

[0134] Perform vector construction processing based on the discretization result: arrange the first value and the second value in order according to the order of attention heads; generate a fixed-length dimensional binary first anchor vector (e.g., [0, 1, 0, 1]), where each bit encodes the group average decision tendency of the corresponding attention head.

[0135] The way to calculate the average value of the first vector matrix by column is to first calculate the average value of multiple other word units on a certain attention head. Specifically, it can be implemented by but not limited to formula (6):

[0136] Formula (6)

[0137] in, is the average of multiple other words on the target attention head, Represents the second attention value of multiple other words on the target attention head. Then calculate the mean of all attention heads to form a mean vector: , select the mean as a first anchor vector.

[0138] The significance of the mean anchor is that it represents the most common retention pattern of a group of words in each attention head. Therefore, when using the first anchor vector for Hamming distance clustering, it can effectively divide the words into the representative group closest to their attention pattern.

[0139] The attention pattern distribution of word groups is objectively extracted through mean calculation. Continuous statistics are converted into binary representations isomorphic to word vectors through threshold binarization to ensure the effectiveness of subsequent Hamming distance calculations. The first anchor vector finally generated serves as the clustering center, providing a unified and interpretable similarity benchmark for efficient compression, significantly improving the semantic representativeness of representative word selection.

[0140] In one implementable method of an embodiment of the present application, when determining multiple first representative word units corresponding to the target network layer, the following methods can also be used but are not limited to: the sorted other word units are evenly distributed according to the representative key-value cache to obtain multiple word unit classifications, wherein the number of classifications of the multiple word unit classifications is equal to the representative key-value cache corresponding to the target network layer; random word unit extraction processing is performed on the multiple word unit classifications respectively to obtain multiple target representative word units corresponding to the target network layer, wherein the number of word units of the multiple target representative word units is equal to the representative key-value cache corresponding to the target network layer.

[0141] In the embodiments of the present application, the calculation The Hamming distance between the first row vector and the first anchor vector. Since the word unit representation is binary, the Hamming distance can be calculated very efficiently by bitwise XOR. Sort the other words (from large to small or from small to large), and then divide the sorted words into Each portion can be called a bucket. From each bucket, another word unit is randomly selected as the representative word unit of the bucket, and the rest of the words in the bucket are discarded.

[0142] Perform an average distribution process on the sorted other tokens (a sequence of tokens ordered by Hamming distance) and cache them according to the representative key value of the target network layer. , divide the ordered sequence into word-unit categories (i.e., cluster buckets); the number of categories is strictly equal to , ensure that the final number of representative words matches the cache constraint; if the total number of other words Cannot be If it is divisible by , the remaining words form a class.

[0143] Perform random word extraction in each word category, randomly select one word from all words in the category as the representative of the category; traverse all categories, a total of word units to form a word unit set, that is, multiple target representative word units.

[0144] The average distribution process ensures that ordered tokens are evenly divided into different categories, ensuring that each bucket contains tokens with similar Hamming distances, maintaining a balanced cluster structure. Random token extraction gives each token equal representation, avoiding systematic bias introduced by fixed rules (such as center selection) and enhancing the robustness and semantic diversity of the compression results. The combination of these two processes maximizes the overall semantic representation capability of the eliminated token population while strictly adhering to cache constraints.

[0145] In one implementable method of an embodiment of the present application, when storing and processing the key key value corresponding to the keyword word, the input key value corresponding to each of the multiple input words, and the representative key value corresponding to the representative word, it can also be implemented in but not limited to the following method: respectively calculating the input key value corresponding to each of the multiple input words according to all attention heads, and storing the multiple input key values ​​calculated for each of the attention heads; respectively calculating the first key value corresponding to each of the multiple first keyword words based on all attention heads, and storing the multiple first key values ​​calculated for each of the attention heads, wherein the key key value includes multiple first key key values; respectively calculating the first representative key value corresponding to each of the multiple first representative words based on all attention heads, and storing the multiple first representative key values ​​of the multiple first representative words corresponding to each of the multiple network layers, wherein the representative key value includes multiple first representative key values; deleting the key value to be discarded from the multiple cached key values, wherein the key value to be discarded is the key value corresponding to the word in the multiple cached words except the keyword word and the representative word.

[0146] In an embodiment of the present application, at least the following steps are included: input word storage, that is, storing the input key values ​​corresponding to the current multiple input words for all attention heads; each attention head independently caches the key (Key) and value (Value) vectors of these words to ensure that the latest task focus information is globally retained, including: all attention heads separately calculate the input key values ​​of the current multiple input words (that is, the key vector and value vector corresponding to each word); each attention head independently generates its own unique key-value pair to ensure the integrity of the multi-head semantic perspective. Each attention head independently stores its own calculated input key value to ensure that the latest task focus information is fully retained in each head.

[0147] Keyword metadata storage: All attention heads store the first key values ​​corresponding to the multiple first keyword metadata they have selected (i.e., key history metadata specific to each head). Each head only caches keyword metadata within its own quota (attention key value cache), respecting the semantic independence of multiple heads. This includes: all attention heads calculate the first key values ​​of the multiple first keyword metadata they have selected; the key value is composed of the set of the first key values ​​of all heads. Each attention head independently stores its own first key value to maintain multi-head independence.

[0148] Representative word storage involves storing the first representative key values ​​corresponding to the multiple first representative words generated by each layer's compression (i.e., the representative set selected by clustering at each layer). Each layer's cache size strictly matches its representative key value cache, enabling differentiated storage across layers. This involves all attention heads separately calculating the first representative key values ​​for the multiple first representative words generated by clustering at the current layer. The representative key value is composed of the set of first representative key values ​​across all layers. Each layer independently stores its own first representative key value, adapting to the information compression requirements of different layers.

[0149] Video memory resource recycling, that is, deleting the key value to be discarded, identifying the remaining words in multiple cached words that have not been selected as keyword words or representative words as the words to be discarded; immediately releasing the key value cache space occupied by the key value to be discarded corresponding to the word to be discarded, and recycling video memory resources.

[0150] By independently caching key words, the core historical context from multiple perspectives is preserved; by independently caching representative words in layers, the compressed semantic representation of redundant information is maintained; by forcibly storing the last input word, the task focus is not lost; and finally, by completely deleting the words to be discarded, the memory usage is substantially reduced. These three factors work together to ensure the quality of model generation while significantly improving system throughput and long-context processing capabilities.

[0151] By separately calculating and storing input keys and key keys, we ensure that task focus and core historical information are fully preserved across multiple attention perspectives. By storing representative keys in layers, we achieve efficient compression of redundant information. Finally, by precisely deleting keys to be discarded, we substantially reduce video memory usage. These three factors work together to significantly improve system throughput and long-context processing capabilities while ensuring generation quality.

[0152] In one possible implementation of an embodiment of the present application, before calculating the target attention value, the following methods may also be used but are not limited to: inputting the request information into a preset model for information processing to obtain a target word sequence corresponding to the request information; determining the input order of the target word sequence, wherein the input order is the order in which the target word sequence is input into the preset model; in the target word sequence, selecting a preset number of input words as multiple input words in the input order from back to front.

[0153] In this embodiment, user-submitted request information (e.g., text queries, instructions, etc.) is input into a pre-defined model (the deployed LLM). The model's tokenizer tokenizes the original text and generates a corresponding target token sequence. The target token sequence consists of multiple tokens arranged in a semantic order. For example, a request for an explanation of quantum mechanics might be tokenized into the sequence ["explanation", "quantum", "mechanics"].

[0154] Identify the input order of the target token sequence into the model, that is, the temporal order in which tokens are fed into the model's inference engine. The input order strictly follows the natural flow of the text: the first token in the sequence is input first, and the last token is input last. For example, in the sequence [T1, T2, T3], T1 is the first token input, and T3 is the last token input.

[0155] In the target word sequence, words are selected from the last word in the input order (i.e., reverse chronological order): Locate the end of the sequence: select the last word input (i.e., the end of the sequence); Backtrack: select the previous word in sequence until the preset number of inputs is reached ( The final selected Tokens are multiple input tokens. For example, in the sequence [T1, T2, T3, T4], if =2, then T4 (the latest input) and T3 (the next latest input) are selected as input tokens.

[0156] A reverse-chronological selection strategy prioritizes capturing the latest key information in user requests (e.g., core questions, latest instructions), ensuring the model's focus on the current task. Combined with a hard constraint on the number of pre-set inputs, redundant selection is avoided, resulting in more efficient allocation of graphics memory resources. This step provides high-value input tokens for subsequent attention calculations, significantly improving the accuracy of keyword token screening.

[0157] Furthermore, in order to facilitate understanding of the implementation process of the embodiment of the present application, the embodiment of the present application also provides an embodiment for illustration, such as Figure 3 As shown, Figure 3 A schematic diagram of the architecture of a storage optimization method provided in an embodiment of the present application, including:

[0158] Step 1: Create an LLM inference service. After the service is started, it starts waiting for user requests to enter the service.

[0159] Step 2: The user starts using the LLM reasoning service and sends a request to the LLM reasoning service. The request enters the LLM reasoning service and the service starts processing the request.

[0160] Step 3: The KV cache management module allocates cache for the request. First, it reserves the last Token cache; then allocate cache for each layer; and divide the cache of each layer into key cache and representative cache; finally allocate key cache for each attention head of each layer;

[0161] Step 4: The keyword metadata module is retained to calculate the keyword metadata from 1 to layer, for each attention head in each layer , we calculate the final input sequence Historical lexicons In each attention head Then sort the key values ​​of all historical word units from large to small, and select the keyword unit from large to small according to the key cache allocated to this attention head;

[0162] Step 5: Select representative word module to first select each remaining historical word In the attention head Normalize the key values ​​on the word, use the threshold method to binarize, and repeat this operation on each attention head. On all attention heads, a binary vector with the total number of attention heads as the dimension is formed; all historical words are normalized and binarized on all attention heads to form a Matrix of

[0163] Step 6: Select the representative word module and calculate The mean of the vectors is selected as an anchor point;

[0164] Step 7: Calculation The Hamming distance between the vector and the anchor point is used to Sort the words (from large to small or from small to large), and then divide the sorted words into representative caches share;

[0165] Step 7: Randomly select a word from each bucket as the representative word of that bucket, and discard the remaining words in the bucket;

[0166] Step 8: Repeat steps 5 to 7, and we will calculate the 1st to 7th values ​​of LLM. All representative tokens of the layer cache.

[0167] In summary, the embodiments of the present application can achieve the following technical effects:

[0168] 1. By adopting a retention and selection strategy, the cache space is compressed. When allocating cache to each network layer, the key-value cache is allocated in a gradually decreasing manner from low to high layers, so as to avoid missing key words at low layers and saving unimportant words at high layers. Then, the attention focus level between the input word and the cached word, i.e., the target attention value, is used to select the most influential key words from the cached words for caching. Then, representative words are selected from the remaining words for storage. Only words that have an impact on the model processing data can be stored, without having to store all cached words. Therefore, the technical problem of how to effectively reduce the storage space occupied by the KV cache without affecting the model accuracy and inference quality can be solved, achieving the technical effect of compressing the space occupied by the KV cache and providing the ability to process more requests.

[0169] 2. The storage strategy of the KV cache has been optimized to reduce the space occupied by the KV cache, allowing the LLM inference service to handle larger batch sizes and long-context tasks, thereby supporting greater model throughput and more model applications.

[0170] 3. When allocating cache to each layer, use a pattern of gradually focusing attention from low to high layers, and allocate it in a gradually decreasing manner to avoid missing key words in the low layers and avoid saving unimportant words in the high layers.

[0171] 4. Using key values ​​and the degree of attention focused on the last token of the input sequence (input word) and previous tokens (cached word), the most influential historical tokens for the current task are precisely selected and cached. This ensures that key information is retained during inference, improving the accuracy and contextual coherence of the model response.

[0172] 5. When selecting representative tokens, based on the different modes of each attention head, historical tokens are represented by binary vectors on the attention heads. This effectively represents the semantic importance of the tokens for each attention head. Furthermore, binary substitution saves computational overhead for the subsequent clustering algorithm, enabling efficient and rapid calculations through masking and bitwise comparisons during clustering.

[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0174] The embodiment of the present application also provides a storage optimization device, Figure 4 A schematic diagram of the structure of a storage optimization device provided in this application is shown as follows: Figure 4Shown, including:

[0175] An allocating unit 41 is configured to perform key-value cache allocation processing in a preset model to obtain first key-value caches corresponding to respective network layers in the preset model, wherein a higher level of the network layers has a smaller corresponding first key-value cache;

[0176] A calculation unit 42 is configured to calculate target attention values ​​of a plurality of cached word-grams corresponding to a plurality of input word-grams and a plurality of cached key values ​​in a preset model;

[0177] A first selection unit 43 is configured to select a word from the plurality of cached word-grams according to the first key-value cache and the target attention value to obtain a keyword word;

[0178] A construction unit 44 is configured to construct word element vectors corresponding to multiple other word elements according to the target attention value;

[0179] a second selection unit 45 for selecting a word from a plurality of other word-grams according to the first key-value cache and the word-gram vector to obtain a representative word-gram, wherein the plurality of other word-grams are cached word-grams other than the keyword word-gram in the plurality of cached word-grams;

[0180] The storage unit 46 is used to store the key value corresponding to the key word, the input key values ​​corresponding to the plurality of input word units, and the representative key value corresponding to the representative word unit.

[0181] In one embodiment of the present application, the allocation unit 41 is further configured to:

[0182] Determine a preset total key-value cache, wherein the preset total key-value cache includes a first total cache of multiple network layers and a reserved total cache of multiple network layers;

[0183] Allocating the reserved total buffer according to the number of layers of the multiple network layers, the number of attention heads corresponding to each of the multiple network layers, and a preset number of inputs to obtain a first reserved buffer corresponding to each of the multiple network layers, wherein the preset number of inputs is the number of pre-configured input tokens, and each network layer includes a plurality of attention heads;

[0184] Allocating the first total cache to obtain initial key-value caches corresponding to the respective multiple network layers according to the number of layers, the layers corresponding to the respective multiple network layers, and a preset layer cache ratio, wherein the preset layer cache ratio is a ratio between the initial key-value cache corresponding to the lowest layer of the pre-configured multiple network layers and the initial key-value cache corresponding to the highest layer of the pre-configured multiple network layers;

[0185] The first reserved cache and the initial key-value cache corresponding to each of the multiple network layers are added together to obtain the first key-value cache corresponding to each of the multiple network layers.

[0186] In one embodiment of the present application, the allocation unit 41 is further configured to:

[0187] The initial key-value caches corresponding to the multiple network layers are divided according to a preset key cache ratio to obtain key key-value caches and representative key-value caches corresponding to the multiple network layers;

[0188] In multiple network layers, the key-value cache is evenly distributed according to the number of attention heads to obtain the attention key-value cache corresponding to multiple attention heads.

[0189] In one embodiment of the present application, the allocation unit 41 is further configured to:

[0190] Obtaining a set cache difference between adjacent network layers in a plurality of network layers, a first set initial key value cache corresponding to a lowest-level network layer, and a second set initial key value cache corresponding to a highest-level network layer, wherein the set cache difference, the first set initial key value cache, and the second set initial key value cache are set assumed values;

[0191] According to the number of layers, layers, preset layer cache ratio, first total cache, set cache difference, first set initial key value cache and second set initial key value cache, key value cache inference of multiple network layers is performed to obtain the initial key value cache corresponding to each of the multiple network layers.

[0192] In one embodiment of the present application, the allocation unit 41 is also used to multiply the preset input quantity by the number of attention heads corresponding to each of the multiple network layers to obtain the first reserved cache corresponding to each of the multiple network layers, wherein the total reserved cache is the product of the first reserved cache corresponding to each of the multiple network layers and the number of levels.

[0193] In one embodiment of the present application, the calculation unit 42 is further configured to:

[0194] In a target network layer, performing attention value calculations on each of the plurality of cached word-units and the plurality of input word-units based on a target attention head to obtain a plurality of first attention values ​​corresponding to each of the plurality of cached word-units, wherein the target network layer is any network layer among the plurality of network layers, and the target attention head is any attention head among the plurality of attention heads included in the target network layer;

[0195] Summing the first attention values ​​corresponding to the plurality of cached word-units respectively to obtain the second attention values ​​corresponding to the plurality of cached word-units calculated based on the target attention head;

[0196] Until the attention value calculation based on all attention heads in the preset model is completed, the target attention value is obtained, wherein the target attention value includes multiple attention values ​​calculated based on all attention heads, and all attention heads correspond to multiple attention values.

[0197] In one embodiment of the present application, the first selection unit 43 is further configured to:

[0198] According to the attention key value cache, among the multiple second attention values ​​calculated based on the target attention head, the attention values ​​are selected in descending order to obtain multiple selected second attention values ​​corresponding to the target attention head;

[0199] Determine the cached word-units corresponding to the plurality of selected second attention values ​​as the plurality of target keyword-units corresponding to the target attention head;

[0200] Until the multiple first keyword elements corresponding to all the attention heads are determined, the keyword elements are obtained, wherein the keyword elements include the multiple first keyword elements corresponding to all the attention heads.

[0201] In one embodiment of the present application, the first selection unit 43 is further configured to:

[0202] Sort the multiple second attention values ​​calculated based on the target attention head in descending order to obtain the sorted second attention value corresponding to the target attention head;

[0203] In the sorting of the second attention values, a plurality of second attention values ​​whose number is the attention key value cache is determined as a plurality of selected second attention values ​​corresponding to the target attention head in a descending order.

[0204] In one embodiment of the present application, the construction unit 44 is further configured to:

[0205] Determining, in the target network layer, a plurality of other attention values ​​corresponding to the target other word-gram, wherein the target other word-gram is any word-gram among the plurality of other word-grams, and the plurality of other attention values ​​are a plurality of attention values ​​calculated for the target other word-gram based on the plurality of attention heads included in the target network layer;

[0206] Multiple other attention values ​​are vector-converted to obtain target word-element vectors corresponding to the target other word-elements, until the first word-element vectors corresponding to multiple other word-elements in multiple network layers are determined respectively to obtain word-element vectors, wherein the word-element vectors include the first word-element vectors corresponding to multiple other word-elements in multiple network layers.

[0207] In one embodiment of the present application, the construction unit 44 is further configured to:

[0208] Comparing the other attention values ​​with the preset conversion threshold respectively to obtain a first comparison result;

[0209] If there are other attention values ​​less than the preset conversion threshold value among the multiple other attention values ​​determined according to the first comparison result, set the other attention values ​​less than the preset conversion threshold value as the first value;

[0210] If there is another attention value not less than the preset conversion threshold among the multiple other attention values ​​determined according to the first comparison result, set the other attention value not less than the preset conversion threshold as the second value;

[0211] A vector construction process is performed according to the first value and the second value to obtain a target word element vector corresponding to the other target word element.

[0212] In one embodiment of the present application, the second selection unit 45 is further configured to:

[0213] In the target network layer, the first word unit vectors corresponding to the multiple other word units are subjected to matrix construction processing to obtain the first vector matrices corresponding to the multiple other word units in the target network layer;

[0214] Perform word unit selection processing from multiple other word units according to the first key value cache and the first vector matrix to obtain multiple target representative word units corresponding to the target network layer;

[0215] Until the multiple first representative word-grams corresponding to the multiple network layers are determined respectively, a representative word-gram is obtained, wherein the representative word-gram includes the multiple first representative word-grams corresponding to the multiple network layers.

[0216] In one embodiment of the present application, the second selection unit 45 is further configured to:

[0217] Perform column-wise mean calculation on the first vector matrix to obtain the first anchor vector corresponding to the target network layer, where the number of columns in the first vector matrix is ​​the number of other word units, and the number of rows in the first vector matrix is ​​the number of attention heads corresponding to the target network layer;

[0218] Calculate the Hamming distance between the first anchor vector and a plurality of first row vectors in the first vector matrix respectively, where the plurality of first row vectors refer to a plurality of rows of the first vector matrix, and the plurality of first row vectors are first word element vectors corresponding to a plurality of other word elements in the target network layer;

[0219] Sorting the other words corresponding to each of the plurality of first row vectors in a preset sorting manner according to the Hamming distance to obtain sorted other words corresponding to the target network layer, wherein the preset sorting manner includes at least sorting from largest to smallest or sorting from smallest to largest;

[0220] The other ranked words are distributed and processed based on the representative key value cache to obtain multiple word categories, and word extraction processes are performed from the multiple word categories respectively to obtain multiple target representative words corresponding to the target network layer.

[0221] In one embodiment of the present application, the second selection unit 45 is further configured to:

[0222] Performing column-wise mean calculation on the first vector matrix to obtain an initial anchor point vector, and comparing multiple anchor point values ​​in the initial anchor point vector with a preset conversion threshold to obtain a second comparison result;

[0223] When there is an anchor point value smaller than a preset conversion threshold value among the multiple anchor point values ​​determined according to the second comparison result, setting the anchor point value smaller than the preset conversion threshold value as the first value;

[0224] If there is an anchor point value not less than a preset conversion threshold value among the multiple anchor point values ​​determined according to the second comparison result, setting the anchor point value not less than the preset conversion threshold value as the second value;

[0225] A vector construction process is performed according to the first value and the second value to obtain a first anchor point vector.

[0226] In one embodiment of the present application, the second selection unit 45 is further configured to:

[0227] The other ranked words are evenly distributed according to the representative key value cache to obtain multiple word classifications, where the number of classifications of the multiple word classifications is equal to the representative key value cache corresponding to the target network layer;

[0228] Random word extraction processing is performed on multiple word classifications to obtain multiple target representative word elements corresponding to the target network layer, wherein the number of word elements in the multiple target representative word elements is equal to the representative key-value cache corresponding to the target network layer.

[0229] In one embodiment of the present application, the storage unit 46 is further configured to:

[0230] Calculate the input key values ​​corresponding to the multiple input words according to all attention heads, and store the multiple input key values ​​calculated by all attention heads respectively;

[0231] Calculating first key values ​​corresponding to each of the plurality of first keyword elements based on all the attention heads, and storing the plurality of first key values ​​calculated for each of the attention heads, wherein the key value includes the plurality of first key values;

[0232] Calculating first representative key values ​​corresponding to each of the plurality of first representative word-grams based on all attention heads, and storing first representative key values ​​corresponding to the plurality of first representative word-grams for each of the plurality of network layers, wherein the representative key value includes the plurality of first representative key values;

[0233] Delete the key values ​​to be discarded from the multiple cache key values, wherein the key values ​​to be discarded are the key values ​​corresponding to the word elements in the multiple cache word elements except the keyword word element and the representative word element.

[0234] In one embodiment of the present application, Figure 5 As shown, the storage optimization device also includes:

[0235] The processing unit 47 is used to input the request information into the preset model for information processing to obtain a target word sequence corresponding to the request information;

[0236] The processing unit 47 is further configured to determine an input order of the target word sequence, wherein the input order is an order in which the target word sequence is input into the preset model;

[0237] The processing unit 47 is further configured to select a preset number of input words from the target word sequence in an input order from back to front as the plurality of input words.

[0238] For the description of the features in the embodiment corresponding to the storage optimization device, please refer to the relevant description of the embodiment corresponding to the storage optimization method, and no further details will be given here.

[0239] An embodiment of the present application further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any of the above-mentioned storage optimization method embodiments.

[0240] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above-mentioned storage optimization method embodiments when running.

[0241] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0242] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of any of the above-mentioned storage optimization method embodiments are implemented.

[0243] An embodiment of the present application also provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned storage optimization method embodiments are implemented.

[0244] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0245] The above is a detailed introduction to a storage optimization method and device, electronic device and storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A storage optimization method, characterized in that: include: Performing key-value cache allocation processing in a preset model to obtain first key-value caches corresponding to respective multiple network layers in the preset model, wherein a higher level of the multiple network layers corresponds to a smaller first key-value cache; Calculating target attention values ​​for a plurality of cached word-grams corresponding to a plurality of input word-grams and a plurality of cached key values ​​in the preset model, and performing word-gram selection processing from the plurality of cached word-grams based on the first key value cache and the target attention values ​​to obtain a keyword word-gram; Constructing word-meta vectors corresponding to each of a plurality of other word-meta based on the target attention value, and performing word-meta selection processing from the plurality of other word-meta based on the first key-value cache and the word-meta vector to obtain a representative word-meta, wherein the plurality of other word-meta are cached word-meta in the plurality of cached word-meta except the keyword word-meta; storing the key value corresponding to the key word, the input key values ​​corresponding to each of the plurality of input word units, and the representative key value corresponding to the representative word unit; The key-value cache allocation process is performed in the preset model to obtain the first key-value cache corresponding to each of the plurality of network layers in the preset model, including: Determine a preset total key-value cache, wherein the preset total key-value cache includes a first total cache of the multiple network layers and a reserved total cache of the multiple network layers; Allocating the reserved total buffer according to the number of levels of the multiple network layers, the number of attention heads corresponding to each of the multiple network layers, and a preset number of inputs to obtain a first reserved buffer corresponding to each of the multiple network layers, wherein the preset number of inputs is a pre-configured number of input tokens, and each network layer includes a plurality of attention heads; Allocating the first total cache according to the number of layers, the layers corresponding to the multiple network layers, and a preset layer cache ratio to obtain an initial key-value cache corresponding to each of the multiple network layers, wherein the preset layer cache ratio is a pre-configured ratio between an initial key-value cache corresponding to a lowest layer of the multiple network layers and an initial key-value cache corresponding to a highest layer of the multiple network layers; The first reserved cache and the initial key-value cache corresponding to each of the multiple network layers are added together to obtain the first key-value cache corresponding to each of the multiple network layers.

2. The storage optimization method according to claim 1, characterized in that: After respectively adding the first reserved cache and the initial key-value cache corresponding to each of the multiple network layers to obtain the first key-value cache, the method further includes: Dividing the initial key-value caches corresponding to the multiple network layers according to a preset key cache ratio to obtain key key-value caches and representative key-value caches corresponding to the multiple network layers; In the multiple network layers, the key-value cache is evenly distributed according to the number of attention heads to obtain the attention key-value caches corresponding to the multiple attention heads.

3. The storage optimization method according to claim 1, characterized in that: The allocating the first total cache according to the number of layers, the layers corresponding to the multiple network layers, and the preset layer cache ratio to obtain the initial key-value cache corresponding to the multiple network layers includes: Obtaining a set cache difference between adjacent network layers in the multiple network layers, a first set initial key value cache corresponding to the lowest-level network layer, and a second set initial key value cache corresponding to the highest-level network layer, wherein the set cache difference, the first set initial key value cache, and the second set initial key value cache are set assumed values; Based on the number of levels, the levels, the preset level cache ratio, the first total cache, the set cache difference, the first set initial key-value cache and the second set initial key-value cache, key-value cache inference is performed on the multiple network layers to obtain the initial key-value cache corresponding to each of the multiple network layers.

4. The storage optimization method according to claim 1, characterized in that: The allocating the reserved total buffer according to the number of levels of the multiple network layers, the number of attention heads corresponding to each of the multiple network layers, and the preset number of inputs to obtain the first reserved buffer corresponding to each of the multiple network layers includes: The preset input quantity is multiplied by the number of attention heads corresponding to each of the multiple network layers to obtain the first reserved cache corresponding to each of the multiple network layers, wherein the reserved total cache is the product of the first reserved cache corresponding to each of the multiple network layers and the number of levels.

5. The storage optimization method according to claim 2, characterized in that: Calculating target attention values ​​of the plurality of input words and the plurality of cached words in the preset model includes: In a target network layer, performing attention value calculations on each of the plurality of cached word-grams and the plurality of input word-grams based on a target attention head to obtain a plurality of first attention values ​​corresponding to each of the plurality of cached word-grams, wherein the target network layer is any one of the plurality of network layers, and the target attention head is any one of the plurality of attention heads included in the target network layer; respectively summing up the plurality of first attention values ​​corresponding to the plurality of cached word-units to obtain a second attention value corresponding to the plurality of cached word-units calculated based on the target attention head; Until the attention value calculation based on all attention heads in the preset model is completed, the target attention value is obtained, wherein the target attention value includes multiple attention values ​​calculated based on each of the all attention heads, and each of the all attention heads corresponds to the multiple attention values.

6. The storage optimization method according to claim 5, characterized in that: The step of performing word-unit selection processing from the plurality of cached word-units according to the first key-value cache and the target attention value to obtain a keyword-unit comprises: Selecting, from the plurality of second attention values ​​according to the attention key value cache, in descending order, a plurality of selected second attention values ​​corresponding to the target attention head; Determine the cached word-grams corresponding to the plurality of selected second attention values ​​as the plurality of target keyword-grams corresponding to the target attention head; Until the multiple first keyword elements corresponding to all the attention heads are determined, the keyword elements are obtained, wherein the keyword elements include the multiple first keyword elements corresponding to all the attention heads.

7. The storage optimization method according to claim 6, characterized in that: The step of selecting, according to the attention key value cache, the plurality of second attention values ​​in descending order to obtain the plurality of selected second attention values ​​corresponding to the target attention head comprises: Sorting the plurality of second attention values ​​in descending order to obtain a sorted second attention value corresponding to the target attention head; In the sorting of the second attention values, a plurality of second attention values ​​whose number is the attention key value cache is determined as the plurality of selected second attention values ​​corresponding to the target attention head in a descending manner.

8. The storage optimization method according to claim 6, characterized in that: The constructing of word element vectors corresponding to the plurality of other word elements according to the target attention value includes: Determining, in the target network layer, a plurality of other attention values ​​corresponding to target other word-grams, wherein the target other word-gram is any word-gram among the plurality of other word-grams, and the plurality of other attention values ​​are the plurality of attention values ​​respectively calculated for the target other word-gram based on the plurality of attention heads included in the target network layer; The multiple other attention values ​​are vector-converted to obtain target word-element vectors corresponding to the target other word-elements, until the first word-element vectors corresponding to the multiple other word-elements in the multiple network layers are respectively determined to obtain the word-element vectors, wherein the word-element vectors include the first word-element vectors corresponding to the multiple other word-elements in the multiple network layers.

9. The storage optimization method according to claim 8, characterized in that: The performing vector conversion processing on the multiple other attention values ​​to obtain the target word element vector corresponding to the target other word element includes: Comparing the plurality of other attention values ​​with the preset conversion threshold value respectively to obtain a first comparison result; If there are other attention values ​​smaller than the preset conversion threshold among the multiple other attention values ​​determined according to the first comparison result, setting the other attention values ​​smaller than the preset conversion threshold as the first value; If there is another attention value not less than the preset conversion threshold among the multiple other attention values ​​determined according to the first comparison result, setting the other attention value not less than the preset conversion threshold as a second value; A vector construction process is performed based on the first value and the second value to obtain the target word element vector corresponding to the other target word element.

10. The storage optimization method according to claim 9, characterized in that: The performing word unit selection processing from the plurality of other word units according to the first key value cache and the word unit vector to obtain a representative word unit includes: In the target network layer, performing matrix construction processing on the plurality of first word element vectors to obtain a first vector matrix corresponding to the target network layer; Perform word unit selection processing from the multiple other word units according to the first key-value cache and the first vector matrix to obtain multiple target representative word units corresponding to the target network layer; Until the multiple first representative word-grams corresponding to the multiple network layers are determined respectively, the representative word-grams are obtained, wherein the representative word-grams include the multiple first representative word-grams corresponding to the multiple network layers.

11. The storage optimization method according to claim 10, characterized in that: The performing word unit selection processing from the multiple other word units according to the first key-value cache and the first vector matrix to obtain multiple target representative word units corresponding to the target network layer includes: Performing column-wise mean calculation on the first vector matrix to obtain a first anchor vector corresponding to the target network layer, wherein the number of columns of the first vector matrix is ​​the number of the multiple other word-units, and the number of rows of the first vector matrix is ​​the number of the attention heads corresponding to the target network layer; Calculating the Hamming distances between the first anchor vector and a plurality of first row vectors in the first vector matrix, respectively, where the plurality of first row vectors refer to a plurality of rows of the first vector matrix, and the plurality of first row vectors are the first word element vectors corresponding to each of the plurality of other word elements in the target network layer; Sorting the other word units corresponding to each of the plurality of first row vectors in a preset sorting manner according to the Hamming distance to obtain sorted other word units corresponding to the target network layer, wherein the preset sorting manner includes at least sorting from large to small or sorting from small to large; The other ranked words are allocated and processed based on the representative key-value cache to obtain a plurality of word-word categories, and word-word extraction processes are performed respectively from the plurality of word-word categories to obtain the plurality of target representative words corresponding to the target network layer.

12. The storage optimization method according to claim 11, characterized in that: The performing column-wise mean calculation on the first vector matrix to obtain the first anchor vector corresponding to the target network layer includes: Performing column-wise mean calculation on the first vector matrix to obtain an initial anchor point vector, and comparing each of the plurality of anchor point values ​​in the initial anchor point vector with the preset conversion threshold to obtain a second comparison result; If there is an anchor point value smaller than the preset conversion threshold among the plurality of anchor point values ​​determined according to the second comparison result, setting the anchor point value smaller than the preset conversion threshold as the first value; If there is an anchor point value not less than the preset conversion threshold among the multiple anchor point values ​​determined according to the second comparison result, setting the anchor point value not less than the preset conversion threshold as the second value; A vector construction process is performed according to the first value and the second value to obtain the first anchor point vector.

13. The storage optimization method according to claim 11, characterized in that: The process of allocating the sorted other words based on the representative key value cache to obtain a plurality of word-word categories, and extracting words from the plurality of word-word categories to obtain the plurality of target representative words corresponding to the target network layer includes: Evenly distributing the other ranked words according to the representative key-value cache to obtain a plurality of word-word categories, wherein the number of categories of the plurality of word-word categories is equal to the representative key-value cache corresponding to the target network layer; Random word extraction processing is performed on the multiple word categories to obtain the multiple target representative word elements corresponding to the target network layer, wherein the number of word elements in the multiple target representative word elements is equal to the representative key-value cache corresponding to the target network layer.

14. The storage optimization method according to claim 10, characterized in that: The storing and processing of the key key value corresponding to the key word, the input key values ​​corresponding to each of the plurality of input word units, and the representative key value corresponding to the representative word unit includes: Calculating the input key values ​​corresponding to the multiple input word-grams respectively according to all the attention heads, and storing the multiple input key values ​​calculated by each of the attention heads respectively; Calculating first key values ​​corresponding to each of the plurality of first keyword elements based on all the attention heads, and storing the plurality of first key values ​​calculated for each of the attention heads, wherein the key value includes the plurality of first key values; Calculating first representative key values ​​corresponding to each of the plurality of first representative word-grams based on all the attention heads, and storing a plurality of first representative key values ​​corresponding to each of the plurality of first representative word-grams for each of the plurality of network layers, wherein the representative key value includes a plurality of the first representative key values; Delete the key values ​​to be discarded from the multiple cache key values, wherein the key values ​​to be discarded are key values ​​corresponding to the word elements in the multiple cache word elements except the keyword word element and the representative word element.

15. The storage optimization method according to claim 5, characterized in that: Before calculating target attention values ​​of a plurality of cached words corresponding to a plurality of input words and a plurality of cached key values ​​in the preset model, the method further includes: Inputting the request information into the preset model for information processing to obtain a target word sequence corresponding to the request information; Determining an input order of the target word sequence, wherein the input order is the order in which the target word sequence is input into the preset model; In the target word sequence, the preset number of words are selected as the multiple input words in the input order from back to front.

16. A storage optimization device, characterized in that: include: an allocation unit, configured to perform key-value cache allocation processing in a preset model to obtain first key-value caches corresponding to respective multiple network layers in the preset model, wherein a higher level of the multiple network layers corresponds to a smaller first key-value cache; A calculation unit, configured to calculate target attention values ​​of a plurality of cached word elements corresponding to a plurality of input word elements and a plurality of cached key values ​​in the preset model; a first selection unit, configured to select a word from the plurality of cached word-grams according to the first key-value cache and the target attention value to obtain a keyword word; A construction unit, configured to construct word element vectors corresponding to a plurality of other word elements according to the target attention value; a second selection unit, configured to perform word-gram selection processing from the plurality of other word-grams according to the first key-value cache and the word-gram vector to obtain a representative word-gram, wherein the plurality of other word-grams are cached word-grams in the plurality of cached word-grams excluding the keyword word-gram; a storage unit, configured to store the key value corresponding to the key word, the input key values ​​corresponding to each of the plurality of input word units, and the representative key value corresponding to the representative word unit; The key-value cache allocation process is performed in the preset model to obtain the first key-value cache corresponding to each of the plurality of network layers in the preset model, including: Determine a preset total key-value cache, wherein the preset total key-value cache includes a first total cache of the multiple network layers and a reserved total cache of the multiple network layers; Allocating the reserved total buffer according to the number of levels of the multiple network layers, the number of attention heads corresponding to each of the multiple network layers, and a preset number of inputs to obtain a first reserved buffer corresponding to each of the multiple network layers, wherein the preset number of inputs is a pre-configured number of input tokens, and each network layer includes a plurality of attention heads; Allocating the first total cache according to the number of layers, the layers corresponding to the multiple network layers, and a preset layer cache ratio to obtain an initial key-value cache corresponding to each of the multiple network layers, wherein the preset layer cache ratio is a pre-configured ratio between an initial key-value cache corresponding to a lowest layer of the multiple network layers and an initial key-value cache corresponding to a highest layer of the multiple network layers; The first reserved cache and the initial key-value cache corresponding to each of the multiple network layers are added together to obtain the first key-value cache corresponding to each of the multiple network layers.

17. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the storage optimization method according to any one of claims 1 to 15 when executing the computer program.

18. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the storage optimization method according to any one of claims 1 to 15.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the storage optimization method according to any one of claims 1 to 15 are implemented.

Images