Cache management method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a cache management method and device, computer equipment, a storage medium and a program product. The method comprises the steps that under the condition that key value caches are distributed for model layers of a target model at least once, the target number of the key value caches needed by all the model layers is obtained; wherein the target number corresponding to the model layer is the number of key value caches required by the model layer when the accumulated attention score of the attention head of the model layer reaches a target value; one attention score is generated every time the attention head based on the model layer executes the calculation task, and the accumulated attention score is an accumulated result of the attention scores after the attention head based on the model layer executes the calculation tasks for multiple times; and adjusting the allocated key value cache of each model layer according to the target number corresponding to each model layer. By adopting the method, the difference of key value cache demands of different model layers can be considered, and the key value cache can be allocated to each model layer more reasonably and accurately, so that the resource utilization rate and the model performance are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a cache management method, apparatus, computer equipment, storage medium and program product. Background Art

[0002] Large language models are a technology that has developed rapidly in the field of natural language processing in recent years and have been widely used in tasks such as code completion and question answering. In the model reasoning process of large language models, in order to speed up the calculation speed, the method of storing the pre-calculated hidden states of keys and values ​​in memory is usually adopted, which is called KV cache. This method can reduce repeated calculations and thus improve the response speed of large language models.

[0003] However, as the length of the processed text increases, the memory resources occupied by the KV cache will become very large. In order to solve the memory problem in long text processing, it is necessary to reduce the KV cache size as much as possible while keeping the reasoning performance of the large language model close to that when using the full KV cache. However, existing optimization methods usually do not take into account the differences in the KV cache requirements of different layers, and uniformly allocate the KV cache of each layer, resulting in a waste of memory resources or a decrease in model performance. Summary of the invention

[0004] Based on this, it is necessary to provide a cache management method, device, computer equipment, storage medium and program product that can more reasonably allocate the key-value cache of each layer of the model, thereby improving resource utilization and model performance.

[0005] In a first aspect, the present application provides a cache management method, comprising:

[0006] When the model layer of the target model has been allocated with a key-value cache at least once, the target number of key-value caches required for each model layer is obtained; wherein the target number corresponding to the model layer is the number of key-value caches required for the model layer when the cumulative attention score of the attention head of the model layer reaches the target value; each time the attention head of the model layer performs a computing task, an attention score is generated, and the cumulative attention score is the cumulative result of the attention scores after the attention head of the model layer performs multiple computing tasks;

[0007] Adjust the key-value cache allocated to each model layer according to the number of targets corresponding to each model layer.

[0008] In one embodiment, the process of allocating at least one key-value cache to a model layer of a target model includes:

[0009] The key-value cache is distributed to each model layer of the target model using a first distribution method; wherein the first distribution method includes at least one of average distribution or distribution according to the level of each model layer.

[0010] In one embodiment, the process of allocating at least one key-value cache to a model layer of a target model includes:

[0011] Using the second allocation method, a basic number of key-value caches are allocated to each model layer of the target model;

[0012] A third allocation method is used to reallocate the key-value cache to each model layer; wherein the second allocation method is different from the third allocation method, the second allocation method includes average allocation, and the third allocation method includes at least one of allocation according to the level of each model layer or allocation according to the quantity ratio between the target quantities corresponding to each model layer.

[0013] In one embodiment, obtaining a target number of key-value caches required by each model layer includes:

[0014] Get the initial number set of key-value caches used by each model layer; the initial number corresponding to the model layer is the number of key-value caches used by the model layer when the cumulative attention score of the attention head of the model layer reaches the target value when the sample data of different context lengths are calculated based on the attention head of the model layer to perform the calculation task;

[0015] Based on the initial number of key-value caches used by each model layer, determine the target number of key-value caches required by each model layer.

[0016] In one embodiment, determining a target number of key-value caches required for each model layer according to an initial number set of key-value caches used by each model layer includes:

[0017] For each model layer, the statistical value of the initial quantity in the initial quantity set corresponding to the model layer is used as the target quantity of the key-value cache required by the model layer.

[0018] In one of the embodiments, when allocating according to the level of each model layer, the number of key-value caches allocated to each model layer decreases as the level increases.

[0019] In one embodiment, the process of allocating according to the quantity ratio between the target quantities corresponding to each model layer includes:

[0020] Determine the remaining number of key-value caches based on the allocated key-value caches;

[0021] Determine the number of items to be allocated for each model layer based on the remaining number of key-value cache items and the ratio of the target number of items for each model layer.

[0022] Re-allocate the key-value cache corresponding to the number to be allocated for each model layer.

[0023] In one embodiment, according to the target quantity corresponding to each model layer, the key-value cache allocated to each model layer is adjusted, including:

[0024] For a model layer, based on the size of the attention score, determine the target number of key-value caches from the key-value caches allocated to the model layer;

[0025] Reclaims all key-value caches except the target number of key-value caches.

[0026] In a second aspect, the present application further provides a cache management device, including:

[0027] A quantity acquisition module is used to obtain the target number of key-value caches required for each model layer when at least one key-value cache has been allocated to the model layer of the target model; wherein the target number corresponding to the model layer is the number of key-value caches required for the model layer when the cumulative attention score of the attention head of the model layer reaches the target value; each time the attention head of the model layer executes a computing task, an attention score is generated, and the cumulative attention score is the cumulative result of the attention scores after the attention head of the model layer executes multiple computing tasks;

[0028] The cache adjustment module is used to adjust the key-value cache allocated to each model layer according to the target quantity corresponding to each model layer.

[0029] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0030] When the model layer of the target model has been allocated with a key-value cache at least once, the target number of key-value caches required for each model layer is obtained; wherein the target number corresponding to the model layer is the number of key-value caches required for the model layer when the cumulative attention score of the attention head of the model layer reaches the target value; each time the attention head of the model layer performs a computing task, an attention score is generated, and the cumulative attention score is the cumulative result of the attention scores after the attention head of the model layer performs multiple computing tasks;

[0031] Adjust the key-value cache allocated to each model layer according to the number of targets corresponding to each model layer.

[0032] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0033] When the model layer of the target model has been allocated with a key-value cache at least once, the target number of key-value caches required for each model layer is obtained; wherein the target number corresponding to the model layer is the number of key-value caches required for the model layer when the cumulative attention score of the attention head of the model layer reaches the target value; each time the attention head of the model layer performs a computing task, an attention score is generated, and the cumulative attention score is the cumulative result of the attention scores after the attention head of the model layer performs multiple computing tasks;

[0034] Adjust the key-value cache allocated to each model layer according to the number of targets corresponding to each model layer.

[0035] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0036] When the model layer of the target model has been allocated with a key-value cache at least once, the target number of key-value caches required for each model layer is obtained; wherein the target number corresponding to the model layer is the number of key-value caches required for the model layer when the cumulative attention score of the attention head of the model layer reaches the target value; each time the attention head of the model layer performs a computing task, an attention score is generated, and the cumulative attention score is the cumulative result of the attention scores after the attention head of the model layer performs multiple computing tasks;

[0037] Adjust the key-value cache allocated to each model layer according to the number of targets corresponding to each model layer.

[0038] The above-mentioned cache management method, device, computer equipment, storage medium and program product, when the key value cache has been allocated to the model layer of the target model at least once, obtains the target number of key value caches required for each model layer; wherein the target number corresponding to the model layer is the number of key value caches required for the model layer when the cumulative attention score of the attention head of the model layer reaches the target value; an attention score is generated each time the attention head of the model layer performs a calculation task, and the cumulative attention score is the cumulative result of the attention score after the attention head of the model layer performs multiple calculation tasks; according to the target number corresponding to each model layer, the key value cache allocated to each model layer is adjusted. This application determines the cumulative attention score of the attention head of each model layer and the target number of key value caches required when the target value is reached by considering the differences in the key value cache requirements of each model layer of the target model, and further adjusts the key value caches allocated to each model layer, so as to achieve more reasonable and accurate allocation of key value caches to each model layer, thereby improving resource utilization and model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0040] FIG1 is an application environment diagram of a cache management method in one embodiment;

[0041] FIG2 is a schematic diagram of a flow chart of a cache management method in one embodiment;

[0042] FIG3 is a schematic diagram of a key-value cache when the accumulated attention score reaches a target value in one embodiment;

[0043] FIG4 is a schematic diagram of a process for allocating key-value caches to each model layer in one embodiment;

[0044] FIG5 is a schematic diagram of a process for determining the number of targets corresponding to each model layer in one embodiment;

[0045] FIG6 is a schematic diagram of a middle layer uncertainty curve according to an embodiment;

[0046] FIG7 is a schematic diagram of a process of reallocating a key-value cache in one embodiment;

[0047] FIG8 is a schematic diagram of a process for recycling a key-value cache in one embodiment;

[0048] FIG9 is a schematic flow chart of a cache management method in another embodiment;

[0049] FIG10 is a block diagram of a cache management device according to an embodiment;

[0050] FIG11 is a block diagram of a cache management device in another embodiment;

[0051] FIG12 is a structural block diagram of a cache management device in yet another embodiment;

[0052] FIG. 13 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] The cache management method provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Specifically, the server 104 obtains the target number of key value caches required for each model layer when the key value cache has been allocated to the model layer of the target model at least once, and adjusts the key value caches allocated to each model layer according to the target number corresponding to each model layer; wherein the target number corresponding to the model layer is the number of key value caches required by the model layer when the cumulative attention score of the attention head of the model layer reaches the target value; an attention score is generated each time the attention head of the model layer executes a calculation task, and the cumulative attention score is the cumulative result of the attention score after the attention head of the model layer executes multiple calculation tasks.

[0055] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0056] In an exemplary embodiment, as shown in FIG2 , a cache management method is provided, including:

[0057] S201, when a key-value cache has been allocated at least once to a model layer of a target model, obtaining a target number of key-value caches required for each model layer.

[0058] The key-value cache is the KV cache, which can be expressed as KV cache. Storing the pre-calculated key-value hidden state in the key-value cache can avoid redundant calculation.

[0059] The target model is the model of the key-value cache to be currently allocated, which can be used to process, identify and / or generate one or more of multiple modal data such as text, images, audio, code, etc., such as natural language processing tasks such as text classification, question and answer, and dialogue, content creation and enhancement tasks such as image recognition or generation, music recognition or generation, as well as multimodal / cross-modal tasks such as sentiment analysis, translation, and question and answer that combine multiple modal data.

[0060] The target model can specifically be a large language model, which is a deep learning model widely used in natural language processing. The target model can be a deep learning model, such as CNN (Convolutional Neural Networks), RNN (Recurrent Neural Networks) and LSTM (Long Short-Term Memory). The target model can also be a generative model, such as GAN (Generative Adversarial Networks).

[0061] The target model includes multiple model layers, and the sizes of key value caches required for calculations in different model layers are different. In an initial state, at least one key value cache is allocated to each model layer, such as one or two key value caches.

[0062] Exemplarily, all key-value caches of the target model are allocated to each model layer at once, or a key-value cache of a basic size is uniformly allocated to each model layer to ensure that each model layer has a minimum key-value cache to maintain the basic functions of the target model, and then the remaining key-value cache is allocated to each model layer in any manner.

[0063] Optionally, the present embodiment may also perform more allocations, each with a different allocation method, so as to perform targeted allocation at a finer granularity. For example, a portion of the key-value cache is evenly allocated to each model layer to maintain basic functions, and then a portion of the key-value cache is allocated to each model layer according to the hierarchy, and then the remaining key-value cache is allocated to each model layer according to the quantity ratio between the target quantities.

[0064] Furthermore, when the model layer of the target model has been allocated at least one key-value cache, the target number of key-value caches required by each model layer is obtained. For each model layer, the target number of key-value caches required by the model layer, that is, the target number corresponding to the model layer, refers to the number of key-value caches required by the model layer when the cumulative attention score of the attention head of the model layer reaches the target value. The target numbers corresponding to different model layers are different and need to be determined separately.

[0065] Among them, attention head and attention score are terms in the attention mechanism of Transformer. Each time the attention head of the model layer performs a calculation task, an attention score is generated. The cumulative attention score is the cumulative result of the attention score after the attention head of the model layer performs multiple calculation tasks. For each model layer, the attention module divides the Query, Key, and Value into several parts, each of which is processed by a separate Attention Head. In other words, each attention head has a separate Query, Key, and Value matrix, and each attention head performs an Attention calculation based on a separate matrix. Then, all Attention calculation results are merged to form a cumulative attention score.

[0066] Specifically, during the historical or real-time model inference process of the target model, the number of key-value caches required by each model layer is determined when the cumulative attention score of the attention head of each model layer reaches the target value, and it is used as the target number corresponding to each model layer. Then, the target number corresponding to each model layer is directly obtained, or the target number corresponding to each model layer is determined in real time.

[0067] The target value corresponding to the cumulative attention score can be a preset empirical value, such as 90%. It can be understood that when the cumulative attention scores of the attention heads of each model layer reach the target value, the reasoning performance of the target model is similar to that when using a complete key-value cache.

[0068] Exemplarily, as shown in FIG3, each row represents all key-value caches occupied by a model layer, each small square represents one of the key-value caches, and the value on the small square represents the attention score of the attention head. When the cumulative value on the small square reaches the target value, such as 90%, the number of small squares counted is the target number corresponding to the model layer. Taking layer 1 as an example, the value on the first small square is 0.50, the value on the second small square is 0.30, and the value on the third small square is 0.12. At this time, the cumulative value of the first three small squares is 0.92, which has reached the target value. Therefore, when it is determined that the cumulative attention score of the attention head of layer 1 reaches the target value, the target number of key-value caches required for layer 1 is 3. The colored area in FIG3 represents the key-value cache whose cumulative attention score reaches the target value, and the colorless area represents the key-value cache whose cumulative attention score exceeds the target value.

[0069] S202: According to the target quantity corresponding to each model layer, the key-value cache allocated to each model layer is adjusted.

[0070] Among them, the key-value caches allocated to each model layer are adjusted, including partially recycling the key-value caches allocated to each model layer, so as to reduce the size of the key-value caches occupied by each model layer as much as possible while ensuring the model performance of the target model. Specifically, the target number of key-value caches is reserved for the model layer, and the key-value caches exceeding the target number are recycled.

[0071] Exemplarily, the target model has three model layers (layer 0, layer 1, and layer 2), and 1,000 allocatable key-value caches. After all or part of these 1,000 key-value caches are allocated to the three model layers in any allocation method, for each model layer, the over-allocated key-value caches are recycled according to the target number corresponding to the model layer.

[0072] Optionally, if the number of key-value caches allocated to any model layer is less than the target number, a corresponding number of key-value caches can be allocated to the model layer again after the key-value caches allocated to other model layers are recycled, so that the number of key-value caches finally used by the model layer reaches the target number.

[0073] In the above-mentioned cache management method, when the key value cache has been allocated to the model layer of the target model at least once, the target number of key value caches required by each model layer is obtained; wherein, the target number corresponding to the model layer is the number of key value caches required by the model layer when the cumulative attention score of the attention head of the model layer reaches the target value; an attention score is generated each time the attention head of the model layer performs a calculation task, and the cumulative attention score is the cumulative result of the attention score after the attention head of the model layer performs multiple calculation tasks; according to the target number corresponding to each model layer, the key value caches allocated to each model layer are adjusted. This embodiment determines the cumulative attention score of the attention head of each model layer and the target number of key value caches required when the target value is reached by considering the differences in the key value cache requirements of each model layer of the target model, and further adjusts the key value caches allocated to each model layer, so as to achieve more reasonable and accurate allocation of key value caches to each model layer, thereby improving resource utilization and model performance.

[0074] In an exemplary embodiment, the process of allocating a key-value cache at least once to the model layer of the target model includes:

[0075] The key-value cache is distributed to each model layer of the target model using a first distribution method; wherein the first distribution method includes at least one of average distribution or distribution according to the level of each model layer.

[0076] In this embodiment, a key value cache is allocated once for the model layer of the target model. Generally speaking, all key value caches of the target model need to be allocated to the model layer. In this embodiment, only part of the key value cache can be allocated to each model layer to reserve a certain number of key value caches for the reliability operation of the target model.

[0077] Optionally, the key-value cache is evenly distributed to each model layer of the target model. Exemplarily, the target model has 5 model layers and 1000 allocatable key-value caches, and 200 key-value caches are allocated to each model layer.

[0078] Optionally, different numbers of key-value caches are allocated to different model layers according to the levels of each model layer. Model layers with lower levels are allocated more key-value caches, and model layers with higher levels are allocated fewer key-value caches, so as to fit the model characteristics in actual application scenarios. The number of key-value caches allocated to model layers of different levels can be a preset constant, such as 200 key-value caches allocated to layer 0, 190 key-value caches allocated to layer 1, etc.

[0079] Optionally, according to the level of each model layer, determine the number of key-value caches corresponding to each model layer; according to the number of key-value caches corresponding to each model layer, allocate key-value caches to each model layer; wherein the number of key-value caches corresponding to each model layer decreases as the level increases. That is, there is an equal proportional relationship between the number of key-value caches allocated to each model layer, such as 200 key-value caches are allocated to layer 0, 180 key-value caches are allocated to layer 1, and 160 key-value caches are allocated to layer 2, etc.

[0080] This embodiment provides a method for allocating key-value cache to each model layer of the target model in an initial state, which can maintain the normal operation of the target model. The first allocation method is used for allocation, which is conducive to reducing the difficulty of subsequent adjustment of the key-value cache.

[0081] In an exemplary embodiment, as shown in FIG4 , the process of allocating a key-value cache at least once to the model layer of the target model includes:

[0082] S401, using the second allocation method to allocate a basic number of key-value caches to each model layer of the target model.

[0083] In this embodiment, the key-value cache is allocated twice to the model layer of the target model, the first allocation is performed using the second allocation method, and the second allocation is performed using the third allocation method, and the second allocation method is different from the third allocation method.

[0084] The second allocation method includes average allocation, and a basic number of key-value caches can be allocated to each model layer of the target model. The basic number is a preset empirical value, and the basic number of key-value caches is a cache resource for maintaining the basic functions of the target model. Specifically, the data items that exceed a specific threshold in the historical performance data of each model layer of the target model are counted. After data accumulation and related experiments, the basic number of key-value caches that meet basic needs can be obtained. This embodiment does not limit the size of the basic number. Exemplarily, the target model has a total of 30 model layers and 900 allocable key-value caches. When the basic number is 10, 300 of the key-value caches are evenly distributed to 30 model layers, that is, 10 key-value caches are allocated to each model layer to meet basic needs.

[0085] S402: Use the third allocation method to allocate key-value cache to each model layer again.

[0086] The third allocation method includes at least one of allocating according to the level of each model layer or allocating according to the quantity ratio between the target quantities corresponding to each model layer.

[0087] Optionally, different numbers of key-value caches can be allocated to different model layers from the remaining allocatable key-value caches according to the level of each model layer. Model layers with lower levels are allocated more key-value caches, and model layers with higher levels are allocated fewer key-value caches, so as to fit the model characteristics in actual application scenarios. Specifically, the empirical value of the number of key-value caches corresponding to each model layer can be determined according to the level of each model layer; key-value caches can be allocated to each model layer according to the empirical value of the number of key-value caches corresponding to each model layer; wherein the empirical value of the number of key-value caches corresponding to each model layer decreases as the level increases.

[0088] Optionally, the quantity ratio between the target quantities corresponding to each model layer is calculated, and then according to the quantity ratio, the key value cache is reallocated to each model layer from the remaining allocatable key value cache, so as to realize the adaptive allocation of key value cache to each model layer. The remaining allocatable key value cache can be all the remaining key value caches, or it can be a preset number of partially remaining key value caches, so as to reserve part of the key value cache for the target model. Exemplarily, according to the quantity ratio, all the remaining allocatable key value caches are allocated to the corresponding model layer. The target model has 3 model layers and 300 remaining allocatable key value caches, and the quantity ratio between the target quantities corresponding to the 3 model layers is 1:3:2, 50 key value caches are allocated to layer 0, 150 key value caches are allocated to layer 1, and 100 key value caches are allocated to layer 2.

[0089] Optionally, when allocating allocable key-value caches to the corresponding model layers according to the quantity ratio, the corresponding allocation quantity of each model layer is calculated, and then the difference between the allocation quantity and the allocated basic quantity is determined as the re-allocated quantity. Exemplarily, the target model has 3 model layers and 600 allocable key-value caches, and the quantity ratio between the target quantities corresponding to the 3 model layers is 1:3:2. Previously, 100 basic quantity key-value caches have been allocated to each of the 3 model layers, so 0 key-value caches are allocated to layer 0, 200 key-value caches are allocated to layer 1, and 100 key-value caches are allocated to layer 2.

[0090] This implementation provides a method for allocating key-value cache to each model layer of the target model in an initial state, which can maintain the normal operation of the target model. The second allocation method and the third allocation method are used for allocation, which is conducive to reducing the difficulty of subsequent adjustments to the key-value cache.

[0091] In an exemplary embodiment, as shown in FIG5 , the above S201 includes:

[0092] S501, obtaining an initial quantity set of key-value caches used by each model layer.

[0093] Among them, for each model layer, the initial number of key-value caches used by the model layer, that is, the initial number corresponding to the model layer, refers to the number of key-value caches used by the model layer when the cumulative attention score of the attention head of the model layer reaches the target value when calculating sample data of different context lengths based on the attention head of the model layer to perform computing tasks.

[0094] It can be understood that sample data of one context length corresponds to an initial quantity, so by calculating the sample quantities of different context lengths, an initial quantity set consisting of multiple initial quantities can be obtained, and one model layer corresponds to an initial quantity set.

[0095] Exemplarily, sample data of three different context lengths are respectively input into the target model to control the attention heads of each model layer of the target model, and the corresponding computing tasks are respectively performed based on the sample data of the three different context lengths. For each model layer, the model layer corresponds to an initial quantity under sample data of each context length, and corresponds to different initial quantities under sample data of different context lengths, forming an initial quantity set including three different initial quantities.

[0096] S502, determining the target number of key-value caches required by each model layer according to the initial number set of key-value caches used by each model layer.

[0097] For each model layer, the target number of key-value caches required by the model layer is determined according to the initial number set of key-value caches used by the model layer, thereby obtaining the target number of key-value caches required by each model layer.

[0098] Optionally, for each model layer, a statistical value of the initial quantity in the initial quantity set corresponding to the model layer is used as a target quantity of key-value cache required by the model layer, wherein the statistical value is one of a mean, a median, a weighted average, and a mode.

[0099] Exemplarily, the initial numbers corresponding to layer 1 under three different context lengths of sample data are 150, 300 and 360 respectively, and the average value of the initial numbers is calculated to be 270, so the target number of key-value caches required for layer 1 is determined to be 270.

[0100] Exemplarily, the minimum number of words (target number of key-value cache) required for the model layer to meet the cumulative attention score of 90% is defined as the layer uncertainty of the model layer. The layer uncertainty curves of the two large language models (Mistral and LLaMa) are shown in Figure 6. When these two large language models are used as target models, the target model may include 32 model layers (Layer Index) from layer 0 to layer 31. By calculating the average of the initial numbers in the initial number set corresponding to each model layer, the target number of key-value caches required for each model layer is obtained, thereby generating the layer uncertainty curve. The layer uncertainty curve shown in Figure 6 can intuitively show the quantity ratio between the target numbers corresponding to each model layer.

[0101] Optionally, for each model layer, after removing the minimum initial quantity and the maximum initial quantity in the initial quantity set corresponding to the model layer, the average of the remaining initial quantities is taken and used as the target quantity of the key-value cache required by the model layer.

[0102] In this embodiment, sample data of different context lengths are calculated based on the attention heads of each model layer to perform computing tasks, thereby determining an initial set of key-value caches used by each model layer, and then determining a target number based on the initial set of key-value caches. This can take into account the impact of sample data of different context lengths on the number of key-value caches used in the calculation process, maximize the accuracy of the target number finally determined, ensure efficient use of cache resources under different workloads and context lengths, and enhance the adaptability and flexibility of the target model.

[0103] In an exemplary embodiment, as shown in FIG7 , the process of allocating according to the quantity ratio between the target quantities corresponding to each model layer includes:

[0104] S701, determining the remaining quantity of the key-value cache according to the allocated key-value cache.

[0105] According to the total amount of key-value cache corresponding to the target model and the amount of key-value cache allocated to each model layer, the remaining amount of key-value cache can be determined.

[0106] Optionally, when the second allocation method and the third allocation method are used for allocation, if the second allocation method used is average allocation, the product of the basic number and the number of model layers of the target model is calculated to obtain the number of allocated key-value caches, and then the difference between the total amount of key-value caches and the allocated number is calculated to obtain the remaining number of key-value caches. The basic number is the number of key-value caches allocated to each model layer of the target model before obtaining the target number of key-value caches required for each model layer. The basic number corresponding to each model layer is the same, and the basic number of key-value caches is a cache resource for maintaining the basic functions of the target model.

[0107] S702, determining the to-be-allocated quantity corresponding to each model layer according to the remaining quantity of the key-value cache and the quantity ratio between the target quantities corresponding to each model layer.

[0108] The remaining number of key-value caches is allocated to each model layer according to the quantity ratio between the target quantities corresponding to each model layer. The proportion of the to-be-allocated quantity corresponding to each model layer in the remaining quantity can be determined based on the quantity ratio. Then, when the remaining number of key-value caches is known, the to-be-allocated quantity corresponding to each model layer can be calculated.

[0109] Optionally, the number of items to be allocated corresponding to each model layer is calculated according to the following formula:

[0110] ;

[0111] Among them, s i is the number of items to be allocated corresponding to the i-th model layer, S is the remaining number of key-value caches, the target model has a total of n model layers (n ≥ i), and the ratio of the number of targets corresponding to each model layer is expressed as x1:x2:…:x n .

[0112] S703: Re-allocate the key-value cache corresponding to the number to be allocated to each model layer.

[0113] For each model layer, the key-value caches to be allocated corresponding to the model layer are allocated to the model layer from the remaining allocatable key-value caches.

[0114] In this embodiment, based on the ratio between the target quantities corresponding to each model layer and the remaining quantity of the key-value cache, the number of key-value caches to be reallocated to each model layer is determined in real time. Taking into account the differences in the key-value cache requirements of each model layer of the target model, different numbers of key-value caches are allocated to different model layers in the reallocation process, so that each model layer can maximize the computing power and enhance the reliability of the target model operation.

[0115] In an exemplary embodiment, as shown in FIG8 , the above S202 includes:

[0116] S801, for a model layer, based on the size of the attention score, determine the target number of key-value caches from the key-value caches allocated to the model layer.

[0117] It should be noted that when performing calculation tasks based on the attention head of the model layer, each time a new token (text unit) is generated, the target model needs to update the key-value cache and recalculate the attention score. Each key vector in the key-value cache corresponds to an attention score, which is obtained by performing a dot product operation between the query vector and the key vector. It is used to indicate the degree of attention of the newly generated token to other tokens in each key-value cache. Therefore, a key-value cache corresponds to an attention score.

[0118] According to the size of the attention score corresponding to the key-value cache, a target number of key-value caches can be determined from the key-value caches allocated in the model layer for retention, such as retaining the key-value caches with larger attention scores.

[0119] Optionally, the key-value caches assigned to the model layer are arranged in descending order according to the attention score to obtain a corresponding key-value cache sequence. As shown in Figure 3, taking layer 1 as an example, when 5 key-value caches are assigned to layer 1, a key-value cache sequence corresponding to layer 1 is formed {0.50, 0.30, 0.12, 0.11, 0.01}.

[0120] S802: Reclaim the key-value caches other than the target number of key-value caches.

[0121] Furthermore, key-value caches with larger attention scores are retained, and key-value caches with smaller attention scores are recycled, so that the number of key-value caches in the model layer is controlled within the target number. For example, when the target number corresponding to layer 1 is 3, key-value caches with attention scores of 0.50, 0.30, and 0.12 are retained, and key-value caches with attention scores of 0.11 and 0.01 are recycled.

[0122] In this embodiment, based on the size of the attention score corresponding to the key-value cache, the key-value cache allocated to the model layer is partially recycled, and the number of key-value caches in the model layer is controlled within the target number. Through the real-time recycling mechanism, the model performance of the target model is guaranteed while allocating as few key-value caches as possible, thereby improving resource utilization.

[0123] In an exemplary embodiment, as shown in FIG9 , a cache management method is provided, including:

[0124] S901, when a basic number of key-value caches have been allocated to each model layer of the target model, an initial number set of key-value caches used by each model layer is obtained.

[0125] Among them, the basic number of key-value caches are cache resources for maintaining the basic functions of the target model, and the initial number corresponding to the model layer is the number of key-value caches used by the model layer when the cumulative attention score of the attention head of the model layer reaches the target value when calculating sample data of different context lengths based on the attention head of the model layer to perform computing tasks.

[0126] S902: Determine the target number of key-value caches required by each model layer according to the initial number set of key-value caches used by each model layer.

[0127] Optionally, for each model layer, an average of the initial quantities in the initial quantity set corresponding to the model layer is used as the target quantity of the key-value cache required for the model layer.

[0128] Among them, the target number corresponding to the model layer is the number of key-value caches required by the model layer when the cumulative attention score of the attention head of the model layer reaches the target value; each time the attention head of the model layer executes a calculation task, an attention score is generated, and the cumulative attention score is the cumulative result of the attention score after the attention head of the model layer executes multiple calculation tasks.

[0129] S903: Determine the remaining quantity of the key-value cache according to the allocated key-value cache.

[0130] S904, determining the to-be-allocated quantity corresponding to each model layer according to the remaining quantity of the key-value cache and the quantity ratio between the target quantities corresponding to each model layer.

[0131] S905: Re-allocate the key-value cache corresponding to the number to be allocated to each model layer.

[0132] S906: Reclaim the key-value cache allocated to each model layer according to the target quantity corresponding to each model layer.

[0133] Optionally, for each model layer, the key-value caches allocated to the model layer are arranged in descending order according to the size of the attention scores to obtain a corresponding key-value cache sequence; wherein, each time the attention head based on the model layer executes a computing task, a key-value cache is required; and in the key-value cache sequence, the key-value caches other than the first target number of key-value caches are recycled.

[0134] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0135] Based on the same inventive concept, the embodiment of the present application also provides a cache management device for implementing the cache management method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more cache management device embodiments provided below can refer to the limitations of the cache management method above, and will not be repeated here.

[0136] In an exemplary embodiment, as shown in FIG10 , a cache management device is provided, including:

[0137] The quantity acquisition module 100 is used to acquire the target quantity of key-value caches required for each model layer when the key-value cache has been allocated to the model layer of the target model at least once.

[0138] Among them, the target number corresponding to the model layer is the number of key-value caches required by the model layer when the cumulative attention score of the attention head of the model layer reaches the target value; each time the attention head of the model layer executes a calculation task, an attention score is generated, and the cumulative attention score is the cumulative result of the attention score after the attention head of the model layer executes multiple calculation tasks.

[0139] The cache adjustment module 200 is used to adjust the key-value cache allocated to each model layer according to the target quantity corresponding to each model layer.

[0140] In an exemplary embodiment, the cache management device further includes:

[0141] The first allocation module is used to allocate the key-value cache to each model layer of the target model by adopting a first allocation method.

[0142] The first allocation method includes at least one of equal allocation and allocation according to the level of each model layer.

[0143] In an exemplary embodiment, the cache management device further includes:

[0144] The second allocation module is used to allocate a basic number of key-value caches to each model layer of the target model by adopting a second allocation method.

[0145] The third allocation module is used to reallocate the key-value cache to each model layer again by adopting a third allocation method.

[0146] Among them, the second allocation method is different from the third allocation method. The second allocation method includes average allocation, and the third allocation method includes at least one of allocation according to the level of each model layer or allocation according to the quantity ratio between the target quantities corresponding to each model layer.

[0147] In an exemplary embodiment, based on FIG. 10 , as shown in FIG. 11 , the quantity acquisition module 100 includes:

[0148] The quantity set acquisition unit 110 is used to acquire an initial quantity set of the key-value cache used by each model layer.

[0149] Among them, the initial number corresponding to the model layer is the number of key-value caches used by the model layer when the cumulative attention score of the attention head of the model layer reaches the target value when calculating sample data of different context lengths based on the attention head of the model layer to perform computing tasks.

[0150] The target quantity determination unit 120 is used to determine the target quantity of key-value caches required by each model layer according to the initial quantity set of key-value caches used by each model layer.

[0151] In an exemplary embodiment, the quantity determination unit 120 is specifically configured to use, for each model layer, a statistical value of the initial quantity in the initial quantity set corresponding to the model layer as a target quantity of the key-value cache required by the model layer.

[0152] In an exemplary embodiment, when allocating according to the level of each model layer, the number of key-value caches allocated to each model layer decreases as the level increases.

[0153] In an exemplary embodiment, the third allocation module includes:

[0154] The remaining quantity determining unit is used to determine the remaining quantity of the key-value cache according to the allocated key-value cache.

[0155] The unit for determining the number to be allocated is used to determine the number to be allocated corresponding to each model layer according to the remaining number of the key-value cache and the quantity ratio between the target numbers corresponding to each model layer.

[0156] The cache allocation unit is used to reallocate the key-value cache of the corresponding number to be allocated to each model layer.

[0157] In an exemplary embodiment, based on FIG. 10 , as shown in FIG. 12 , the cache adjustment module 200 includes:

[0158] The cache screening unit 210 is used to determine a target number of key-value caches from the key-value caches allocated to the model layer based on the size of the attention score for a model layer.

[0159] The cache recycling unit 220 is used to recycle the key-value caches other than the target number of key-value caches.

[0160] Each module in the cache management device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0161] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG13. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store sample data of different context lengths, etc. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a cache management method is implemented.

[0162] Those skilled in the art will understand that the structure shown in FIG. 13 is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0163] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the above cache management method when executing the computer program.

[0164] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the cache management method described above are implemented.

[0165] In one embodiment, a computer program product is provided, including a computer program, which implements the steps of the cache management method when executed by a processor.

[0166] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0167] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0168] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the present application. It should be noted that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A cache management method, characterized in that: The method comprises: In the case where a key value cache has been allocated at least once to the model layer of the target model, the target number of key value caches required for each model layer is obtained; wherein the target number corresponding to the model layer is the number of key value caches required for the model layer when the cumulative attention score of the attention head of the model layer reaches the target value; an attention score is generated each time the attention head of the model layer executes a computing task, and the cumulative attention score is the cumulative result of the attention scores after the attention head of the model layer executes multiple computing tasks; Adjust the key-value cache allocated to each model layer according to the number of targets corresponding to each model layer.

2. The method according to claim 1, characterized in that The process of assigning at least one key-value cache to the model layer of the target model includes: The key-value cache is distributed to each model layer of the target model using a first distribution method; wherein the first distribution method includes at least one of average distribution or distribution according to the hierarchy of each model layer.

3. The method according to claim 1, characterized in that The process of assigning at least one key-value cache to the model layer of the target model includes: Using the second allocation method, a basic number of key-value caches are allocated to each model layer of the target model; A third allocation method is used to reallocate the key-value cache to each model layer; wherein the second allocation method is different from the third allocation method, the second allocation method includes average allocation, and the third allocation method includes at least one of allocation according to the level of each model layer or allocation according to the quantity ratio between the target quantities corresponding to each model layer.

4. The method according to claim 2 or 3, characterized in that: The target number of key-value caches required for each model layer is obtained, including: Obtain an initial number set of key-value caches used by each model layer; wherein the initial number corresponding to the model layer is the number of key-value caches used by the model layer when the cumulative attention score of the attention head of the model layer reaches the target value when calculating sample data of different context lengths based on the attention head of the model layer to perform a computing task; Based on the initial number of key-value caches used by each model layer, determine the target number of key-value caches required by each model layer.

5. The method according to claim 4, characterized in that Determining the target number of key-value caches required for each model layer according to the initial number set of key-value caches used by each model layer includes: For each model layer, the statistical value of the initial quantity in the initial quantity set corresponding to the model layer is used as the target quantity of the key-value cache required by the model layer.

6. The method according to claim 2 or 3, characterized in that: When allocating according to the levels of each model layer, the number of key-value caches allocated to each model layer decreases as the level increases.

7. The method according to claim 3, characterized in that The process of allocating according to the quantity ratio between the target quantities corresponding to each model layer includes: Determining the remaining quantity of the key-value cache according to the allocated key-value cache; Determine the to-be-allocated quantity corresponding to each model layer according to the remaining quantity of the key-value cache and the quantity ratio between the target quantities corresponding to each model layer; Re-allocate the key-value cache corresponding to the number to be allocated for each model layer.

8. The method according to claim 1, characterized in that The step of adjusting the key-value cache allocated to each model layer according to the target quantity corresponding to each model layer includes: For a model layer, based on the size of the attention score, determine a target number of key-value caches from the key-value caches allocated to the model layer; The key-value caches other than the target number of key-value caches are recycled.

9. A cache management device, characterized in that: The device comprises: A quantity acquisition module, used to obtain the target quantity of key value caches required for each model layer when the model layer of the target model has been allocated at least one key value cache; wherein the target quantity corresponding to the model layer is the quantity of key value caches required for the model layer when the cumulative attention score of the attention head of the model layer reaches the target value; an attention score is generated each time the attention head of the model layer executes a computing task, and the cumulative attention score is the cumulative result of the attention scores after the attention head of the model layer executes multiple computing tasks; The cache adjustment module is used to adjust the key-value cache allocated to each model layer according to the target quantity corresponding to each model layer.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images