Key value cache compression method and electronic equipment
By distinguishing behavioral patterns in the attention head of a multimodal large language model and adopting a personalized key-value caching strategy, the problem of low performance of multimodal large language models in the prior art is solved, and more efficient key-value caching compression and decoding speed is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- HANGZHOU HIGH-TECH ZONE (BINJIANG) INSTITUTE OF BLOCKCHAIN & DATA SECURITY
- Filing Date
- 2025-12-25
- Publication Date
- 2026-05-05
AI Technical Summary
In existing technologies, the methods for key-value caching compression in multimodal large language models fail to effectively consider the attention heads of different behavioral patterns, resulting in low performance.
By determining the behavior patterns of each attention head in the decoding stage of the multimodal large language model, namely static mode and dynamic mode, different key-value caching budgets and compression strategies are adopted to perform personalized key-value caching and compression processing on each attention head.
It improves the key-value caching compression effect of multimodal large language models, reduces memory usage, and improves decoding speed and performance.
Smart Images
Figure CN121979810A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of large language model technology, and in particular relates to a key-value caching compression method and electronic device. Background Technology
[0002] With the rapid development of multimodal large language models, they have made significant progress in performance when handling inference tasks involving text, images, and videos. However, during inference tasks, multimodal large language models face challenges due to the rapid growth of key-value (KV) caches. Specifically, during inference tasks, each visual input (e.g., an image or video) of a multimodal large language model expands into thousands of tokens, causing the size of the key-value cache to grow with the context and continuously consuming processor (e.g., graphics processing unit) memory during decoding.
[0003] Currently, in order to solve the above problems, some key-value cache compression methods have been proposed in existing technologies. However, these methods only focus on the allocation of cache budgets for key-value caches at different granularities, without considering the use of different key-value cache budget determination methods for attention heads with different behavioral patterns, nor do they use different key-value cache compression processing methods for attention heads with different behavioral patterns. This results in poor compression performance of existing key-value cache compression methods, which reduces the performance of multimodal large language models. Summary of the Invention
[0004] In view of this, embodiments of this application provide a key-value caching compression method and an electronic device to solve the technical problem of low performance of multimodal large language models in the prior art.
[0005] In a first aspect, embodiments of this application provide a key-value cache compression method, including: Determine the behavior pattern of each attention head in the decoding phase of the trained multimodal large language model; Based on the behavior pattern corresponding to each attention head, determine the target cache budget corresponding to each attention head; For each attention head, key-value cache compression is performed on the attention head based on the target cache budget and the behavior pattern corresponding to that attention head.
[0006] Optionally, the behavior pattern includes a static mode and a dynamic mode; the step of performing key-value cache compression processing on each attention head according to the target cache budget corresponding to the attention head and the behavior pattern includes: For each attention head, if the behavior mode corresponding to the attention head is a static mode, then the attention head is subjected to key-value cache compression processing according to the target cache budget corresponding to the attention head and the first key-value cache compression strategy corresponding to the static mode; if the behavior mode corresponding to the attention head is a dynamic mode, then the attention head is subjected to key-value cache compression processing according to the target cache budget corresponding to the attention head and the second key-value cache compression strategy corresponding to the dynamic mode.
[0007] Optionally, determining the behavioral pattern of each attention head in the decoder phase of the trained multimodal large language model includes: For each attention head, determine the text-centric attention sparsity of that attention head in the pre-filling stage, and determine the behavior pattern corresponding to that attention head based on the attention sparsity.
[0008] Optionally, determining the text-centric attention sparsity of the attention head during the pre-filling stage includes: In the pre-filling stage, preset text is input into the attention head, and the preset text is used as a query vector. The attention distribution of the attention head on each key vector in the preset text is calculated. Based on the attention distribution of the attention head on each key vector in the preset text, the attention sparsity of the attention head is determined.
[0009] Optionally, determining the attention sparsity of the attention head based on the attention distribution of each key vector in the preset text includes: For each key vector, based on the attention distribution corresponding to the key vector, N key key vectors corresponding to the key vector are selected, and based on the first attention distribution score of each key key vector corresponding to the key vector, the second attention distribution score corresponding to the key vector is determined; the key key vector is the key vector whose attention distribution score ranks in the top N among all key vectors except the key vector itself, where N>0; The attention sparsity of the attention head is determined based on the second attention distribution score corresponding to each of the key vectors.
[0010] Optionally, determining the target cache budget for each attention head based on the corresponding behavior pattern includes: The average cache budget for each attention head is determined based on the preset total cache budget and the number of all attention heads in the multimodal large language model. Based on the total cache budget, the average cache budget, the sharing coefficient, and the number of first attention heads, the sum of the first cache budgets for all first attention heads and the sum of the second cache budgets for all second attention heads are determined; the behavior pattern corresponding to the first attention head is a static mode, and the behavior pattern corresponding to the second attention head is a dynamic mode. For each first attention head, the target cache budget corresponding to the first attention head is determined based on the sum of the first cache budgets, the number of first attention heads, and the attention sparsity corresponding to the first attention head. For each second attention head, the target cache budget corresponding to that second attention head is determined based on the sum of the second cache budgets and the number of second attention heads.
[0011] Optionally, the step of performing key-value cache compression processing on the attention head according to the target cache budget corresponding to the attention head and the first key-value cache compression strategy corresponding to the static mode includes: For each key vector corresponding to the attention head, determine whether the key vector is the target key vector; the target key vector includes any one or more of the following vectors: all words in the preset observation window, all text words in the historical context, and the top M visual words in the historical context with the highest relevance, where M>0; Based on the target cache budget corresponding to the attention head and each of the target key vectors, the attention head is subjected to key-value cache compression processing.
[0012] Optionally, the step of performing key-value cache compression processing on the attention head based on the target cache budget corresponding to the attention head and the second key-value cache compression strategy corresponding to the dynamic mode includes: The key-value cache sequence of the attention head is divided into several sub-sequences, and the average value of all key vectors in each sub-sequence is determined. Based on the average value of all key vectors in each sub-sequence, metadata corresponding to each sub-sequence is generated. The key-value cache sequence of the attention focus is migrated from the current processor to a preset target processor, and the index corresponding to each sub-sequence is stored in the current processor; wherein, the current processor is a processor for running the multimodal large language model, the target processor is a processor for storing the key-value cache sequence, and the index corresponding to the sub-sequence is composed of the metadata corresponding to the sub-sequence.
[0013] Optionally, after performing key-value cache compression processing on each attention head based on the target cache budget and the behavior pattern corresponding to that attention head, the method further includes: Each time the multimodal large language model is used for decoding, a first target subsequence with the highest relevance to the current query vector is determined, and for each attention head, according to the target cache budget corresponding to the attention head, several second target subsequences, including the first target subsequence, are loaded from the target processor into the current processor. Using the multimodal large language model, attention is calculated based on each target key vector and each second target subsequence to generate a target vector.
[0014] Secondly, embodiments of this application provide a key-value cache compression apparatus, comprising: The first determining unit is used to determine the behavior pattern of each attention head in the decoding stage in the trained multimodal large language model. The second determining unit is used to determine the target cache budget corresponding to each attention head according to the behavior pattern corresponding to each attention head; The key-value cache compression unit is used to perform key-value cache compression processing on each attention head according to the target cache budget and the behavior pattern corresponding to that attention head.
[0015] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the key-value cache compression method as described in any of the first aspects above.
[0016] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the key-value cache compression method as described in any of the first aspects above.
[0017] Fifthly, embodiments of this application provide a computer program product that, when run on a control device, causes the control device to perform the steps of the key-value cache compression method as described in any of the first aspects above.
[0018] The key-value cache compression method and electronic device provided in this application have the following beneficial effects: In the key-value cache compression method provided in this application embodiment, the behavior pattern corresponding to each attention head in the decoding stage of the trained multimodal large language model is first determined. Then, based on the behavior pattern corresponding to each attention head, the target cache budget corresponding to each attention head is determined. Finally, for each attention head, key-value cache compression processing is performed on the attention head according to the target cache budget and behavior pattern. Through the key-value cache compression provided in this application, different key-value cache budget determination methods can be adopted for attention heads with different behavior patterns, and different key-value cache compression processing methods can also be adopted for attention heads with different behavior patterns. This improves the effect of key-value cache compression for multimodal large language models, thereby reducing the memory footprint of multimodal large language models, increasing the decoding speed of multimodal large language models, and ultimately improving the performance of multimodal large language models. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0020] Figure 1 A flowchart illustrating the implementation of the key-value caching compression method provided in this application embodiment; Figure 2 This is a schematic diagram of a key-value cache compression structure provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0021] It should be noted that the terminology used in the embodiments of this application is only for explaining specific embodiments of this application and is not intended to limit this application. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, "at least one" or "one or more" means one, two or more. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.
[0022] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.
[0023] The execution subject of the key-value caching compression method provided in this application embodiment can be an electronic device that has deployed a multimodal large language model. The electronic device may include, but is not limited to, laptops, desktop computers, tablets, and mobile phones.
[0024] The key-value caching compression method provided in this application can be applied to any scenario requiring key-value caching compression of a multimodal large language model. For example, when a user inputs text, images, and / or videos into a multimodal large language model, instructing the model to output based on the user's input, the electronic device can execute the various steps of the key-value caching compression method provided in this application, thereby improving the compression effect of key-value caching for the multimodal large language model.
[0025] Please see Figure 1 , Figure 1 The flowchart below shows the implementation of the key-value cache compression method provided in this application embodiment. The key-value cache compression method provided in this application embodiment may include S101~S103, which are described in detail below: In S101, the behavior pattern of each attention head in the trained multimodal large language model during the decoding phase is determined.
[0026] In the embodiments of this application, the behavior pattern of each attention head in the trained multimodal large language model during the decoding stage may include a static mode and a dynamic mode.
[0027] Among them, the static mode can be used to describe that the attention head is stably focused on a small number of fixed context words during the decoding process, while the dynamic mode can be used to describe that the attention head's attention shifts among multiple context words during the decoding process.
[0028] In one possible implementation, the electronic device can determine the behavior pattern of each attention head in the trained multimodal large language model during the decoding phase by following step a, as detailed below: In step a, for each attention head, the attention sparsity centered on the text in the pre-filling stage is determined, and the behavior pattern corresponding to the attention head is determined based on the attention sparsity.
[0029] In this implementation, the electronic device can first determine the text-centric attention sparsity of the attention head during the pre-filling stage by: In the pre-filling stage, the electronic device can input preset text into the attention head and use the preset text as a query vector to calculate the attention distribution of the attention head on each key vector in the preset text. Based on the attention distribution of the attention head on each key vector in the preset text, the attention sparsity of the attention head is determined.
[0030] The electronic device can calculate the attention distribution of the attention head on each key vector in the preset text according to the following formula:
[0031] in, This is a query vector matrix containing preset text suggestion words. This is the key vector matrix of the entire context corresponding to the preset text. Let be the hidden state dimension of the key vector matrix. As a scaling factor Let be the attention distribution for any key vector.
[0032] Furthermore, the electronic device can achieve "determining the attention sparsity of the attention head based on the attention distribution of the attention head to each key vector in the preset text" through steps b to c, as detailed below: In step b, for each key vector, based on the attention distribution corresponding to the key vector, N key vectors corresponding to the key vector are selected, and based on the first attention distribution score of each key vector corresponding to the key vector, the second attention distribution score corresponding to the key vector is determined; the key vector is the key vector whose attention distribution score ranks in the top N among all key vectors except the key vector itself, where N>0;
[0033] The purpose of "selecting N key vectors corresponding to the key vector based on the attention distribution corresponding to the key vector" is to suppress interference. In practical applications, for each key vector, the attention distribution corresponding to the key vector is usually long-tailed. That is, in the attention distribution corresponding to the key vector, usually only a few key vectors get high scores, while a large number of other key vectors get low scores. Therefore, electronic devices can first select N key vectors corresponding to the key vector based on the attention distribution corresponding to the key vector, so as to focus only on the few key vectors that get high scores and eliminate the interference caused by other key vectors that get low scores.
[0034] The following is an example for understanding step b. For example, it can include key vectors 1 to 10 (in actual applications, the number of key vectors is much greater than 10, but this example reduces the number of key vectors for ease of description), and N equals 3. For key vector 1, three key key vectors (e.g., key vector 2, key vector 3, and key vector 4) can be selected based on the attention distribution corresponding to key vector 1. Then, the second attention distribution score corresponding to key vector can be determined based on the first attention distribution score of the three key key vectors corresponding to key vector 1. (For example, if the first attention distribution score of key vector 2 is point1, the first attention distribution score of key vector 3 is point2, and the first attention distribution score of key vector 4 is point3, then the second attention distribution score corresponding to key vector 1 can be determined as point1, point2, and point3.)
[0035] It should be noted that electronic devices can determine the second attention distribution scores corresponding to each of the other key vectors (key vectors 2 to 9) by using the method described above for determining the second attention distribution score corresponding to key vector 1.
[0036] In step c, the attention sparsity of the attention head is determined based on the second attention distribution score corresponding to each key vector.
[0037] In this implementation, the electronic device can determine the attention sparsity of the attention head by averaging the sum of the scores of the second attention distribution corresponding to each key vector.
[0038] For example, if the second attention distribution scores corresponding to key vector 1 are point1, point2, and point3, then the sum of the second attention distribution scores corresponding to key vector 1 is (point1 + point2 + point3). In the above way, the sum of the second attention distribution scores corresponding to each of the other key vectors (key vectors 2 to 9) can be calculated respectively. Then, the average value of the sum of the second attention distribution scores corresponding to each key vector (key vectors 1 to 10) can be used to determine the attention sparsity corresponding to the attention head.
[0039] After determining the attention sparsity corresponding to a particular attention head, the electronic device can "determine the behavioral pattern corresponding to the attention head based on the attention sparsity" in the following manner, detailed below: The electronic device can compare the attention sparsity corresponding to the attention head with a preset attention sparsity threshold. If the attention sparsity corresponding to the attention head is greater than or equal to the attention sparsity threshold, it can be considered that the attention sparsity corresponding to the attention head is high sparsity, thereby determining that the behavior pattern corresponding to the attention head is a static mode. If the attention sparsity corresponding to the attention head is less than the attention sparsity threshold, it can be considered that the attention sparsity corresponding to the attention head is low sparsity, thereby determining that the behavior pattern corresponding to the attention head is a dynamic mode.
[0040] In this implementation, the attention sparsity of the attention head specifically refers to the text-centric attention sparsity during the pre-filling stage. If the attention sparsity of the attention head is high, then the attention of the attention head is highly concentrated, meaning that when processing text, the attention head only pays high attention to a very small number of key vectors in the context, while paying low attention to most other key vectors. Therefore, it can be determined that the corresponding behavior mode of the attention head in the decoding stage is a static model. If the attention sparsity of the attention head is low, then the attention of the attention head is scattered, meaning that when processing text, the attention head pays attention to various key vectors in the context extensively, without paying particularly high attention to any key vector. Therefore, it can be determined that the corresponding behavior mode of the attention head in the decoding stage is a dynamic model.
[0041] In practical applications, after multiple experiments, it was found that the method of determining the behavior pattern of the attention head in the decoding stage by determining the text-centered attention sparsity of the attention head in the pre-filling stage has a high accuracy.
[0042] In S102, the target cache budget for each attention head is determined according to the behavior pattern corresponding to each attention head.
[0043] In this embodiment of the application, after determining the behavior pattern of each attention head in the decoding stage in the trained multimodal large language model, the electronic device can determine the target cache budget corresponding to each attention head in steps d to g, as detailed below: In step d, the average cache budget for each attention head is determined based on the preset total cache budget and the number of all attention heads in the multimodal large language model.
[0044] In this implementation, the electronic device can determine the preset total cache budget based on the performance parameters of the electronic device (such as the parameters of the image processor).
[0045] Specifically, electronic devices can determine the average cache budget for each attention head by the quotient of the total cache budget and the number of all attention heads in the multimodal large language model.
[0046] In step e, the sum of the first cache budgets for all first attention heads is determined based on the total cache budget, the average cache budget, the sharing coefficient, and the number of first attention heads, and the sum of the second cache budgets for all second attention heads is determined.
[0047] The behavior pattern corresponding to the first attention head is the static mode, and the behavior pattern corresponding to the second attention head is the dynamic mode.
[0048] Specifically, the electronic device can determine the sum of the first buffer budgets for all first attention heads and the sum of the second buffer budgets for all second attention heads using the following formulas:
[0049] in, This is represented as the sum of the second cache budget. Let r represent the total budget of the first cache, and r represent the sharing coefficient. This is expressed as the average cache budget. This represents the number of heads that receive the first attention. This represents the total cache budget.
[0050] It should be noted that the sharing coefficient r can be used to adjust the trade-off between efficiency and accuracy in multimodal large language models.
[0051] Because the first attention head only pays high attention to a very small number of key vectors in the context when processing text, while the second attention head pays attention to a wide range of key vectors in the context, the higher the sum of the first cache budgets of all first attention heads (i.e., the lower the sum of the second cache budgets of all second attention heads), the higher the efficiency and the lower the accuracy of the multimodal large language model; conversely, the lower the sum of the first cache budgets of all first attention heads (i.e., the higher the sum of the second cache budgets of all second attention heads), the lower the efficiency and the higher the accuracy of the multimodal large language model.
[0052] Based on this, when it is necessary to improve the accuracy of a multimodal large language model, the value of r can be increased to increase the sum of the second cache budgets of all second attention heads, thereby enabling each second attention head to retrieve key information and ultimately improving the accuracy of the multimodal large language model. When it is necessary to improve the efficiency of a multimodal large language model, the value of r can be decreased to increase the sum of the first cache budgets of all first attention heads, thereby improving the efficiency of the multimodal large language model.
[0053] In step f, for each first attention head, the target cache budget corresponding to the first attention head is determined based on the sum of the first cache budgets, the number of first attention heads, and the attention sparsity corresponding to the first attention head.
[0054] In this implementation, after determining the sum of the first cache budgets for all first attention heads, the electronic device can also determine the target cache budget for each first attention head based on the sum of the first cache budgets, the number of first attention heads, and the attention sparsity corresponding to that first attention head, thereby obtaining the target cache budget corresponding to each first attention head.
[0055] For example, an electronic device can achieve "determining the target cache budget corresponding to the first attention head based on the sum of the first cache budgets, the number of first attention heads, and the attention sparsity corresponding to the first attention head" using the following formula:
[0056] in, This is represented as the target cache budget corresponding to the first attention head. Represented as the balance factor, Represented as the sum of the first cache budget, It is expressed as the negative first power of the number of primary attention heads. This is represented as the attention sparsity corresponding to the first attention head.
[0057] As can be seen from the above formula, the target cache budget corresponding to the first attention head can be composed of two parts. The first part is the base budget that is evenly distributed. It can be understood that the base budget corresponding to each first attention head is the same. The second part is the additional budget that is proportional to the attention sparsity corresponding to the first attention head. It can be understood that the higher the attention sparsity corresponding to the first attention head, the more additional budget corresponding to the first attention head, and the more target cache budget corresponding to the first attention head.
[0058] Among them, the balance factor It can be used to balance the allocation ratio between the basic budget and the supplementary budget. In practical applications, the balancing factor... The specific value can be set according to actual needs, and is not limited here.
[0059] By determining the target cache budget corresponding to the first attention head in the above way, we can ensure that the first attention head, which is more important in the current task (i.e., has higher attention sparsity), can obtain more cache budget. This allows us to retain key information to the maximum extent while compressing, thereby improving the accuracy of the multimodal large language model on different multimodal tasks.
[0060] In step g, for each second attention head, the target cache budget corresponding to that second attention head is determined based on the sum of the second cache budgets and the number of second attention heads.
[0061] In this implementation, after determining the sum of the second cache budgets for all second attention heads, the electronic device can also determine the target cache budget for each second attention head based on the sum of the second cache budgets and the number of second attention heads, thereby obtaining the target cache budget corresponding to each second attention head.
[0062] For example, the electronic device can determine the target cache budget corresponding to each second attention head by dividing the sum of the second cache budgets by the number of second attention heads. That is, the electronic device can distribute the sum of the second cache budgets equally among each second attention head to obtain the target cache budget corresponding to each second attention head.
[0063] As can be seen, the above method for determining the target cache budget corresponding to the second attention head is relatively simple. This simple cache budget allocation strategy can improve the efficiency of multimodal large language models in the decoding process.
[0064] In S103, for each attention head, key-value cache compression is performed on the attention head according to the target cache budget and behavior pattern corresponding to that attention head.
[0065] In this implementation, since the behavior modes corresponding to attention heads include static and dynamic modes, the electronic device can perform key-value cache compression processing on each attention head if the behavior mode corresponding to the attention head is static, based on the target cache budget corresponding to the attention head and the first key-value cache compression strategy corresponding to the static mode; if the behavior mode corresponding to the attention head is dynamic, then the attention head can perform key-value cache compression processing on the target cache budget corresponding to the attention head and the second key-value cache compression strategy corresponding to the dynamic mode.
[0066] In one possible implementation, if the behavior mode corresponding to the attention head is a static mode, the electronic device can perform key-value caching and compression processing on the attention head through steps h to i, as detailed below: In step h, for each key vector corresponding to the attention head, it is determined whether the key vector is the target key vector.
[0067] The target key vector includes any one or more of the following vectors: all lexical units within the preset observation window, all textual lexical units in the historical context, and the top M visual lexical units in the historical context with the highest relevance, where M > 0.
[0068] The purpose of determining all lexical units within the preset observation window as target key vectors is to preserve the most recent context before decoding, thereby ensuring the integrity of recent messages obtained by the multimodal large language model.
[0069] The purpose of identifying all text lexical units in the historical context as the target key vector is that all text lexical units in the historical context include all text instruction lexical units before the observation window, which are the core vectors guiding the inference process of the multimodal large language model.
[0070] The purpose of identifying the top M visual terms in the historical context as the target key vector is to preserve visual information to the greatest extent possible while performing key-value caching and compression, and to avoid excessive loss of visual information.
[0071] In practical applications, the value of M can be determined based on the target cache budget allocated to the attention head. Specifically, the cache budget corresponding to the visual word can be determined first based on the target cache budget allocated to the attention head, the cache budget corresponding to all words in the preset observation window, and the cache budget corresponding to all text words in the historical context. Then, the value of M can be determined based on the cache budget corresponding to the visual word.
[0072] In step i, the attention head is subjected to key-value cache compression processing based on the target cache budget and each target key vector.
[0073] In this implementation, after determining the target cache budget and each target key vector corresponding to the attention head, the attention head can be subjected to key-value cache compression processing through a preset key-value cache compression processing method.
[0074] In practical applications, the specific key-value caching compression method can be set according to actual needs, and no limitation is made here.
[0075] In one possible implementation, if the behavior pattern corresponding to the attention head is a dynamic pattern, the electronic device can perform key-value caching and compression processing on the attention head through steps j to i, as detailed below: In step j, the key-value cache sequence of the attention head is divided into several subsequences, and the average value of all key vectors in each subsequence is determined. Based on the average value of all key vectors in each subsequence, the metadata corresponding to each subsequence is generated.
[0076] In this implementation, the number of tokens included in each subsequence can be set according to actual needs. For example, each subsequence can include 8 tokens. After dividing the sequence into several subsequences, the electronic device can calculate the average value of all key vectors in each subsequence using a preset method. Then, the electronic device can generate a compact metadata for the subsequence based on the average value of all key vectors in the subsequence, thereby obtaining the metadata corresponding to each subsequence.
[0077] In step k, the key-value cache sequence of the attention head is migrated from the current processor to the preset target processor, and the index corresponding to each subsequence is stored in the current processor.
[0078] The current processor is used to run a multimodal large language model, the target processor is used to store key-value cache sequences, and the index corresponding to the subsequence is composed of the metadata corresponding to the subsequence.
[0079] In this embodiment of the application, after obtaining the metadata corresponding to each subsequence, the electronic device can migrate the key-value cache sequence of the attention head originally stored in the current processor to the preset target processor, and construct the index corresponding to each subsequence of the key-value cache sequence according to the metadata corresponding to each subsequence, and store the index corresponding to each subsequence in the current processor.
[0080] In practical applications, the current processor can be the image processor of an electronic device, and the target processor can be the central processing unit of the electronic device.
[0081] By using steps j to k, the key-value cache sequence of the attention head can be stored in the current processor, and only the index corresponding to each subsequence is retained in the current processor. Therefore, the purpose of key-value cache compression of the attention head can be achieved.
[0082] If the behavior pattern corresponding to the attention head is dynamic, the reason for performing key-value caching compression on the attention head through steps j to i is as follows: Since the attention of the attention head corresponding to the dynamic behavior pattern will change continuously during the decoding process, if steps h to i are used, it will cause serious information loss. However, through steps j to k, since the key-value cache sequence of the attention head is still stored in the target processor, and the key-value cache sequence of the attention head can be obtained through the index stored in the current processor, the information loss can be reduced, thereby improving the performance of the multimodal large language model.
[0083] In one possible implementation, after the electronic device performs key-value caching and compression on the attention head, it can further execute steps 1 to m to enable the multimodal large language model to generate the target vector. Details are as follows: In step 1, each time decoding is performed using a multimodal large language model, the first target subsequence with the highest relevance to the current query vector is determined, and for each attention head, several second target subsequences, including the first target subsequence, are loaded from the target processor into the current processor according to the target cache budget corresponding to that attention head.
[0084] In this implementation, each time the electronic device decodes using a multimodal large language model, it can perform an inner product calculation between the current query vector and the index (i.e., the metadata corresponding to each subsequence) stored by the current processor, thereby determining the first target subsequence that is most relevant to the current query vector.
[0085] After determining the first target subsequence that is most relevant to the current query vector, the electronic device can, for each attention head, load several second target subsequences, including the first target subsequence, from the target processor into the current processor according to the target cache budget corresponding to that attention head. For example, each second target subsequence can be loaded into the unified cache area of the current processor.
[0086] It should be noted that the several second target subsequences corresponding to each attention head may not be exactly the same.
[0087] In step m, attention is calculated using a multimodal large language model based on each target key vector and each second target subsequence to generate a target vector.
[0088] In this implementation, after the current processor obtains the target key vector corresponding to each first attention head (i.e., the attention head with a static behavior model) and the second target subsequence corresponding to each second attention head (i.e., the attention head with a dynamic behavior model), the electronic device can use a multimodal large language model to perform attention calculations based on each target key vector and each second target subsequence in a preset vector generation method to generate target vectors.
[0089] In practical applications, the vector generation method can be set according to actual needs, and no restrictions are imposed here. Furthermore, the target vector can be the next vector that the multimodal large language model needs to generate.
[0090] As can be seen from the above, in the key-value cache compression method provided in this application embodiment, the behavior pattern of each attention head in the trained multimodal large language model during the decoding stage is first determined. Then, based on the behavior pattern of each attention head, the target cache budget corresponding to each attention head is determined. Finally, for each attention head, key-value cache compression processing is performed on the attention head according to the target cache budget and behavior pattern. Through the key-value cache compression provided in this application, different key-value cache budget determination methods can be adopted for attention heads with different behavior patterns, and different key-value cache compression processing methods can also be adopted for attention heads with different behavior patterns. This improves the effect of key-value cache compression for multimodal large language models, thereby reducing the memory footprint of multimodal large language models, increasing the decoding speed of multimodal large language models, and ultimately improving the performance of multimodal large language models.
[0091] Based on the key-value cache compression method provided in the above embodiments, this application further provides a key-value cache compression apparatus for implementing the above method embodiments. Please refer to [link to relevant documentation]. Figure 2 , Figure 2 This is a schematic diagram of a key-value cache compression device provided in an embodiment of this application. Figure 2 As shown, the key-value cache compression device 20 may include: a first determining unit 21, a second determining unit 22, and a key-value cache compression unit 23. Wherein: The first determining unit 21 is used to determine the behavior pattern of each attention head in the decoding stage in the trained multimodal large language model. The second determining unit 22 is used to determine the target cache budget corresponding to each attention head according to the behavior pattern corresponding to each attention head; The key-value cache compression unit 23 is used to perform key-value cache compression processing on each attention head according to the target cache budget and behavior pattern corresponding to that attention head.
[0092] Optionally, the key-value cache compression unit 23 is specifically used for: For each attention head, if the behavior pattern corresponding to the attention head is a static pattern, then the attention head is subjected to key-value cache compression processing according to the target cache budget corresponding to the attention head and the first key-value cache compression strategy corresponding to the static pattern. If the behavior pattern corresponding to the attention head is a dynamic pattern, then the attention head is subjected to key-value cache compression processing according to the target cache budget corresponding to the attention head and the second key-value cache compression strategy corresponding to the dynamic pattern.
[0093] Optionally, the first determining unit 21 is specifically used for: For each attention head, determine the text-centric attention sparsity of that attention head in the pre-filling stage, and determine the corresponding behavior pattern of that attention head based on the attention sparsity.
[0094] Optionally, the first determining unit 21 is specifically used for: In the pre-filling stage, the preset text is input into the attention head and used as the query vector. The attention distribution of the attention head on each key vector in the preset text is calculated. Based on the attention distribution of the attention head on each key vector in the preset text, the attention sparsity of the attention head is determined.
[0095] Optionally, the first determining unit 21 is specifically used for: For each key vector, based on the attention distribution corresponding to the key vector, N key vectors corresponding to the key vector are selected, and based on the first attention distribution score of each key vector corresponding to the key vector, the second attention distribution score corresponding to the key vector is determined; the key vector is the key vector whose attention distribution score ranks in the top N among all key vectors except the key vector itself, where N>0; The attention sparsity of the attention head is determined based on the second attention distribution score corresponding to each key vector.
[0096] Optionally, the second determining unit 22 is specifically used for: Based on the preset total cache budget and the number of all attention heads in the multimodal large language model, determine the average cache budget for each attention head; Based on the total cache budget, average cache budget, sharing coefficient, and the number of first attention heads, determine the sum of the first cache budgets for all first attention heads, and determine the sum of the second cache budgets for all second attention heads; the behavior pattern corresponding to the first attention head is the static mode, and the behavior pattern corresponding to the second attention head is the dynamic mode; For each first attention head, the target cache budget corresponding to that first attention head is determined based on the total first cache budget, the number of first attention heads, and the attention sparsity corresponding to that first attention head. For each second attention head, the target cache budget corresponding to that second attention head is determined based on the total second cache budget and the number of second attention heads.
[0097] Optionally, the key-value cache compression unit 23 is specifically used for: For each key vector corresponding to the attention head, determine whether the key vector is a target key vector; the target key vector includes any one or more of the following vectors: all words in the preset observation window, all text words in the historical context, and the top M visual words in the historical context with the highest relevance, where M>0; based on the target cache budget corresponding to the attention head and each target key vector, perform key-value cache compression processing on the attention head.
[0098] Optionally, the key-value cache compression unit 23 is specifically used for: The key-value cache sequence of the attention head is divided into several subsequences, and the average value of all key vectors in each subsequence is determined. Based on the average value of all key vectors in each subsequence, metadata corresponding to each subsequence is generated. The key-value cache sequence of the attention head is migrated from the current processor to a preset target processor, and the index corresponding to each subsequence is stored in the current processor. The current processor is the processor used to run the multimodal large language model, and the target processor is the processor used to store the key-value cache sequence.
[0099] Optionally, the key-value cache compression 20 may also include a vector generation unit, wherein: The vector generation unit is specifically used for: Each time the multimodal large language model is used for decoding, the first target subsequence with the highest relevance to the current query vector is determined. For each attention head, according to the target cache budget corresponding to the attention head, several second target subsequences, including the first target subsequence, are loaded from the target processor into the current processor. Using a multimodal large language model, attention is calculated based on each target key vector and each second target subsequence to generate the target vector.
[0100] It should be noted that the information interaction and execution process between the above-mentioned units are based on the same concept as the method embodiments of this application. Their specific functions and technical effects can be referred to the method embodiments section, and will not be repeated here.
[0101] Please see Figure 3 , Figure 3This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 3 As shown, the electronic device 3 provided in this embodiment may include: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a program for a key-value cache compression method. When the processor 30 executes the computer program 32, it implements the steps described above in the embodiment of the key-value cache compression method, for example... Figure 1 S101~S103 are shown. Alternatively, when processor 30 executes computer program 32, it implements the functions of each module / unit in the above-described key-value cache compression device embodiment, for example... Figure 2 The functions of units 21-23 shown.
[0102] For example, computer program 32 can be divided into one or more modules / units, one or more of which are stored in memory 31 and executed by processor 30 to complete this application. One or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of computer program 32 in electronic device 3. For example, computer program 32 can be divided into a first determining unit 21, a second determining unit 22, and a key-value cache compression unit 23. For the specific functions of each unit, please refer to... Figure 2 The relevant descriptions in the corresponding embodiments are not repeated here.
[0103] Those skilled in the art will understand that Figure 3 This is merely an example of electronic device 3 and does not constitute a limitation on electronic device 3. It may include more or fewer components than shown, or combine certain components, or use different components.
[0104] The processor 30 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.
[0105] The memory 31 can be an internal storage unit of the electronic device 3, such as a hard disk or RAM. The memory 31 can also be an external storage device of the electronic device 3, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, or flash card. Furthermore, the memory 31 can include both internal and external storage units of the electronic device 3. The memory 31 is used to store computer programs and other programs and data required by the electronic device. The memory 31 can also be used to temporarily store data that has been output or will be output.
[0106] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units is merely an example. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure of the key-value cache compression device can be divided into different functional units to complete all or part of the functions described above. The functional units in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0107] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the various method embodiments described above.
[0108] This application provides a computer program product that, when run on a terminal device, enables the terminal device to implement the steps described in the various method embodiments above.
[0109] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, refer to the relevant descriptions of other embodiments.
[0110] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0111] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A key-value caching compression method, characterized in that, include: Determine the behavior pattern of each attention head in the decoding phase of the trained multimodal large language model; Based on the behavior pattern corresponding to each attention head, determine the target cache budget corresponding to each attention head; For each attention head, key-value cache compression is performed on the attention head based on the target cache budget and the behavior pattern corresponding to that attention head.
2. The method according to claim 1, characterized in that, The behavior patterns include static and dynamic modes; the step of performing key-value cache compression processing on each attention head based on the target cache budget and the behavior pattern includes: For each attention head, if the behavior mode corresponding to the attention head is a static mode, then the attention head is subjected to key-value cache compression processing according to the target cache budget corresponding to the attention head and the first key-value cache compression strategy corresponding to the static mode; if the behavior mode corresponding to the attention head is a dynamic mode, then the attention head is subjected to key-value cache compression processing according to the target cache budget corresponding to the attention head and the second key-value cache compression strategy corresponding to the dynamic mode.
3. The method according to claim 2, characterized in that, Determining the behavior pattern of each attention head in the decoding stage of the trained multimodal large language model includes: For each attention head, determine the text-centric attention sparsity of that attention head in the pre-filling stage, and determine the behavior pattern corresponding to that attention head based on the attention sparsity.
4. The method according to claim 3, characterized in that, Determining the text-centric attention sparsity of the attention head during the pre-filling phase includes: In the pre-filling stage, preset text is input into the attention head, and the preset text is used as a query vector. The attention distribution of the attention head on each key vector in the preset text is calculated. Based on the attention distribution of the attention head on each key vector in the preset text, the attention sparsity of the attention head is determined.
5. The method according to claim 4, characterized in that, Determining the attention sparsity of the attention head based on the attention distribution of the attention head to each key vector in the preset text includes: For each key vector, based on the attention distribution corresponding to the key vector, N key key vectors corresponding to the key vector are selected, and based on the first attention distribution score of each key key vector corresponding to the key vector, the second attention distribution score corresponding to the key vector is determined; the key key vector is the key vector whose attention distribution score ranks in the top N among all key vectors except the key vector itself, where N>0; The attention sparsity of the attention head is determined based on the second attention distribution score corresponding to each of the key vectors.
6. The method according to claim 3, characterized in that, The step of determining the target cache budget for each attention head based on the behavior pattern corresponding to each attention head includes: The average cache budget for each attention head is determined based on the preset total cache budget and the number of all attention heads in the multimodal large language model. Based on the total cache budget, the average cache budget, the sharing coefficient, and the number of first attention heads, the sum of the first cache budgets for all first attention heads and the sum of the second cache budgets for all second attention heads are determined; the behavior pattern corresponding to the first attention head is a static mode, and the behavior pattern corresponding to the second attention head is a dynamic mode. For each first attention head, the target cache budget corresponding to the first attention head is determined based on the sum of the first cache budgets, the number of first attention heads, and the attention sparsity corresponding to the first attention head. For each second attention head, the target cache budget corresponding to that second attention head is determined based on the sum of the second cache budgets and the number of second attention heads.
7. The method according to any one of claims 2 to 6, characterized in that, The step of performing key-value cache compression processing on the attention head according to the target cache budget corresponding to the attention head and the first key-value cache compression strategy corresponding to the static mode includes: For each key vector corresponding to the attention head, determine whether the key vector is the target key vector; the target key vector includes any one or more of the following vectors: all words in the preset observation window, all text words in the historical context, and the top M visual words in the historical context with the highest relevance, where M>0; Based on the target cache budget corresponding to the attention head and each of the target key vectors, the attention head is subjected to key-value cache compression processing.
8. The method according to claim 7, characterized in that, The step of performing key-value cache compression processing on the attention head according to the target cache budget corresponding to the attention head and the second key-value cache compression strategy corresponding to the dynamic mode includes: The key-value cache sequence of the attention head is divided into several sub-sequences, and the average value of all key vectors in each sub-sequence is determined. Based on the average value of all key vectors in each sub-sequence, metadata corresponding to each sub-sequence is generated. The key-value cache sequence of the attention focus is migrated from the current processor to a preset target processor, and the index corresponding to each sub-sequence is stored in the current processor; wherein, the current processor is a processor for running the multimodal large language model, the target processor is a processor for storing the key-value cache sequence, and the index corresponding to the sub-sequence is composed of the metadata corresponding to the sub-sequence.
9. The method according to claim 8, characterized in that, After performing key-value cache compression processing on each attention head based on the target cache budget and the behavior pattern corresponding to that attention head, the method further includes: Each time the multimodal large language model is used for decoding, a first target subsequence with the highest relevance to the current query vector is determined, and for each attention head, according to the target cache budget corresponding to the attention head, several second target subsequences, including the first target subsequence, are loaded from the target processor into the current processor. Using the multimodal large language model, attention is calculated based on each target key vector and each second target subsequence to generate a target vector.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements each step of the key-value cache compression method as described in any one of claims 1 to 9.