Self-adaptive prefix key value cache compression method and device and electronic equipment

Abstract

The invention provides a self-adaptive prefix key value cache compression method and device and electronic equipment, relates to the technical field of computers, and aims to solve the defects of an existing key value cache compression method in the aspect of ensuring high reasoning performance and high task precision. The method comprises the steps that mark sequences corresponding to image information and text information are input into a large model for multi-layer attention interaction, and the attention score of each mark in each layer is obtained; according to the attention score of each mark, determining the importance proportion of each key value cache vector of each layer; taking the importance proportion of the key value cache vectors as a priority numerical value, and performing descending sort on the key value cache vectors in each layer to obtain a priority sequence of the key value cache vectors of each layer; determining a target information retention ratio based on the target compression ratio budget; and according to the target information retention ratio, determining a retained key value cache vector from the priority sequence of the key value cache vector of each layer.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an adaptive prefix key value cache compression method, device and electronic device. Background Art

[0002] Large-scale visual-language models (LVLMs) demonstrate powerful information processing and generation capabilities by integrating visual modalities based on large-scale language models (LLMs). However, they have high inference costs due to their large number of parameters and complex calculations, which hinders the deployment of large-scale visual-language models in practical applications.

[0003] Existing methods to improve model processing efficiency (e.g., distillation, pruning, etc.) usually focus on compression at the model level to reduce the inference overhead caused by parameters. At the same time, key-value cache at the data level also brings significant memory usage and GPU communication overhead during the inference process, but existing key-value cache compression methods ignore the different importance distributions across layers and retain the same number of key-value cache vectors at each layer of the model, which leads to significant information loss and seriously affects the model's generation ability during the decoding process. Therefore, it is impossible to take into account both high inference performance and high task accuracy, and it is difficult to achieve high-precision and efficient key-value cache compression for large models, and it is difficult to carry out actual landing applications. Summary of the invention

[0004] In view of the above problems, embodiments of the present application provide an adaptive prefix key value cache compression method, device and electronic device to overcome the above problems or at least partially solve the above problems.

[0005] In a first aspect of an embodiment of the present application, an adaptive prefix key value cache compression method is disclosed, the method comprising: The tag sequences corresponding to the image information and text information are input into the large model for multi-layer attention interaction to obtain the attention scores of each tag in each layer; Determine the importance ratio of each key-value cache vector of each layer according to the attention score of each tag, wherein the key-value cache vector represents the key vector and the value vector corresponding to the tag; The importance ratio of the key-value cache vector is used as a priority value, and the key-value cache vectors are arranged in descending order at each layer to obtain a priority sequence of the key-value cache vectors at each layer; Determining a target information retention ratio based on a target compression ratio budget, wherein the target information retention ratio represents a cumulative priority threshold; According to the target information retention ratio, the retained key-value cache vectors are determined from the priority sequence of the key-value cache vectors of each layer.

[0006] Optionally, determining the reserved key-value cache vector from the priority sequence of the key-value cache vectors of each layer according to the target information retention ratio includes: For each layer, according to the target information retention ratio, determine the minimum prefix cache value that meets the first constraint condition as the retention ratio of the key value cache vector; wherein the first constraint condition represents that the cumulative priority value retained based on the minimum prefix cache value is greater than or equal to the cumulative priority threshold corresponding to the target information retention ratio; According to the retention ratio of the key-value cache vectors, the retained key-value cache vectors are determined from the priority sequence of the key-value cache vectors.

[0007] Optionally, the retention ratio of the key-value cache vectors of each layer satisfies a second constraint condition; the second constraint condition represents that the sum of the key-value cache vectors retained based on the retention ratio of the key-value cache vectors of each layer is equal to the key-value cache vector threshold determined based on the target compression ratio budget.

[0008] Optionally, based on the target compression ratio budget, determining a target information retention ratio includes: Determine a third constraint condition according to the target compression ratio budget; wherein the third constraint condition represents the sum of the retention ratios of all layers of key value cache vectors determined based on the information retention ratio, which is equal to the sum of the retention ratios of all layers of key value cache vectors determined by the target compression ratio preset; Based on the binary search method, an information retention ratio is determined, and when the information retention ratio satisfies the third constraint condition, the information retention ratio is used as the target information retention ratio.

[0009] Optionally, based on a binary search method, determining an information retention ratio, and when the information retention ratio satisfies the third constraint condition, using the information retention ratio as a target information retention ratio, includes: Determine the information retention ratio based on the upper limit and lower limit of the search area; When the information retention ratio does not satisfy the third constraint condition, updating the search area according to the information retention ratio; Determining a new information retention ratio according to the updated upper limit and lower limit of the search area; When the information retention ratio satisfies the third constraint condition, the current information retention ratio is used as the target information retention ratio.

[0010] Optionally, each layer of attention interaction performs multi-head attention interaction based on the token sequence; the attention score of each token is determined as follows: Obtain a total attention score received by the tag in each attention head, wherein the total attention score represents the sum of the attention scores of all tags in the tag sequence received by the tag in the attention head; According to the total attention scores received in each attention head, the average of the total attention scores of all attention heads is obtained as the labeled attention score.

[0011] Optionally, determining the importance ratio of each key-value cache vector of each layer according to the attention score of each tag includes: According to the attention scores of all tags in each layer, the attention score of each tag is normalized to obtain the normalized result of the attention score of each tag; The attention score of each tag is normalized as the importance ratio of the corresponding key-value cache vector.

[0012] Optionally, the method further comprises: Decoding is performed according to the key-value cache vectors retained in each layer to obtain generation results corresponding to the image information and the text information.

[0013] A second aspect of an embodiment of the present application discloses an adaptive prefix key value cache compression device, the device comprising: The first input module is used to input the tag sequences corresponding to the image information and text information into the large model for multi-layer attention interaction to obtain the attention score of each tag in each layer; A first determination module is used to determine the importance ratio of each key-value cache vector of each layer according to the attention score of each tag, wherein the key-value cache vector represents the key vector and the value vector corresponding to the tag; A first sorting module is used to use the importance ratio of the key-value cache vector as a priority value, and arrange the key-value cache vectors in descending order at each layer to obtain a priority sequence of the key-value cache vectors at each layer; A second determination module is used to determine a target information retention ratio based on a target compression ratio budget, wherein the target information retention ratio represents a cumulative priority threshold; The third determination module is used to determine the reserved key-value cache vectors from the priority sequence of the key-value cache vectors of each layer according to the target information retention ratio.

[0014] A third aspect of an embodiment of the present application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the adaptive prefix key value cache compression method described in the first aspect of the embodiment of the present application when executing the computer program.

[0015] The fourth aspect of the embodiments of the present application discloses a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the adaptive prefix key value cache compression method described in the first aspect of the embodiments of the present application are implemented.

[0016] The fifth aspect of the embodiments of the present application discloses a computer program product, including a computer program, which, when executed by a processor, implements the steps of the adaptive prefix key value cache compression method described in the first aspect of the embodiments of the present application.

[0017] The embodiments of the present application include the following advantages: In the embodiment of the present application, the importance ratio of the key-value cache vector is determined based on the attention score of each tag, and the importance ratio of the key-value cache vector is used as the priority value to obtain the priority sequence of the key-value cache vectors of each layer, so that the key-value cache vectors with high importance can be retained at each layer based on the priority sequence; and, compared with the related art, the size of the same key-value cache vector is not directly included in each layer. This method takes into account the different importance distributions of key-value cache vectors of different layers, and determines the retained key-value cache vectors from the priority sequence of the key-value cache vectors of each layer based on the target information retention ratio, so that the key-value cache can be adaptively compressed at each layer to retain some key-value cache vectors with high importance ratios that meet the target information retention ratio. In this way, a key-value cache vector compression method that takes into account both high reasoning performance and high task accuracy is realized, which can maximize the reasoning speed of large models while maintaining the processing power of the model well. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 It is a flowchart of the steps of an adaptive prefix key value cache compression method provided by an embodiment of the present application; Figure 2 It is an overall architecture diagram of an adaptive prefix key value cache compression method provided in an embodiment of the present application; Figure 3 It is a structural diagram of an adaptive prefix key value cache compression device provided in an embodiment of the present application; Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] In order to solve the memory usage and GPU communication overhead caused by the key-value cache at the data level during the model inference process, one method of compressing the key-value cache is to use the distance to the output token (tag / token) as the importance indicator, retain the starting key-value cache vector and the key-value cache vector adjacent to the output token. Another method of compressing the key-value cache is to use the attention score as the importance measure of the key-value cache vector and retain important key-value cache vectors during the generation process. Another method of compressing the key-value cache is to divide the key-value cache into multiple buckets according to the location of important key-value caches, and merge the vectors in the same bucket into one.

[0022] Although the above-mentioned compressed key-value cache methods have shown good results in improving the model reasoning speed, these methods have ignored the different importance distributions across layers and simply retain the same key-value cache size at each layer, which leads to significant information loss, thus seriously affecting the generation ability of large models during the decoding process. Therefore, it is impossible to take into account both high reasoning performance and high task accuracy, and it is difficult to achieve high-precision and efficient large model key-value cache compression, and it is difficult to carry out actual landing applications.

[0023] In order to overcome the limitations of the related technology, an embodiment of the present application provides an adaptive prefix key-value cache compression method, which determines the importance ratio of the key-value cache vector based on the attention score of each tag, and uses the importance ratio of the key-value cache vector as the priority value to obtain the priority sequence of the key-value cache vectors of each layer, so that the key-value cache vectors with high importance can be retained at each layer based on the priority sequence; and, taking into account the different importance distributions of key-value cache vectors in different layers, the method determines the retained key-value cache vectors from the priority sequence of the key-value cache vectors of each layer based on the target information retention ratio, so that the key-value cache can be adaptively compressed at each layer to retain some key-value cache vectors with high importance ratios that meet the target information retention ratio.

[0024] The following is a description of the adaptive prefix key value cache compression method in an embodiment of the present application in conjunction with the accompanying drawings.

[0025] Reference Figure 1 As shown, Figure 11 is a flowchart of the steps of an adaptive prefix key value cache compression method provided by an embodiment of the present application. Figure 1 As shown, an adaptive prefix key value cache compression method provided by an embodiment of the present application may include steps S110 to S150: Step S110: Input the tag sequence corresponding to the image information and text information into the large model for multi-layer attention interaction to obtain the attention score of each tag in each layer.

[0026] In the embodiment of the present application, image information refers to visual information used for large model processing, and the image information can be one or more images, or a video; text information refers to the descriptive information of the question, for example, the text information can be a question description based on the image information; the large model refers to a large visual language model, and the large model includes multiple attention interaction layers (i.e., multiple transformer layers).

[0027] The image information and text information are input into the big model. The big model processes the image information and text information into a tag sequence composed of multiple tags through a tokenization mechanism. The tag sequence is sequentially input into multiple attention interaction layers of the big model for attention interaction processing to obtain the attention score of each tag in each layer, where the attention score of the tag represents the attention value received by the tag in the attention interaction.

[0028] Step S120: Determine the importance ratio of each key-value cache vector of each layer according to the attention score of each tag, wherein the key-value cache vector represents the key vector and value vector corresponding to the tag.

[0029] In the embodiment of the present application, the key-value cache vector represents the key vector and value vector corresponding to the tag, that is, the key-value cache vector refers to the key vector (K) and value vector (V) generated by the tag during the attention interaction process; the larger the attention score of the tag, the higher the value of the information contained in the tag, so for the key-value cache vector corresponding to each tag, the importance ratio of the key-value cache vector is determined according to the attention score of the tag. Among them, the importance ratio of the key-value cache vector refers to the importance ratio of the key-value cache vector in all the key-value cache vectors of the layer. The larger the importance ratio of the key-value cache vector, the more important the key-value cache vector is, and the larger the amount of context information it contains.

[0030] It is understandable that the distribution of the marked attention scores in different layers is different, so the distribution of the importance ratio of the key-value cache vectors in different layers is also different, that is, the importance ratio of each key-value cache vector in each layer will be different between layers.

[0031] Step S130: taking the importance ratio of the key-value cache vector as a priority value, arranging the key-value cache vectors in descending order at each layer, and obtaining a priority sequence of the key-value cache vectors at each layer.

[0032] In an embodiment of the present application, for each layer, the importance ratio of the key-value cache vector is used as the priority value, and all the key-value cache vectors of the layer are arranged in descending order to obtain a priority sequence of the key-value cache vector. In the priority sequence of the key-value cache vector, the key-value cache vector with a higher ranking has a greater importance ratio, which makes the key-value cache vector compression equivalent to retaining the prefix of the key-value cache vector, wherein retaining the prefix of the key-value cache vector refers to the key-value cache vector retained in the priority sequence from the beginning to the end according to the sorting, that is, the retained key-value cache vector is equivalent to retaining the key-value cache vector of the prefix ratio in the priority sequence.

[0033] Step S140: Determine a target information retention ratio based on the target compression ratio budget, where the target information retention ratio represents a cumulative priority threshold.

[0034] In the embodiment of the present application, the target compression ratio budget is a parameter flexibly determined according to the inference speed of the model and the performance of the large model, and the target information retention ratio refers to the threshold of the amount of context information retained at each layer, that is, the cumulative priority threshold. For example, if the target information retention ratio is 0.9, the cumulative priority value (or amount of context information) corresponding to the key-value cache vector retained at each layer cannot be less than the target information retention ratio.

[0035] Step S150: According to the target information retention ratio, the retained key-value cache vectors are determined from the priority sequence of the key-value cache vectors of each layer.

[0036] In an embodiment of the present application, each key-value cache vector in the priority sequence is arranged according to a priority value (i.e., an importance ratio), and the target information retention ratio is a cumulative priority threshold. For the priority sequence of the key-value cache vectors of each layer, the cumulative priority value at each key-value cache vector position is calculated in sequence from the beginning to the end according to the sorting, and then the cumulative priority value is not less than the target information retention ratio, and the corresponding key-value cache vector is used as the retained key-value cache vector.

[0037] It is understandable that, since the importance ratio distribution of key-value cache vectors in different layers is also different, the priority values ​​corresponding to each key-value cache vector in the priority sequence are also different. Therefore, the number of key-value cache vectors finally retained in each layer may also be different, but the total importance ratio (or amount of context information) corresponding to the retained key-value cache vectors may be the same. For example, if the target information retention ratio is 0.8, if the cumulative priority values ​​of the first two key-value cache vectors in the priority sequence of a certain layer are not less than 0.8, then the first two key-value cache vectors are retained, and if the cumulative priority values ​​of the first three key-value cache vectors in the priority sequence of a certain layer are not less than 0.8, then the first three key-value cache vectors are retained. In this way, there is no need to retain the same number of key-value cache vectors in all layers of the large model, but to adaptively retain the most important key-value cache vectors in each layer.

[0038] By adopting the technical solution of the embodiment of the present application, the importance ratio of the key-value cache vector is determined based on the attention score of each tag, and the importance ratio of the key-value cache vector is used as the priority value to obtain the priority sequence of the key-value cache vectors of each layer, so that the key-value cache vectors with high importance can be retained at each layer based on the priority sequence; and compared with the related art, it does not directly include the same size of the key-value cache vector at each layer. This method takes into account the different importance distributions of the key-value cache vectors of different layers, and determines the retained key-value cache vectors from the priority sequence of the key-value cache vectors of each layer based on the target information retention ratio, so that the key-value cache can be adaptively compressed at each layer to retain some key-value cache vectors with high importance ratios that meet the target information retention ratio. In this way, a key-value cache vector compression method that takes into account both high reasoning performance and high task accuracy is realized, which can maximize the reasoning speed of large models while maintaining the processing power of the model well.

[0039] In combination with the above embodiments, in one implementation, the present application embodiment further provides an adaptive prefix key value cache compression method. In this method, after the above step S150, the following steps are included: Step S160: Decoding is performed according to the key-value cache vectors retained in each layer to obtain generation results corresponding to the image information and the text information.

[0040] In the embodiment of the present application, the key-value cache vectors in each layer are retained based on the priority value (importance ratio), so the key-value cache vector retained in each layer is the key-value cache vector with the highest importance among all key-value cache vectors, that is, the retained key-value cache vector contains the main context information. Therefore, the large model can be directly decoded according to the key-value cache vectors retained in each layer to obtain the generation results corresponding to the image information and text information. Among them, the generation result can be the answer to the question about the text information.

[0041] In this way, there is no need to use all the key-value cache vectors in the large model for decoding. Only the key-value cache vectors retained in each layer are used for decoding, which improves the reasoning speed of the large model. In addition, the key-value cache vectors retained in each layer contain the main contextual information, and there is no significant information loss, so the processing capability of the model can be well maintained.

[0042] In combination with the above embodiments, in one implementation, the embodiment of the present application also provides an adaptive prefix key value cache compression method. In this method, each layer of attention interaction performs multi-head attention interaction based on the tag sequence, and the attention score of each tag in the above step S110 is determined according to the following steps A1 to A2: Step A1: Obtain a total attention score received by a tag in each attention head, wherein the total attention score represents the sum of the attention scores of all tags in the tag sequence received by the tag in the attention head.

[0043] Step A2: According to the total attention scores received in each attention head, the average of the total attention scores of all attention heads is obtained as the labeled attention score.

[0044] In the embodiment of the present application, each layer of attention interaction performs multi-head attention interaction based on the tag sequence, and each attention head performs attention interaction based on the tag sequence. For example, if the large model includes Layer attention interaction, in the The input tag sequence is ,in is the number of tags; multi-head attention interaction is performed in the attention interaction layer, and each tag receives the attention scores of all tags in the tag sequence in the attention head.

[0045] For Tags , in In the attention head, the query vector, key vector and value vector are represented as , and , then the attention scores of all tags in the tag sequence received by the attention head can be expressed as the attention score matrix : , , in, Indicated in The first Mark pair The attention scores of the tags.

[0046] The total attention score received by the marker in each attention head can be obtained by The attention score for each tag is obtained as the average of the total attention scores of all attention heads ,Right now: , in, It is understood that the attention scores of the labels obtained in this way will vary between layers, but will be the same in all heads.

[0047] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides an adaptive prefix key value cache compression method. In this method, the "determining the importance ratio of each key value cache vector at each layer according to the attention score of each tag" in the above step S110 specifically includes: According to the attention scores of all tags in each layer, the attention score of each tag is normalized to obtain the normalized result of the attention score of each tag; the normalized result of the attention score of each tag is used as the importance ratio of the corresponding key-value cache vector.

[0048] In an embodiment of the present application, the attention score of a tag represents the importance of the key-value cache vector corresponding to the tag. In order to more accurately reflect the importance of each key-value cache vector and accurately identify key-value cache vectors with better importance, the attention score of each tag is normalized as the importance ratio of the corresponding key-value cache vector, and then the key-value cache vectors retained in each layer are subsequently calculated based on the importance ratio of the key-value cache vector.

[0049] For example, for Layer Key-value cache vectors, the importance ratio of key-value cache vectors It can be expressed as: , Furthermore, the importance ratio of the key-value cache vector is used as the priority value, and the key-value cache vector is prioritized at each layer. Arrange in descending order to obtain a priority sequence of the key-value cache vectors of each layer, in which a key-value cache vector with a higher ranking has a greater importance ratio, which makes the key-value cache vector compression equivalent to retaining the vector prefix of the key-value cache.

[0050] In combination with the above embodiments, in one implementation, the embodiment of the present application further provides an adaptive prefix key value cache compression method. In this method, the above step S150 of "determining the reserved key value cache vector from the priority sequence of the key value cache vectors of each layer according to the target information retention ratio" specifically includes steps B1 to B2: Step B1: For each layer, according to the target information retention ratio, determine the minimum prefix cache value that meets the first constraint condition as the retention ratio of the key-value cache vector; wherein the first constraint condition represents the cumulative priority value retained based on the minimum prefix cache value, which is greater than or equal to the cumulative priority threshold corresponding to the target information retention ratio.

[0051] In the embodiment of the present application, the minimum prefix cache value is adaptively derived based on the target information retention ratio. In the priority sequence, the target information retention ratio represents the cumulative priority threshold, and the retention ratio of the key-value cache vector is the minimum prefix cache value, which needs to meet the first constraint condition.

[0052] For example, the first constraint is expressed as: , in, Indicates The retention ratio of the key-value cache vector of the layer, Indicates the minimum prefix cache value, Indicates the cumulative priority value of the minimum prefix cache value. Target information retention ratio.

[0053] Assume that the sorted index is , for the The minimum prefix cache value of the priority sequence of the layer can be obtained by the following formula to obtain the cumulative priority value retained by the minimum prefix cache value: , in, Indicates the cumulative priority value of the minimum prefix cache value, that is, the minimum prefix cache value is retained. After the important key-value cache vector, The amount of context information retained by the layer.

[0054] Furthermore, the retention ratio of the key-value cache vectors of each layer satisfies a second constraint condition; the second constraint condition represents that the sum of the key-value cache vectors retained based on the retention ratio of the key-value cache vectors of each layer is equal to the key-value cache vector threshold determined based on the target compression ratio budget.

[0055] For example, the retention ratio of the key-value cache vector of each layer is ,in, Indicates the retention ratio of the key-value cache vector at the first layer, Indicates the retention ratio of the key-value cache vector at the second layer, Indicates The retention ratio of the layer's key-value cache vector.

[0056] The second constraint is expressed as: , in, represents the target compression ratio budget, represents the number of layers of attention interaction, Indicates the number of key-value cache vectors, Indicates a key-value cache vector threshold determined based on the target compression ratio budget.

[0057] Step B2: According to the retention ratio of the key-value cache vectors, the retained key-value cache vectors are determined from the priority sequence of the key-value cache vectors.

[0058] In the embodiment of the present application, each key value cache vector in the priority sequence is arranged according to the priority value, and for each layer of the priority sequence, the key value cache vector corresponding to the retention ratio of the first key value cache vector in the priority sequence is retained, and the remaining key value cache vectors are discarded. For example, if the retention ratio of the key value cache vector is 30%, the first 30% of the key value cache vectors in the priority sequence are retained.

[0059] By adopting the technical solution of the embodiment of the present application, the retention ratio of the key-value cache vector is adaptively determined according to the target information retention ratio, so that based on the retention ratio of the key-value cache vector, some key-value cache vectors with a high importance ratio that meet the target information retention ratio are retained in the priority sequence.

[0060] In combination with the above embodiments, in one implementation, the present application embodiment further provides an adaptive prefix key value cache compression method. In this method, the "determining the target information retention ratio based on the target compression ratio budget" in the above step S140 specifically includes steps C1 to C2: Step C1: Determine a third constraint condition based on the target compression ratio budget; wherein the third constraint condition represents the sum of the retention ratios of all layers of key value cache vectors determined based on the information retention ratio, which is equal to the sum of the retention ratios of all layers of key value cache vectors determined by the target compression ratio preset.

[0061] Step B2: Based on the binary search method, determine the information retention ratio, and if the information retention ratio satisfies the third constraint condition, use the information retention ratio as the target information retention ratio.

[0062] In the embodiment of the present application, a target information retention ratio is determined given a target compression ratio budget, so as to determine the retained key value cache vectors from the priority sequence of the key value cache vectors of each layer based on the target information retention ratio. In the priority sequence, the target information retention ratio represents the cumulative priority threshold, and the target information retention ratio needs to meet the requirements of the compression ratio budget, that is, the third constraint condition.

[0063] For example, the third constraint can be expressed as: , in, represents the target compression ratio budget, represents the number of layers of attention interaction, represents the sum of the retention ratios of all layer key-value cache vectors determined based on the information retention ratio, Indicates the retention ratio of the key-value cache vector, Represents the total retention ratio of the key-value cache vector.

[0064] Then, an information retention ratio search is performed based on a binary search method, so as to take the information retention ratio that satisfies the third constraint condition as the target information retention ratio.

[0065] Specifically, based on the binary search method, an information retention ratio is determined, and when the information retention ratio satisfies the third constraint condition, the information retention ratio is used as the target information retention ratio, including: determining the information retention ratio based on the upper limit value and the lower limit value of the search area; when the information retention ratio does not satisfy the third constraint condition, updating the search area according to the information retention ratio; determining a new information retention ratio according to the upper limit value and the lower limit value of the updated search area; when the information retention ratio satisfies the third constraint condition, using the current information retention ratio as the target information retention ratio.

[0066] The following search area is ,in and As an example, the method of determining the target information retention ratio is described. First, calculate the information retention ratio , and determine the information retention ratio at this time Whether the third constraint is met, that is, calculating the information retention ratio The corresponding compression budget difference .if Equal to zero, indicating the information retention ratio The third constraint is met, and the current information retention ratio is used as the target information retention ratio; if Less than zero, set ; Otherwise, set ,Then, a new information retention ratio is calculated and the above process is repeated until the ,information retention ratio satisfies the third constraint condition, and the current information retention ratio is ,taken as the target information retention ratio.

[0067] By adopting the technical solution of the embodiment of the present application, the target information retention ratio can be determined based on a given target compression ratio budget, that is, the optimal retained key value cache vector size for each layer can be determined, so that some key value cache vectors with a high importance ratio that meet the target information retention ratio can be retained based on the target information retention ratio.

[0068] Reference Figure 2 As shown, Figure 2 It is an overall architecture diagram of an adaptive prefix key-value cache compression method provided by an embodiment of the present application. Specifically, the large model includes a multi-layer attention interaction layer (transformer layer), and the tag sequence corresponding to the image information and the text information is input into the large model for multi-layer attention interaction to obtain the attention score of each tag in each layer, and then the attention score of each tag is normalized according to the attention scores of all tags in each layer to obtain the normalized result of the attention score of each tag, and the normalized result of the attention score of each tag is used as the importance ratio of the corresponding key-value cache vector. Then, the importance ratio of the key-value cache vector is used as the priority value, and the key-value cache vectors are arranged in descending order at each layer to obtain the priority sequence of the key-value cache vectors of each layer; given the target compression ratio budget, the target information retention ratio is determined based on the binary search method, and according to the target information retention ratio, the retained key-value cache vector is determined from the priority sequence of the key-value cache vectors of each layer. Finally, the large model decodes the key-value cache vectors retained in each layer to obtain the generation results corresponding to the image information and the text information.

[0069] In this way, a key-value cache vector compression method is implemented that takes into account both high inference performance and high task accuracy. It can maximize the inference speed of large models while maintaining the processing capability of the model, demonstrating good potential in practical applications.

[0070] Based on the same technical concept, the embodiment of the present application also provides an adaptive prefix key value cache compression device, referring to Figure 3 As shown, Figure 3 : is a structural diagram of an adaptive prefix key value cache compression device provided in an embodiment of the present application, the device comprising: The first input module 310 is used to input the tag sequence corresponding to the image information and the text information into the large model for multi-layer attention interaction to obtain the attention score of each tag in each layer; A first determination module 320, configured to determine the importance ratio of each key-value cache vector of each layer according to the attention score of each tag, wherein the key-value cache vector represents a key vector and a value vector corresponding to the tag; A first sorting module 330 is used to use the importance ratio of the key-value cache vector as a priority value, and to sort the key-value cache vectors in descending order at each layer to obtain a priority sequence of the key-value cache vectors at each layer; A second determination module 340 is used to determine a target information retention ratio based on a target compression ratio budget, wherein the target information retention ratio represents a cumulative priority threshold; The third determination module 350 is configured to determine the reserved key-value cache vectors from the priority sequence of the key-value cache vectors of each layer according to the target information reservation ratio.

[0071] In an optional embodiment, the third determining module includes: A first determination submodule is used to determine, for each layer, a minimum prefix cache value that satisfies a first constraint condition according to the target information retention ratio, as the retention ratio of the key value cache vector; wherein the first constraint condition represents that the cumulative priority value retained based on the minimum prefix cache value is greater than or equal to the cumulative priority threshold corresponding to the target information retention ratio; The second determining submodule is used to determine the reserved key-value cache vector from the priority sequence of the key-value cache vector according to the reserved ratio of the key-value cache vector.

[0072] In an optional embodiment, the retention ratio of the key-value cache vectors of each layer satisfies a second constraint condition; the second constraint condition represents that the sum of the key-value cache vectors retained based on the retention ratio of the key-value cache vectors of each layer is equal to the key-value cache vector threshold determined based on the target compression ratio budget.

[0073] In an optional embodiment, the second determining module includes: A third determination submodule is used to determine a third constraint condition according to the target compression ratio budget; wherein the third constraint condition represents the sum of the retention ratios of all layers of key value cache vectors determined based on the information retention ratio, which is equal to the sum of the retention ratios of all layers of key value cache vectors determined by the target compression ratio preset; The fourth determination submodule is used to determine the information retention ratio based on a binary search method, and when the information retention ratio satisfies the third constraint condition, use the information retention ratio as the target information retention ratio.

[0074] In an optional embodiment, the fourth determination submodule is also used to determine the information retention ratio based on the upper limit value and the lower limit value of the search area; when the information retention ratio does not satisfy the third constraint condition, update the search area according to the information retention ratio; determine a new information retention ratio according to the updated upper limit value and the lower limit value of the search area; when the information retention ratio satisfies the third constraint condition, use the current information retention ratio as the target information retention ratio.

[0075] In an optional embodiment, each layer of attention interaction performs multi-head attention interaction based on a tag sequence; the first input module includes: A score determination module is used to obtain a total attention score received by the tag in each attention head, wherein the total attention score represents the sum of the attention scores of all tags in the tag sequence received by the tag in the attention head; based on the total attention score received in each attention head, an average value of the total attention scores of all attention heads is obtained as the attention score of the tag.

[0076] In an optional embodiment, the first determination module is also used to normalize the attention score of each tag according to the attention scores of all tags in each layer to obtain a normalized result of the attention score of each tag; and use the normalized result of the attention score of each tag as the importance ratio of the corresponding key-value cache vector.

[0077] In an optional embodiment, the device further includes: The decoding module is used to decode according to the key-value cache vectors retained in each layer to obtain the generation results corresponding to the image information and the text information.

[0078] The present application also provides an electronic device, referring to Figure 4 , Figure 4 Schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device 400 includes: a memory 410 and a processor 420. The memory 410 and the processor 420 are connected via a bus communication. A computer program is stored in the memory 410. The computer program can be run on the processor 420 to implement the steps of the adaptive prefix key value cache compression method described in the embodiment of the present application.

[0079] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the adaptive prefix key value cache compression method described in the embodiment of the present application are implemented.

[0080] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the adaptive prefix key value cache compression method described in the embodiment of the present application.

[0081] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0082] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods and devices according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0083] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0084] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0085] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0086] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0087] The above is a detailed introduction to an adaptive prefix key value cache compression method, device and electronic device provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An adaptive prefix key value cache compression method, characterized in that: The method comprises: The tag sequences corresponding to the image information and text information are input into the large model for multi-layer attention interaction to obtain the attention scores of each tag in each layer; Determine the importance ratio of each key-value cache vector of each layer according to the attention score of each tag, wherein the key-value cache vector represents the key vector and the value vector corresponding to the tag; The importance ratio of the key-value cache vector is used as a priority value, and the key-value cache vectors are arranged in descending order at each layer to obtain a priority sequence of the key-value cache vectors at each layer; Determining a target information retention ratio based on a target compression ratio budget, wherein the target information retention ratio represents a cumulative priority threshold; According to the target information retention ratio, the retained key-value cache vectors are determined from the priority sequence of the key-value cache vectors of each layer.

2. The method according to claim 1, characterized in that According to the target information retention ratio, the retained key value cache vector is determined from the priority sequence of the key value cache vectors of each layer, including: For each layer, according to the target information retention ratio, determine the minimum prefix cache value that meets the first constraint condition as the retention ratio of the key value cache vector; wherein the first constraint condition represents that the cumulative priority value retained based on the minimum prefix cache value is greater than or equal to the cumulative priority threshold corresponding to the target information retention ratio; According to the retention ratio of the key-value cache vectors, the retained key-value cache vectors are determined from the priority sequence of the key-value cache vectors.

3. The method according to claim 2, characterized in that The retention ratio of the key-value cache vectors of each layer satisfies a second constraint condition; the second constraint condition represents that the sum of the key-value cache vectors retained based on the retention ratio of the key-value cache vectors of each layer is equal to the key-value cache vector threshold determined based on the target compression ratio budget.

4. The method according to claim 1, characterized in that Based on the target compression ratio budget, determine the target information retention ratio, including: Determine a third constraint condition according to the target compression ratio budget; wherein the third constraint condition represents the sum of the retention ratios of all layers of key value cache vectors determined based on the information retention ratio, which is equal to the sum of the retention ratios of all layers of key value cache vectors determined by the target compression ratio preset; Based on the binary search method, an information retention ratio is determined, and when the information retention ratio satisfies the third constraint condition, the information retention ratio is used as the target information retention ratio.

5. The method according to claim 4, characterized in that Based on the binary search method, determining the information retention ratio, and when the information retention ratio satisfies the third constraint condition, using the information retention ratio as the target information retention ratio, includes: Determine the information retention ratio based on the upper limit and lower limit of the search area; When the information retention ratio does not satisfy the third constraint condition, updating the search area according to the information retention ratio; Determining a new information retention ratio according to the updated upper limit and lower limit of the search area; When the information retention ratio satisfies the third constraint condition, the current information retention ratio is used as the target information retention ratio.

6. The method according to claim 1, characterized in that Each layer of attention interaction performs multi-head attention interaction based on the tag sequence; the attention score of each tag is determined according to the following steps: Obtain a total attention score received by the tag in each attention head, wherein the total attention score represents the sum of the attention scores of all tags in the tag sequence received by the tag in the attention head; According to the total attention scores received in each attention head, the average of the total attention scores of all attention heads is obtained as the labeled attention score.

7. The method according to claim 1, characterized in that According to the attention score of each tag, the importance ratio of each key-value cache vector of each layer is determined, including: According to the attention scores of all tags in each layer, the attention score of each tag is normalized to obtain the normalized result of the attention score of each tag; The attention score of each tag is normalized as the importance ratio of the corresponding key-value cache vector.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: Decoding is performed according to the key-value cache vectors retained in each layer to obtain generation results corresponding to the image information and the text information.

9. An adaptive prefix key value cache compression device, characterized in that: The device comprises: The first input module is used to input image information and text information into the large model for multi-layer attention interaction to obtain the attention score of each tag in each layer; A first determination module is used to determine the importance ratio of each key-value cache vector of each layer according to the attention score of each tag, wherein the key-value cache vector represents the key vector and the value vector corresponding to the tag; A first sorting module is used to use the importance ratio of the key-value cache vector as a priority value, and arrange the key-value cache vectors in descending order at each layer to obtain a priority sequence of the key-value cache vectors at each layer; A second determination module is used to determine a target information retention ratio based on a target compression ratio budget, wherein the target information retention ratio represents a cumulative priority threshold; The third determination module is used to determine the reserved key-value cache vectors from the priority sequence of the key-value cache vectors of each layer according to the target information retention ratio.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the adaptive prefix key value cache compression method described in any one of claims 1-8 are implemented.

Images