Key-value cache quantification method and apparatus, device, and storage medium

Abstract

The application belongs to the technical field of large models, and discloses a key-value cache quantization method and device, equipment and a storage medium. According to the sensitivity score corresponding to the key channel in the large model and the importance score of the current query word element on the key channel, the significance score corresponding to the key channel is determined, then the key degree corresponding to the key channel is determined according to the significance score, the key quantization mode corresponding to the key cache is determined based on the key degree, and then key-value cache quantization is performed according to the key quantization mode and the value quantization mode corresponding to the value cache. The application can allocate precision by combining the internal quantization difficulty of the key channel and the dynamic query importance, determine the key quantization mode corresponding to the key cache based on the key degree, focus high-bit-width resources on the key channel, and then perform key-value cache quantization according to the key quantization mode and the value quantization mode corresponding to the value cache, so as to maintain the performance of long-chain reasoning of the large model under extremely low bit width.

Description

Technical Field

[0001] This application relates to the field of large model technology, and in particular to a key-value cache quantization method, apparatus, device and storage medium. Background Technology

[0002] Currently, large models generate extremely long intermediate thought processes during long-chain inference, causing the key-value cache to snowball and easily lead to memory explosion. There are two main methods for key-value cache quantization: fixed-precision quantization and mixed-precision quantization. Fixed-precision quantization uses the same quantization bit width (e.g., 4-bit or 2-bit) across the entire key-value cache, providing the most direct way to uniformly compress all keys and values. However, it cannot handle channels with extremely uneven numerical distributions or outliers in the key cache. The quantization requirements of outliers and normal channels cannot be distinguished; a channel containing outliers may require 4 bits or even higher precision for faithful representation, while other channels may only need 2 bits. Mixed-precision quantization allocates different quantization bit widths to different parts of the key-value cache, using higher precision for critical channels and lower precision for non-critical channels, aiming for a better performance-compression balance within a limited memory budget. However, its bit width allocation target is fundamentally flawed, resulting in the wasteful allocation of high-bit-width resources. The aforementioned key-value caching quantization method suffers severe performance degradation under complex inference tasks. The root cause is that it ignores the dynamic correlation between the key channel and the current query, resulting in high-bit-width resources being incorrectly allocated to channels that have little impact on attention calculation. Summary of the Invention

[0003] The main purpose of this application is to provide a key-value caching quantization method, apparatus, device and storage medium, which aims to solve the technical problem that existing large models suffer from high bit width resources and severe performance degradation in long chain inference.

[0004] To achieve the above objectives, this application provides a key-value cache quantization method, which includes the following steps: The significance score of the key channel is determined based on the sensitivity score of the key channel in the large model and the importance score of the current query term on the key channel. The key level of the key channel is determined based on the significance score, and the key quantization method corresponding to the key cache is determined based on the key level. Key-value cache quantization is performed according to the key quantization method and the corresponding value quantization method of the value cache.

[0005] Optionally, determining the saliency score corresponding to the key channel based on the sensitivity score corresponding to the key channel in the large model and the importance score of the current query term on the key channel includes: Identify the key channels within the current buffer of the large model, and determine the sensitivity scores corresponding to the key channels; Determine the activation value of the current query term on the key channel, and determine the importance score of the current query term on the key channel based on the activation value; The significance score corresponding to the key channel is calculated based on the product of the sensitivity score and the importance score.

[0006] Optionally, determining the key channel within the current buffer of the large model and determining the sensitivity score corresponding to the key channel includes: Determine the key channel within the current buffer of the large model, and determine the set of numerical values ​​for all tokens on the key channel; The sensitivity score corresponding to the key channel is determined based on the set of values ​​and the target quantization bit width.

[0007] Optionally, determining the activation value of the current query term on the key channel and determining the importance score of the current query term on the key channel based on the activation value includes: Determine the sequence length of the query term in the current buffer, and determine the activation value of the current query term on the key channel; The importance score of the current query term on the key channel is calculated based on the sequence length and the activation value.

[0008] Optionally, determining the key level corresponding to the key channel based on the saliency score, and determining the key quantization method corresponding to the key cache based on the key level, includes: The criticality of the key channel is determined based on the significance score, the first threshold, and the second threshold. When the criticality level is critical channel, the key quantization method corresponding to the key cache is determined to be no quantization; In the case where the criticality level is medium criticality, the key quantization method corresponding to the key cache is determined to be the first asymmetric quantization method; When the criticality level is non-critical channel, the key quantization method corresponding to the key cache is determined to be the second asymmetric quantization method.

[0009] Optionally, determining the criticality of the key channel based on the saliency score, the first threshold, and the second threshold includes: If the significance score is greater than the first threshold, the key channel is determined to be a key channel. If the significance score is less than or equal to the first threshold and greater than the second threshold, the key channel is determined to be a medium key channel. If the significance score is less than the second threshold, the key channel is determined to be a non-key channel.

[0010] Optionally, after performing key-value cache quantization according to the key quantization method and the value quantization method corresponding to the value cache, the method further includes: After the key-value cache quantization is completed in the current buffer of the large model, the current buffer is cleared to obtain a new buffer; After the new buffer is filled with terms, the step of determining the saliency score of the key channel based on the sensitivity score of the key channel in the large model and the importance score of the current query term on the key channel is returned to iterate and perform key-value cache quantization on all buffers.

[0011] Furthermore, to achieve the above objectives, this application also provides a key-value cache quantization device, the key-value cache quantization device comprising: The score determination module is used to determine the significance score of the key channel based on the sensitivity score of the key channel in the large model and the importance score of the current query term on the key channel; The quantization method determination module is used to determine the criticality of the key channel based on the significance score, and to determine the key quantization method corresponding to the key cache based on the criticality. The key-value cache quantization module is used to perform key-value cache quantization according to the key quantization method and the value quantization method corresponding to the value cache.

[0012] In addition, to achieve the above objectives, this application also proposes a key-value cache quantization device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the key-value cache quantization method as described above.

[0013] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the key-value cache quantization method described above.

[0014] This application determines the saliency score of a key channel based on its sensitivity score and the importance score of the current query term on the key channel within the large model. Then, it determines the key level of the key channel based on the saliency score, and determines the key quantization method for the key cache based on the key level. Finally, it performs key-value cache quantization based on both the key quantization method and the value quantization method for the value cache. This application first determines the saliency score of the key channel based on its sensitivity score and the importance score of the current query term on the key channel, enabling the allocation of precision by combining the inherent quantization difficulty of the key channel with dynamic query importance. Then, it determines the key quantization method for the key cache based on the key level, allowing high-bit-width resources to be focused on key channels. Finally, it performs key-value cache quantization based on both the key quantization method and the value quantization method for the value cache, thus maintaining the performance of long-chain inference in a large model with extremely low bit-width conditions. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating the first embodiment of the key-value caching quantization method of this application; Figure 2 This is a flowchart illustrating the second embodiment of the key-value caching quantization method of this application; Figure 3 This is a flowchart illustrating the third embodiment of the key-value caching quantization method of this application; Figure 4 This is a structural block diagram of the first embodiment of the key-value caching quantization device of this application; Figure 5 This is a schematic diagram of the key-value caching quantization device of the hardware operating environment involved in the embodiments of this application.

[0018] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0020] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0021] The main solution of this application embodiment is: to determine the saliency score of the key channel based on the sensitivity score of the key channel in the large model and the importance score of the current query term on the key channel; to determine the key degree of the key channel based on the saliency score, and to determine the key quantization method corresponding to the key cache based on the key degree; and to perform key-value cache quantization based on the key quantization method and the value quantization method corresponding to the value cache.

[0022] Currently, large models generate extremely long intermediate thought processes during long-chain inference, causing the key-value cache to snowball and easily lead to memory explosion. There are two main methods for key-value cache quantization: fixed-precision quantization and mixed-precision quantization. Fixed-precision quantization uses the same quantization bit width (e.g., 4-bit or 2-bit) across the entire key-value cache, providing the most direct way to uniformly compress all keys and values. However, it cannot handle channels with extremely uneven numerical distributions or outliers in the key cache. The quantization requirements of outliers and normal channels cannot be distinguished; a channel containing outliers may require 4 bits or even higher precision for faithful representation, while other channels may only need 2 bits. Mixed-precision quantization allocates different quantization bit widths to different parts of the key-value cache, using higher precision for critical channels and lower precision for non-critical channels, aiming for a better performance-compression balance within a limited memory budget. However, its bit width allocation target is fundamentally flawed, resulting in the wasteful allocation of high-bit-width resources. The aforementioned key-value caching quantization method suffers severe performance degradation under complex inference tasks. The root cause is that it ignores the dynamic correlation between the key channel and the current query, resulting in high-bit-width resources being incorrectly allocated to channels that have little impact on attention calculation.

[0023] This application determines the saliency score of a key channel based on its sensitivity score and the importance score of the current query term on the key channel within the large model. Then, it determines the key level of the key channel based on the saliency score, and determines the key quantization method for the key cache based on the key level. Finally, it performs key-value cache quantization based on both the key quantization method and the value quantization method for the value cache. This application first determines the saliency score of the key channel based on its sensitivity score and the importance score of the current query term on the key channel, enabling the allocation of precision by combining the inherent quantization difficulty of the key channel with dynamic query importance. Then, it determines the key quantization method for the key cache based on the key level, allowing high-bit-width resources to be focused on key channels. Finally, it performs key-value cache quantization based on both the key quantization method and the value quantization method for the value cache, thus maintaining the performance of long-chain inference in a large model with extremely low bit-width conditions.

[0024] It should be noted that the executing entity of this application can be a computing service device with data processing, network communication, and program execution functions, such as a computer, or an electronic device or key-value caching quantization device capable of performing the above functions. The following description uses a key-value caching quantization device as an example to illustrate this embodiment and the subsequent embodiments. The key-value caching quantization method proposed in this application can be applied to fields involving large models, such as intelligent customer service and dialogue systems, education, and academic research.

[0025] Based on this, embodiments of this application provide a key-value caching quantization method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the key-value caching quantization method of this application.

[0026] In this embodiment, the key-value caching quantization method includes the following steps: Step S10: Determine the saliency score of the key channel based on the sensitivity score of the key channel in the large model and the importance score of the current query term on the key channel.

[0027] Understandably, in the attention mechanism of large models, each token generates a key vector, which serves as an index label for the current query to match. A key channel refers to a fixed dimension within the key vector. For example, if a large model's key vector is 128-dimensional, then this vector has 128 channels.

[0028] It should be understood that the sensitivity score refers to the inherent difficulty of quantizing the key channel without considering the query interaction; the higher the value, the more difficult it is to quantize into a low-bit representation. The importance score refers to the activation value of the current query token on a key channel, used to measure the relevance of a key channel to the current query token. This score quantifies the contribution of the key channel to the current attention score; that is, the larger the query activation value, the more significant the amplification effect of the error in the key channel, and therefore, the more worthwhile it is to retain a high bit width.

[0029] In practice, the product of the sensitivity score and the importance score can be used as the significance score of the key channel to measure its criticality.

[0030] Step S20: Determine the criticality of the key channel based on the saliency score, and determine the key quantization method corresponding to the key cache based on the criticality.

[0031] Understandably, the keyness can be determined based on the significance score of each key channel. The higher the significance, the higher the keyness. For different keyness levels, the corresponding key channels will have different bond quantization methods.

[0032] Step S30: Perform key-value cache quantization according to the key quantization method and the value quantization method corresponding to the value cache.

[0033] It should be understood that the key cache can include multiple key channels, and differential quantization can be performed column by column. This can automatically focus high-bit-width resources on channels where even small errors can severely impact inference. The value cache has a relatively uniform error distribution and lacks a clear outlier structure. This means that regardless of whether channel-by-channel or token-by-token quantization is used, the error magnitude of each element is roughly the same. Therefore, the complete value vector for each token can be asymmetrically uniformly quantized and compressed to 2 bits.

[0034] This embodiment determines the saliency score of a key channel based on its sensitivity score and the importance score of the current query term on that key channel within the large model. Then, it determines the key level of the key channel based on the saliency score, and determines the key quantization method for the key cache based on the key level. Finally, it performs key-value cache quantization based on both the key quantization method and the value quantization method for the value cache. This embodiment first determines the saliency score of the key channel based on its sensitivity score and the importance score of the current query term on that key channel, enabling the allocation of precision by combining the inherent quantization difficulty of the key channel with the dynamic importance of the query. Then, it determines the key quantization method for the key cache based on the key level, focusing high-bit-width resources on key channels. Finally, it performs key-value cache quantization based on both the key quantization method and the value quantization method for the value cache, thus maintaining the performance of long-chain inference in a large model with extremely low bit-width conditions.

[0035] refer to Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the key-value caching quantization method of this application.

[0036] Based on the first embodiment described above, in this embodiment, step S10 includes: Step S101: Determine the key channels in the current buffer of the large model, and determine the sensitivity scores corresponding to the key channels.

[0037] Understandably, the current buffer is a dynamic observation window, a pre-allocated high-speed video memory space with a fixed size of R=128 tokens, which can be used to temporarily store newly generated, unquantized key and value caches. The key channels within the current buffer can be determined, and the sensitivity scores corresponding to each key channel can be identified.

[0038] Furthermore, in order to accurately calculate the sensitivity score corresponding to the key channel, in this embodiment, step S101 includes: determining the key channel in the current buffer of the large model, and determining the numerical set of all lexical terms on the key channel; and determining the sensitivity score corresponding to the key channel based on the numerical set and the target quantization bit width.

[0039] It should be understood that the formula for calculating the sensitivity score corresponding to the key channel is: , Indicates the key channel in the current buffer. The set of values ​​for all tokens. Quantize the target bit width, i.e., the bit width you want to quantize. Key channel Sensitivity score.

[0040] Step S102: Determine the activation value of the current query term on the key channel, and determine the importance score of the current query term on the key channel based on the activation value.

[0041] Furthermore, in order to accurately calculate the importance score of the current query term on the key channel, in this embodiment, step S102 includes: determining the sequence length of the query term in the current buffer and determining the activation value of the current query term on the key channel; and calculating the importance score of the current query term on the key channel based on the sequence length and the activation value.

[0042] Understandably, the formula for calculating importance scores is: , This indicates the length of the query token sequence in the current buffer. This represents the activation value of the i-th query token on the d-th key channel. This indicates that the i-th query token is in the key channel. Importance score.

[0043] Step S103: Calculate the significance score corresponding to the key channel based on the product between the sensitivity score and the importance score.

[0044] In the specific implementation, the product of the sensitivity score and the importance score is the significance score corresponding to the key channel, i.e. , Key channel The corresponding significance score.

[0045] This embodiment identifies key channels within the current buffer of the large model and determines their corresponding sensitivity scores. It then determines the activation value of the current query term on the key channel and, based on this activation value, determines its importance score. Finally, it calculates the significance score of the key channel by multiplying the sensitivity and importance scores. This embodiment considers both the inherent quantification difficulty of key channels and dynamic query relevance, accurately measuring the criticality of key channels based on the obtained significance scores.

[0046] refer to Figure 3 , Figure 3 This is a flowchart illustrating the third embodiment of the key-value caching quantization method of this application.

[0047] Based on the above embodiments, in this embodiment, step S20 includes: Step S201: Determine the criticality of the key channel based on the significance score, the first threshold, and the second threshold.

[0048] Understandably, the first threshold can be the BF16 threshold, and the second threshold can be the UNIT4 threshold. The significance score can be compared with the first and second thresholds, and the keyness of the key channel can be determined based on the comparison results.

[0049] Furthermore, in order to effectively determine the criticality of the key channel, in this embodiment, step S201 includes: if the saliency score is greater than a first threshold, determining the criticality of the key channel as a critical channel; if the saliency score is less than or equal to the first threshold and greater than a second threshold, determining the criticality of the key channel as a moderately critical channel; and if the saliency score is less than the second threshold, determining the criticality of the key channel as a non-critical channel.

[0050] It should be understood that when the significance score is greater than the first threshold, i.e., greater than the BF16 threshold, the key channel can be determined to be a key channel; when the significance score is less than or equal to the first threshold and greater than the second threshold, i.e., less than or equal to the BF16 threshold and greater than the UNIT4 threshold, the key channel can be determined to be a moderately key channel; when the significance score is less than the second threshold, i.e., less than the UNIT4 threshold, the key channel can be determined to be a non-key channel.

[0051] Step S202: When the criticality level is critical channel, determine that the key quantization method corresponding to the key cache is not quantized.

[0052] Understandably, when the criticality level is critical channel, it can be determined that the key quantization method corresponding to the inter-caching is no quantization, that is, the original BF16 value is directly retained without any quantization compression.

[0053] Step S203: When the criticality level is medium critical channel, determine that the key quantization method corresponding to the key cache is the first asymmetric quantization method.

[0054] It should be understood that, in the case of a medium-critical channel, the key quantization method corresponding to the key cache can be determined to be the first asymmetric quantization method, that is, using the asymmetric UINT4 quantization method to perform 4-bit asymmetric quantization and retain medium details.

[0055] Step S204: When the criticality level is non-critical channel, determine that the key quantization method corresponding to the key cache is the second asymmetric quantization method.

[0056] In the specific implementation, when the criticality level is non-critical channel, the key quantization method corresponding to the key cache can be determined to be the second asymmetric quantization method, that is, using asymmetric UINT2 quantization to perform 2-bit asymmetric quantization for extreme compression.

[0057] Furthermore, in this embodiment, after step S30, the method further includes: after the current buffer in the large model completes key-value cache quantization, the current buffer is cleared to obtain a new buffer; after the new buffer is filled with tokens, the method returns to the step of determining the saliency score corresponding to the key channel based on the sensitivity score corresponding to the key channel in the large model and the importance score of the current query token on the key channel, so as to iteratively perform key-value cache quantization on all buffers.

[0058] Understandably, after the key-value cache quantization is completed in the current buffer—that is, when the buffer is first filled to R=128—the key-value cache quantization is completed in the manner described above. After quantization, the current buffer can be cleared, and 129 to 256 tokens can be received to fill the buffer again, resulting in a new buffer. The key-value cache quantization process is then repeated, recalculating the saliency score based on the latest query information in the new buffer. At this point, the precision allocation will be applied to all historical tokens and tokens in the current buffer. This means that as the semantics of the query change, the precision level of some historical channels may be adjusted up or down.

[0059] In the implementation, newly generated tokens are first written to a full-precision buffer. Once the buffer is full, the group is uniformly quantized. This mechanism delays the accumulation of quantization errors in early tokens. The update strategy adopted is to perform a saliency score recalculation and precision configuration update only after storing each group (R tokens, R=128), avoiding the computational overhead of reallocating precision for each token. Initially, tokens are stored in the buffer with full precision. When the buffer reaches R tokens, the first quantization is triggered: the average amplitude of the Query activation value in the current window within the buffer is calculated as an estimate of I; the quantization is then calculated based on the numerical range of the Key channel in the buffer. ;Calculate for all channels d ;according to The values ​​are assigned BF16 / UNIT4 / UNIT2 quantization precision levels; mixed progress quantization is performed on the Key Cache by channel, and 2-bit quantization is performed on the Value Cache.

[0060] This embodiment sets up a full-precision buffer (size R=128). Newly generated tokens are first stored in the buffer. Once the buffer is full, the saliency score is dynamically updated based on the query information in the current window, and quantization is performed. This maintains the performance of complex inference with long contexts with extremely low bit width (average 2.3–2.7 bits). Compared with other quantization methods on long text inference datasets, such as the KIVI quantization method, which reduces accuracy by 8.95%, and the RotateKV quantization method, which reduces accuracy by 3.33%, the quantization method of this invention achieves an accuracy reduction of only 1.80% compared to the full-precision model with an average bit width of only 2.3 bits, which is superior to existing quantization technologies.

[0061] This embodiment determines the criticality of a key channel based on its significance score, a first threshold, and a second threshold. If the channel is critical, the key quantization method for the key cache is no quantization. If the channel is moderately critical, the key quantization method for the key cache is a first asymmetric quantization method. If the channel is not critical, the key quantization method for the key cache is a second asymmetric quantization method. This embodiment determines different key quantization methods based on the criticality of the key channel, automatically focusing high-bit-width resources on channels where even small errors can significantly impact inference.

[0062] Reference Figure 4 , Figure 4 This is a structural block diagram of the first embodiment of the key-value caching quantization device of this application.

[0063] like Figure 4 As shown, the key-value caching quantization device proposed in this application includes: The score determination module 10 is used to determine the significance score of the key channel based on the sensitivity score of the key channel in the large model and the importance score of the current query term on the key channel; The quantization method determination module 20 is used to determine the criticality of the key channel based on the significance score, and to determine the key quantization method corresponding to the key cache based on the criticality. The key-value cache quantization module 30 is used to perform key-value cache quantization according to the key quantization method and the value quantization method corresponding to the value cache.

[0064] This embodiment determines the saliency score of a key channel based on its sensitivity score and the importance score of the current query term on that key channel within the large model. Then, it determines the key level of the key channel based on the saliency score, and determines the key quantization method for the key cache based on the key level. Finally, it performs key-value cache quantization based on both the key quantization method and the value quantization method for the value cache. This embodiment first determines the saliency score of the key channel based on its sensitivity score and the importance score of the current query term on that key channel, enabling the allocation of precision by combining the inherent quantization difficulty of the key channel with the dynamic importance of the query. Then, it determines the key quantization method for the key cache based on the key level, focusing high-bit-width resources on key channels. Finally, it performs key-value cache quantization based on both the key quantization method and the value quantization method for the value cache, thus maintaining the performance of long-chain inference in a large model with extremely low bit-width conditions.

[0065] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this application. In practical applications, those skilled in the art can select some or all of it to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0066] In addition, for technical details not described in detail in this embodiment, please refer to the key-value caching quantization method provided in any embodiment of this application, which will not be repeated here.

[0067] Based on the first embodiment of the key-value cache quantization device described above in this application, a second embodiment of the key-value cache quantization device of this application is proposed.

[0068] In this embodiment, the score determination module 10 is further configured to determine the key channel in the current buffer of the large model, and determine the sensitivity score corresponding to the key channel; determine the activation value of the current query term on the key channel, and determine the importance score of the current query term on the key channel based on the activation value; and calculate the saliency score corresponding to the key channel based on the product between the sensitivity score and the importance score.

[0069] Furthermore, the score determination module 10 is also used to determine the key channel in the current buffer of the large model, and to determine the numerical set of all lexical terms on the key channel; and to determine the sensitivity score corresponding to the key channel based on the numerical set and the target quantization bit width.

[0070] Furthermore, the score determination module 10 is also used to determine the sequence length of the query term in the current buffer and determine the activation value of the current query term on the key channel; and calculate the importance score of the current query term on the key channel based on the sequence length and the activation value.

[0071] Furthermore, the quantization method determination module 20 is also used to determine the criticality of the key channel based on the saliency score, the first threshold, and the second threshold; when the criticality is a critical channel, the key quantization method corresponding to the key cache is determined to be no quantization; when the criticality is a moderately critical channel, the key quantization method corresponding to the key cache is determined to be a first asymmetric quantization method; when the criticality is a non-critical channel, the key quantization method corresponding to the key cache is determined to be a second asymmetric quantization method.

[0072] Furthermore, the quantization method determination module 20 is also used to determine the criticality level corresponding to the key channel as a key channel when the significance score is greater than the first threshold; to determine the criticality level corresponding to the key channel as a moderately critical channel when the significance score is less than or equal to the first threshold and greater than the second threshold; and to determine the criticality level corresponding to the key channel as a non-critical channel when the significance score is less than the second threshold.

[0073] Furthermore, the key-value cache quantization module 30 is also used to clear the current buffer after the key-value cache quantization is completed in the current buffer of the large model, and obtain a new buffer; after the new buffer is filled with tokens, it returns to the step of determining the significance score corresponding to the key channel based on the sensitivity score corresponding to the key channel in the large model and the importance score of the current query token on the key channel, so as to iteratively perform key-value cache quantization on all buffers.

[0074] Other embodiments or specific implementations of the key-value caching quantization device of this application can be found in the above-described method embodiments, and will not be repeated here.

[0075] This application provides a key-value cache quantization device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the key-value cache quantization method in the first embodiment described above.

[0076] The following is for reference. Figure 5This document illustrates a schematic diagram of a key-value caching quantization device suitable for implementing embodiments of this application. The key-value caching quantization device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The key-value caching quantization device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0077] like Figure 5 As shown, the key-value cache quantization device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 1002 or a program loaded from storage device 1003 into random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the key-value cache quantization device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the key-value caching quantization device to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows key-value caching quantization devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0078] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0079] The key-value caching quantization device provided in this application, employing the key-value caching quantization method in the above embodiments, can solve the technical problem of high bit-width resources and severe performance degradation in long-chain inference of existing large models. Compared with the prior art, the beneficial effects of the key-value caching quantization device provided in this application are the same as those of the key-value caching quantization method provided in the above embodiments, and other technical features in this key-value caching quantization device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0080] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0081] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0082] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the key-value cache quantization method in the above embodiments.

[0083] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0084] The aforementioned computer-readable storage medium may be included in a key-value cache quantization device; or it may exist independently and not assembled into a key-value cache quantization device.

[0085] The aforementioned computer-readable storage medium carries one or more programs. When the one or more programs are executed by the key-value cache quantization device, the key-value cache quantization device: determines the saliency score corresponding to the key channel based on the sensitivity score corresponding to the key channel in the large model and the importance score of the current query term on the key channel; determines the key degree corresponding to the key channel based on the saliency score, and determines the key quantization method corresponding to the key cache based on the key degree; and performs key-value cache quantization based on the key quantization method and the value quantization method corresponding to the value cache.

[0086] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Python, Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0087] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0088] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0089] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described key-value caching quantization method. This addresses the technical problem of high bit-width resources and severe performance degradation in long-chain inference for existing large models. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the key-value caching quantization method provided in the above embodiments, and will not be elaborated upon here.

[0090] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A key-value caching quantization method, characterized in that, The key-value cache quantization method includes the following steps: The significance score of the key channel is determined based on the sensitivity score of the key channel in the large model and the importance score of the current query term on the key channel. The key level of the key channel is determined based on the significance score, and the key quantization method corresponding to the key cache is determined based on the key level. Key-value cache quantization is performed according to the key quantization method and the corresponding value quantization method of the value cache.

2. The key-value caching quantization method as described in claim 1, characterized in that, The determination of the saliency score corresponding to the key channel based on the sensitivity score corresponding to the key channel in the large model and the importance score of the current query term on the key channel includes: Identify the key channels within the current buffer of the large model, and determine the sensitivity scores corresponding to the key channels; Determine the activation value of the current query term on the key channel, and determine the importance score of the current query term on the key channel based on the activation value; The significance score corresponding to the key channel is calculated based on the product of the sensitivity score and the importance score.

3. The key-value caching quantization method as described in claim 2, characterized in that, The process of determining the key channels within the current buffer of the large model and determining the sensitivity scores corresponding to the key channels includes: Determine the key channel within the current buffer of the large model, and determine the set of numerical values ​​for all lexical terms on the key channel; The sensitivity score corresponding to the key channel is determined based on the set of values ​​and the target quantization bit width.

4. The key-value caching quantization method as described in claim 2, characterized in that, Determining the activation value of the current query term on the key channel, and determining the importance score of the current query term on the key channel based on the activation value, includes: Determine the sequence length of the query term in the current buffer, and determine the activation value of the current query term on the key channel; The importance score of the current query term on the key channel is calculated based on the sequence length and the activation value.

5. The key-value caching quantization method as described in any one of claims 1 to 4, characterized in that, The step of determining the criticality of the key channel based on the saliency score, and determining the key quantization method corresponding to the key cache based on the criticality, includes: The criticality of the key channel is determined based on the significance score, the first threshold, and the second threshold. When the criticality level is critical channel, the key quantization method corresponding to the key cache is determined to be no quantization; In the case where the criticality level is medium criticality, the key quantization method corresponding to the key cache is determined to be the first asymmetric quantization method; When the criticality level is non-critical channel, the key quantization method corresponding to the key cache is determined to be the second asymmetric quantization method.

6. The key-value caching quantization method as described in claim 5, characterized in that, The determination of the criticality of the key channel based on the significance score, the first threshold, and the second threshold includes: If the significance score is greater than the first threshold, the key channel is determined to be a key channel. If the significance score is less than or equal to the first threshold and greater than the second threshold, the key channel is determined to be a medium key channel. If the significance score is less than the second threshold, the key channel is determined to be a non-key channel.

7. The key-value caching quantization method as described in any one of claims 1 to 4, characterized in that, After performing key-value cache quantization according to the key quantization method and the value quantization method corresponding to the value cache, the process further includes: After the key-value cache quantization is completed in the current buffer of the large model, the current buffer is cleared to obtain a new buffer; After the new buffer is filled with terms, the step of determining the saliency score of the key channel based on the sensitivity score of the key channel in the large model and the importance score of the current query term on the key channel is returned to iterate and perform key-value cache quantization on all buffers.

8. A key-value cache quantization device, characterized in that, The key-value cache quantization device includes: The score determination module is used to determine the significance score of the key channel based on the sensitivity score of the key channel in the large model and the importance score of the current query term on the key channel; The quantization method determination module is used to determine the criticality of the key channel based on the significance score, and to determine the key quantization method corresponding to the key cache based on the criticality. The key-value cache quantization module is used to perform key-value cache quantization according to the key quantization method and the value quantization method corresponding to the value cache.

9. A key-value caching quantization device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the key-value cache quantization method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the key-value cache quantization method as described in any one of claims 1 to 7.

Images