Quantization method and computing kernel of large language model key-value cache

Abstract

The application provides a quantization method for a large language model key-value cache, comprising: obtaining key-value cache activation values of each layer attention module in a large language model inference process, establishing a sliding window along a word time dimension, and determining a window statistical quantity index; in a quantization writing process in an inference stage, online recursive updating is performed on zero point states and scaling factor states of each channel, and the key-value cache activation values are quantized into low-bit code words and written into a cache compression, so that the quantization code words are implicitly embedded in the recursive process; in an attention calculation process in the inference stage, the quantization code words are read from the compressed key-value cache data, the zero point states and the scaling factor states corresponding to each historical word are recovered according to the recursive rules which are isomorphic to the quantization writing process, the quantization code words are dequantized, and the dequantized data is directly used for attention calculation. The method reduces the memory overhead and bandwidth pressure caused by explicit metadata storage.

Description

Technical Field

[0001] This invention relates to the field of data compression and decompression processing technology, and in particular to a quantization method and computation kernel for key-value caching of large language models. Background Technology

[0002] Inference in large language models is divided into two stages: pre-filling and generation. In the pre-filling stage, the large language model processes the user-input prompts in one go and predicts the next lexical unit. In the generation stage, the large language model uses the lexical units generated in the previous step as input and predicts the next lexical unit using a neural network and a Transformer attention mechanism. During this process, a key-value cache can store historical keys and values ​​from the attention mechanism to avoid unnecessary redundant computation. However, modern large language models can handle millions of historical lexical unit contexts, requiring a large amount of key-value cache data to be repeatedly read and written between memory and computing devices with limited data bandwidth, severely limiting the inference throughput of large language models.

[0003] Existing technologies propose a method and system for key-value cache compression and decompression based on static grouped quantization. This method includes: quantizing the key cache in groups of K terms per channel; quantizing the value cache in groups of K channels per term; independently calculating and storing the zeros and scaling factors of each quantization group's quantization interval; and reading the quantized values ​​of the corresponding group, along with the zeros and scaling factors, for dequantization during the dequantization stage. Different grouping strategies are adopted for the key-value cache data because the key cache contains obvious outlier channels, while the value cache does not. This static grouped quantization ensures that no significant quantization noise is introduced into the key-value cache, while reducing the data bandwidth requirements of the memory for reading and writing key-value cache data, ultimately improving the inference throughput of large language models. However, the aforementioned traditional static grouping quantization method has two obvious technical problems and defects: First, static grouping quantization requires independent grouping of key-value cache data. Each quantization group needs to independently calculate and store the zero point and scaling factor of the quantization interval. These additional metadata (zero point and scaling factor) occupy part of the space of the quantized data, resulting in the effective quantization bit depth being greater than the theoretical quantization bit depth. Second, if the computation and storage overhead of metadata is reduced by increasing the amount of data in the quantization group, the rotation position encoding introduces sinusoidal numerical changes into the key cache data, increasing the dynamic range of the quantization interval. This will introduce significant quantization noise into the key-value cache, ultimately deteriorating the reasoning ability of large language models. Summary of the Invention

[0004] This invention provides a quantization method and computation kernel for key-value caching of large language models. The zero point and scaling factor are implicitly embedded in the quantization sequence, eliminating the need to explicitly store group statistics. This decouples metadata overhead from context length and can significantly reduce the overall space occupied by key-value cache data.

[0005] This invention also provides a quantization method for key-value caching of large language models, comprising:

[0006] The key-value cache activation values ​​of each attention module in the inference process of the large language model are obtained, a sliding window is established along the lexical time dimension, and the window statistics indicators are determined. The window statistics indicators include the window temporal zero point index and the window temporal scaling factor index.

[0007] Based on the window statistics index, calibration samples are used to learn the quantization parameters of each layer, each attention point, and each channel. The quantization parameters include at least the zero-point state, scaling factor state, zero-point update coefficient, and scaling factor update coefficient.

[0008] During the quantization writing process in the inference phase, the zero-point state and scaling factor state of each channel are updated online recursively, and the key-value cache activation value is combined with the quantization into low-bit codewords and written to the cache for compression, so that the quantized codewords are implicitly embedded in the recursive process.

[0009] During the attention calculation process in the inference phase, quantized codewords are read from the compressed key-value cache data. Based on the recursive rules that are isomorphic to the quantization writing process, the zero-point state and scaling factor state corresponding to each historical word are restored. The quantized codewords are dequantized, and the dequantized data is directly used in the attention calculation.

[0010] In one embodiment of the present invention, the key-value cache activation values ​​of each attention module in the large language model inference process are obtained, a sliding window is established along the lexical time dimension, and window statistics indicators are determined. These window statistics indicators include window temporal zero-point indicators and window temporal scaling factor indicators, specifically including:

[0011] In the autoregressive inference process of the large language model, the input sequence is pre-filled and generated, and the corresponding key-value cache is stored in each layer;

[0012] For any attention head and hidden dimension, construct the corresponding channel time series to represent the key-value cache activation value corresponding to each word. For each attention module, extract key cache data and value cache data according to the word generation order.

[0013] A sliding window is established along the lexical generation time dimension. The local mean of the activation value in each window is calculated as the window zero point, and the average absolute deviation of the activation value in each window relative to the window zero point is calculated as the window scaling factor.

[0014] The window zero point and the window scaling factor are recursively estimated using an exponential moving average model to obtain the prediction error of the window zero point and the prediction error of the window scaling factor.

[0015] Calculate the window zero-point prediction error and the window scaling factor prediction error, and define the window temporal zero-point index and the window temporal scaling factor index based on the window zero-point prediction error and the window scaling factor prediction error, respectively.

[0016] In one embodiment of the present invention, the step of calibrating and learning the quantization parameters of each layer, each attention point, and each channel using calibration samples based on the window statistics index includes:

[0017] Obtain the target model configuration, quantization configuration, and running mode. The target model configuration includes the number of layers, number of attention heads, single-head dimension, and positional encoding method of the target model. The quantization configuration includes the quantization bit width, quantization upper and lower bounds, and scaling factor boundaries.

[0018] Construct a quantization parameter index table, a cache layout table, and a channel mapping table; allocate parameter storage locations for each layer, each key-value header, and each channel; and establish a mapping relationship between layer number, attention header number, channel dimension number, parameter address, cache address, page number, and offset.

[0019] Determine whether a calibration sample set corresponding to the target model already exists. If not, sample continuous word sequences from the calibration corpus and perform forward propagation of the target model on the continuous word sequences to obtain the query tensor, key tensor, value tensor, and attention mask corresponding to each layer.

[0020] Perform a differentiable adaptive quantization forward process on the calibration sample: Divide a certain cache tensor in a certain layer of the calibration sample into several sub-blocks according to a preset truncation length. For each sub-block, each word, and each channel, perform residual generation, codeword quantization, reconstruction, and state update in sequence. The state update is performed based on the quantization parameters to be learned.

[0021] For each calibration sample, the reference attention output is calculated in the full-precision path, the reconstructed attention output is calculated in the quantization path, and the attention perception loss is calculated based on the difference between the two.

[0022] The quantization parameters are updated based on the attention perception loss, and a valid parameter table is output when the convergence condition is met.

[0023] In one embodiment of the present invention, before performing a differentiable adaptive quantization forward process on the calibration sample, the method further includes:

[0024] Initialize the quantization parameters for each layer, each head, and each channel;

[0025] The initial quantization parameters are mapped to valid parameters according to the preset reparameterization relationship, so as to ensure that the scaling factor state is positive, the zero-point update coefficient is in the preset stable range, and the scaling factor update coefficient is in the preset update rate range.

[0026] In one embodiment of the present invention, updating the quantization parameter based on the attention perception loss includes:

[0027] The back-inverse gradient of the rounding operation is approximated by a pass-through estimator. The long sequence is divided into multiple sub-blocks by truncating backpropagation. Only the forward state is retained at the boundaries of the sub-blocks while the gradient propagation is truncated.

[0028] The quantization parameters are updated using an optimizer based on the gradient. If a preset number of rounds is reached or the loss meets the convergence condition, a valid parameter table is output and written to the corresponding storage location to generate the kernel configuration.

[0029] In one embodiment of the present invention, the online recursive update of the zero-point state and scaling factor state of each channel, combined with the quantization of the key-value cache activation value into a low-bit codeword for buffer compression, includes:

[0030] Receive user requests and generate input word sequences, and create the corresponding session's cache context, page table structure, layer state descriptor, and write pointer;

[0031] The model forward pass is performed during the pre-filling phase to obtain the query, key, and value tensors of each layer. For input prompts that are longer than the preset length, the process is divided into blocks according to the preset block length. The quantization recursion is performed on the tokens in each block in natural order.

[0032] Perform adaptive quantization on the newly generated key and value tensors, including:

[0033] Read the zero-point state and scaling factor state of the current channel at the current moment, and calculate the difference between the activation value of the key cache or value cache of the current word and the current zero-point state as the residual;

[0034] The residual is subjected to integer codeword generation to obtain quantized codewords;

[0035] The reconstructed residual is obtained based on the quantized codeword and the current scaling factor, and the zero-point state at the next moment is updated. The scaling factor state at the next moment is updated based on the current scaling factor, the scaling factor update coefficient, and the deviation between the absolute value of the quantized codeword and the scale offset threshold.

[0036] Write the quantization codeword into the quantization cache.

[0037] In one embodiment of the present invention, before performing adaptive quantization on the newly generated key tensor and value tensor, the method further includes:

[0038] Determine whether the current word is the first word of the session, the first word of the page, or the first word of the block, or whether the system has been rebuilt due to sliding window elimination, segment migration, or abnormal recovery;

[0039] If any of the above conditions are met, the zero point state and scaling factor state of the current channel will be set to the initial zero point state, the initial scaling factor state, or the most recent checkpoint state.

[0040] In one embodiment of the present invention, the step of generating an integer codeword from the residual to obtain a quantized codeword includes:

[0041] The residual is divided by the current scaling factor and then rounded to the nearest integer. It is then clipped to between the lower and upper bounds of the quantization interval to obtain the quantized codeword.

[0042] In one embodiment of the present invention, after performing integer codeword generation on the residual, the method further includes:

[0043] Determine whether the pruning ratio of several consecutive words exceeds the preset pruning ratio. If it does, execute one of the following branch processes: temporarily increase the quantization bit width of the current channel; write the current word to the outlier bypass buffer; shorten the update frequency and reinitialize the scaling factor state; or return to full-precision or high-precision block storage.

[0044] In one embodiment of the present invention, updating the zero-point state at the next moment includes:

[0045] The reconstruction residual is obtained by multiplying the quantized codeword by the current scaling factor, and the zero-point state at the next moment is updated based on the reconstruction residual and the zero-point update coefficient.

[0046] The update of the scaling factor state for the next time step includes:

[0047] The difference between the absolute value of the quantized codeword and the scale offset threshold is calculated as the deviation. The scaling factor state at the next moment is updated based on the current scaling factor, the scaling factor update coefficient, and the deviation. The updated scaling factor state is then clipped to the scaling factor boundary range.

[0048] In one embodiment of the present invention, writing the quantized codeword into the quantization buffer for compression includes:

[0049] The codeword with a preset number of bits is compressed into the storage unit. If the remaining space of the current cache page is insufficient, a new page is allocated and the page table is updated. If the current word is the end of a block but the packing unit is not full, zero codewords are written for unused bits, codewords are reserved, or the codewords are delayed until the next word to be merged and written.

[0050] In one embodiment of the present invention, the step of restoring the zero-point state and scaling factor state corresponding to each historical lexical unit according to a recursive rule isomorphic to the quantization writing process, performing dequantization on the quantized codeword, and directly participating the dequantized data in attention calculation includes:

[0051] The quantized key cache is scanned in the order of historical lexical terms and online dequantization is performed: For each historical position, the target storage unit is located according to the page table and packing index, the corresponding bit segment is extracted according to the parity of the lexical terms or the packing position, the integer codeword is recovered, and the dequantized key value is calculated to be equal to the integer codeword multiplied by the current scaling factor plus the current zero point state. Local point accumulation is performed on the channel segments responsible for by each thread in the same thread block to obtain the local sum. The attention log score of the historical position is obtained by reduction within the thread block, and the zero point state and scaling factor state in the register are updated according to the current codeword to ensure that the dequantization state of the next historical position is consistent with that of the writing end.

[0052] Perform causal masking, scaling, and normalization operations on the attention log score to obtain the weight vector of the attention score;

[0053] The value cache is scanned using the same recursive method as the key cache. For each historical position, the quantized value cache codeword is read and unpacked, dequantized, and then multiplied and accumulated with the attention score to obtain the context vector of the current head.

[0054] In one embodiment of the present invention, before scanning the quantized key cache in historical lexical order and performing online dequantization, the method further includes:

[0055] The generation phase receives the query vector of the current layer and starts query-value fusion dequantization calculation, and selects one or more kernel instances based on batch size, number of heads, history length, packing factor, page table layout and hardware topology;

[0056] For each attention head pair, the query vector and the initial zero-point state, initial scaling factor state, zero-point update coefficient, scaling factor update coefficient, and scale offset threshold of the corresponding channel are loaded into a register or on-chip cache. If the most recent checkpoint is used, the checkpoint state is loaded in addition and used as the starting point for recursion.

[0057] In one embodiment of the present invention, it further includes:

[0058] The context vectors of each head are concatenated or projected back to the model dimension, and the attention output projection, residual connection, normalization and feedforward network of the subsequent layers are continued until the last layer is completed, the log probability distribution of the current step is obtained, and the next word is obtained according to the preset decoding strategy.

[0059] In one embodiment of the present invention, it further includes:

[0060] During inference, when the video memory usage reaches a threshold, cold pages are migrated to host memory, solid-state drives, or remote nodes.

[0061] When using sliding windows or segmented attention, discard the oldest page and update the page table synchronously.

[0062] When a checkpointing mechanism is used, the zero-point state and scaling factor state are stored once every preset number of tokens.

[0063] In one embodiment of the present invention, the step of directly participating the dequantized data in attention calculation includes:

[0064] Within a thread block, state recovery, dequantization, and vector dot product operations are performed sequentially according to the lexical time order. Each thread block is responsible for the dot product operations of a preset number of channels.

[0065] The dequantized data directly participates in the attention matrix-vector multiplication in the on-chip cache, without generating a separate intermediate copy in full-precision format;

[0066] After the current lexical unit completes dequantization, the updated zero-point state and scaling factor state are immediately passed to the next lexical unit.

[0067] Another aspect of the present invention provides a fusion quantization key-value cache attention computation kernel, deployed within a GPU stream processor, AI accelerator array, or dedicated tensor computation unit, the kernel comprising:

[0068] The key-value cache unit is used to store the quantization sequence corresponding to the historical tokens of each channel of each attention head in each layer in a compressed format, as well as the quantization parameters corresponding to each channel.

[0069] The L0 instruction cache is used to store kernel-executed instructions;

[0070] A thread bundle scheduler, connected to the L0 instruction cache, is used to schedule thread bundles based on batch size, number of heads, history length, and page table layout.

[0071] The distribution unit, connected to the thread bundle scheduler, is used to distribute the scheduled thread bundles to the corresponding arithmetic logic unit array.

[0072] The register file is used to temporarily store the query vector block, current zero-point state, current scaling factor state, and current token dequantization result of each thread in each thread bundle at runtime, wherein the current zero-point state and the current scaling factor state are dynamically updated as the token recursion process proceeds.

[0073] An array of arithmetic logic units, each arithmetic logic unit corresponding to a channel, is connected to the key-value cache unit and the register file, and is used to perform the following operations:

[0074] The quantization codeword corresponding to the current historical position and the quantization parameters of the corresponding channel are read from the key-value cache unit. The current zero-point state and current scaling factor state of the current channel at the current historical position are restored according to the adaptive incremental recursive process that is isomorphic to the quantization writing process. The quantization codeword is dequantized according to the current zero-point state and current scaling factor state, and the dequantization result is temporarily stored in the register file. The current zero-point state and current scaling factor state in the register file are updated.

[0075] The arithmetic logic unit array is executed sequentially according to the word time order.

[0076] As can be seen from the above solutions, the advantages of the present invention are:

[0077] The quantization method for key-value caching in large language models provided by this invention analyzes the statistical continuity of key-value caching in the time dimension during the inference process of large language models, constructs window temporal zero points and window temporal scaling factor indices, and realizes dynamic traceability modeling of the statistical characteristics of key-value caching. Furthermore, based on the exponential moving average mechanism, the zero point state and scaling factor state are updated online recursively, and the quantization metadata is implicitly embedded in the low-bit quantization sequence, thereby reducing the memory overhead and bandwidth pressure caused by explicit metadata storage in traditional static group quantization methods. Attached Figure Description

[0078] Figure 1 This is a flowchart illustrating the quantization method for key-value caching of a large language model provided in Embodiment 1 of the present invention.

[0079] Figure 2 The smooth change characteristics of key-value cache statistics in different layers of the LLaMA-2-7B model over time in the Wikipedia dataset;

[0080] Figure 3 for Figure 1 A detailed flowchart of step S2 is shown below;

[0081] Figure 4 for Figure 1 A detailed flowchart of step S3 is shown below;

[0082] Figure 5 for Figure 1 A detailed flowchart of step S4 is shown below;

[0083] Figure 6 This is a schematic diagram of the overall structure of the fused quantization key-value cache attention computation kernel provided in Embodiment 2 of the present invention;

[0084] Figure 7 A block diagram of a computing system that enables the implementation of several embodiments of the present disclosure.

[0085] The attached figures are labeled as follows:

[0086] 400: Computational kernel;

[0087] 410: Key-value cache unit;

[0088] 420: L0 instruction cache;

[0089] 430: Thread bundle scheduler;

[0090] 440: Distribution Unit;

[0091] 450: Register file;

[0092] 460: Arithmetic Logic Unit Array;

[0093] 500: Computing system;

[0094] 510: Memory module;

[0095] 520: Processor and Interaction Interface;

[0096] 530: Computational kernel. Detailed Implementation

[0097] It should be noted that, in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0098] In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0099] To further improve the effectiveness of quantization methods, it is necessary to reduce the metadata overhead generated by quantization while ensuring that no significant quantization noise is introduced into the key-value cache. This invention observes that key-value cache data possesses a natural temporal structure, and an adaptive incremental algorithm can be applied to dynamically track the zeros and scaling factors required for quantization. By mimicking the behavior of the dequantizer through the quantizer, the zeros and scaling factors are implicitly embedded into the quantized data, achieving extremely low metadata overhead. Simultaneously, this adaptive quantization algorithm can effectively adapt to the sinusoidal variation trend of the key-value cache data, reducing quantization noise in the key-value cache to maintain the inference capability of large language models.

[0100] refer to Figure 1 As shown, Figure 1 The diagram illustrates a specific flowchart of a quantization method for key-value caching of a large language model according to an embodiment of the present invention.

[0101] A quantization method for key-value caching of large language models includes the following steps:

[0102] Step S1: Obtain key-value cache data of each attention module in the large language model inference process, establish a sliding window along the lexical time dimension, and determine the window statistics indicators, which include the window temporal zero point index and the window temporal scaling factor index.

[0103] Step S2: Based on the window statistics index, use calibration samples to calibrate and learn the quantization parameters of each layer, each attention point, and each channel. The quantization parameters include at least the zero-point state, scaling factor state, zero-point update coefficient, and scaling factor update coefficient.

[0104] Step S3: During the quantization writing process in the inference phase, the zero-point state and scaling factor state of each channel are updated online recursively, and the key-value cache activation value is quantized into a low-bit codeword and written to the cache for compression, so that the quantized codeword is implicitly embedded in the recursive process.

[0105] Step S4: During the attention calculation process in the inference phase, quantized codewords are read from the compressed key-value cache data. Based on the recursive rules that are isomorphic to the quantization writing process, the zero-point state and scaling factor state corresponding to each historical word are restored. The quantized codewords are dequantized, and the dequantized data is directly used in the attention calculation.

[0106] In a preferred implementation, in order to analyze the continuous change characteristics of key-value cache statistics in the time dimension during the inference process of a large language model, the Windowed Temporal Zero-Point (WTZP) and Windowed Temporal Scaling Factor (WTS) indices are constructed to quantify the smoothness of key-value cache statistics in the time dimension, thereby providing a basis for the design of the subsequent adaptive differential quantizer.

[0107] In the autoregressive inference process of the large language model, the input sequence is pre-filled and generated, and the corresponding key-value cache is stored at each layer, forming key-value cache data that increases along the time dimension. During inference, the KV cache stores the key tensors and value tensors generated by each attention head at each layer.

[0108] For any attention head h and hidden dimension d, construct the time series of the corresponding channels. This represents the key-value cache activation value corresponding to the t-th lexical unit. Since the key-value cache activation values ​​are generated sequentially according to the lexical unit order, the above sequence has a natural temporal order structure.

[0109] Establish a sliding window of length w along the lexical time dimension By using a sliding window approach, the dynamic changes in key-value cache activation value statistics over a local time range can be analyzed.

[0110] For each time window, the local mean of the activation values ​​within the window is calculated as the window zero point. Due to rotational position encoding and long context accumulation effects, some channels will drift slowly with the word position. Therefore, this local zero point can reflect the dynamic changes of channel distribution over time.

[0111] Furthermore, for each window, a local dynamic range statistic is calculated as a window scaling factor. The window scaling factor is used for subsequent analysis to quantify the temporal continuity of the scaling factor.

[0112] To analyze whether the window zero point and window scaling factor can be dynamically predicted, an exponential moving average tracking model is introduced to recursively estimate the window zero point and window scaling factor respectively. For the window zero point: ,in This is the zero-point update coefficient. For the window scaling factor: ,in The scaling factor updates the coefficients. This exponential moving average tracking model is used to simulate the dynamic state tracking mechanism in the subsequent quantizer.

[0113] To measure whether the window zero point has smooth temporal continuity, the window zero point prediction error is calculated. T represents the statistical duration used in the prediction error calculation, i.e., the total number of tokens; the window temporal zero-point index is defined. When WTZP is close to 1, it indicates that the window zero point changes smoothly, has strong time continuity, and can be accurately predicted by a low-overhead dynamic tracker.

[0114] Similarly, establish the window scaling factor prediction error for the window scaling factor. Further define the window temporal scaling factor index When WTS is close to 1, it indicates that the scaling factor changes slowly and continuously over time, making it suitable for dynamic maintenance using a recursive update method, without needing to explicitly store the quantization parameters for each window.

[0115] refer to Figure 2 As shown, Figure 2 The diagram illustrates the smooth variation of key-value cache statistics across different layers of the LLaMA-2-7B model on the Wikipedia dataset over time. The horizontal axis represents the model layer number (layer 0 to layer 30); the vertical axis represents the values ​​of the Window Temporal Zero Point Index (WTZP) and the Window Temporal Scaling Factor Index (WTS). Figure 2 Statistical analysis of key-value caches at different layers, with different attention heads, and with different context lengths revealed that: the window zero point exhibits significant low-frequency drift characteristics; the window scaling factor changes relatively smoothly over time; WTZP and WTS are significantly higher in the original word order than after random shuffling; and high temporal continuity is maintained even in long context scenarios. Therefore, key-value cache data statistics possess a predictable and traceable temporal structure, enabling the use of dynamic recursive quantization mechanisms to replace explicit metadata storage in traditional static grouping quantization methods.

[0116] For each channel, a zero-point state variable and a scaling factor state variable are maintained to track the mean drift and amplitude changes of the current channel, respectively. The zero-point state is used to compensate for the sinusoidal dynamic offset of the key-value buffer data, and the scaling factor state is used to adaptively adjust the quantization range.

[0117] In a preferred implementation, refer to Figure 3 As shown, Figure 3 A schematic diagram of the specific process for step S2 is shown.

[0118] Based on the aforementioned window statistics, calibration learning is performed on the quantization parameters of each layer, each attention focus, and each channel using calibration samples. Specifically, this includes the following steps:

[0119] Step S205: Obtain the target model configuration, quantization configuration, and operating mode. Specifically, the target model configuration includes the number of layers l, the number of attention heads h, the single-head dimension d, the positional encoding method, and the deployment hardware type; the quantization configuration includes the quantization bit width b, the quantization upper and lower bounds, and the scaling factor boundaries.

[0120] Step S210 involves constructing a quantization parameter index table, a cache layout table, and a channel mapping table. Specifically, parameter storage locations are allocated for each layer, each key-value header, and each channel, and a mapping relationship is established from (l, h, d) to parameter address, cache address, page number, and offset. Separate index tables can be created for keys and values. For shared parameter scenarios, multiple channels can share some parameters. Furthermore, the parameter table uses a contiguous memory layout to support vectorized loading. Cache pages use a fixed-page-length or variable-page-length layout to be compatible with paginated key-value management frameworks.

[0121] Step S215: Determine whether a calibration sample set corresponding to the target model already exists. If a pre-prepared calibration sample set already exists, skip step S220. Otherwise, proceed to step S220.

[0122] Step S220: Construct a calibration sample set. Specifically, sample continuous lexical sequences from the calibration corpus and perform forward propagation of the target model on these sequences to obtain the query tensor, key tensor, value tensor, and attention mask for each layer. The continuous lexical sequences preferably retain their original temporal order to preserve the statistical continuity of the key-value cache along the lexical dimension. Optionally, the calibration sample length can be set to 512, 1024, 2048, 4096, or longer, or dynamically set according to the maximum context length supported by the model.

[0123] Step S225: Initialize the parameters to be learned and perform constraint reparameterization. For each layer, each head, and each channel, initialize the quantization parameters to obtain the initial zero-point state. Initial scaling factor state Zero-point update coefficient Scaling factor update coefficient And the scale offset threshold T. The initial quantization parameters are mapped to effective parameters according to a preset reparameterization relationship. Through the above reparameterization, it can be guaranteed that... , Located within the preset stable range, It is located within the preset update rate range, thereby enhancing the numerical stability of the system.

[0124] Step S230: Perform a differentiable adaptive quantization forward process on the calibration sample. Specifically, let X be a cache tensor of a certain layer in the calibration sample, and divide it into several sub-blocks with a truncation length of k. For each sub-block, each word, and each channel, perform residual generation, codeword quantization, reconstruction, and state update sequentially. The key-value cache activation value of a certain channel for the t-th word is then calculated. First calculate the residuals Then, integer codeword generation is performed on the residual to obtain the quantized codeword. Here, Round represents rounding to the nearest integer, and Clamp represents clipping to the quantization interval (L, U), where L and U are the lower and upper bounds of the quantization interval, respectively. The reconstructed residual is then calculated based on the quantized codewords. And update the zero-point state for the next moment: ; Calculate the deviation And update the scaling factor state for the next time step. ,in These represent the minimum and maximum values ​​of the scaling factor state. In one embodiment of the invention, the scaling factor is updated in the logarithmic domain to approximate the multiplicative update into an additive update.

[0125] Step S235: Perform truncated backpropagation and gradient estimation. Since the quantization rounding operation is non-differentiable, in one embodiment of the invention, a pass-through estimator (STE) is used to approximate the backpropagation gradient of the quantization rounding operation. For the scaling factor state... The same gradient estimation is used. For long sequence recursive states, truncated backpropagation is adopted, dividing the sequence of length T into multiple sub-blocks of length k, and retaining only the forward state at the sub-block boundaries while truncating gradient propagation. This can maintain the correctness of causal recursion while reducing the length of the backpropagation graph and the memory usage.

[0126] Step S240: Calculate the attention perception loss. Specifically, for each calibration sample, calculate the reference attention output along the full-precision path. Calculate the reconstructed attention output under the quantization path Furthermore, a mean squared error loss function is constructed based on the difference between the two. By constraining the attention output rather than just the individual key-value reconstruction error, the final attention calculation accuracy can be optimized more directly.

[0127] Step S245 involves parameter updating, convergence judgment, and deployment and solidification. Specifically, based on the gradients obtained in steps S235 and S240, the original parameters are updated using Adam, SGD, AdamW, or other optimizers. Gradient pruning, learning rate annealing, or early stopping judgment is performed if necessary. If a preset number of rounds is reached or the loss meets the convergence condition, a valid parameter table is output; otherwise, the process returns to step S230 to continue iteration. In one embodiment of the invention, several consecutive samples, several rounds of training, a preset truncation length k, and a preset learning rate are used for parameter learning, but the invention is not limited to a specific number of training rounds or learning rate.

[0128] Step S250: Write the final quantization parameter table to the deployment storage and generate the kernel configuration. Specifically, the initial zero-point state is configured by layer, by header, and by channel. Initial scaling factor state Zero-point update coefficient Scaling factor update coefficient The scale offset threshold T is stored in a location that can be efficiently accessed by the device, such as contiguous memory, constant memory, texture cache, on-chip static random access memory, or static random access memory in a field-programmable gate array. Simultaneously, kernel-level configuration, packaging rules, page table format, and communication routing information are generated.

[0129] In a preferred implementation, refer to Figure 4 As shown, Figure 4 A schematic diagram of the specific process for step S3 is shown.

[0130] During the quantization write process in the inference phase, the zero-point state and scaling factor state of each channel are updated online recursively, and the key-value cache activation value is quantized into a low-bit codeword and written to the cache for compression, so that the quantized codeword is implicitly embedded in the recursive process. Specifically, it includes the following steps:

[0131] Step S305: Receive the user request and generate an input lexical sequence. Specifically, encode the original text, code, image / text input, or other modal input to obtain the input lexical sequence, and create the corresponding session's cache context, page table structure, layer state descriptor, and write pointer. If it is a new session, the initial state of each layer and channel of that session is set to the initial zero-point state. Initial scaling factor state If the session is resumed from a breakpoint, it can start from the most recent checkpoint state.

[0132] Step S310: Perform the model forward pass in the pre-filling phase to obtain the query, key, and value tensors for each layer. For input prompts longer than 1, the model execution module can execute in blocks of length k to reduce peak memory usage. For each lexical unit within a block, quantization recursion can be performed in natural order. If page-based caching is used, one or more cache pages are allocated for each block, and the page number, block offset, and session number are recorded.

[0133] Step S315: Perform adaptive quantization on the newly generated key tensor and value tensor. Let the current layer be l, the current sequence be n, the absolute position of the current term in the layer's cache be t, and the current key or value element be... First, read the current status of the corresponding channel. If the term is a conversation start term, page start term, block start term, or the system is rebuilt due to sliding window eviction, segment migration, or abnormal recovery, then the current state can be set to the initial state. Or the status of the most recent checkpoint. The residuals are then calculated. .

[0134] Step S320: Perform integer codeword generation on the residual to obtain quantized codewords. Specifically, calculate... .like If the value exceeds the representable range, it will be automatically pruned to the boundary codeword. If the pruning ratio of several consecutive words exceeds a preset threshold, it is determined that the current channel or the current block is at risk of overload. In this case, branch processing can be performed: first, temporarily increase the bit width of the channel; second, write the current word into the outlier bypass buffer; third, shorten the update frequency and reinitialize. Fourth, return to full-precision or high-precision block storage. If no such anomaly exists, proceed to step S325.

[0135] Step S325: Update the zero-point state and scaling factor state of the current channel. Based on the quantization codeword... Calculate the reconstruction residual And update the zero-point state for the next moment: ; Calculate the deviation And update the scaling factor state for the next time step. ,in These represent the minimum and maximum values ​​of the scaling factor state. If the system uses a fixed-point implementation, then... , , Quantization is performed to a preset format and implemented via shift addition or a digital signal processing unit. If the system uses floating-point implementation, updates can be completed in FP16, BF16, or FP32. To prevent state drift, numerical guard rules can be set, for example, when... season ,when When the threshold is exceeded, either limiting or normalization is applied.

[0136] Step S330: Write the codewords to the quantization cache and maintain the index. In one embodiment of the invention, taking b=4 as an example, the codewords of two terms can be compressed into the same byte. If t is even, the lower four bits are written; if t is odd, the higher four bits are written. More generally, 8 / b codewords can be written to one byte, 16 / b codewords to one half-word, or 32 / b codewords to one word. If the current cache page has insufficient remaining space, a new page is allocated and the page table is updated. If the current term is at the end of a block but the packing unit is not full, unused bits can be written as zero codewords, reserved codewords, or deferred to the next term for merging and writing.

[0137] Step S335: Determine whether pre-padding or append writing is complete. If there are still unprocessed tokens in the current block, return to step S315. If all layers have completed quantization writing, proceed to the generation stage.

[0138] In a preferred implementation, in a preferred implementation, refer to Figure 5 As shown, Figure 5 A schematic diagram of the specific process for step S4 is shown.

[0139] During the attention calculation process in the inference phase, quantized codewords are read from the compressed key-value cache data. Based on the recursive rules isomorphic to the quantization writing process, the zero-point state and scaling factor state corresponding to each historical lexical unit are restored. The quantized codewords are then dequantized, and the dequantized data is directly used in the attention calculation. Specifically, the process includes the following steps:

[0140] Step S405: The generation phase receives the query vector of the current layer and initiates query-value fusion dequantization calculation. Specifically, for the currently generated tokens in layer l, the model execution module outputs q. The fusion kernel scheduling module selects one or more kernel instances based on batch size, number of heads, history length T, packing factor p, page table layout, and hardware topology. In the GPU implementation, a thread block is allocated for each (n, h) pair. If the dimension d is greater than the number of channels that a single thread block can cover, multiple thread blocks can be allocated for a head and reduced within the block.

[0141] Step S410: Load the query vector and channel parameters into registers or on-chip cache. Specifically, for each (n, h) pair, load the initial zero-point state of the corresponding channel. Initial scaling factor state Zero-point update coefficient Scaling factor update coefficient The scale offset threshold T is loaded into the register array. If the most recent checkpoint is used, the checkpoint state is additionally loaded. And use this as the starting point for recursion. For cases where the dimension d is not divisible by the thread block size, this can be handled by disabling the tail thread channel or filling invalid thread channels with zeros.

[0142] Step S415: Scan the quantized key cache in historical word order and perform online dequantization. For historical positions t=0, 1, ..., T-1, first locate the target storage unit based on the page table and packing index, then extract the corresponding bit segment based on word parity or packing position to recover the integer codeword. In one embodiment of the present invention, taking 4-bit signed quantization as an example, unsigned quantization can be mapped to a signed integer in the range [-8, 7] by subtracting the bias 2b-1. The dequantized key value is then calculated. Local point accumulation is performed on the channel segments handled by each thread within the same thread block to obtain the local sum. Then, the attention log score for that historical position is obtained through thread block reduction or tree reduction. Immediately afterwards, without waiting for the entire sequence to end, the local register is updated based on the current codeword. This ensures that the dequantization state of the next historical position is consistent with that of the writing end.

[0143] Step S420 involves performing causal masking, scaling, and softmax normalization on the attention log score obtained in step S415. Specifically, a causal mask, padding mask, window mask, or retrieval constraint mask can be applied to the attention log score, followed by scaling and softmax normalization to obtain the weight vector of the attention score. If the system uses distributed parallelism and historical lexical units are distributed across multiple devices, a local softmax preparation operation can be performed on the local attention log score fragments first, and then the cross-device aggregation required for normalization can be completed through all-reduce, all-gather, or other communication methods.

[0144] Step S425: Scan the value cache using the same recursive method as in step S415, and calculate the attention output. Specifically, for each historical position t, read and unpack the quantized value cache codewords. ,according to Dequantization is performed, and the result is multiplied and accumulated with the attention score obtained in step S420 to obtain the context vector of the current head. The state recursion of the value cache can be the same as that of the key cache, or it can be configured with an independent parameter table and update rate. If a fused attention score-value kernel is used, dequantization, weighted accumulation, and state update can be completed simultaneously in a single scan.

[0145] In this embodiment, during the dequantization phase, the zero point and scaling factor are recursively recovered from the quantization sequence using the same state update rules as in the quantization phase, eliminating the need to store each set of quantization parameters separately. Subsequently, the key-value cache data is reconstructed based on the current state, achieving implicit embedding and recovery of quantization metadata.

[0146] In step S430, the context vectors of each head are concatenated or projected back to the model dimension, and the attention output projection, residual connection, normalization, and feedforward network of subsequent layers are continued until the last layer is completed, obtaining the log probability distribution of the current step. Then, the next word is obtained according to the preset decoding strategy (e.g., greedy search, maximum k, bundle search, or restricted sampling).

[0147] Step S435: Perform cache maintenance and exception handling. Specifically, when the video memory usage reaches a threshold, cold pages can be migrated to host memory, solid-state drives, or remote nodes; when using a sliding window or segmented attention mechanism, the oldest page can be evicted and the page table updated synchronously; when using a checkpointing mechanism, the state can be stored every F tokens. This allows for recursion from the most recent checkpoint during random access. If a page miss, checksum error, or communication timeout occurs during migration, recovery can be achieved using the most recent consistent snapshot, or the process can degrade to replaying the quantized sequence to rebuild the state.

[0148] Step S440: Output the generated result and release or persist the session state. For ending a session, the corresponding page table, parameter handle, and temporary buffer can be released; for sessions that need to be continued, the current write pointer, page table, random seed, and optional state checkpoint can be persisted for subsequent recovery.

[0149] Through the above steps, the writing end and the reading end share the same isomorphic recursive rules, making it unnecessary to explicitly store zero points and scaling factors in groups. Thus, the metadata overhead of key-value caching can be reduced while maintaining quantization accuracy. Furthermore, since state updates continuously track dynamic distribution drift at the lexical granularity, robustness in long context scenarios can be improved.

[0150] In this embodiment, the quantizer is treated as a recursive structure, and the stochastic gradient descent method is used to train and optimize the quantization parameters. By minimizing the mean square error between the attention output before and after quantization, the zero-point tracking rate, scaling factor tracking rate, target quantization magnitude, and initial state parameters are jointly optimized, thereby improving the inference accuracy of the quantized model.

[0151] In one embodiment of the present invention, the dequantization process of key-value cache data is integrated with matrix-vector multiplication calculation during the inference stage, so as to realize the calculation of attention output while dequantizing, thereby reducing the number of GPU memory accesses and improving GPU hardware utilization and large-scale inference throughput.

[0152] Through the aforementioned adaptive incremental quantization mechanism, the zero point, scaling factor, and outlier index information that need to be explicitly stored in the traditional static grouping quantization method are implicitly encoded into the quantization sequence, so that the metadata overhead no longer increases with the context length, thereby significantly reducing the overall space occupation of key-value cache data and improving the efficiency of long context inference.

[0153] Compared to existing technologies, conventional static grouping quantization methods require additional storage of auxiliary metadata such as zero points, scaling factors, and outlier indexes. This metadata overhead increases linearly with the context length. This invention, through a dynamic state tracking mechanism based on exponential moving averages, implicitly embeds zero points and scaling factors into the quantization sequence, eliminating the need for explicit storage of grouping statistics. This decouples metadata overhead from context length, significantly reducing the overall space occupied by key-value cache data. In long context scenarios, metadata overhead is reduced by approximately 12.8 times compared to existing methods. Traditional static grouping quantization uses fixed window statistics, which struggles to adapt to the sinusoidal dynamic distribution drift caused by rotational position encoding, easily leading to quantization error accumulation during long context inference. This invention, through analysis of the temporal correlation of key-value cache data, uses an exponential moving average mechanism to dynamically track zero points and scaling factors, enabling real-time adaptation to changes in activation distribution. This effectively reduces quantization errors in long sequence inference, improving the inference accuracy and stability of large language models in long context tasks.

[0154] Furthermore, it should be noted that in this method implementation, the steps are not limited to a sequential description, and the execution order is not strictly limited. Those skilled in the art can adjust the logical order of the steps according to the actual situation. There is no strict sequential dependency relationship, as long as it does not violate the internal logic of the technical solution.

[0155] Example 2:

[0156] In one embodiment, reference Figure 6 As shown, Figure 6 A schematic diagram of the overall structure of a fused quantized key-value cache attention computing kernel 400 is shown.

[0157] A fusion quantization key-value cache attention computing kernel 400 is deployed within a GPU stream processor, AI accelerator array, or dedicated tensor computing unit. The kernel includes:

[0158] The key-value cache unit 410 is used to store, in a compressed format, the quantization sequence corresponding to the historical tokens of each channel of each attention head in each layer, as well as the quantization parameters corresponding to each channel. In one embodiment, the key-value cache unit 410 is video memory.

[0159] The L0 instruction cache is 420 bytes long and is used to store kernel-executed instructions.

[0160] The thread bundle scheduler 430, connected to the L0 instruction cache, is used to schedule thread bundles based on batch size, number of heads, history length, and page table layout.

[0161] The distribution unit 440 is connected to the thread bundle scheduler and is used to distribute the scheduled thread bundles to the corresponding arithmetic logic unit array.

[0162] Register file 450 is used to temporarily store the query vector block, current zero-point state, current scaling factor state, and current token dequantization result of each thread in each thread bundle at runtime, wherein the current zero-point state and the current scaling factor state are dynamically updated as the token recursion process proceeds.

[0163] Arithmetic logic unit array 460, each arithmetic logic unit corresponding to one channel, is connected to the key-value cache unit and the register file, and is used to perform the following operations:

[0164] The quantization codeword corresponding to the current historical position and the quantization parameters of the corresponding channel are read from the key-value cache unit. The current zero-point state and current scaling factor state of the current channel at the current historical position are restored according to the adaptive incremental recursive process that is isomorphic to the quantization writing process. The quantization codeword is dequantized according to the current zero-point state and current scaling factor state, and the dequantization result is temporarily stored in the register file. The current zero-point state and current scaling factor state in the register file are updated.

[0165] The arithmetic logic unit array is executed sequentially according to the word time order.

[0166] In one embodiment, the L0 instruction cache 420, thread bundle scheduler 430, dispatch unit 440, register file 450, and arithmetic logic unit array 460 are located in the GPU computing unit.

[0167] During the decoding phase, when attention calculation is required for the current term, the compressed key-value cache data of the corresponding historical term is first read from the video memory. This compressed key-value cache data includes: the low-bit quantization residual sequence; the initial zero-point state corresponding to the current channel; the initial scaling factor state corresponding to the current channel; and quantization parameter update coefficients. Because this invention employs an implicit state recursion mechanism, it eliminates the need to read a large amount of additional group scaling factor and zero-point metadata as required by traditional static group quantization methods, thereby reducing video memory bandwidth pressure.

[0168] The thread block recursively restores the quantization state according to the lexical time sequence. Through an adaptive incremental recursive process, the computation kernel can dynamically restore the quantization parameters corresponding to each lexical without accessing additional metadata. After restoring the current state, the computation kernel immediately performs dequantization on the current quantization value. Since both dequantization and state restoration are completed in registers or shared memory, the memory access overhead caused by frequent loading of block metadata in traditional schemes is avoided.

[0169] After dequantizing the current term, the computation kernel no longer writes the reconstruction result back to GPU memory. Instead, it performs matrix-vector multiplication directly in the on-chip cache. Specifically, 32 channels are mapped to thread blocks (32 threads) for dequantization and vector dot product operations. Each thread block uses intra-block reduction to handle the dot product operation of 32 elements. The dequantized data directly participates in attention calculations, instead of generating a separate complete FP16 format copy.

[0170] Because state updates have a time-recursive relationship, thread blocks are executed pipelined in lexical order. After the current lexical completes dequantization, the updated zero-point state and scaling factor state are immediately passed to the next lexical. The next lexical continues with state recovery and fusion quantization key-value cache attention calculation.

[0171] In one embodiment of the present invention, the current zero-point state, the current scaling factor state, the current term dequantization result, and the query vector block are all cached in registers or shared memory. Therefore, the video memory only needs to read the compressed low-bit key-value cache data, without having to repeatedly load large-scale quantization metadata. This significantly reduces the video memory bandwidth pressure and improves the computational utilization of the computing unit.

[0172] Because the fused quantization key-value cache attention computation kernel reduces key-value cache memory usage, metadata memory access overhead, and dequantization intermediate cache overhead, it can support larger batch sizes and longer context lengths under fixed memory conditions.

[0173] Experiments show that, in a 48GB GPU environment, this invention supports larger inference batch sizes compared to traditional static block quantization key-value caching, and achieves higher end-to-end throughput in high-volume inference scenarios. While maintaining model inference capabilities, it reduces metadata overhead by approximately 12.8 times compared to existing methods in long-context scenarios, and achieves approximately 1.96 times higher inference throughput compared to the FP16 baseline.

[0174] In some embodiments, the present invention provides a key-value caching compression and attention computation system based on adaptive differential quantization, such as the overall system of the present invention. Figure 7As shown, the system 500 includes a memory module 510, a processor and interaction interface 520, and a fused quantized key-value cache attention computing kernel 530. The modules are connected through a high-speed on-chip interconnect bus to achieve high-speed transmission and collaborative computing of key-value cache data during the inference process of large language models.

[0175] In some implementations, the memory module includes one or more of the following: cache, on-chip static random access memory, high-bandwidth video memory, double-rate synchronous dynamic random access memory, or non-volatile memory devices.

[0176] The processor is used to perform large language model inference tasks, adaptive differential quantization tasks, and attention computation tasks. The processor may include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processor (TPU), a neural network accelerator, or other dedicated AI processing units that support parallel tensor computation.

[0177] The interactive interface is used to receive text data, inference requests, model configuration parameters, and quantization control commands input by the user, and output model generation results, inference status information, and system operation information. The interactive interface may include a network communication interface, a human-computer interaction interface, a PCIe interface, an Ethernet interface, or a high-speed data exchange interface.

[0178] The fusion quantization key-value cache attention calculation kernel adopts the calculation kernel 400 of the above embodiment 2, which will not be described in detail here.

[0179] In some implementations, the fused quantized key-value cache attention computing kernel is deployed inside a GPU stream processor, AI accelerator array, or dedicated tensor computing unit, enabling low-latency data access through on-chip shared cache and register files.

[0180] The high-speed on-chip interconnect bus is used to transfer key-value cache data, quantization state parameters, and attention intermediate results between the memory module, the processor, and the fused quantization key-value cache attention computing core. The high-speed on-chip interconnect bus includes on-chip networks, NVLink interconnects, AXI buses, on-chip cross switches, or other high-bandwidth, low-latency interconnect structures.

[0181] Therefore, this invention integrates the key-value cache data state recovery, dequantization, and attention calculation processes into a unified data path by fusing the quantized key-value cache attention calculation kernel with a high-speed on-chip interconnect architecture. This reduces the number of GPU memory accesses, lowers off-chip bandwidth pressure, and improves throughput and GPU memory utilization efficiency in long context reasoning scenarios for large language models.

[0182] In summary, this invention proposes a quantization method for key-value caches in large language models based on adaptive incremental quantization. By analyzing the statistical continuity of the key-value cache in the time dimension during the inference process of large language models, it constructs window temporal zero-point and window temporal scaling factor indices to achieve dynamic traceability modeling of the statistical characteristics of the key-value cache. Furthermore, based on the exponential moving average mechanism, it performs online recursive updates of the zero-point state and scaling factor state, and implicitly embeds the quantization metadata into the low-bit quantization sequence, thereby reducing the memory overhead and bandwidth pressure caused by explicit metadata storage in traditional static block quantization methods.

[0183] Meanwhile, this invention further proposes a fusion attention computing kernel, which integrates quantization state recovery, low-bit dequantization, and attention matrix vector multiplication processes into a unified computing pipeline. It achieves low-latency access and pipelined scheduling of key-value cache data through registers, shared caches, and high-speed on-chip interconnect buses, thereby reducing the number of off-chip memory accesses and improving the utilization of computing resources of GPUs and AI accelerators.

[0184] Compared to existing key-value caching quantization techniques, this invention can significantly reduce the metadata overhead and overall memory usage of key-value caching quantization while maintaining the inference accuracy of large language models. It can also improve the throughput and concurrency in long context and large-batch inference scenarios, and has good hardware compatibility, engineering deployability, and large-scale industrial application value. It can be widely used in large language model inference systems, edge AI devices, high-performance computing platforms, and generative artificial intelligence service systems.

[0185] Furthermore, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0186] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of the present invention.

Claims

1. A quantization method for key-value caching of a large language model, characterized in that, include: The key-value cache activation values ​​of each attention module in the inference process of the large language model are obtained, a sliding window is established along the lexical time dimension, and the window statistics indicators are determined. The window statistics indicators include the window temporal zero point index and the window temporal scaling factor index. Based on the window statistics index, calibration samples are used to learn the quantization parameters of each layer, each attention point, and each channel. The quantization parameters include at least the zero-point state, scaling factor state, zero-point update coefficient, and scaling factor update coefficient. During the quantization writing process in the inference phase, the zero-point state and scaling factor state of each channel are updated online recursively, and the key-value cache activation value is combined with the quantization into low-bit codewords and written to the cache for compression, so that the quantized codewords are implicitly embedded in the recursive process. During the attention calculation process in the inference phase, quantized codewords are read from the compressed key-value cache data. Based on the recursive rules that are isomorphic to the quantization writing process, the zero-point state and scaling factor state corresponding to each historical word are restored. The quantized codewords are dequantized, and the dequantized data is directly used in the attention calculation.

2. The method as described in claim 1, characterized in that, The key-value cache activation values ​​of each attention module in the large language model inference process are obtained. A sliding window is established along the lexical time dimension, and window statistics indicators are determined. These window statistics indicators include window temporal zero-point indicators and window temporal scaling factor indicators, specifically including: In the autoregressive inference process of the large language model, the input sequence is pre-filled and generated, and the corresponding key-value cache is stored in each layer; For any attention head and hidden dimension, construct the corresponding channel time series to represent the key-value cache activation value corresponding to each word. For each attention module, extract key cache data and value cache data according to the word generation order. A sliding window is established along the lexical generation time dimension. The local mean of the activation value in each window is calculated as the window zero point, and the average absolute deviation of the activation value in each window relative to the window zero point is calculated as the window scaling factor. The window zero point and the window scaling factor are recursively estimated using an exponential moving average model to obtain the prediction error of the window zero point and the prediction error of the window scaling factor. Calculate the window zero-point prediction error and the window scaling factor prediction error, and define the window temporal zero-point index and the window temporal scaling factor index based on the window zero-point prediction error and the window scaling factor prediction error, respectively.

3. The method as described in claim 1, characterized in that, Based on the aforementioned window statistics, the steps of calibrating and learning the quantization parameters of each layer, each attention point, and each channel using calibration samples include: Obtain the target model configuration, quantization configuration, and running mode. The target model configuration includes the number of layers, number of attention heads, single-head dimension, and positional encoding method of the target model. The quantization configuration includes the quantization bit width, quantization upper and lower bounds, and scaling factor boundaries. Construct a quantization parameter index table, a cache layout table, and a channel mapping table; allocate parameter storage locations for each layer, each key-value header, and each channel; and establish a mapping relationship between layer number, attention header number, channel dimension number, parameter address, cache address, page number, and offset. Determine whether a calibration sample set corresponding to the target model already exists. If not, sample continuous word sequences from the calibration corpus and perform forward propagation of the target model on the continuous word sequences to obtain the query tensor, key tensor, value tensor, and attention mask corresponding to each layer. Perform a differentiable adaptive quantization forward process on the calibration sample: Divide a certain cache tensor in a certain layer of the calibration sample into several sub-blocks according to a preset truncation length. For each sub-block, each word, and each channel, perform residual generation, codeword quantization, reconstruction, and state update in sequence. The state update is performed based on the quantization parameters to be learned. For each calibration sample, the reference attention output is calculated in the full-precision path, the reconstructed attention output is calculated in the quantization path, and the attention perception loss is calculated based on the difference between the two. The quantization parameters are updated based on the attention perception loss, and a valid parameter table is output when the convergence condition is met.

4. The method according to claim 3, characterized in that, Before performing a differentiable adaptive quantization forward process on the calibration sample, the process further includes: Initialize the quantization parameters for each layer, each head, and each channel; The initial quantization parameters are mapped to valid parameters according to the preset reparameterization relationship, so as to ensure that the scaling factor state is positive, the zero-point update coefficient is in the preset stable range, and the scaling factor update coefficient is in the preset update rate range.

5. The method according to claim 3, characterized in that, Updating the quantization parameters based on the attentional perception loss includes: The back-inverse gradient of the rounding operation is approximated by a pass-through estimator. The long sequence is divided into multiple sub-blocks by truncating backpropagation. Only the forward state is retained at the boundaries of the sub-blocks while the gradient propagation is truncated. The quantization parameters are updated using an optimizer based on the gradient. If a preset number of rounds is reached or the loss meets the convergence condition, a valid parameter table is output and written to the corresponding storage location to generate the kernel configuration.

6. The method according to claim 1, characterized in that, The online recursive update of the zero-point state and scaling factor state of each channel, combined with the quantization of the key-value cache activation value into low-bit codewords for cache compression, includes: Receive user requests and generate input word sequences, and create the corresponding session's cache context, page table structure, layer state descriptor, and write pointer; The model forward pass is performed during the pre-filling phase to obtain the query, key, and value tensors of each layer. For input prompts that are longer than the preset length, the process is divided into blocks according to the preset block length. The quantization recursion is performed on the tokens in each block in natural order. Perform adaptive quantization on the newly generated key and value tensors, including: Read the zero-point state and scaling factor state of the current channel at the current moment, and calculate the difference between the activation value of the key cache or value cache of the current word and the current zero-point state as the residual; The residual is subjected to integer codeword generation to obtain quantized codewords; The reconstructed residual is obtained based on the quantized codeword and the current scaling factor, and the zero-point state at the next moment is updated. The scaling factor state at the next moment is updated based on the current scaling factor, the scaling factor update coefficient, and the deviation between the absolute value of the quantized codeword and the scale offset threshold. Write the quantization codeword into the quantization cache.

7. The method according to claim 6, characterized in that, Before performing adaptive quantization on the newly generated key tensors and value tensors, the process also includes: Determine whether the current word is the first word of the session, the first word of the page, or the first word of the block, or whether the system has been rebuilt due to sliding window elimination, segment migration, or abnormal recovery; If any of the above conditions are met, the zero point state and scaling factor state of the current channel will be set to the initial zero point state, the initial scaling factor state, or the most recent checkpoint state.

8. The method according to claim 6, characterized in that, The step of generating quantized codewords from the residual by performing integer codeword generation includes: The residual is divided by the current scaling factor and then rounded to the nearest integer. It is then clipped to between the lower and upper bounds of the quantization interval to obtain the quantized codeword.

9. The method according to claim 6, characterized in that, After performing integer codeword generation on the residual, the method further includes: Determine whether the pruning ratio of several consecutive words exceeds the preset pruning ratio. If it does, execute one of the following branch processes: temporarily increase the quantization bit width of the current channel; write the current word to the outlier bypass buffer; shorten the update frequency and reinitialize the scaling factor state; or return to full-precision or high-precision block storage.

10. The method according to claim 6, characterized in that, The update of the zero-point state at the next moment includes: The reconstruction residual is obtained by multiplying the quantized codeword by the current scaling factor, and the zero-point state at the next moment is updated based on the reconstruction residual and the zero-point update coefficient. The update of the scaling factor state for the next time step includes: The difference between the absolute value of the quantized codeword and the scale offset threshold is calculated as the deviation. The scaling factor state at the next moment is updated based on the current scaling factor, the scaling factor update coefficient, and the deviation. The updated scaling factor state is then clipped to the scaling factor boundary range.

11. The method according to claim 6, characterized in that, The step of writing the quantized codeword into the quantization buffer for compression includes: The codeword with a preset number of bits is compressed into the storage unit. If the remaining space of the current cache page is insufficient, a new page is allocated and the page table is updated. If the current word is the end of a block but the packing unit is not full, zero codewords are written for unused bits, codewords are reserved, or the codewords are delayed until the next word to be merged and written.

12. The method according to claim 1, characterized in that, The process involves restoring the zero-point state and scaling factor state corresponding to each historical lexical unit based on a recursive rule isomorphic to the quantization writing process, performing dequantization on the quantized codewords, and directly participating the dequantized data in attention calculations, including: The quantized key cache is scanned in the order of historical lexical terms and online dequantization is performed: For each historical position, the target storage unit is located according to the page table and packing index, the corresponding bit segment is extracted according to the parity of the lexical terms or the packing position, the integer codeword is recovered, and the dequantized key value is calculated to be equal to the integer codeword multiplied by the current scaling factor plus the current zero point state. Local point accumulation is performed on the channel segments responsible for by each thread in the same thread block to obtain the local sum. The attention log score of the historical position is obtained by reduction within the thread block, and the zero point state and scaling factor state in the register are updated according to the current codeword to ensure that the dequantization state of the next historical position is consistent with that of the writing end. Perform causal masking, scaling, and normalization operations on the attention log score to obtain the weight vector of the attention score; The value cache is scanned using the same recursive method as the key cache. For each historical position, the quantized value cache codeword is read and unpacked, dequantized, and then multiplied and accumulated with the attention score to obtain the context vector of the current head.

13. The method according to claim 12, characterized in that, Before scanning the quantized key cache in historical lexical order and performing online dequantization, the process also includes: The generation phase receives the query vector of the current layer and starts query-value fusion dequantization calculation, and selects one or more kernel instances based on batch size, number of heads, history length, packing factor, page table layout and hardware topology; For each attention head pair, the query vector and the initial zero-point state, initial scaling factor state, zero-point update coefficient, scaling factor update coefficient, and scale offset threshold of the corresponding channel are loaded into a register or on-chip cache. If the most recent checkpoint is used, the checkpoint state is loaded in addition and used as the starting point for recursion.

14. The method according to claim 12, characterized in that, Also includes: The context vectors of each head are concatenated or projected back to the model dimension, and the attention output projection, residual connection, normalization and feedforward network of the subsequent layers are continued until the last layer is completed, the log probability distribution of the current step is obtained, and the next word is obtained according to the preset decoding strategy.

15. The method according to claim 13, characterized in that, Also includes: During inference, when the video memory usage reaches a threshold, cold pages are migrated to host memory, solid-state drives, or remote nodes. When using sliding windows or segmented attention, discard the oldest page and update the page table synchronously. When a checkpointing mechanism is used, the zero-point state and scaling factor state are stored once every preset number of tokens.

16. The method according to claim 13, characterized in that, The step of directly using the dequantized data in attention calculation includes: Within a thread block, state recovery, dequantization, and vector dot product operations are performed sequentially according to the lexical time order. Each thread block is responsible for the dot product operations of a preset number of channels. The dequantized data directly participates in the attention matrix-vector multiplication in the on-chip cache, without generating a separate intermediate copy in full-precision format; After the current lexical unit completes dequantization, the updated zero-point state and scaling factor state are immediately passed to the next lexical unit.

17. A fusion quantization key-value cache attention computation kernel, characterized in that, Deployed within a GPU stream processor, AI accelerator array, or dedicated tensor computing unit, the kernel includes: The key-value cache unit is used to store the quantization sequence corresponding to the historical tokens of each channel of each attention head in each layer in a compressed format, as well as the quantization parameters corresponding to each channel. The L0 instruction cache is used to store kernel-executed instructions; A thread bundle scheduler, connected to the L0 instruction cache, is used to schedule thread bundles based on batch size, number of heads, history length, and page table layout. The distribution unit, connected to the thread bundle scheduler, is used to distribute the scheduled thread bundles to the corresponding arithmetic logic unit array. The register file is used to temporarily store the query vector block, current zero-point state, current scaling factor state, and current token dequantization result of each thread in each thread bundle at runtime, wherein the current zero-point state and the current scaling factor state are dynamically updated as the token recursion process proceeds. An array of arithmetic logic units, each arithmetic logic unit corresponding to a channel, is connected to the key-value cache unit and the register file, and is used to perform the following operations: The quantization codeword corresponding to the current historical position and the quantization parameters of the corresponding channel are read from the key-value cache unit. The current zero-point state and current scaling factor state of the current channel at the current historical position are restored according to the adaptive incremental recursive process that is isomorphic to the quantization writing process. The quantization codeword is dequantized according to the current zero-point state and current scaling factor state, and the dequantization result is temporarily stored in the register file. The current zero-point state and current scaling factor state in the register file are updated. The arithmetic logic unit array is executed sequentially according to the word time order.

Images