Key-value cache compression method and system for accelerating large language model inference

Abstract

Provided is an accelerator for accelerating batch large language model (LLM) inference via key-value cache compression. An accelerator, according to one embodiment, may comprise a plurality of compute cores. Here, each of the plurality of compute cores may comprise a plurality of processing units for processing an LLM inference operation on a per-token basis, and a direct memory access unit for managing operations of reading weights from a memory and writing key-value activation data back to the memory. In addition, the direct memory access unit may comprise a compression engine for managing online compression of the key-value activation data when the key-value activation data is written to the memory, a decompression engine for decompressing the compressed key-value activation data retrieved from the memory, and a memory management unit for managing reading and writing of the compressed key-value activation data in the memory.

Description

Key-value cache compression method and system for accelerating large-scale language model inference

[0001] Embodiments of the present invention relate to a technique for compressing a key-value cache to accelerate Large Language Model (LLM) inference and an accelerator applying such a compression technique.

[0002] To improve the throughput of a Large Language Model (LLM) inference service system, batching multiple prompts is essential. However, LLM's multi-head attention layer cannot support batch processing because each prompt must be computed independently. This limits on-chip data reuse and makes bandwidth a serious bottleneck in batched LLM inference. This results in significantly underutilized compute cores, resulting in poor performance.

[0003] A key-value cache compression method for accelerating large language model (LLM) inference and an accelerator applying the key-value cache compression method can be provided.

[0004] The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0005] An accelerator is provided, comprising: a plurality of compute cores, each of the plurality of compute cores comprising: a plurality of processing units for processing a Large Language Model (LLM) inference operation on a token basis; and a direct memory access unit for managing operations of reading weights from a memory and writing key-value activation data back to the memory, wherein the direct memory access unit comprises: a compression engine for managing online compression of the key-value activation data when the key-value activation data is written to the memory; a decompression engine for decompressing the compressed key-value activation data retrieved from the memory; and a memory management unit for managing reading and writing of the compressed key-value activation data to the memory.

[0006] According to one aspect, the compression engine may be characterized by including a decomposition unit that separates the key-value activation data into normal values ​​and outliers based on a predefined threshold value; a normal value quantization module that quantizes keys and values ​​of the key-value activation data of the normal values; and an outlier quantization module that quantizes keys and values ​​by adjusting a distribution through an outlier threshold approximation for the key-value activation data of the outliers.

[0007] According to another aspect, the normal value quantization module may be characterized in that it quantizes the keys of the normal value key-value activation data channel by channel using a predefined scaling factor derived from fixed minimum and maximum values, dynamically calculates the scaling factor during runtime from the minimum and maximum values ​​of each token, and quantizes the buffered values ​​after all values ​​of the normal value key-value activation data are buffered.

[0008] According to another aspect, the outlier quantization module may be characterized in that it adjusts the distribution by subtracting an upper threshold from an outlier greater than an upper threshold in the key-value activation data of the outlier, and subtracting a lower threshold from an outlier less than a lower threshold.

[0009] According to another aspect, the outlier quantization module may be characterized by calculating a minimum value, a maximum value, and a scaling factor for a key and a value, then performing quantization, transmitting at least some bits of the quantized data to the normal quantization module for integration, generating an index for transforming a coordinate list storing the outlier, removing 0 from the key-value activation data of the outlier using a zero-removal shifter, and shifting the remaining data of the key-value activation data of the outlier to be stored in the memory.

[0010] According to another aspect, the list of coordinates may be characterized by including at least one bit for storing the value of the outlier and at least one bit for indicating the position of the value of the outlier.

[0011] According to another aspect, the decompression engine may be characterized by including a normal value dequantization module that identifies outliers in key-value activation data compressed through the compression engine, passes the outliers to an outlier dequantization module, and dequantizes normal values; and an outlier dequantization module that restores the data structure of the passed outliers by reinserting 0s removed during compression of the key-value activation data based on sparse coordinate list data that stores the outliers and dequantizes the outliers.

[0012] According to another aspect, the memory management unit may include a management table for each of dense data and sparse data among the key-value activation data, and the management table may include a memory address and a transfer size for the key-value activation data.

[0013] According to another aspect, the memory management unit may be characterized in that it supports burst access for reading all key-value activation data of a previous token in burst mode by dividing key-value activation data generated in the current layer into heads and writing them to other pages of the memory as specified in the management table.

[0014] According to another aspect, in the summary step of the large language model inference operation, the input tokens of each request may be scheduled to be processed in parallel across the plurality of compute cores, and in the generation step of the large language model inference operation, each of the plurality of compute cores may process an output token of a single request.

[0015] Specific details of other embodiments are included in the detailed description and drawings.

[0016] A key-value cache compression method for accelerating large language model (LLM) inference and an accelerator applying the key-value cache compression method can be provided.

[0017] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

[0018] Figures 1 to 3 are diagrams showing examples of KV value distributions of the OPT-6.7B and Llama-7B models using the Wikitext2, PIQA, and Hellaswag data sets.

[0019] FIG. 4 is a diagram illustrating an example of a quantization method of an accelerator according to an embodiment of the present invention.

[0020] FIG. 5 is a diagram illustrating an example of ideal value quantization of an accelerator according to an embodiment of the present invention.

[0021] FIG. 6 is a drawing showing an example of an accelerator structure according to one embodiment of the present invention.

[0022] FIG. 7 is a diagram illustrating an example of a compression engine and a decompression engine for efficiently supporting KV compression in one embodiment of the present invention.

[0023] FIG. 8 is a diagram illustrating an example of a memory management unit (MMU) for dense and sparse data in one embodiment of the present invention.

[0024] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms. These embodiments are provided only to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined only by the scope of the claims. Like reference numerals designate like elements throughout the specification.

[0025] When one component is referred to as being "connected to" or "coupled to" another component, it includes both cases where it is directly connected or coupled to the other component, or cases where there is another component intervening therebetween. Conversely, when one component is referred to as being "directly connected to" or "directly coupled to" another component, it indicates that there is no other component intervening therebetween. "And / or" includes each and any combination of one or more of the mentioned items.

[0026] The terminology used herein is for the purpose of describing embodiments only and is not intended to limit the present invention. In this specification, the singular also includes the plural unless specifically stated otherwise. As used herein, the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components, steps, operations, and / or elements.

[0027] Although terms like "first" and "second" are used to describe various components, these components are not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it should be understood that a "first" component referred to below may also be a "second" component within the technical scope of the present invention.

[0028] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in their common sense to those of ordinary skill in the art to which the present invention pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0029] Embodiments of the present invention describe an end-to-end accelerator designed to efficiently service batched Large Language Model (LLM) inference even with large batches and long prompt lengths. According to one embodiment, the accelerator can mitigate memory bandwidth and capacity issues caused by non-shared key-value caches in batched LLM inference through an algorithm-hardware code design. This approach not only reduces attention layer latency but also improves the throughput and scalability of batched LLM inference services.

[0030] First, embodiments of the present invention address memory bandwidth bottlenecks through key-value cache compression. A hybrid offline-online compression technique is provided that utilizes observations of key-value distributions to effectively quantize key-values ​​of unpredictable future tokens. Furthermore, an outlier quantization algorithm is provided that provides a high compression ratio while minimizing accuracy loss. Furthermore, an encoding method for compressed data that explores dense and sparse matrices is provided. The compression algorithm of an accelerator according to one embodiment can significantly improve system throughput by alleviating memory bandwidth and capacity bottlenecks caused by the KV cache.

[0031] Furthermore, the accelerator can efficiently utilize compression and decompression algorithms in hardware while minimizing associated overhead, and the proposed quantization algorithm can be assigned to the compression and decompression engines. These compression and decompression engines, built into the Direct Memory Access (DMA) unit, operate independently from other accelerator units and inference logic, allowing for seamless integration into other alternative accelerators. Furthermore, a Memory Management Unit (MMU) is provided that separates virtual and physical address spaces in page segmentation and maximizes memory bandwidth for sparse data of variable length, efficiently utilizing the given capacity.

[0032] Key-value compression algorithm

[0033] A. Observations in the key-value distribution

[0034] Prior art in LLM quantization emphasizes that handling outliers with larger sizes than other values ​​significantly impacts model accuracy. Quantization methods that do not account for outliers can lower quantization precision because outliers distort the range of the quantization target. Therefore, analyzing the distribution and characteristics of the quantization target is crucial for designing an effective quantization technique. In the embodiments of the present invention, we observed and analyzed the key-value distributions for various LLM models and various data sets, providing insights for designing quantization techniques.

[0035] Figures 1 to 3 are diagrams showing examples of KV value distributions of the OPT-6.7B and Llama-7B models using the Wikitext2, PIQA, and Hellaswag data sets.

[0036] Figure 1 shows the minimum and maximum ranges of KV values ​​for each decoder layer in different LLM models. Specifically, the key-value size range varies across models and decoder layers. This variation arises from variations in model weights across each model and decoder layer. This observation provides insight into the need for individual quantization scales and outlier thresholds for each model and decoder layer.

[0037] Figure 2 shows the minimum and maximum ranges of KV values ​​obtained from the Llama-7B model using the Wikitext2, PIQA, and Hellaswag datasets. This means that the range and distribution of key-value pairs are consistent across different datasets, even when using the same model. This suggests that a custom quantization scale is not required for each dataset.

[0038] Figure 3 shows the size distribution of KV values ​​for the 16th decoder layer of the Llama-7B model for each channel. Certain channels of the key exhibit relatively larger sizes than other channels, indicating the presence of outlier channels. In contrast, for the values, a uniform size distribution is observed across all channels. Based on this insightful observation, we can determine the outlier threshold for each key channel while simultaneously using a common outlier threshold for the values ​​to preserve quantization precision.

[0039] In summary, the observations yield the following three insights for designing quantization techniques for accelerators:

[0040] · The accelerator can individually determine the quantization scale and outlier threshold for each model and decoder layer.

[0041] · Accelerators can use a common quantization scale for diverse data sets because the input data has minimal impact on the key-value distribution.

[0042] · The accelerator can use per-channel outlier thresholds for keys and per-tensor outlier thresholds for values.

[0043] B. Offline Outlier Threshold Approximation

[0044] B-1. Quantization method

[0045] Previous research on LLM quantization has demonstrated that determining the quantization scale per vector results in less accuracy loss than using a tensor-wise approach. Furthermore, recent research has suggested that keys should be quantized per channel and values ​​should be quantized per token. As previously observed, channel-wise quantization is required when keys contain outlier channels. Based on previous work and observations, accelerators can quantize key-value pairs generated by the attention layer, employing channel-wise quantization for keys and token-wise quantization for values.

[0046] Figure 4 is a diagram illustrating an example of a quantization method of an accelerator according to an embodiment of the present invention. Figure 4(a) shows the KV quantization method used in the accelerator. Following the approach of previous studies, the accelerator distinguishes between normal and outlier values ​​of the quantization target and applies different quantization methods as described in Figure 4(b). Normal values ​​are Thr, which is the lower threshold value of outliers, respectively. low and the upper threshold Thr up It can be defined as a value between . To obtain this threshold, the topK operation is typically used to maintain a constant proportion of outliers. The separated normal values ​​and outliers can be stored in dense and sparse matrices, respectively. In an accelerator, outliers can be stored in a coordinate list (COO), which is one of the sparse matrix formats.

[0047] B-2. Uniform quantization

[0048] The accelerator adopts uniform quantization, where the scaling factor, denoted by σ, can be calculated as shown in Equation 1 below using only simple statistics to minimize hardware complexity.

[0049]

[0050] Here, m represents the number of bits of the quantized value, and Max and Min represent the maximum and minimum values ​​of the value to be quantized, respectively. The uniform quantization function that converts the value x into the quantized value Q(x) can be defined as shown in Equation 2 below using the scaling factor of Equation 1.

[0051]

[0052] B-3. ​​Hybrid Quantization

[0053] As previously explained, uniform quantization can calculate the scaling factor by finding the maximum (Max) and minimum (Min) values ​​for each group. However, the problem is that outliers within a group can increase the scaling factor and widen the range, reducing accuracy.

[0054] In this embodiment, accuracy can be improved by separately managing outliers within a group in a codebook-style table, narrowing the range by collecting only inliers, and applying uniform quantization. In this case, hardware-specific logic was designed to efficiently process quantization (streamline), improving overall speed performance. The designed logic has a very simple structure, resulting in low hardware costs.

[0055] Here, codebook quantization may be a non-uniform method of quantizing by defining representative values ​​in the codebook, rather than a uniform method of quantizing data types such as floating point (FP) and integer (INT) values. In this case, the representative values ​​can be selected to maximize accuracy.

[0056] B-4. Offline Outlier Threshold Approximation

[0057] Although only simple statistics (Min and Max) are required to calculate the scaling factor, determining the outlier threshold and quantization scaling factor online is challenging because it is difficult to accurately know the key-value distribution during runtime. First, the cost of the topK operation for determining the outlier threshold is high. This topK operation is essentially a sort with time complexity of O(n log n), and performing it during inference adds significant performance overhead.

[0058] Second, due to the inherent unpredictability of generated tokens in LLM inference, the scaling factor in per-channel quantization can change depending on the tokens to be generated in the future. If the scaling factor changes, values ​​already quantized with the previous scaling factor must be re-quantized using the new scaling factor. This computational cost scales proportionally with the number of tokens processed so far, making it infeasible for large batches and long sequence lengths. This is particularly a major challenge when quantizing keys online.

[0059] To address this issue, an accelerator according to an embodiment performs offline outlier threshold approximation to exploit the consistent properties of the key-value distribution discussed above. The accelerator applies Thr for each decoder layer for keys and each decoder layer for channels, values. low and Thr up can be obtained.

[0060] C. Online KV cache quantization

[0061] As discussed earlier, the accelerator adopts token-wise quantization for values, which can only perform quantization on newly generated value vectors from the attention layer. In token-wise quantization, the accelerator uses two outlier thresholds obtained from offline approximation to separate normal and outlier values ​​from the new vector, and Min, the minimum and maximum values ​​of the inliers. in and Max in can be searched for respectively. The accelerator can then compute the scaling factor and quantize the values ​​online.

[0062] On the other hand, for channel-wise quantization applied to keys, it is difficult to obtain the distribution of a specific channel online. In addition, there is another problem that the scaling factor may change due to the nature of LLM inference, which cannot predict the tokens to be generated. To solve this problem, channel-wise quantization in the accelerator is applied in an offline approximation. in and Max in In addition, two outlier thresholds must be obtained. These outlier thresholds can be used during inference to separate normal and outlier values, and quantization can be performed using the profiled minimum and maximum values.

[0063] D. Outlier quantization due to distribution shift

[0064] The COO format used in an accelerator according to one embodiment requires additional bits to indicate the location of a value, in addition to the bits representing the value itself. The accelerator uses a 7-bit index to specify the location of an outlier within the original matrix. If the outlier is not quantized, the value is represented with 16-bit precision, resulting in a total of 23 bits per outlier element. This capacity overhead is negligible when the key-value sparsity is low, but becomes significant as sparsity increases. This capacity overhead complicates the trade-off between performance and accuracy by adjusting the sparsity of the KV cache.

[0065] One simple way to alleviate this problem is to quantize outliers. However, as discussed previously, quantizing outliers is difficult due to their large size. To address this issue, embodiments of the present invention utilize online outlier distribution shifting, a hardware-friendly algorithm that minimizes accuracy loss while quantizing outliers.

[0066] Fig. 5 is a diagram illustrating an example of outlier quantization of an accelerator according to an embodiment of the present invention. As in Fig. 5(a), initially, Thr is used to move the outlier. up Thr in larger outliers up Thr except low Thr in smaller outliers low can be subtracted. Consequently, the shifted outliers represent a narrower range of values ​​than before, enabling low-bit quantization. Unlike outlier quantization, the accelerator can compute the quantization measure online, token-by-token, for both keys and values. Outlier distribution shifting does not require additional statistical information and can be performed using only the statistics obtained from the offline profiling described above. For example, 5-bit outlier quantization can improve performance while minimizing accuracy loss.

[0067] E. Dense and Sparse Encoding

[0068] To further reduce capacity overhead, the 0 elements of the dense matrix can be utilized. The 0 elements of the dense matrix correspond to the portions of the original KV cache where outliers existed, and remain unused even after the KV cache is split into dense and sparse matrices. Embodiments of the present invention can utilize these unused 4 bits to provide a fused dense and sparse encoding that stores a portion of the outliers, as shown in FIG. 5(b). Specifically, the accelerator can store the 4 bits excluding the sign bit of the quantized 5-bit outlier as 0s in the dense matrix, and store the remaining sign bits and 7 index bits in a sparse COO format. Given that the index bits in the COO format indicate the location of the outlier in the dense matrix, a dedicated flag indicating the presence of the outlier is not required. This fused dense and sparse encoding, combined with an outlier distribution shift, can maximize the compression ratio of key-value pairs by reducing the number of bits required per outlier element from 23 to 8.

[0069] Accelerator architecture

[0070] A. Overall architecture

[0071] FIG. 6 is a diagram illustrating an example of an accelerator structure according to an embodiment of the present invention. The accelerator (Oaken Accelerator (600)) according to the present embodiment may mainly be composed of a compute core (Compute Core (610)), a memory controller (MC (620)), a host interface (Host Interface (630)), and an interconnect (Interconnect (640)). The compute core is designed for end-to-end LLM inference tasks and may include a control unit (Control Unit (650)), a vector processing unit (Vector Processing Unit (660)), a register file (Register File (670)), a matrix processing unit (Matrix Processing Unit (680)), and a DMA (Direct Memory Access, 690). Additionally, the DMA (690) may include a Compression Engine (Comp.Engine (691)), a Decompression Engine (Decomp.Engine (692)), and a Memory Management Unit (MMU) (693). The memory controller may access the LPDDR5X to read elements such as weights, keys, and values, and may also be responsible for writing keys and values ​​back to memory. The host interface may use a PCIe-based connection to communicate with the host system. This interface may also manage the scheduling of requests coming from the host and distribute them to various compute cores for efficient processing. The interconnect may connect all these modules and be optimized to maximize bandwidth utilization during each memory read, ensuring efficient data transfer to each compute core. This design may use all memory controllers to read weights and distribute them to the compute cores.Conversely, memory writes from compute cores are infrequent and small in data size, which minimizes bandwidth consumption and reduces logical design size without compromising performance.

[0072] B. Compute cores of the accelerator

[0073] B-1. Overview

[0074] An accelerator according to one embodiment may include multiple compute cores to enable end-to-end LLM inference. Each compute core may include a Matrix Processing Unit (MPU) and a Vector Processing Unit (VPU) designed to process LLM inference operations on a token-by-token basis. These processing units are designed to maintain high utilization throughout the LLM inference process, minimizing inefficient logic and ensuring efficient computation with low latency. Specifically, the MPU is designed to stream weights read from memory to support efficient matrix-vector multiplication, while the VPU handles minor vector operations that occur between major matrix-vector calculations, minimizing the impact on overall latency. The DMA unit may manage the tasks of reading weights from memory and writing key-value pairs back to memory. The DMA unit may also include a compression engine, a decompression engine, and an MMU, which are key components for efficiently supporting the proposed KV compression.

[0075] B-2. Compression Engine and Decompression Engine

[0076] FIG. 7 is a diagram illustrating an example of a compression engine and a decompression engine for efficiently supporting KV compression in one embodiment of the present invention.

[0077] The compression engine illustrated in Figure 7(a) is embedded within the DMA unit and can manage online compression when key-value pairs are written to memory. This compression engine can be composed of three modules that simplify activation processing. The decomposer can divide incoming activations into normal and outliers based on a predefined threshold. Normal activations are directed to the normal quantization path (normal quantization module), and outliers are directed to the outlier quantization path (outlier quantization module). Accordingly, zeros can be inserted into the alternative path.

[0078] The normal quantization module can handle quantization of each key and value. For keys, a predefined scaling factor derived from fixed minimum and maximum values ​​can be utilized for 4-bit quantization per channel. For values, the scaling factor can be dynamically calculated at runtime from the minimum and maximum values ​​per token, facilitating immediate 4-bit quantization after all values ​​are buffered. The outlier quantization module can process both keys and values ​​equally while adjusting the distribution by subtracting a streaming threshold. After calculating the minimum and maximum values ​​and the scaling factor, 5-bit quantization can be performed. After quantization, the least significant 4 bits can be passed to the normal quantization module for integration using an OR gate. The outlier quantization module can also generate indices for the COO transform and optimize memory storage by using a zero-eliminating shifter to remove zeros and shift valid data. Overall, these modules can maximize effective memory bandwidth.

[0079] The decompression engine illustrated in Figure 7(b) can operate within the DMA unit to decompress compressed key-value data retrieved from memory. The decompression engine can also be composed of three modules. The normalized dequantization module can first buffer incoming dense data (compressed key-values; Dense KV) to synchronize it with the sparse data processed by the outlier dequantization module. When the data leaves the buffer, it undergoes a separation process that identifies outliers based on index information and passes them to the outlier dequantization module, which can guide extraction and subsequent dequantization using a predefined scaling factor. Meanwhile, the outlier dequantization module processes sparse COO data and can initially perform a zero-insertion operation to restore the original data structure by reinserting zeros removed during compression. This restored data can be buffered to align with the output of the normalized quantization module to facilitate the final concatenation process. Ultimately, the outputs of the two dequantization modules can be combined via an OR operation and passed to the processing unit.

[0080] Figure 8 is a diagram illustrating an example of an MMU for dense and sparse data, according to one embodiment of the present invention. Figure 8 illustrates an MMU that manages reads and writes of compressed key-value activations to memory. The MMU has two tasks: the first is managing addresses and transfer sizes, and the second is determining access units and order for memory reads and writes.

[0081] For the first challenge, dense data has a predictable size and can be aligned within memory, whereas sparse data has variable sizes and may have alignment issues. Managing this variability requires management tables for both dense and sparse data: one for dense data and one for sparse data. These tables contain memory addresses and transfer sizes for activations (key-value activation data), which can be considered up to the maximum sequence length per head. These addresses and transfer sizes can be dynamically calculated at runtime by checking for available pages as needed.

[0082] The second challenge is to maximize effective memory bandwidth when reading and writing activations. To increase effective memory bandwidth, it is crucial to minimize access times by using burst accesses whenever possible. Write activations only involve the activations of the current token, so they contain a smaller data size, while read activations require retrieving all activations from previous tokens. The most efficient management strategy is to write in a way that supports burst reads. For example, activations generated in the current layer can be split into heads and written to different memory pages as specified in the management table. When the next token's activation is generated, it can be similarly split and written immediately after the activation of the previous token. This sequential arrangement allows all activations from previous tokens to be read in burst mode, thus efficiently utilizing memory bandwidth.

[0083] C. Token-level scheduling

[0084] To achieve high throughput, efficient scheduling for batch processing on the proposed hardware is crucial. Each compute core is optimized to efficiently process a single token. In the summary phase, the input tokens of each request can be scheduled to be processed in parallel across multiple compute cores. In the generation phase, each compute core can process the output tokens of a single request. This architecture and scheduling method efficiently manage parallel computation in the summary phase for a single batch, but its repetitive nature makes it underutilized in the generation phase. However, for larger batches, the generation phase can process multiple requests in parallel, making this architecture and scheduling method ideally suited for large-scale batch scenarios.

[0085] In this way, according to embodiments of the present invention, a key-value cache compression method for accelerating large-scale language model inference and an accelerator to which the key-value cache compression method is applied can be provided.

[0086] Although the embodiments of the present invention have been described with reference to the attached drawings, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without altering the technical concept or essential features thereof. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

Claims

1. Multiple compute cores Including, Each of the above multiple compute cores, Multiple processing units that process Large Language Model (LLM) inference operations on a token-by-token basis; and A direct memory access unit that manages the task of reading weights from memory and writing key-value activation data back to said memory. Including, The above direct memory access unit is, A compression engine that manages online compression of the key-value activation data when the key-value activation data is written to the memory; A decompression engine for decompressing compressed key-value activation data retrieved from the above memory; and A memory management unit that manages reading and writing of compressed key-value activation data to the above memory. Including Accelerator featuring.

2. In paragraph 1, The above compression engine, A decomposer that separates the above key-value activation data into normal values ​​and outliers based on a predefined threshold; A normal value quantization module that quantizes the keys and values ​​of the key-value activation data of the above normal value; and An outlier quantization module that quantizes keys and values ​​by adjusting the distribution through an outlier threshold approximation for the key-value activation data of the above outliers. Including An accelerator featuring:

3. In paragraph 2, The above normal value quantization module is, Quantize the key of the key-value activation data of the above normal value channel-wise using a predefined scaling factor derived from fixed minimum and maximum values, Dynamically calculating a scaling factor at runtime from the minimum and maximum values ​​per token, and quantizing the buffered values ​​after all values ​​of the key-value activation data of the normal value are buffered. An accelerator featuring:

4. In paragraph 2, The above outlier quantization module is, Adjusting the distribution by subtracting the upper threshold from outliers larger than the upper threshold in the key-value activation data of the above outliers and subtracting the lower threshold from outliers smaller than the lower threshold. An accelerator featuring:

5. In paragraph 2, The above outlier quantization module is, After calculating the minimum, maximum, and scaling factors for the keys and values, quantization is performed, transmitting at least some bits of the quantized data to the normal quantization module for integration, Generating an index for transforming a list of coordinates storing outliers, removing 0 from the key-value activation data of the outliers using a zero-removal shifter, and shifting the remaining data of the key-value activation data of the outliers to be stored in the memory. An accelerator featuring:

6. In paragraph 5, An accelerator characterized in that the above coordinate list includes at least one bit for storing the value of the outlier and at least one bit for indicating the position of the value of the outlier.

7. In paragraph 1, The above decompression engine, A normal value dequantization module that identifies outliers in the compressed key-value activation data through the above compression engine and passes them to an outlier dequantization module, and dequantizes normal values; and The outlier dequantization module restores the data structure of the transmitted outlier by reinserting 0s removed during compression of the key-value activation data based on the sparse coordinate list data storing the outliers and dequantizes the outlier. Including An accelerator featuring:

8. In paragraph 1, The above memory management unit, Includes a management table for each of the dense and sparse data among the above key-value activation data, The above management table includes a memory address and transfer size for the key-value activation data. An accelerator featuring:

9. In paragraph 8, An accelerator characterized in that the above memory management unit supports burst access for reading all key-value activation data of a previous token in burst mode by dividing key-value activation data generated in the current layer into heads and writing them to other pages of the memory as specified in the management table.

10. In paragraph 1, In the summary step of the above-mentioned large language model inference operation, the input tokens of each request are scheduled to be processed in parallel across the plurality of compute cores, In the generation step of the above-mentioned large language model inference operation, each of the plurality of compute cores processes the output token of a single request. An accelerator featuring:

Images