Method for Compressing and Reconstructing Key-Value Cache Using Sparse Dictionary in Large Language Model Environment, and Cache Processing Apparatus Therefor

Abstract

A method for compressing and restoring a sparse dictionary-based key-value cache in a large-scale language model environment according to one technical aspect of the present invention is performed in a cache processing device operating based on a large-scale language model and comprises: a step of collecting key vectors and value vectors generated from multiple layers of a large-scale language model for a plurality of input sequences; a step of learning a global dictionary based on the collected key vectors and value vectors; a step of generating sparse expression information for each input key vector or value vector by applying a sparse expression generation algorithm based on the global dictionary; a step of storing the generated sparse expression information; a step of loading the stored sparse expression information in response to a restoration request at the time of inference; and a step of restoring the key vector or value vector based on the loaded sparse expression information and the global dictionary.

Description

Technology Field

[0001] The present invention relates to a technology for improving the inference efficiency of a large-scale language model, and more specifically, to a cache optimization method for an LLM and a cache processing device therefor that can improve memory usage and response delay by compressing a key-value cache based on a sparse dictionary using a global dictionary learned based on deep learning and rapidly restoring it through matrix operations capable of GPU parallel processing. Background Technology

[0002] Recently, Large Language Models (LLMs) have demonstrated innovative performance in the field of Natural Language Processing and are being utilized in various applications. In particular, the ability to efficiently process long contexts is emerging as a key element in high-dimensional tasks such as generating personalized responses, understanding long documents, and multi-turn conversations.

[0003] LLM performs attention operations on each token of an input sequence, and in this process, uses a Key-Value cache (hereinafter KV cache) that stores key vectors and value vectors for previous tokens for each layer. Such a KV cache prevents redundant calculations during the decoding process and contributes to improving inference speed. However, as the length of the input sequence increases, the size of the KV cache increases linearly, which leads to a surge in GPU memory resource consumption and acts as a cause of reduced efficiency due to bottlenecks in the system processing.

[0004] To address these issues, various KV cache compression techniques, such as quantization, pruning, or fixed transformations, have recently been studied.

[0005] However, these conventional technologies have the following limitations.

[0006] These conventional technologies suffer from storage and GPU I / O bottlenecks, as well as delays in the Time to First Token (TTFT). The large volume of pre-computed document KV-caches utilized in RAG systems is so massive that transferring them from storage to GPU memory takes a significant amount of time; this is a major cause of degraded user experience, particularly by delaying the Time to First Token (TTFT).

[0007] Furthermore, conventional technology has limitations in the scalability of existing learning-based compression methods for large-scale datasets. In other words, traditional methods that improve representation efficiency by learning a dictionary from data face scalability limitations due to high computational complexity when applied to large datasets. In particular, methods requiring computational power proportional to the cube of the number of training data points are practically very difficult to apply to the vast KV-cache of LLMs.

[0008] Furthermore, conventional technologies suffer from issues such as reduced compression efficiency and performance, as well as a lack of adaptability. Existing compression techniques struggle to strike a balance between compression ratio and performance; for instance, the complex decompression process slows down overall inference speed, while increasing the compression ratio significantly degrades model performance. In particular, they exhibit limitations in adaptability, failing to provide optimal compression efficiency for various types of KV-cache data by being restricted to specific patterns or using fixed methods.

[0009] Accordingly, a new method of KV cache compression and recovery technology capable of simultaneously satisfying high compression ratio and low recovery delay is required. Prior art literature

[0010] Republic of Korea Published Patent Application No. 10-2025-0096320 (Date of publication: June 27, 2025) The problem to be solved

[0011] The present invention aims to solve the aforementioned problems. An embodiment according to the present disclosure aims to provide a technology that effectively generates a global dictionary applicable to various input contexts by applying a deep neural network-based end-to-end dictionary learning method, so that key-value cache data of a large-scale language model can be stored at a high compression rate while being restored quickly. Through this, the invention aims to resolve the scalability issues of predefined fixed transformation methods or traditional SVD-based dictionary learning methods and to implement a learning-based compression method adaptable to large-scale KV-cache data.

[0012] In addition, one embodiment according to the present disclosure aims to provide a high-efficiency compression technology that can drastically reduce memory and storage usage by enabling the compressed KV-cache to be composed of a combination of very small index and coefficient pairs by applying a sparse coding-based compression method that utilizes a learned global dictionary to represent each KV-cache vector as a linear combination of a few dictionary vectors, and an orthogonal matching pursuit (OMP) algorithm optimized for GPU parallel processing.

[0013] One embodiment according to the present disclosure aims to provide a high-speed restoration technology that minimizes restoration delay, shortens Time-To-First-Token (TTFT), and maximizes the efficiency of the prefill step by configuring the original KV-cache vector to be restored using only a single matrix multiplication operation with a pre-learned global dictionary during inference.

[0014] Furthermore, an embodiment according to the present disclosure aims to provide a data-adaptive sparse representation-based cache compression framework that can achieve a high compression ratio and minimize model performance degradation by effectively capturing structural characteristics inherent in various types of documents or input contexts through a learned dictionary, thereby resolving the problem of insufficient adaptability that occurred in existing fixed transformation methods or simple statistical compression methods.

[0015] However, the problems to be solved in this disclosure are not limited to those mentioned above, and may be expanded in various ways without departing from the spirit and scope of this disclosure. means of solving the problem

[0016] One technical aspect of the present invention proposes a method for compressing and restoring a sparse dictionary-based key-value cache in a large-scale language model environment. The method is performed in a cache processing device operating based on a large-scale language model and comprises: a step of collecting key vectors and value vectors generated from multiple layers of the large-scale language model for a plurality of input sequences; a step of learning a global dictionary based on the collected key vectors and value vectors; a step of generating sparse expression information for each input key vector or value vector by applying a sparse expression generation algorithm based on the global dictionary; a step of storing the generated sparse expression information; a step of loading the stored sparse expression information in response to a restoration request at the time of inference; and a step of restoring the key vector or value vector based on the loaded sparse expression information and the global dictionary.

[0017] In one embodiment, the step of learning a global dictionary based on the collected key vectors and value vectors may include the step of iteratively updating the global dictionary by applying a numerical optimization technique based on a loss function that takes the collected key vectors and value vectors as inputs and minimizes the reconstruction error between the dictionary matrix and the sparse representation vectors.

[0018] In one embodiment, the sparse expression generation algorithm is an Orthogonal Matching Pursuit (OMP) algorithm configured to repeatedly select the atom with the largest inner product value with the input vector among a plurality of atoms of a global dictionary for the input key vector or value vector, and approximate the input vector with a linear combination of the selected atoms.

[0019] In one embodiment, the sparse representation information may be represented as a set of index-coefficient pairs consisting of identification indices of some selected atoms among a plurality of atoms in a global dictionary and coefficient values ​​corresponding to each index.

[0020] In one embodiment, the step of storing the generated sparse representation information may include storing the generated sparse representation information in external memory of the GPU (Graphics Processing Unit) or non-volatile storage.

[0021] In one embodiment, the step of restoring a key vector or value vector based on the loaded sparse representation information and the global dictionary may include the step of restoring the key vector or value vector prior to compression by performing a single matrix multiplication operation on the indices and coefficients included in the loaded sparse representation information and the corresponding atoms of the global dictionary.

[0022] In one embodiment, the restored key vector or value vector can be used as initial context information during the prefill phase of a large-scale language model.

[0023] In one embodiment, the method may further include the step of storing a plurality of sparse representation information corresponding to a plurality of documents, and selectively loading and restoring only some of the sparse representation information related to the user's query according to the user's query.

[0024] Another technical aspect of the present invention proposes a cache processing device. The cache processing device is a cache processing device that operates based on a large-scale language model and includes at least one processor; and a memory for storing instructions. When the instructions are executed individually or collectively by the at least one processor, the processor is configured to perform: an operation of collecting key vectors and value vectors generated from multiple layers of the large-scale language model for a plurality of input sequences; an operation of learning a global dictionary based on the collected key vectors and value vectors; an operation of generating sparse expression information for each of the input key vectors or value vectors by applying a sparse expression generation algorithm based on the global dictionary; an operation of storing the generated sparse expression information; an operation of loading the stored sparse expression information in response to a restoration request at the time of inference; and an operation of restoring the key vector or value vector based on the loaded sparse expression information and the global dictionary. Effects of the invention

[0025] According to various embodiments of the present disclosure, by storing and restoring the key-value cache of a large-scale language model with a high compression rate, it is possible to effectively alleviate GPU memory bottlenecks and storage I / O bottlenecks that occur in application environments with long context lengths, such as RAG systems.

[0026] According to various embodiments of the present disclosure, by utilizing a globally learned deep learning-based dictionary to compress the KV-cache into a sparse representation and rapidly restoring it with only a single matrix multiplication operation, the effect of minimizing the latency of the inference step and shortening the Time-To-First-Token (TTFT) is provided.

[0027] According to various embodiments of the present disclosure, the compression and restoration operations of the KV-cache are not performed during the decoding stage, and the compressed cache is restored and used only during the prefill stage. This configuration provides the effect of maximizing the efficiency of context preprocessing without affecting the real-time decoding speed.

[0028] According to various embodiments of the present disclosure, by effectively capturing patterns or structural characteristics that appear repeatedly in various input sequences and document types through a learned global dictionary, the effect of resolving the problem of insufficient adaptability that occurred in fixed transformation methods or simple statistical compression methods is provided.

[0029] According to various embodiments of the present disclosure, the effect of effectively achieving a balance between compression ratio and model performance is provided by achieving a higher compression ratio at the same performance level compared to existing methods (xKV, etc.) or by securing superior benchmark results at the same compression ratio.

[0030] According to various embodiments of the present disclosure, the effect is to provide a practical inference optimization framework that can improve overall system throughput and enhance user-perceived response performance in various LLM-based services, such as search augmentation generation (RAG), long document understanding, and streaming conversational AI response.

[0031] The effects obtainable from the present disclosure are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art to which the present disclosure belongs from the description below. Brief explanation of the drawing

[0032] FIG. 1 is a diagram that step-by-step compares and explains how a key-value cache processing method according to one embodiment of the present invention has structural and functional differences from an existing RAG system. FIG. 2 is a schematic diagram illustrating a sparse dictionary-based key-value cache compression and restoration device according to one embodiment of the present invention. FIG. 3 is a flowchart illustrating a sparse dictionary-based key-value cache compression and restoration method according to an embodiment of the present invention. FIG. 4 is a schematic diagram illustrating a comparative example of a RAG inference scenario to which a sparse dictionary-based key-value cache processing method according to one embodiment of the present invention is applied. FIG. 5 is a block diagram illustrating the configuration of a cache processing device according to one embodiment of the present invention. FIG. 6 is a diagram illustrating an example of sparse dictionary-based compression and restoration processing according to an embodiment of the present invention. FIG. 7 is a schematic diagram showing an example of an experimental configuration performed by a cache processing device (100). Figure 8 shows experimental results comparing the average performance between various cache compression techniques and SparKV configurations for multiple benchmark evaluation items (N-S1, QA-1, etc.) based on the Qwen2.5-7B-Instruct-1M model. Specific details for implementing the invention

[0033] Hereinafter, embodiments of the present disclosure are described in detail with reference to the drawings so that those skilled in the art can easily practice them. However, the present disclosure may be embodied in various different forms and is not limited to the embodiments described herein. In relation to the description of the drawings, the same or similar reference numerals may be used for identical or similar components. Furthermore, in the drawings and related descriptions, descriptions of well-known functions and configurations may be omitted for clarity and brevity.

[0034] The various embodiments of this document and the terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutions of said embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of said items unless the relevant context clearly indicates otherwise. In this document, phrases such as "A or B," "at least one of A and B," "at least one of A or B," "A, B or C," "at least one of A, B and C," and "at least one of A, B, or C" may each include any one of the items listed together in the corresponding phrase, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used simply to distinguish said components from other said components and do not limit said components in any other aspect (e.g., importance or order). Where any (e.g., 1st) component is referred to as "coupled" or "connected" to another (e.g., 2nd) component, with or without the terms "functionally" or "communicationly," it means that said any component may be connected to said other component directly (e.g., via a wire), wirelessly, or through a third component.

[0035] The term “module” as used in the various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be a component formed integrally, or a minimum unit of said component or a part thereof that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0036] Various embodiments of this document may be implemented as software (e.g., a program) comprising one or more instructions stored in a storage medium (e.g., memory) readable by a machine or device. For example, the processor of the machine or device may call at least one of the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The storage medium readable by a machine may be provided in the form of a non-transitory storage medium. Here, "non-transitory" simply means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where it is stored temporarily in the storage medium.

[0037] According to one embodiment, the method according to the various embodiments disclosed herein may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or distributed online (e.g., download or upload) through an application store or directly between two user devices (e.g., smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created on a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0038] According to various embodiments, each component (e.g., module or program) of the components described above may include a singular or multiple entities, and some of the multiple entities may be separated and placed in other components. According to various embodiments, one or more of the components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Generally or additionally, multiple components (e.g., module or program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the multiple components in the same or similar manner as those performed by the corresponding component among the multiple components prior to integration. According to various embodiments, operations performed by the module, program, or other components may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0039] In this disclosure, the term "processor" may refer to hardware capable of performing functions and operations according to each designation described herein, computer program code capable of performing specific functions and operations, or an electronic recording medium loaded with computer program code capable of performing specific functions and operations. According to an embodiment, the operation of the processor may be defined and / or interpreted as the operation of a knowledge graph adjustment device, but is not limited thereto. The processor may refer to a functional and / or structural combination of hardware for carrying out the technical concept of this disclosure and / or software for driving said hardware.

[0041] FIG. 1 is a diagram that step-by-step compares and explains how a key-value cache processing method according to one embodiment of the present invention has structural and functional differences from an existing RAG system.

[0042] First, the Normal RAG structure illustrated at the top of Fig. 1 represents a configuration generally adopted in existing Retrieve-and-Generate systems. In this case, multiple text chunks are vectorized through a pre-trained embedding model and stored in an Embedding DB, and the structure is configured to perform an embedding similarity-based search in response to a user query.

[0043] However, in this structure, since documents retrieved after a user query are fed into a Language Model (LLM) and the LLM must perform decoding for each document from scratch, the cost of iterative inference is very high. This leads to a problem where the amount of computation and response latency in the LLM inference stage increase sharply, particularly when context lengths are long and the number of documents increases.

[0044] Accordingly, a Pre-computed RAG structure is proposed to compensate for these shortcomings. This structure adopts a method in which, while embedding each text fragment, each document is input via a Large Language Model (LLM) in advance, and the Key-Value Cache (KV Cache) generated during that input is stored separately. In other words, text documents are input into the LLM, the process proceeds to the prefill stage to extract the generated internal KV cache, and the corresponding KV cache is stored in advance.

[0045] Since the KV cache stored in this way can be quickly retrieved at the time of a user query and utilized for decoding, the overall inference time can be effectively reduced.

[0046] However, even in this case, because the KV cache itself is highly sophisticated and has high memory occupancy, there remains an unresolved problem regarding I / O bottlenecks and storage burdens arising from the pre-save and load processes.

[0047] One embodiment of the present invention is a Pre-computed RAG with compressed KV-Cache shown at the bottom of FIG. 1, which schematically describes the core structure of a sparse dictionary-based key-value cache processing system proposed by the present invention.

[0048] In the present invention, the advantages of Pre-computed RAG are maintained, but a deep learning-based sparse dictionary is additionally introduced to enable high-efficiency compression of a pre-generated KV-cache set.

[0049] Specifically, a set of KV-caches generated in advance in LLM is collected through multiple text fragments, and a global dictionary is learned based on the KV vectors, and this global dictionary is generalized so that it can be commonly applied to all prior input KV vectors.

[0050] Subsequently, by applying sparse coding techniques, such as the Orthogonal Matching Pursuit (OMP) algorithm, to each KV vector based on a pre-trained global dictionary, each KV vector is approximated as a linear combination of a small number of prior atoms. In addition to this Orthogonal Matching Pursuit (OMP) algorithm, various other sparse coding techniques can be applied.

[0051] Since such sparse representations generally consist only of dictionary indices and coefficients that each vector references, the use of memory and storage space can be drastically reduced compared to storing the entire original KV vector.

[0052] In addition, fine-grained reconstruction can also be performed very quickly. When a user query is entered, only the relevant sparse KV cache is selectively loaded onto the GPU, and the index and coefficient information of the loaded sparse representation is calculated through a global dictionary and a single matrix multiplication operation, thereby quickly securing the reconstructed KV cache. Importantly, these compression and reconstruction operations are performed only during the prefill phase, and since the already reconstructed KV cache is used during the decoding phase, it does not affect the decoding speed.

[0053] Consequently, one embodiment of the present invention can effectively improve the response delay (TTFT) and I / O bottlenecks of LLM compared to conventional RAG, and can also provide a high-efficiency KV cache compression / recovery system that achieves several times the compression rate in terms of storage while maintaining original performance without degrading the precision of LLM.

[0055] Hereinafter, with reference to FIGS. 2 to 8, various embodiments of a sparse dictionary-based key-value cache compression and restoration method in a large-scale language model environment and a cache processing device for the same according to the present invention will be described in more detail.

[0057] FIG. 2 is a schematic diagram illustrating a sparse dictionary-based key-value cache compression and restoration device according to one embodiment of the present invention.

[0058] As shown in FIG. 2, a cache processing device (100) is shown with a large-scale language model (200) and a cache processing device (100) that assists it.

[0059] The cache processing unit (100) includes a processor (101) and memory (102) internally and is implemented in a structure capable of communicating with an external large-scale language model (200).

[0060] For example, the large-scale language model (200) may be a Transformer-type multilayer neural network model and may perform inference functions that generate key vectors and value vectors in each layer according to prompt input. The large-scale language model (200) performs KV-cache generation in the prefill stage and response generation in the decoding stage.

[0061] The cache processing unit (100) performs the aforementioned sparse representation-based cache compression and restoration functions through interaction with the large-scale language model (200).

[0062] The processor (101) performs various processing logics of the cache processing unit (100) described below, and, for example, can perform global dictionary learning, application of a sparse representation algorithm, matrix multiplication operation, key-value vector restoration, etc.

[0063] Such a processor (101) may include, as an example, at least one of a microprocessor, a central processing unit, a processor core, a multi-core processor, a multiprocessor, an ASIC (Application-Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array), but is not limited thereto.

[0064] The memory (102) can store data required for various processing steps performed by the processor (101). For example, it can temporarily store compressed sparse representation information or store a restored cache so that it can be referenced during the LLM decoding step.

[0065] Memory (102) can store instructions (or programs) that can be executed by the processor (110). Memory (120) may include volatile memory or non-volatile memory. Volatile memory may be implemented as dynamic random access memory (DRAM), static random access memory (SRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), or twin transistor RAM (TTRAM). Non-volatile memory can be implemented as EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, MRAM (Magnetic RAM), Spin-Transfer Torque (STT)-MRAM, Conductive Bridging RAM (CBRAM), FeRAM (Ferroelectric RAM), PRAM (Phase change RAM), Resistive RAM (RRAM), Nanotube RRAM, Polymer RAM (PoRAM), Nano Floating Gate Memory (NFGM), holographic memory, Molecular Electronic Memory Device, or Insulator Resistance Change Memory.

[0066] For example, a cache processing unit (100) can learn a global dictionary based on a plurality of key vectors and value vectors collected from a large-scale language model.

[0067] For example, a cache processing unit (100) can apply a sparse representation generation algorithm to each input key vector or value vector based on a learned global dictionary to compress them into the form of index-count pairs.

[0068] For example, the cache processing unit (100) may store the generated sparse representation information in memory (102) or external storage. For example, the sparse representation may be stored in GPU external memory or NVMe-based high-speed storage based on KV-ID.

[0069] For example, at the time of inference, the cache processing unit (100) can load the stored sparse representation information and restore the original key / value vector through a single matrix multiplication operation with the global dictionary.

[0070] This cache processing device (100) can reduce the overall inference delay time and effectively minimize memory resources and I / O bottlenecks by selectively retrieving and restoring only the sparse representation caches of related documents in response to user queries.

[0071] According to an embodiment, the cache processing device (100) can be realized as an independent device located outside the GPU or as a module included in a parallel processing unit within the GPU, and can be usefully utilized in various LLM-based application environments such as a RAG (Retrieval-Augmented Generation) system, a document summarizer, and a complex question-answering system.

[0073] FIG. 3 is a flowchart illustrating a sparse dictionary-based key-value cache compression and restoration method according to an embodiment of the present invention.

[0074] One embodiment of FIG. 3 is performed by a cache processing device (100), and aims to improve overall inference performance by pre-compressing and storing a key / value cache through interaction with a large-scale language model (200) and rapidly restoring it at the time of inference.

[0075] First, in step S310, the cache processing unit (100) collects key vectors and value vectors generated from multiple layers of a large-scale language model (200) for multiple input sequences.

[0076] For example, the key / value vectors collected at this stage are internal cache values ​​generated through the self-attention operations of each layer corresponding to the input sequence during the prefill phase, and these are subsequently used as input data for global pre-training and compression algorithms.

[0077] Next, in step S320, the cache processing unit (100) learns a global dictionary based on the collected key vectors and value vectors.

[0078] According to one embodiment, a cache processing device (100) can iteratively update atoms of a global dictionary by taking collected key / value vectors as input and applying a numerical optimization technique based on a loss function that minimizes the reconstruction error between a global dictionary matrix and a sparse representation vector.

[0079] This learning can be implemented in a way optimized for GPU parallel computing, and unlike the general K-SVD method, it can be realized as a deep learning-based end-to-end pipeline.

[0080] Subsequently, in step S330, the cache processing unit (100) generates sparse representation information by applying a sparse representation generation algorithm based on a global dictionary to each input key vector or value vector.

[0081] In one embodiment, the cache processing device (100) can generate a sparse representation by using an Orthogonal Matching Pursuit (OMP) algorithm to repeatedly select the prior atom with the largest inner product value with the corresponding input vector and approximate the input vector with a linear combination of the selected atoms.

[0082] The sparse representation information generated as a result can be structured into a set of index-coefficient pairs consisting of the index of a selected prior atom and a coefficient (scale) value corresponding to each index.

[0083] Subsequently, in step S340, the cache processing unit (100) stores the generated sparse representation information.

[0084] According to one embodiment, this storage may be performed on GPU external memory or non-volatile storage and may be managed separately as a pre-cache area or query-corresponding cache memory.

[0085] This storage process is particularly effective in RAG systems or long-context QA models where the total number of documents increases, and provides structural advantages that can reduce total storage space requirements by several times.

[0086] In step S350, the cache processing unit (100) loads the stored sparse representation information in response to a restoration request at the time of inference. For example, this loading operation is performed on related documents selected by a user query.

[0087] In one embodiment, the cache processing device (100) can minimize memory usage and I / O bottlenecks by selectively selecting and loading only some sparse representation information corresponding to the document most relevant to the query among a plurality of sparse representation information corresponding to a plurality of documents.

[0088] Finally, at step S360, the cache processing unit (100) restores the input key vector or value vector based on the loaded sparse representation information and the global dictionary.

[0089] In one embodiment, restoration is implemented by performing a single matrix multiplication operation with atoms of a global dictionary through selected index and coefficient values, and this operation can be processed in parallel on a GPU and performed very quickly.

[0090] The restored key / value cache can be fed into the decoding stage of a large-scale language model and utilized as initial context information to replace the prefill stage.

[0091] Therefore, by replacing the prefill step, which caused the largest delay in the entire inference pipeline, with a pre-computation, compression, and rapid restoration structure, the Time to First Token (TTFT) can be drastically reduced.

[0093] FIG. 4 is a schematic diagram illustrating a comparative example of a RAG inference scenario to which a sparse dictionary-based key-value cache processing method according to one embodiment of the present invention is applied.

[0094] The left side of the drawing illustrates the method of utilizing the KV-cache of the existing Pre-computed RAG, and the right side illustrates the method of utilizing the sparse dictionary-based key-value cache (Sparse KV-Cache) according to the present invention.

[0095] Both methods are identical in that they search for related documents based on a query and retrieve the KV cache corresponding to each document for decoding, but they differ structurally in the processing of storing, loading, and restoring the KV cache.

[0096] First, according to the flow on the left, when a user query is entered, the Retriever module selects multiple documents most similar to the query from the embedding DB. The selected set of documents is {i1, i₂, ..., i k It is represented as}, and for each document i, there exists a KV-Cache Set that has been created and stored in advance.

[0097] In existing pre-computed RAG systems, this KV-Cache Set is directly loaded into GPU memory or HBM, and the decoding process is performed by inputting it along with user queries.

[0098] At this point, since the entire KV cache is highly dimensional and continuous I / O load accumulates as the number of documents increases, there is a limitation in that the Time to First Token (TTFT) is drastically delayed.

[0099] On the other hand, the right side of the drawing illustrates the inference processing flow based on a sparse dictionary-based key-value cache (Sparse KV-Cache) proposed by the present invention.

[0100] The cache processing device (100) according to the present invention stores a pre-generated KV-Cache Set by compressing it into a sparse representation (Sparse Coding) based on a global dictionary. Accordingly, the KV-Cache information corresponding to document i is stored as a Sparse KV-Cache, which has a size several times smaller than the original KV vector.

[0101] At the time of inference, the set of document indices {i1, i₂, ..., i} selected by the Retriever k Only Sparse KV-Caches corresponding to} are selectively loaded, and these Sparse KV-Caches can be utilized in two ways.

[0102] For example, if the compressed form is loaded directly into system memory, it can be utilized to load directly without decompression. On the other hand, if only the compressed form is stored, it can be decompressed and restored based on the stored sparse representation information and global dictionary.

[0103] This restoration process can be composed of a single matrix multiplication operation with the global dictionary according to the present invention, and is optimized for GPU parallel processing to secure the restored KV-cache within a very short time.

[0104] The restored KV-cache is then fed into the LLM's decoding stage along with the query to generate the final output.

[0105] As such, in one embodiment of the present invention, since KV restoration or computation is not involved in the decoding process, compression / restoration processing occurs only at the prefill stage, and there is no effect whatsoever on decoding performance.

[0106] In addition, as illustrated on the right side of Fig. 4, by selectively restoring only the Sparse KV cache of documents related to the user query rather than loading the entire KV cache,

[0107] A query-based selective restore loading method is implemented, providing highly favorable results in terms of memory usage, I / O bottlenecks, power consumption, and throughput.

[0108] In one embodiment of the present invention, sparse coding and dictionary learning may be performed as follows. That is, a signal set Y=[y1,y2,...,yp]∈R mХp is a vector yi ∈ R of dimension m m It consists of, which can be composed of p samples where each column is one sample.

[0109] The goal of dictionary learning is a dictionary D=[d1,d2,...,dn] ∈ R consisting of n atoms. mХn and, sparse representation matrix X=[x1,x2,...,xp] ∈ R nХp It is to find...

[0110] Here, each column xi is a sparse coefficient for approximating the vector yi, and the coefficient is constructed by linearly combining some columns (atoms) of the prior D.

[0111] This goal is formulated as an optimization problem as shown in [Equation 1] below.

[0112] [Mathematical Formula 1]

[0113]

[0114] Here, ∥·∥0 is the 0-norm (L0norm), which represents the number of non-zero items in the vector. s is the maximum number of allowed non-zero terms and serves to limit the sparsity level.

[0115] In one embodiment of the present invention, an orthogonal matching inference-based K-SVD algorithm (K-SVD with Orthogonal Matching Pursuit) may be applied. K-SVD is an iterative algorithm that learns a dictionary in a generalized manner of classic K-means clustering.

[0116] This K-SVD algorithm operates by alternately performing the following two steps.

[0117] Sparse Coding Stage

[0118] Fix the current prior D and estimate the sparse coefficient vector xi for each signal yi.

[0119] Dictionary Update Stage

[0120] For a fixed sparse coefficient X, the prior D is updated. Each atomic dk is individually optimized to minimize the associated residual.

[0121] This algorithm has excellent computational efficiency and interpretability, and has the advantage of performing well even on actual large-scale datasets.

[0122] In one embodiment of the present invention, a sparse coding stage using OMP can be performed. The sparse coding stage using OMP estimates a sparse coefficient vector xi for each signal yi when the prior D is fixed.

[0123] This can be expressed as an optimization problem like the following [Equation 2].

[0124] [Mathematical Formula 2]

[0125]

[0126] Since this is an NP-hard problem, finding an exact solution can be computationally impractical. Therefore, greedy algorithms such as Orthogonal Matching Pursuit (OMP) are commonly used.

[0127] OMP operates on the input signal yi with the following procedure.

[0128] The dot product between the columns of the prior D and yi is calculated to select the most similar atom. Then, the selected atoms are linearly combined to calculate the current approximation vector, the residuals are updated, and this process is repeated until the target sparsity s is reached.

[0129] Because this method offers high computational efficiency and is suitable for GPU parallel processing, it can be effectively applied in real-world application environments, such as key-value cache compression in Large Language Models (LLM).

[0131] FIG. 5 is a block diagram illustrating the configuration of a cache processing device according to one embodiment of the present invention.

[0132] Referring to FIG. 5, the cache processing unit (100) mainly includes a global pre-learner (110), a KV encoder (120), and a KV decoder (130).

[0133] The cache processing unit (100) configures a sparse dictionary-based compression / recovery framework so as to be able to represent and recover key vectors and value vectors generated during the inference process of a large-scale language model with a high compression rate, and this can be designed to be particularly suitable for KV-cache management in a RAG system.

[0134] Key vectors (e.g., yi,K) and value vectors (e.g., yi,V) can be pre-collected from the attention layer of a Large Language Model (LLM) by processing the training dataset. These vectors are respectively the training matrix Y, such as [Equation 3] and [Equation 4] below. K train and Y V train forms.

[0135] [Mathematical Formula 3]

[0136]

[0137] [Mathematical Formula 4]

[0138]

[0139] In this case, for each key vector and value vector, a separate global dictionary D K and D V is learned. The sparse target value s defines the maximum number of non-zero coefficients allowed in the sparse representation of each vector.

[0140] The global pre-learner (110) performs a learning process. The learning process follows an iterative optimization strategy that alternately performs a sparse encoding step and a pre-update step, which is a general structure common to algorithms such as K-SVD. However, SparKV according to the present invention specifically adjusts the pre-update step to suit large-scale KV cache data.

[0141] Specifically, in each iteration, the sparse encoding step is performed first. A mini-batch (Y) of key vectors sampled from the training set. K,batch Mini-batch of ) and value vectors (Y V,batch Regarding ), current dictionary D K and D V Using this, the Orthogonal Matching Pursuit (OMP) algorithm is applied, and accordingly, the sparse coefficient matrix X K,batch and XV,batch can calculate.

[0142] Subsequently, in the dictionary update step, the global dictionary learner (110) can refine the dictionary itself while fixing the previously calculated sparse encoding result. The existing K-SVD algorithm sequentially updates each dictionary atom through singular value decomposition (SVD) on a limited error matrix, but this is computationally very expensive and is not suitable for modern GPU-based large-scale batch learning methods. Therefore, in one embodiment of the present invention, a simultaneous update rule based on mini-batch gradient descent can be applied to effectively process a large amount of KV vectors.

[0143] That is, the entire dictionary D K and D V It can be updated to minimize the reconstruction loss function as shown in [Equation 5] and [Equation 6] below.

[0144] [Mathematical Formula 5]

[0145]

[0146] [Mathematical Formula 6]

[0147]

[0148] After each gradient step, all prior atoms can be normalized again so that their L2-norm becomes 1 to ensure the stability of the learning. This sparse encoding and iterative refinement process of prior atoms continues until convergence is achieved, as described above.

[0149] The KV encoder (120) can process KV-cache compression for RAG corpus preprocessing.

[0150] Global Dictionary D K *Once trained, it can be used to perform preprocessing on documents or document chunks within the RAG corpus. For each document (or document chunk), the corresponding KV-cache vector (individual key vector y j,K} and value vector y j,V It is compressed. This may correspond to the offline stage.

[0151] For each vector, the OMP algorithm identifies a sparse representation. The compressed output—which we call the sparse code—consists of two components: indices for s selected dictionary atoms and their corresponding non-zero coefficient values. More specifically, for a given vector y, compression provides a tuple in the form of (ids, scale), where ids contains the integer indices of the atoms selected from dictionary D and scale contains the floating-point coefficients corresponding to those indices. This tuple is the value stored for each vector.

[0152] These symbols (e.g., C doc,K , C doc,V ) can be stored in an indexed structure.

[0153] The KV decoder (130) can perform RAG query processing through KV-cache decompression.

[0154] When it is time for a RAG query, relevant document chunks are retrieved from the preprocessed corpus. The sparse codes corresponding to the corresponding document chunks are the entire compressed corpus C KV It consists of a subset of, and retrieves it by searching.

[0155] The code set found in this way is C retrieved It is denoted as such, and the KV decoder (130) performs decompression using the stored sparse code.

[0156] For each code, the approximated original dense key vector ( or ) can be reconstructed by calculating the weighted sum of dictionary atoms. Here, the indices (ids) specify the selected atoms, and the scale can be the weight value to be multiplied by each atom. The reconstructed vectors, along with the query's own KV vectors, are used by the LLM's attention mechanism to generate the final response.

[0158] FIG. 6 is a diagram illustrating an example of sparse dictionary-based compression and restoration processing according to an embodiment of the present invention.

[0159] With reference to Fig. 6, the aforementioned sparse dictionary-based compression and restoration process will be explained in more detail.

[0160] Figure 6 (a) illustrates the Offline Global Dictionary Learning step.

[0161] The cache processing device (100) first collects a number of Key vectors and Value vectors generated from the attention layer of a large-scale language model (LLM) for a training corpus. These are respectively Y K train =[y1K,y2K,...,yPK] and Y V train It can be defined as =[y1V,y2V,...,yPV].

[0162] Subsequently, the cache processing unit (100) uses the global dictionary D based on the optimization problem as shown in the following [Equation 7] through the global dictionary learner (110). K ,D V Each can be learned.

[0163] [Mathematical Formula 7]

[0164]

[0165] Here, ∥xi∥0 is the 0-norm, representing the sparsity constraint, and s signifies the maximum number of selectable dictionary atoms. The above optimization problem is solved through a combination of the K-SVD algorithm and the Orthogonal Matching Pursuit (OMP) method, and upon convergence, the global dictionary D K and D V The is completed.

[0166] Figure 6 (b) illustrates the KV-Cache Compression step.

[0167] After the global dictionary is learned, the cache processing unit (100) extracts the KV-cache vector y again from the document (or document chunk). K ,y V Sparse code is generated by applying the OMP algorithm to each.

[0168] For each vector, OMP iteratively performs the optimization problem as shown in [Equation 8] below to select the most suitable prior atoms.

[0169] [Mathematical Formula 8]

[0170]

[0171] The sparse representation information generated through this process is stored in the form of a tuple (ids,scale)(\text{ids}, \text{scale})(ids,scale), where ids is the index (integer) of a selected prior atom and scale is the floating-point coefficient value corresponding to that index.

[0172] These sparse representations dramatically improve storage efficiency, and each block at the bottom of Fig. 6(b) is also represented exemplarily in the form ids = [2, 7], scale = [a, b].

[0173] Figure 6 (c) illustrates the KV-Cache Decompression step.

[0174] At the time of inference, the cache processing unit (100) loads the stored sparse representation (ids, scale) and performs a restoration operation through the KV decoder (130).

[0175] Restoration is performed in the form of a single matrix multiplication, and selected prior atoms d i1 ,d i2 ,...,d is Coefficients a1, a2, ... corresponding to s It is calculated as shown in [Equation 9] below through a linear combination with.

[0176] [Mathematical Formula 9]

[0177]

[0178] Through this, the original key vector y K or value vector y V It is restored approximately, and large-scale language models can utilize it as a prefill cache to improve inference speed.

[0180] FIG. 7 is a schematic diagram showing an example of an experimental configuration performed by a cache processing device (100).

[0181] Figure 7 illustrates the experimental procedure for measuring how effectively the SparKV technique actually operates on an LLM-based RAG system.

[0182] A cache processing device (100) according to one embodiment of the present invention applies a sparse dictionary-based KV-cache compression method, thereby providing a function that can systematically analyze the pattern of performance change under various compression rate settings.

[0183] Accordingly, the cache processing device (100) provides reference information to a model designer or system operator to search for an optimal compression-performance balance point by generating a standard that allows for visual comparison of accuracy changes according to different compression levels and structure settings, as shown in FIG. 7.

[0184] More specifically, the cache processing device (100) can quantitatively collect and visualize the correlation between the compression ratio and average accuracy according to various techniques, as shown in the graph in FIG. 7. In one embodiment of the present invention, the existing Minicache and SVD-based compression methods and the SparKV method of the present invention are compared based on the compression ratio, and among them, the SparKV method demonstrates performance that maintains a high compression ratio while stably maintaining average accuracy.

[0185] In particular, the cache processing unit (100) analyzes the shift of the compression-accuracy curve according to different layer selection methods such as SparKV-L1, SparKV-L2, and SparKV-L4 and changes in the setting of the s value (sparseness criterion). In this process, a general trend is observed in which accuracy gradually decreases as the compression rate increases, and an optimal point is identified in which a relatively high compression rate is achieved while maintaining accuracy close to the baseline level in a specific setting (L1-s64 or L2-s48, etc.).

[0186] Accordingly, the cache processing device (100) according to the present invention can effectively derive a sparse dictionary-based cache compression strategy suitable for a RAG-based application environment based on the comparison results of FIG. 7. This visual information provision function can contribute to realizing strategic model optimization in a real-world usage environment by supporting the system operator to intuitively determine the trade-off between accuracy and resource efficiency.

[0188] Figure 8 shows experimental results comparing the average performance between various cache compression techniques and SparKV configurations for multiple benchmark evaluation items (N-S1, QA-1, etc.) based on the Qwen2.5-7B-Instruct-1M model.

[0189] As shown in the data of FIG. 8, the cache processing device (100) according to the present invention can be used to prove the effectiveness and generalizability of a cache compression technique by quantitatively evaluating performance under various parameter settings using SparKV, a sparse dictionary-based KV cache compression method.

[0190] More specifically, the cache processing unit (100) analyzes performance in the following way.

[0191] Comparative evaluation with existing methods

[0192] As shown in Figure 8, it is confirmed that existing Minicache or SVD-based compression methods cause severe loss of accuracy as the compression ratio increases. For example, when the compression ratio of Minicache is 1.3, the average accuracy is only 5.7%, and when the compression ratio of the SVD method is 6.4, the accuracy drops sharply to 24.8%. On the other hand, the SparKV method of the present invention achieves an excellent compression-performance balance by maintaining an accuracy of over 80% even at compression ratios of 8 to 16 times.

[0193] SparKV Structure and Performance Changes by Level

[0194] SparKV has a structure in which performance varies depending on the compression target layer (L1, L2, L4) and sparsity(s). For example, SparKV-L1-s64 achieved an average accuracy of 88.0% at a compression ratio of 5.3, and SparKV-L2-s64 maintained an accuracy of 76.1% even at a compression ratio of 10.6. These results suggest that the information in a specific layer (L1) is more critical to the model's compression, and that its influence may be limited even at high compression ratios.

[0195] Analysis of performance trends based on scarcity(s)

[0196] Figure 8 also shows the trend of performance improvement as the sparsity s increases. At the same level (L4), the average accuracy was 35.6% when s=32, whereas it increased to 63.9% when s=64 and 84.3% when s=128. This means that as the expressiveness of the sparse code increases, the restoration of the original KV information is performed more precisely.

[0197] Preserving relative performance compared to the reference model

[0198] Several configurations of SparKV maintain an average accuracy of over 95% relative to the baseline, and some configurations even outperform the baseline in certain items (e.g., QA-2, FWE). These results experimentally demonstrate that SparKV is a practical compression technique that can be applied to actual RAG systems.

[0199] Accordingly, through the experiment shown in FIG. 8, the cache processing device (100) of the present invention has proven that the performance of the compression technique under various conditions is superior to that of the conventional one, and it can be seen that the inference speed can be improved and resources saved by using this.

[0201] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. In addition, other processing configurations, such as parallel processors, are also possible.

[0202] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or command the processing unit independently or collectively. Software and / or data may be permanently or temporarily embodied in any type of machine, component, physical device, virtual equipment, computer storage medium or device, or transmitted signal wave so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0203] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

[0204] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0205] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

[0206] Although specific embodiments have been described in the detailed description of this document, it will be obvious to those skilled in the art that various modifications are possible within the scope of this document. Explanation of the symbols

[0207] 100: Cache processing unit 200: Large-scale language models

Claims

Claim 1 A method performed in a cache processing device operating based on a large-scale language model, comprising: a step of collecting key vectors and value vectors generated from multiple layers of the large-scale language model for a plurality of input sequences; a step of learning a global dictionary based on the collected key vectors and value vectors; a step of generating sparse expression information for each of the input key vectors or value vectors by applying a sparse expression generation algorithm based on the global dictionary; a step of storing the generated sparse expression information; and a step of loading the stored sparse expression information in response to a restoration request at the time of inference. A method for compressing and restoring a key-value cache based on a sparse dictionary in a large-scale language model environment, comprising: a step of restoring a key vector or a value vector based on the loaded sparse representation information and the global dictionary; wherein the step of generating the sparse representation information comprises: a step of generating a sparse representation by iteratively selecting the dictionary atom with the largest inner product value with the input key vector or value vector using an Orthogonal Matching Pursuit (OMP) algorithm, and approximating the input vector with a linear combination of the selected dictionary atoms; and wherein the sparse representation information is structured as a set of index-coefficient pairs consisting of the index of the selected dictionary atom and a coefficient (scale) value corresponding to each index. Claim 2 A method for compressing and restoring a sparse dictionary-based key-value cache in a large-scale language model environment, wherein, in claim 1, the step of learning a global dictionary based on the collected key vectors and value vectors comprises: a step of iteratively updating the global dictionary by applying a numerical optimization technique based on a loss function that takes the collected key vectors and value vectors as inputs and minimizes the reconstruction error between the dictionary matrix and the sparse representation vectors. Claim 3 delete Claim 4 delete Claim 5 A method for compressing and restoring a sparse dictionary-based key-value cache in a large-scale language model environment, wherein, in claim 1, the step of storing the generated sparse representation information comprises the step of storing the generated sparse representation information in GPU (Graphics Processing Unit) external memory or non-volatile storage. Claim 6 A method for compressing and restoring a sparse dictionary-based key-value cache in a large-scale language model environment, wherein, in claim 1, the step of restoring a key vector or value vector based on the loaded sparse representation information and the global dictionary comprises the step of restoring the key vector or value vector prior to compression by performing a single matrix multiplication operation on the indices and coefficients included in the loaded sparse representation information and the corresponding atoms of the global dictionary. Claim 7 A method for compressing and restoring a sparse dictionary-based key-value cache in a large-scale language model environment, wherein the restored key vector or value vector is utilized as initial context information in the prefill stage of a large-scale language model. Claim 8 A method for compressing and restoring a sparse dictionary-based key-value cache in a large-scale language model environment, further comprising the step of storing multiple sparse representation information corresponding to multiple documents in claim 1, and selectively loading and restoring only some sparse representation information related to the user's query according to the user's query. Claim 9 A cache processing device operating based on a large-scale language model, comprising: at least one processor; and a memory for storing instructions, wherein when the instructions are executed individually or collectively by the at least one processor, the processor comprises: an operation of collecting key vectors and value vectors generated from multiple layers of the large-scale language model for multiple input sequences; an operation of learning a global dictionary based on the collected key vectors and value vectors; an operation of generating sparse expression information for each input key vector or value vector by applying a sparse expression generation algorithm based on the global dictionary; an operation of storing the generated sparse expression information; an operation of loading the stored sparse expression information in response to a restoration request at the time of inference; and an operation of restoring the key vector or value vector based on the loaded sparse expression information and the global dictionary. A cache processing device comprising: performing the operation to generate the sparse representation information, wherein the operation to generate the sparse representation information includes the operation of generating the sparse representation by repeatedly selecting the prior atom with the largest inner product value with the input vector using an Orthogonal Matching Pursuit (OMP) algorithm and approximating the input vector with a linear combination of the selected prior atoms; and wherein the sparse representation information is structured as a set of index-coefficient pairs consisting of the index of the selected prior atom and a coefficient (scale) value corresponding to each index. Claim 10 A computer-readable recording medium having a computer program stored therein for performing a sparse dictionary-based key-value cache compression and restoration method in a large-scale language model environment according to any one of claims 1 to 2 and 5 to 8, combined with hardware.

Images