Method and Apparatus for KV Cache Compression Using Candidate Atom Selection by MLP Encoder and OMP-Based Refinement

Abstract

A method for compressing a KV cache of a large-scale language model, performed by a computing device comprising at least one processor according to an embodiment of the present invention, comprises: a step of obtaining a KV cache for a token input to the large-scale language model; a step of inputting the KV cache into an MLP encoder to calculate a coefficient corresponding to each of a plurality of atoms included in a sparse dictionary; a step of selecting a candidate atom among the plurality of atoms based on the coefficient; and a step of performing an OMP on the candidate atom to calculate a final atom and a coefficient for the final atom among the candidate atoms for representing the KV cache.

Description

Technology Field

[0001] The present invention relates to a method and apparatus for compressing a KV cache using an encoder model and a sparse dictionary, and more specifically, to a compression technology for efficiently storing and managing a Key-Value Cache (KV cache) generated in a Large Language Model (LLM). Background Technology

[0002] Recently, large-scale language models have demonstrated excellent performance in various natural language processing tasks, and various techniques to improve inference efficiency are being applied. KV caches are widely used as a technique to reduce the repetition of identical operations and improve inference speed by storing key vectors and value vectors for previous tokens during the attention operation and reusing them in subsequent operations.

[0003] However, since the KV cache has the characteristic of continuously increasing in proportion to the length of the input sequence, storage space and memory usage can increase significantly as the input sentence becomes longer or the number of tokens to be processed increases. In addition, since large-scale language models generally include multiple layers and multiple attention heads, the size of the entire KV cache can become very large if all key vectors and value vectors generated for each layer and each head are stored.

[0004] Such an increase in KV cache can increase the memory burden on the inference system, raise data transfer costs between storage and memory, and degrade the overall processing efficiency of the system. Particularly in environments with limited memory resources, processable context lengths or concurrent throughput may be restricted, and the costs required for cache storage and management can become a significant constraint on overall system performance.

[0005] Accordingly, there is an increasing need for compression technology that can reduce the storage capacity of the KV cache while maintaining its informational utility as much as possible. The problem to be solved

[0006] The objective of the present invention, devised to solve the aforementioned problems, is to provide a KV cache compression method and apparatus capable of reducing the storage capacity and memory usage of a KV cache by compressing the KV cache generated during the inference process of a large-scale language model using an MLP encoder and a sparse dictionary.

[0007] The objective of the present invention is to provide a KV cache compression method and apparatus that can reduce computational load and processing time compared to performing OMP on the entire sparse dictionary by using an MLP encoder to preferentially select candidate atoms among a plurality of atoms included in a sparse dictionary and performing OMP only on the selected candidate atoms.

[0008] The objective of the present invention is to provide a KV cache compression method and apparatus capable of improving both the compression efficiency and reconstruction accuracy of a KV cache by using the final atom and the coefficients for the final atom as the compression representation according to the result of performing OMP, instead of directly using the coefficients calculated by an MLP encoder as the final compression representation.

[0009] The objective of the present invention is to provide a KV cache compression method and apparatus that enables an MLP encoder to select atoms that are highly likely to be selected by OMP as candidate atoms by training the MLP encoder using OMP labels generated by performing OMP based on a sparse dictionary on a KV cache for training. means of solving the problem

[0010] A method for compressing a KV cache of a large-scale language model, performed by a computing device including at least one processor according to the present invention for achieving the above-described purpose, comprises: a step of obtaining a KV cache for a token input to the large-scale language model; a step of inputting the KV cache into an MLP encoder to calculate a coefficient corresponding to each of a plurality of atoms included in a sparse dictionary; a step of selecting a candidate atom among the plurality of atoms based on the coefficient; and a step of performing an OMP on the candidate atom to calculate a final atom and a coefficient for the final atom among the candidate atoms for representing the KV cache.

[0011] In addition, the step of selecting the candidate atoms is characterized by selecting the top N atoms among the plurality of atoms as candidate atoms in order of the magnitude of the absolute value of the coefficients calculated by the MLP encoder.

[0012] In addition, the above N is characterized by being set to a value greater than or equal to the number of final atoms finally selected by the above OMP, and less than or equal to the number of multiple atoms K included in the above sparse dictionary.

[0013] In addition, the coefficients calculated by the MLP encoder are used as selection criteria for the candidate atoms, and the coefficients for the final atoms based on the result of performing the OMP are used to represent the KV cache in a compressed manner.

[0014] Additionally, the method is characterized by further including the step of generating a sparse dictionary using a training KV cache; the step of generating an OMP label by performing OMP on the training KV cache based on the sparse dictionary; and the step of training an MLP encoder using the OMP label.

[0015] In addition, the step of training the MLP encoder is characterized by training the MLP encoder to select an atom identical to the atom selected by performing OMP on the training KV cache.

[0016] In addition, the step of training the MLP encoder is characterized by constructing a loss function by assigning a weight greater than the error of the position corresponding to the atom not selected by the OMP to the error of the position not selected by the OMP.

[0017] In addition, the values ​​of the plurality of atoms included in the above sparse dictionary are characterized by being fixed while the result of performing an OMP on the above learning KV cache is used for the training of the MLP encoder.

[0018] In addition, the KV cache corresponds to at least one of a plurality of transformer layers or a plurality of attention heads of the large-scale language model, and each of the steps is performed for each of the plurality of transformer layers or for each of the plurality of attention heads.

[0019] An apparatus for compressing a KV cache of a large-scale language model according to the present invention comprises: a memory in which one or more instructions are stored; and at least one processor configured to execute one or more instructions stored in the memory, wherein the processor performs the steps of: acquiring a KV cache for a token input to the large-scale language model; inputting the KV cache into an MLP encoder to calculate a coefficient corresponding to each of a plurality of atoms included in a sparse dictionary; selecting a candidate atom among the plurality of atoms based on the coefficient; and performing an OMP on the candidate atom to calculate a final atom and a coefficient for the final atom among the candidate atoms for representing the KV cache. Effects of the invention

[0020] According to the present invention as described above, a KV cache compression method and apparatus can be provided that can reduce the storage capacity and memory usage of the KV cache by compressing the KV cache generated during the inference process of a large-scale language model using an MLP encoder and a sparse dictionary.

[0021] The present invention can provide a KV cache compression method and apparatus that can reduce computational load and processing time compared to performing OMP on the entire sparse dictionary by using an MLP encoder to preferentially select candidate atoms among a plurality of atoms included in a sparse dictionary and performing OMP only on the selected candidate atoms.

[0022] The present invention can provide a KV cache compression method and apparatus that can improve both the compression efficiency and reconstruction accuracy of the KV cache by using the final atom and the coefficients for the final atom as the compression representation according to the result of performing OMP, instead of directly using the coefficients calculated by the MLP encoder as the final compression representation.

[0023] The present invention can provide a KV cache compression method and apparatus that enables an MLP encoder to select atoms that are highly likely to be selected by OMP as candidate atoms by training the MLP encoder using OMP labels generated by performing OMP based on a sparse dictionary on a KV cache for training. Brief explanation of the drawing

[0024] FIG. 1 is a diagram illustrating a KV cache compression and restoration system according to one embodiment of the present invention. FIG. 2 is a diagram illustrating the candidate atom selection and final atom determination process according to one embodiment of the present invention. FIG. 3 is a flowchart illustrating a KV cache compression method according to an embodiment of the present invention. FIG. 4 is a diagram illustrating the overall processing flow of an MLP encoder learning method according to an embodiment of the present invention. FIG. 5 is a diagram illustrating the process of generating a sparse dictionary using a learning KV cache according to an embodiment of the present invention and freezing the generated sparse dictionary. FIG. 6 is a diagram illustrating a method for training an MLP encoder by comparing an OMP label and the output of an MLP encoder according to an embodiment of the present invention. FIG. 7 is a diagram illustrating the overall processing flow of a KV cache compression method using an MLP encoder and OMP according to an embodiment of the present invention. FIG. 8 is a diagram illustrating a method for selecting candidate atoms using an MLP encoder according to an embodiment of the present invention. FIG. 9 is a diagram illustrating a method for calculating the final atom and the coefficients for the final atom by performing OMP on candidate atoms according to one embodiment of the present invention. FIG. 10 is a configuration diagram for explaining a computing device according to an embodiment of the present invention. Specific details for implementing the invention

[0025] In the following, embodiments related to the present invention are illustrated in the drawings and described in detail through the detailed description. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms and should be understood to include all modifications, equivalents, and substitutions that fall within the spirit and scope of the present invention.

[0026] In describing the components of the present invention, terms such as first, second, A, B, (a), (b), etc., may be used. These terms are intended merely to distinguish the component from other components, and the essence, order, or sequence of the component is not limited by such terms. Furthermore, where it is stated in this specification that a component is "connected," "coupled," or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but that another component may also be "connected," "coupled," or "connected" between each component. In the case of "connection," "coupled," or "connected," it may be understood not only as being physically "connected," "coupled," or "connected," but also, if necessary, electrically "connected," "coupled," or "connected."

[0027] Terms such as "~part (unit)," "~device," "~part," and "~module" as used in this specification refer to a unit that processes at least one function or operation, and may be implemented in hardware, software, or a combination of hardware and software. Furthermore, terms such as "include," "compose," or "have" as used in this specification, unless specifically stated otherwise, mean that the relevant component may be inherent; therefore, they should be interpreted as allowing for the inclusion of additional components rather than excluding other components.

[0028] Furthermore, it is intended to clarify that the classification of components in this specification is merely based on the primary function each component is responsible for. That is, two or more components described below may be combined into a single component, or a single component may be divided into two or more components based on more subdivided functions. Additionally, each component described below may additionally perform some or all of the functions of other components in addition to its primary function, and it is obvious that some of the primary functions of each component may be exclusively performed by other components.

[0029] Hereinafter, the present invention will be described with reference to the drawings related to embodiments of the present invention.

[0030] FIG. 1 is a diagram illustrating a KV cache compression and restoration system according to one embodiment of the present invention.

[0031] Referring to FIG. 1, the KV cache compression and restoration system may include a large-scale language model (20), a KV cache compression device (10), a storage unit (30), and a KV cache restoration device (40). The KV cache may refer to a set of key vectors and value vectors generated during the inference process of the large-scale language model. Additionally, the key vector may refer to a vector for calculating the degree of relevance with a query vector in an attention operation, and the value vector may refer to a vector containing information that is actually referenced according to the degree of relevance.

[0032] A large-scale language model (20) can generate token-specific hidden states, key vectors, and value vectors based on input data. The hidden states may be the result of operations of each transformer layer of the large-scale language model (20). The hidden states may be intermediate representation vectors prior to KV projection. The KV cache compression device (10) can distinguish between important tokens and non-important tokens based on information generated from the large-scale language model (20), and selectively compress the KV cache corresponding to the important tokens. At this time, important tokens may be tokens that are highly likely to be used relatively significantly in the subsequent inference process, and non-important tokens may be tokens with relatively low contribution. Accordingly, the KV cache compression device (10) can improve compression efficiency by performing compression processing on important tokens and removing or excluding non-important tokens from the compression target.

[0033] The storage unit (30) can store the compressed KV cache generated by the KV cache compression device (10). By storing the compressed KV cache instead of the original KV cache, the storage unit (30) can reduce storage space and reduce the input / output time required during the process of reading the compressed KV cache.

[0034] The KV cache recovery device (40) can generate a recovered KV cache by decoding the compressed KV cache stored in the storage unit (30). The KV cache recovery device (40) can provide the recovered KV cache to the large-scale language model (20). Accordingly, the KV cache stored in a compressed state can be recovered at the time of actual computation and used for attention computation or subsequent inference processes.

[0035] That is, the KV cache compression device (10) distinguishes between important tokens and non-important tokens and selectively compresses and restores the KV cache corresponding to the important tokens, thereby increasing the compression efficiency of the entire KV cache and reducing memory usage, storage capacity, and I / O burden.

[0037] FIG. 2 is a diagram illustrating the candidate atom selection and final atom determination process according to one embodiment of the present invention.

[0038] Referring to FIG. 2, the KV cache compression device (10) may first refer to a plurality of atoms included in a sparse dictionary. The sparse dictionary may be a set of reference atoms that are pre-generated to represent the KV cache as a combination of a small number of atoms and coefficients.

[0039] The KV cache compression device (10) can input the KV cache to be compressed into an MLP encoder. The MLP encoder can calculate coefficients corresponding to each of the multiple atoms included in the sparse dictionary, and the KV cache compression device (10) can select N candidate atoms among the multiple atoms based on the calculated coefficients. Here, the candidate atoms may be atoms that are likely to be used to represent the KV cache.

[0040] Subsequently, the KV cache compression device (10) may perform OMP only on N candidate atoms selected by the MLP encoder, rather than performing OMP on all atoms included in the entire sparse dictionary. Accordingly, the search range of OMP can be reduced from all atoms to candidate atoms.

[0041] The KV cache compression device (10) can determine s final atoms out of N candidate atoms based on the results of performing OMP on the candidate atoms. The final atoms may be atoms actually used to represent the compressed KV cache, and coefficients for the final atoms may be used to construct the compressed representation of the KV cache.

[0042] That is, according to the embodiment illustrated in FIG. 2, the KV cache compression device (10) can determine the final atoms by first selecting candidate atoms among a plurality of atoms of a sparse dictionary using an MLP encoder and performing OMP only on the selected candidate atoms. Accordingly, compared to the method of performing OMP on the entire sparse dictionary, the amount of computation is reduced, and the final atoms for representing the KV cache can be precisely determined through OMP.

[0044] FIG. 3 is a flowchart illustrating a KV cache compression method according to an embodiment of the present invention. Each step illustrated in FIG. 3 can be performed by a KV cache compression device (10).

[0045] In step 310, the KV cache compression device (10) can obtain a KV cache for a token input to a large-scale language model. Here, the large-scale language model may be a transformer-based language model that generates a subsequent token based on a plurality of input tokens. The KV cache may include at least one of a key vector and a value vector generated during the attention operation process of the large-scale language model. For example, the KV cache compression device (10) can obtain a KV cache to be compressed by directly receiving a KV cache generated by the large-scale language model during the process of processing the input token, or by loading a KV cache stored in memory or storage.

[0046] In step 320, the KV cache compression device (10) inputs the KV cache into an MLP encoder to calculate coefficients corresponding to each of the multiple atoms included in the sparse dictionary. Here, the MLP encoder may be an encoder model having a multilayer perceptron structure, and the sparse dictionary may be a set of reference atoms generated in advance to represent the KV cache as a combination of a few atoms and coefficients. Each atom included in the sparse dictionary may be a reference vector for approximately representing the vector constituting the KV cache. The KV cache compression device (10) may obtain coefficient vectors corresponding to each of the multiple atoms of the sparse dictionary from the output of the MLP encoder, and the coefficient vectors may indicate the possibility or contribution of each of the multiple atoms to be used to represent the currently input KV cache.

[0047] In step 330, the KV cache compression device (10) can select candidate atoms among a plurality of atoms included in the sparse dictionary based on coefficients. In one embodiment, the KV cache compression device (10) can select the top N atoms among a plurality of atoms as candidate atoms in order of the magnitude of the absolute value of the coefficient calculated by the MLP encoder. Here, N can be set to a value greater than or equal to s, the number of final atoms finally selected by the Orthogonal Matching Pursuit (OMP), and less than or equal to K, the number of atoms included in the sparse dictionary. That is, the KV cache compression device (10) can first select N atoms that are highly likely to be used to represent the KV cache from a total of K atoms as candidate atoms. Accordingly, in the subsequent OMP execution step, the search target can be reduced to candidate atoms rather than all atoms in the entire sparse dictionary.

[0048] In step 340, the KV cache compression device (10) can perform OMP on candidate atoms to calculate final atoms and coefficients for final atoms to represent the KV cache among the candidate atoms. Here, OMP stands for Orthogonal Matching Pursuit and may refer to a sparse approximation algorithm that iteratively selects atoms that reduce residuals to approximate an input vector as a combination of atoms in a sparse dictionary. The KV cache compression device (10) can perform OMP based on a partial sparse dictionary composed of candidate atoms and can select s final atoms among the candidate atoms based on the result of performing OMP. Additionally, the KV cache compression device (10) can calculate coefficients for each final atom so that the selected final atoms approximate the KV cache.

[0049] In one embodiment, the coefficients calculated by the MLP encoder can be used as criteria for selecting candidate atoms, and the coefficients for the final atoms based on the result of performing OMP can be used to compress and represent the KV cache. That is, the coefficients calculated by the MLP encoder are used as values ​​to quickly select candidate atoms from the entire sparse dictionary, and the coefficients included in the actual compressed representation of the KV cache can be calculated based on the result of performing OMP on the candidate atoms. Accordingly, the KV cache compression device (10) can reduce the search range of OMP using the MLP encoder, while also precisely determining the final atoms and final coefficients for representing the KV cache using OMP.

[0050] In addition, in one embodiment of the present invention, the MLP encoder can be trained using OMP labels. Specifically, the KV cache compression device (10) can generate a sparse dictionary using a KV cache for training, generate OMP labels by performing OMP on the KV cache for training based on the sparse dictionary, and train the MLP encoder using the generated OMP labels. Here, the OMP labels may include an atom selected by OMP to represent the KV cache for training and a coefficient for that atom. The MLP encoder can be trained to select an atom identical to the atom selected by performing OMP on the KV cache for training.

[0051] Additionally, during the process of training the MLP encoder, if the atom selected by performing OMP on the training KV cache and the atom selected by the MLP encoder are the same, the KV cache compression device (10) can construct a loss function by weighting the difference between the coefficients for the atom selected by performing OMP and the coefficients for the atom selected by the MLP encoder. Accordingly, the MLP encoder can be trained to increase the probability of selecting the atom actually selected by OMP as a candidate atom.

[0052] Additionally, the values ​​of multiple atoms included in the sparse dictionary can be fixed while the results of performing OMP on the training KV cache are used for training the MLP encoder. When OMP labels are generated and the MLP encoder is trained while the atomic values ​​of the sparse dictionary are fixed, the meaning of the atoms indicated by the same atomic index can be maintained during the training process. Accordingly, the MLP encoder can stably learn the atoms selected by the OMP based on the fixed sparse dictionary.

[0053] Additionally, the KV cache may correspond to at least one of a plurality of transformer layers or a plurality of attention heads of a large-scale language model. In this case, each step illustrated in FIG. 3 may be performed for each of the plurality of transformer layers or for each of the plurality of attention heads. For example, the KV cache compression device (10) may acquire a KV cache corresponding to a specific transformer layer and a specific attention head, and perform candidate atomic selection and OMP-based final atomic calculation using a sparse dictionary and an MLP encoder corresponding to the layer or the head. In another embodiment, each step of FIG. 3 may be performed using a common sparse dictionary and an MLP encoder for a plurality of layers or a plurality of heads.

[0054] As such, according to the embodiment illustrated in FIG. 3, the KV cache compression device (10) can first select candidate atoms among a plurality of atoms included in a sparse dictionary using an MLP encoder, and perform OMP only on the selected candidate atoms to calculate the final atoms and coefficients for the final atoms for representing the KV cache. Accordingly, the amount of OMP computation can be reduced compared to when OMP is performed on the entire sparse dictionary, and the accuracy of calculating the final atoms and coefficients can be improved compared to when the KV cache is compressed using only the coefficients calculated by the MLP encoder.

[0056] FIG. 4 is a diagram illustrating the overall processing flow of an MLP encoder learning method according to an embodiment of the present invention.

[0057] Referring to FIG. 4, the KV cache compression device (10) may perform steps of generating and freezing a sparse dictionary, generating an OMP label, and training the MLP encoder in order to train the MLP encoder. Meanwhile, in FIG. 4, the OMP label generation step and the MLP encoder training step are shown separately for convenience of explanation, but the present invention is not limited thereto. In one embodiment, the KV cache compression device (10) may perform OMP on the KV cache for training to generate and store an OMP label in advance, and then train the MLP encoder using the stored OMP label. In another embodiment, the KV cache compression device (10) may perform OMP on the KV cache for training to generate an OMP label during the training process of the MLP encoder, and train the MLP encoder using the generated OMP label. In the former case, the burden of performing OMP can be reduced during the training process of the MLP encoder, and in the latter case, the storage space required to separately store OMP labels for the entire training KV cache can be reduced.

[0058] First, the KV cache compression device (10) can generate a sparse dictionary using a training KV cache. Here, the training KV cache may include key vectors and / or value vectors generated by inputting training data or a text corpus into a large-scale language model. The sparse dictionary may be a set of reference atoms for representing the training KV cache as a combination of a few atoms and coefficients. In one embodiment, the sparse dictionary may be represented by a matrix D containing a plurality of atoms, and D ∈ R^(d × K). Here, d may represent the dimension of a vector included in the training KV cache, and K may represent the number of atoms included in the sparse dictionary.

[0059] The KV cache compression device (10) can freeze the sparse dictionary after generating the sparse dictionary. Here, freezing the sparse dictionary means maintaining the values ​​of multiple atoms included in the sparse dictionary so that they do not change during the subsequent OMP label generation step and MLP encoder training step. When the sparse dictionary is frozen, the same atomic index can point to the same atomic throughout the training process. Therefore, the meaning of the atomic index and coefficient generated as a result of OMP execution can be consistently maintained during the training process.

[0060] Next, the KV cache compression device (10) can generate OMP labels by performing OMP on the training KV cache based on a frozen sparse dictionary. The OMP labels may include atomic indices selected by OMP to represent the training KV cache and coefficients corresponding to the selected atomic indices. Specifically, the KV cache compression device (10) can perform OMP greedy selection and Least Squares (LS) operations based on a frozen sparse dictionary D for the training KV cache. OMP greedy selection may be a process of selecting atoms that best reduce the current residual, and the least squares operation may be a process of calculating coefficients for each atom so that the error between the training KV cache and the reconstruction vector is reduced given a set of selected atoms. Accordingly, the KV cache compression device (10) can generate OMP labels corresponding to the training KV cache.

[0061] Next, the KV cache compression device (10) can input the training KV cache into an MLP encoder to produce raw_codes. Here, raw_codes may be a vector containing coefficients or scores corresponding to each of the multiple atoms included in the sparse dictionary. For example, if the sparse dictionary contains K atoms, the MLP encoder may output K-dimensional raw_codes. Each element of raw_codes may represent the possibility or contribution of the corresponding atom being used to represent the training KV cache.

[0062] The KV cache compression device (10) can calculate a loss function by comparing raw_codes, which are the outputs of the MLP encoder, with the OMP labels, and train the MLP encoder based on the calculated loss function. In one embodiment, the KV cache compression device (10) can assign a relatively large weight to the atomic positions selected by the OMP in the OMP labels. Accordingly, the MLP encoder can be trained to predict atoms likely to be selected by the OMP with a high priority.

[0063] Specifically, the KV cache compression device (10) can generate a full label vector omp_full corresponding to the OMP label. omp_full may have dimensions equal to the number of atoms K of the sparse dictionary, and the coefficient c* calculated by the OMP is placed at the position corresponding to the support S* (set of selected atomic indices) selected by the OMP, and 0 is placed at the remaining positions. That is, omp_full can be defined as follows.

[0064] omp_full[S*] = c*

[0065] omp_full[k] = 0, if k S*

[0066] Here, S* may represent the set of correct atomic indices selected by the OMP for the training KV cache, and c* may represent the correct coefficients corresponding to each atom included in S*.

[0067] The KV cache compression device (10) can calculate sparse_code_loss based on the error between the output raw_codes and omp_full of the MLP encoder. In one embodiment, sparse_code_loss can be calculated based on the following mathematical formula.

[0068] sparse_code_loss = mean( w(k) · (raw_codes[k] - omp_full[k])²)

[0069] Here, w(k) may represent a weight corresponding to an atomic index k. In one embodiment, w(k) may be set to a relatively large value for atomic indices selected by the OMP and to a relatively small value for atomic indices not selected by the OMP. For example, w(k) may be defined as follows.

[0070] w(k) = α, if k ∈S*

[0071] w(k) = 1, if k S*

[0072] Here, α can be a value greater than 1, and can be set to, for example, 5, 10, 20, etc. However, the specific value of α may be changed depending on the embodiment. In this way, by giving greater weight to the error of the position corresponding to the atomic index selected by the OMP, the MLP encoder can be trained to prioritize selecting the atom selected by the OMP as a candidate atom.

[0073] In addition, in one embodiment, the KV cache compression device (10) can train an MLP encoder using a reconstruction loss along with sparse_code_loss. In this case, the total training loss total_loss can be calculated based on the following mathematical formula.

[0074] total_loss = sparse_code_loss + λ_recon · recon_loss

[0075] Here, λ_recon may be a weight that controls the degree of reflection of the reconstruction loss, and recon_loss may be a loss calculated based on the difference between the vector reconstructed by the combination of atoms selected based on the output of the MLP encoder and the training KV cache. In one embodiment, recon_loss may be calculated using a straight-through estimator (STE) over the top-s mask. STE may be a technique for using the actual selection result in forward propagation and transmitting an approximate gradient in backpropagation for selection operations that are difficult to differentiate, such as top-s selection.

[0076] As such, according to the embodiment illustrated in FIG. 4, the KV cache compression device (10) can generate OMP labels based on a frozen sparse dictionary and train an MLP encoder using the OMP labels. Accordingly, the MLP encoder can be trained to select an atom that is identical or similar to the atom selected by the OMP for the training KV cache as a candidate atom, and can efficiently select a candidate atom to be reviewed by the OMP from among a plurality of atoms in the entire sparse dictionary during the inference step.

[0078] FIG. 5 is a diagram illustrating the process of generating a sparse dictionary using a learning KV cache according to an embodiment of the present invention and freezing the generated sparse dictionary.

[0079] Referring to FIG. 5, the KV cache compression device (10) can generate a sparse dictionary D using a training KV cache Y. The training KV cache (Y) may include a plurality of KV cache vectors obtained by inputting training data or a text corpus into a large-scale language model. In one embodiment, Y may be represented as a matrix containing p vectors of m dimensions. Here, m may be the dimension of each training KV cache vector, and p may be the number of training KV cache vectors used for generating the sparse dictionary.

[0080] The KV cache compression device (10) can learn a sparse dictionary D using a learning KV cache (Y). The sparse dictionary D consists of multiple atoms d0, d₁, ..., d k- It may include ₁. In one embodiment, D may be represented as a matrix containing K atoms of d dimension. That is, D may be represented as follows.

[0081] D = {d0, d₁, ..., d k- ₁}

[0082] Here, each atom d can be a reference vector for sparsely representing the training KV cache vector, and K can represent the number of atoms included in the sparse dictionary. The sparse dictionary D can be expressed as D∈R^(d×K).

[0083] The KV cache compression device (10) can learn a sparse dictionary D such that vectors contained in the training KV cache (Y) can be well approximated by a small number of atomic combinations contained in the sparse dictionary D. For example, the KV cache compression device (10) can generate a sparse dictionary D using at least one of OMP, K-SVD, neural network-based pre-learning, or reconstruction loss-based learning methods. However, the specific algorithm for generating the sparse dictionary D is not limited thereto.

[0084] When a sparse dictionary D is generated, the KV cache compression device (10) can freeze the sparse dictionary D. Freezing the sparse dictionary D may mean that the values ​​of multiple atoms included in D are maintained during the OMP label generation step and the MLP encoder training step. That is, while the MLP encoder is being trained, the atoms d0, d₁, ..., d of the sparse dictionary D k- ₁ may not be updated.

[0085] The reason for freezing the sparse dictionary D is that the OMP labels depend on the sparse dictionary D. For example, even if specific atomic indices and coefficients are generated as OMP labels by performing OMP on a training KV cache vector, if the atomic value of the sparse dictionary D is subsequently changed, the meaning of the atoms indicated by the same atomic index may change. In this case, the previously generated OMP labels may no longer have the same meaning. Therefore, the KV cache compression device (10) can freeze the sparse dictionary D after generating the sparse dictionary D, generate OMP labels based on the frozen sparse dictionary D, and train an MLP encoder based on the same sparse dictionary D.

[0086] As such, according to the embodiment illustrated in FIG. 5, the KV cache compression device (10) can generate a sparse dictionary D using a learning KV cache (Y) and freeze the generated sparse dictionary D. Accordingly, the correspondence between the atomic index and the atomic value can be consistently maintained in the subsequent OMP label generation step and the MLP encoder learning step, and the MLP encoder can stably learn the atomic pattern selected by the OMP based on the frozen sparse dictionary D.

[0088] FIG. 6 is a diagram illustrating a method for training an MLP encoder by comparing an OMP label and the output of an MLP encoder according to an embodiment of the present invention.

[0089] Referring to FIG. 6, the KV cache compression device (10) can generate OMP labels by performing OMP using a training KV cache and a frozen sparse dictionary D. Additionally, the KV cache compression device (10) can generate an output of the MLP encoder by inputting the same training KV cache into the MLP encoder. Subsequently, the KV cache compression device (10) can train the MLP encoder by comparing the OMP labels with the output of the MLP encoder and calculating a loss based on the comparison result.

[0090] First, the KV cache compression device (10) can perform OMP based on a sparse dictionary D that is frozen for the training KV cache. The OMP can iteratively select the atoms included in the sparse dictionary that best approximate the input vector based on the current residual. Additionally, the OMP can perform least squares operations on the selected atoms to calculate coefficients corresponding to each atom. Accordingly, the KV cache compression device (10) can generate OMP labels.

[0091] In the example illustrated in FIG. 6, the OMP label may include ids and scale. ids may be indices of atoms included in a sparse dictionary, and scale may be a coefficient corresponding to each atomic index. For example, in the OMP label of FIG. 6, the scale values ​​corresponding to atomic indices 2, 4, 6, and 7 may be indicated as 0.3, 0.4, 0.2, and 0.6, respectively. In this case, the OMP can be understood as having selected atomic indices 2, 4, 6, and 7 to represent a KV cache for training. Thus, the support S* of the OMP label may be {2, 4, 6, 7}, and the corresponding coefficient c* may be {0.3, 0.4, 0.2, 0.6}.

[0092] Meanwhile, the KV cache compression device (10) can input the training KV cache into the MLP encoder to generate an MLP encoder output. The MLP encoder can output raw_codes containing coefficients corresponding to each of the multiple atoms included in the sparse dictionary D. In the example illustrated in FIG. 6, the MLP encoder output is performed for all atoms, and scale values ​​corresponding to each atom index can be output as 0.1, 0.2, 0.8, 0.3, 0.1, 0.2, 0.4, 0.5, and 0.7, respectively.

[0093] The KV cache compression device (10) can construct a loss function by comparing the OMP label and the MLP encoder output. In one embodiment, the KV cache compression device (10) can train the MLP encoder so that the atom selected by the OMP in the OMP label matches the atom selected by the MLP encoder or assigned a high coefficient. Additionally, if the atom selected by the OMP and the atom selected by the MLP encoder are the same, the KV cache compression device (10) can construct a loss function by assigning weights to the difference between the coefficient for the atom calculated by performing the OMP and the coefficient for the corresponding atom calculated by the MLP encoder.

[0094] In the example of FIG. 6, the positions corresponding to atomic indices 2, 4, 6, and 7 selected in the OMP label may be positions to which weights are applied during MLP encoder training. That is, the KV cache compression device (10) may apply a relatively large learning weight so that the scale values ​​corresponding to atomic indices 2, 4, 6, and 7 in the MLP encoder output become close to the scale values ​​of the OMP label. For example, for atomic index 2, the difference between the scale value of the OMP label 0.3 and the scale value of the MLP encoder output 0.8 may be reflected in the loss calculation, and for atomic index 4, the difference between the scale value of the OMP label 0.4 and the scale value of the MLP encoder output 0.1 may be reflected in the loss calculation. At this time, a relatively large weight may be applied to the atomic positions selected by the OMP.

[0095] As such, according to the embodiment illustrated in FIG. 6, the KV cache compression device (10) can use an OMP label generated based on a frozen sparse dictionary D as the correct answer and train the MLP encoder to select an atom that is identical or similar to the atom selected by the OMP. Additionally, by assigning a relatively large weight to the atom position selected by the OMP, the MLP encoder can be trained to select a candidate atom that is highly likely to be selected by the OMP as the final atom during the inference phase. Accordingly, during the inference phase, the MLP encoder can efficiently select a candidate atom among a plurality of atoms included in the entire sparse dictionary, and the KV cache compression device (10) can perform OMP only on the selected candidate atom to calculate the final atom and the coefficient for the final atom.

[0097] FIG. 7 is a diagram illustrating the overall processing flow of a KV cache compression method using an MLP encoder and OMP according to an embodiment of the present invention.

[0098] Referring to FIG. 7, the KV cache compression device (10) can sequentially perform a candidate atom selection step and an OMP execution step. The candidate atom selection step may be a step of selecting candidate atoms to be subject to OMP execution among a plurality of atoms included in a sparse dictionary using an MLP encoder. The OMP execution step may be a step of performing OMP only on the selected candidate atoms to calculate the final atoms and coefficients for the final atoms to represent the KV cache.

[0099] First, in the candidate atom selection step, the KV cache compression device (10) may input the KV cache to be compressed into an MLP encoder. Here, the KV cache may include at least one of a key vector and a value vector generated for a token input into a large-scale language model. In one embodiment, the KV cache to be compressed may be a vector corresponding to one token, one transformer layer, or one attention head.

[0100] A sparse dictionary may be a set of reference atoms to represent a KV cache as a combination of a few atoms and coefficients. A sparse dictionary may be represented by a matrix D containing multiple atoms, where D∈R^(d×K). Here, d may represent the vector dimension of the KV cache to be compressed or the dimension of each atom in the sparse dictionary, and K may represent the total number of atoms included in the sparse dictionary.

[0101] The KV cache compression device (10) can represent the KV cache to be compressed as x. In this case, x can be a d-dimensional vector and can be expressed as D∈R^d. An MLP encoder can receive x as input and produce K-dimensional raw_codes corresponding to each of the K atoms included in the sparse dictionary D. That is, raw_codes can be expressed as raw_codes∈R^K. Each element of raw_codes can be a coefficient or score representing the likelihood or contribution of the corresponding atom included in the sparse dictionary to be used to represent the KV cache to be compressed.

[0102] In one embodiment, the candidate atom selection step can be expressed as follows.

[0103] Stage 1 ― MLP Candidate Selection (K→N)

[0104] The input vector x is passed through an MLP encoder to obtain raw_codes∈R^K, and the set of the top N indices C of |raw_codes| is extracted as a set of candidate atoms.

[0105] That is, the KV cache compression device (10) can select the top N atoms out of K atoms as candidate atoms based on the size of the absolute value of raw_codes. At this time, the set of candidate atom indices can be represented as C, and C can be a subset of the entire set of atom indices {1, ⪋, K}. Also, the size of C can be N. This can be expressed as follows.

[0106] C ⊂ {1, ..., K}, |C| = N

[0107] Here, N may be the number of candidate atoms that the OMP examines in subsequent steps. N can be set to a value greater than or equal to the number of atoms to be finally selected, s, and less than or equal to the total number of atoms included in the sparse dictionary, K. That is, s≤N≤K.

[0108] The candidate atom selection step of Fig. 7 may be a process of first narrowing down the K atoms to N candidate atoms using an MLP encoder, rather than using all K atoms included in the entire sparse dictionary D as OMP targets. Accordingly, the search range in the OMP execution step may be reduced.

[0109] Next, in the OMP execution step, the KV cache compression device (10) can generate a partial dictionary composed of candidate atoms. Specifically, the KV cache compression device (10) can construct a partial dictionary D_sub by extracting only the atoms corresponding to the set of candidate atom indices C from the entire sparse dictionary D. The partial dictionary D_sub can be expressed as follows.

[0110] D_sub = D[:, C] ∈ R^(d×N)

[0111] Here, D[:, C] may mean selecting columns corresponding to the set of candidate atomic indices C from all rows of the entire sparse dictionary D. Therefore, D_sub may be a partial sparse dictionary containing N atoms of d dimension.

[0112] The KV cache compression device (10) can perform OMP on a partial dictionary D_sub. That is, the KV cache compression device (10) can perform OMP only on candidate atoms selected by the MLP encoder, rather than performing OMP on the entire sparse dictionary D. In one embodiment, the OMP may include a greedy atom selection process and a least squares (LS) operation. The OMP greedy process may be a process of selecting an atom from among the candidate atoms that best reduces the current residual, and the LS operation may be a process of calculating a coefficient for each atom so that the error between the KV cache and the reconstruction vector is reduced given the set of selected atoms.

[0113] In one embodiment, the OMP execution step can be expressed as follows.

[0114] Stage 2 ― OMP Precision Selection (N →s)

[0115] We perform OMP on the token-specific subdictionary D_sub = D[:, C]∈R^(d×N) to obtain the final support S⊂C, |S|=s, and coefficients c∈R^s.

[0116] Here, S can be a set of atomic indices finally selected by the OMP and can be a subset of C. Also, the size of S can be s. That is, S ⊂ C, and |S| = s. c can be a coefficient vector corresponding to the final atoms and can be expressed as c ∈ R^s.

[0117] Accordingly, in the embodiment illustrated in FIG. 7, the KV cache compression device (10) may not perform OMP on all K atoms, but may perform OMP on only N candidate atoms selected through an MLP encoder. Accordingly, the number of targets for the inner product operation performed in each step of OMP may be reduced from K to N. For example, when K=1024 and N=128, the number of atoms to be compared in each step of OMP may be reduced from 1024 to 128. Accordingly, the inner product cost per step of OMP may be reduced by the ratio of N / K.

[0118] Finally, the KV cache compression device (10) can compress the KV cache using the final atomic index S and coefficient c calculated according to the result of the OMP operation. The compressed data may include s atomic indices and s coefficients per token. That is, the compressed representation stored can be described as follows.

[0119] Storage: s atom indices + s coefficients per token = 2·s values

[0120] However, since the atom index is identification information of an atom and the coefficient is a real coefficient, the actual storage capacity may vary depending on the number of bits of each data. In this embodiment, the final storage format may include s atomic indices and s coefficients, identical to the OMP-only method that performs OMP on the entire sparse dictionary. Therefore, since the present invention reduces the number of candidate atoms from K to N through an MLP encoder before performing OMP, the amount of computation required for OMP execution can be reduced while maintaining the same storage format.

[0121] That is, according to the embodiment illustrated in FIG. 7, the KV cache compression device (10) can first select candidate atoms among the atoms of the entire sparse dictionary using an MLP encoder, and perform OMP only on the selected candidate atoms to calculate the final atoms and counts. Accordingly, the present invention can efficiently compress the KV cache by utilizing the fast candidate selection capability of the MLP encoder and the precise atom selection and count calculation capability of OMP together.

[0123] FIG. 8 is a diagram illustrating a method for selecting candidate atoms using an MLP encoder according to an embodiment of the present invention.

[0124] Referring to FIG. 8, the KV cache compression device (10) can input the KV cache to be compressed into an MLP encoder. A sparse dictionary D may contain multiple atoms, and each atom may have a unique index. For example, in FIG. 8, the indices of atoms included in the sparse dictionary D are represented as ids=0, 1, 2, ..., 8. In the example of FIG. 8, the total atomic indices are represented as 0 to 8, but this is merely an example for illustrative purposes, and the actual sparse dictionary D may contain a larger number of atoms.

[0125] The KV cache compression device (10) can calculate a coefficient or score corresponding to each of a plurality of atoms included in a sparse dictionary D using an MLP encoder. In FIG. 8, the output of the MLP encoder can be expressed as ids and scale. ids is an index of an atom included in the sparse dictionary, and scale is a coefficient or atom score calculated by the MLP encoder corresponding to each atom index.

[0126] For example, as shown in Fig. 8, the MLP encoder can produce the following scale values ​​for each atomic index.

[0127] ids = [0, 1, 2, 3, 4, 5, 6, 7, 8]

[0128] scale = [0.1, 0.2, 0, 0.8, 0.1, 0.5, 0.3, 0.5, 0]

[0129] In this case, the KV cache compression device (10) can select candidate atoms based on the magnitude of the scale value. In one embodiment, the KV cache compression device (10) can select the top N atoms as candidate atoms in order of the magnitude of the absolute value of the scale value. In the example of FIG. 8, when N=4, atomic indices 3, 5, 6, and 7 with large scale values ​​can be selected as candidate atoms. Here, the scale value of atomic index 3 may be 0.8, the scale values ​​of atomic indices 5 and 7 may each be 0.5, and the scale value of atomic index 6 may be 0.3.

[0130] In one embodiment, the raw_codes or scale values, which are the outputs of the MLP encoder, are not directly used as final coefficients included in the final compressed representation, but can be used as criteria for selecting candidate atoms. Subsequently, the final atoms and coefficients for the final atoms can be calculated by an OMP performed on the candidate atoms. Accordingly, the MLP encoder can perform the role of quickly selecting candidate atoms from the entire sparse dictionary that are likely to be reviewed by the OMP.

[0131] That is, according to the embodiment illustrated in FIG. 8, the KV cache compression device (10) can select N candidate atoms out of K atoms included in the sparse dictionary D based on the output value of the MLP encoder. Accordingly, in the subsequent OMP execution step, the search target of the OMP can be reduced from the total K atoms to N candidate atoms.

[0133] FIG. 9 is a diagram illustrating a method for calculating the final atom and the coefficients for the final atom by performing OMP on candidate atoms according to one embodiment of the present invention.

[0134] Referring to FIG. 9, the KV cache compression device (10) can compress the KV cache using candidate atoms selected based on the output value of the MLP encoder in FIG. 8. For example, in FIG. 8, the MLP encoder calculates a scale value corresponding to each of the atomic indices 0 to 8, and the KV cache compression device (10) can select atomic indices 3, 5, 6, and 7 as candidate atoms based on the magnitude of the scale value.

[0135] The KV cache compression device (10) can form a partial sparse dictionary (D_sub) by extracting only the atoms corresponding to atomic indices 3, 5, 6, and 7 from all the atoms included in the sparse dictionary D. That is, in the embodiment of FIG. 9, the target of OMP execution may be limited not to all atoms included in the sparse dictionary D, but to atoms corresponding to atomic indices 3, 5, 6, and 7 selected as candidate atoms by the MLP encoder.

[0136] Subsequently, the KV cache compression device (10) can perform OMP only on candidate atoms corresponding to atomic indices 3, 5, 6, and 7. During the OMP process, the KV cache compression device (10) can sequentially select atoms that best represent the KV cache among the candidate atoms and calculate coefficients for the selected atoms. For example, some atoms among atomic indices 3, 5, 6, and 7 may be selected as final atoms, and coefficients corresponding to the selected final atoms may be calculated.

[0137] Finally, the KV cache compression device (10) can compress and represent the KV cache using the final atoms and coefficients for the final atoms calculated according to the OMP execution result. For example, if atomic indices 3 and 7 are selected as final atoms, the KV cache compression device (10) can store atomic indices 3 and 7 and coefficients corresponding to each of them as the compressed representation of the KV cache.

[0138] In this way, according to the embodiment illustrated in FIG. 9, the KV cache compression device (10) can calculate the final atoms and coefficients for representing the KV cache while reducing the amount of computation compared to performing OMP on the entire sparse dictionary by performing OMP only on candidate atoms selected in FIG. 8, e.g., atomic indices 3, 5, 6, and 7.

[0140] FIG. 10 is a configuration diagram for explaining a computing device according to an embodiment of the present invention.

[0141] Referring to FIG. 10, the computing device (1000) includes a processor (1010) and a memory (1020). The computing device (1000) may represent a KV cache compression device (10) or a KV cache restoration device (40), etc. Additionally, the computing device (1000) may include all components included in a KV cache compression and restoration system, such as an MLP encoder and an OMP execution module.

[0142] The processor (1010) can process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Here, the instructions may be provided from memory (1020) or an external device.

[0143] For example, the functions performed by each module included in the processor (1010) may be performed by a single processor or by separate processors. The processor (1010) may perform operations or data processing regarding the control and / or communication of at least one other component.

[0144] The processor (1010) may be implemented as an array of multiple logic gates, or as a combination of a general-purpose microprocessor and memory storing a program that can be executed on the microprocessor. For example, the processor (1010) may include a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, the processor (1010) may include an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. For example, the processor (1010) may refer to a combination of processing devices such as a combination of a digital signal processor (DSP) and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors combined with a digital signal processor (DSP) core, or any other combination of such configurations.

[0145] The memory (1020) may include any non-transient computer-readable recording medium. As an example, the memory (1020) may include a permanent mass storage device such as random access memory (RAM), read-only memory (ROM), a disk drive, a solid state drive (SSD), or flash memory. As another example, a permanent mass storage device such as ROM, an SSD, flash memory, or a disk drive may be a separate permanent storage device distinct from the memory. Additionally, the memory (1020) may store an operating system (OS) and at least one program code (e.g., code for the processor (1010) to execute).

[0146] These software components may be loaded from a computer-readable recording medium separate from the memory (1020). This separate computer-readable recording medium may be a recording medium that can be directly connected to a computer and may include, for example, computer-readable recording media such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. Alternatively, the software components may be loaded into the memory (1020) via a communication device other than a computer-readable recording medium. For example, at least one program may be loaded into the memory (1020) based on a computer program installed by files provided through a communication device by developers or a file distribution system that distributes installation files for applications.

[0147] Meanwhile, the above-described method can be written as a program executable on a computer and can be implemented on a general-purpose digital computer that operates the program using a computer-readable recording medium. In addition, the structure of the data used in the above-described method can be recorded on a computer-readable recording medium through various means. The computer-readable recording medium includes storage media such as magnetic storage media (e.g., ROM, RAM, USB, floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD, etc.).

[0148] A person skilled in the art to which the present invention pertains will understand that the present invention may be implemented in other specific forms without altering its technical concept or essential features. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts should be interpreted as being included within the scope of the present invention. Explanation of the symbols

[0150] 10: KV Cache Compressor 20: Large-scale language models 30: Storage section 40: KV Cache Restorer 1000: Computing device 1010: Processor 1020: Memory

Claims

Claim 1 A method for compressing a Key-Value Cache (KV) of a large-scale language model, performed by a computing device comprising at least one processor, comprising: a step of obtaining a KV cache for a token input to the large-scale language model; a step of inputting the KV cache into an MLP encoder to calculate a coefficient corresponding to each of a plurality of atoms included in a sparse dictionary; a step of selecting a candidate atom among the plurality of atoms based on the coefficient; and a step of performing an Orthogonal Matching Pursuit (OMP) on the candidate atom to calculate a final atom and a coefficient for the final atom among the candidate atoms to represent the KV cache, and further comprising: a step of generating a sparse dictionary using a training KV cache; a step of generating an OMP label by performing an OMP on the training KV cache based on the sparse dictionary; and a step of training the MLP encoder using the OMP label. Claim 2 A KV cache compression method according to claim 1, wherein the step of selecting the candidate atoms is characterized by selecting the top N atoms among the plurality of atoms as candidate atoms in order of the magnitude of the absolute value of the coefficients calculated by the MLP encoder. Claim 3 A KV cache compression method according to claim 2, wherein N is set to a value greater than or equal to the number of final atoms finally selected by the OMP and less than or equal to the number of multiple atoms K included in the sparse dictionary. Claim 4 A KV cache compression method according to claim 1, wherein the coefficient calculated by the MLP encoder is used as a selection criterion for the candidate atom, and the coefficient for the final atom based on the result of performing the OMP is used to represent the KV cache in a compressed manner. Claim 5 delete Claim 6 A KV cache compression method according to claim 1, wherein the step of training the MLP encoder is characterized by training the MLP encoder to select an atom identical to the selected atom by performing OMP on the training KV cache. Claim 7 A KV cache compression method according to claim 1, wherein the step of training the MLP encoder comprises constructing a loss function by assigning a weight greater than the error of the position corresponding to an atom not selected by the OMP to the error of the position corresponding to an atom not selected by the OMP. Claim 8 A KV cache compression method according to claim 1, characterized in that the values ​​of the plurality of atoms included in the sparse dictionary are fixed while the result of performing an OMP on the learning KV cache is used for the learning of the MLP encoder. Claim 9 A KV cache compression method according to claim 1, wherein the KV cache corresponds to at least one of a plurality of transformer layers or a plurality of attention heads of the large-scale language model, and each of the steps is performed for each of the plurality of transformer layers or for each of the plurality of attention heads. Claim 10 A device for compressing a Key-Value Cache (KV) of a large-scale language model, comprising: a memory in which one or more instructions are stored; and at least one processor configured to execute one or more instructions stored in the memory, wherein the processor performs the steps of: generating a sparse dictionary using a training KV cache; generating an OMP label by performing an Orthogonal Matching Pursuit (OMP) on the training KV cache based on the sparse dictionary; training an MLP encoder using the OMP label; obtaining a KV cache for a token input to the large-scale language model; inputting the KV cache into the MLP encoder to calculate a coefficient corresponding to each of a plurality of atoms included in the sparse dictionary; selecting a candidate atom among the plurality of atoms based on the coefficient; and performing an OMP on the candidate atom to calculate a final atom and a coefficient for the final atom among the candidate atoms to represent the KV cache.

Images