Method and Apparatus for KV Cache Compression Using an Encoder Model and a Sparse Dictionary

Abstract

A method for compressing a KV cache of a large-scale language model according to an embodiment of the present invention comprises the steps of: receiving a KV cache corresponding to a plurality of tokens included in training data; generating a sparse dictionary using the KV cache; training an encoder model to receive the KV cache and output a sparse code including identification information and coefficients of some elements among a plurality of elements of the sparse dictionary; inputting a KV cache for an input token corresponding to the input data into the encoder model to output a sparse code; and compressing the KV cache based on the sparse code. According to the present invention, a method and apparatus for compressing a KV cache can be provided, which train an encoder model to receive a KV cache and output a sparse code including identification information and coefficients of some elements among a plurality of elements of the sparse dictionary, and efficiently compress a KV cache corresponding to the input data using the trained encoder model.

Description

Technology Field

[0001] The present invention relates to a method and apparatus for compressing a KV cache using an encoder model and a sparse dictionary, and more specifically, to a compression technology for efficiently storing and managing a Key-Value Cache (KV cache) generated in a Large Language Model (LLM). Background Technology

[0002] Recently, large-scale language models have demonstrated excellent performance in various natural language processing tasks, and various techniques to improve inference efficiency are being applied. KV caches are widely used as a technique to reduce the repetition of identical operations and improve inference speed by storing key vectors and value vectors for previous tokens during the attention operation and reusing them in subsequent operations.

[0003] However, since the KV cache has the characteristic of continuously increasing in proportion to the length of the input sequence, storage space and memory usage can increase significantly as the input sentence becomes longer or the number of tokens to be processed increases. In addition, since large-scale language models generally include multiple layers and multiple attention heads, the size of the entire KV cache can become very large if all key vectors and value vectors generated for each layer and each head are stored.

[0004] Such an increase in KV cache can increase the memory burden on the inference system, raise data transfer costs between storage and memory, and degrade the overall processing efficiency of the system. Particularly in environments with limited memory resources, processable context lengths or concurrent throughput may be restricted, and the costs required for cache storage and management can become a significant constraint on overall system performance.

[0005] Accordingly, there is an increasing need for compression technology that can reduce the storage capacity of the KV cache while maintaining its informational utility as much as possible. The problem to be solved

[0006] The objective of the present invention, devised to solve the aforementioned problems, is to provide a KV cache compression method and apparatus capable of receiving a KV cache as input, training an encoder model that outputs a sparse code including identification information and coefficients of some elements among a plurality of elements of a sparse dictionary, and efficiently compressing a KV cache corresponding to input data using the trained encoder model.

[0007] The objective of the present invention is to provide a KV cache compression method and apparatus capable of generating sparse codes more precisely than a method of compressing the KV cache all at once by having the encoder model be performed repeatedly for a preset number of iterations and determining a preset number of selected elements at each iteration step.

[0008] The objective of the present invention is to provide a KV cache compression method and apparatus capable of efficiently generating sparse codes while improving reconstruction accuracy for the KV cache by performing element determination of subsequent iteration steps using residual values ​​calculated in each iteration step.

[0009] The objective of the present invention is to provide a KV cache compression method and apparatus capable of improving the reconstruction performance of a KV cache based on sparse codes by correcting coefficients for elements through least squares optimization on elements determined by an encoder model.

[0010] The objective of the present invention is to provide a KV cache compression method and apparatus capable of reducing the storage capacity and computational load of the KV cache by excluding compression of low-importance input non-important tokens through the selection of input important tokens based on residual values ​​calculated at each iteration step or residual values ​​calculated during the process of performing Orthogonal Matching Pursuit (OMP) on a sparse dictionary.

[0011] The objective of the present invention is to provide a KV cache compression method and apparatus capable of reducing the influence of input non-important tokens in subsequent attention operations by replacing the key vector corresponding to the input non-important token with a fake key. means of solving the problem

[0012] A method for compressing a KV cache of a large-scale language model according to the present invention for achieving the above-mentioned purpose comprises the steps of: receiving a KV cache corresponding to a plurality of tokens included in training data; generating a sparse dictionary using the KV cache; training an encoder model to receive the KV cache and output a sparse code including identification information and coefficients of some elements among a plurality of elements of the sparse dictionary; inputting a KV cache for an input token corresponding to the input data into the encoder model to output a sparse code; and compressing the KV cache based on the sparse code.

[0013] In addition, the step of outputting the sparse code is performed repeatedly for a preset number of iterations, and is characterized by determining a preset number of selected elements among a plurality of elements of the sparse dictionary at each iteration step.

[0014] In addition, the step of outputting the sparse code is characterized in that, in each iteration step, the residual value calculated in the previous iteration step is input into the encoder model.

[0015] In addition, the step of training the encoder model is characterized by using at least one of a first method of training the encoder model while the sparse dictionary is fixed and a second method of training the sparse dictionary and the encoder model together.

[0016] Additionally, the method further includes a step of selecting important input tokens among a plurality of input tokens included in the input data, wherein the step of selecting important input tokens selects important input tokens by determining the importance of each input token based on residual values ​​calculated in each iteration step, and the step of compressing the KV cache is performed only for important input tokens.

[0017] Additionally, the method further includes a step of selecting input important tokens among a plurality of input tokens included in the input data, wherein the step of selecting input important tokens is characterized by performing an Orthogonal Matching Pursuit (OMP) on the sparse dictionary a preset number of times and selecting input important tokens based on residual values ​​calculated during the OMP execution process, and the step of compressing the KV cache is performed only for the input important tokens.

[0018] In addition, the step of outputting the sparse code further includes the step of correcting coefficients for the determined elements by performing least squares optimization on the elements determined by the encoder model.

[0019] In addition, the above least squares optimization is characterized by being performed only in the last iteration step among each iteration step.

[0020] In addition, the method is characterized by further including the step of selecting important input tokens among a plurality of input tokens included in the input data, and the step of replacing key vectors for non-important input tokens with fake keys such that the inner product value with the query vector is less than or equal to a preset negative threshold.

[0021] An apparatus for compressing a KV cache of a large-scale language model according to an embodiment of the present invention comprises a memory in which one or more instructions are stored and at least one processor configured to execute one or more instructions stored in the memory, wherein the processor performs the steps of: receiving a KV cache corresponding to a plurality of tokens included in training data; generating a sparse dictionary using the KV cache; training an encoder model to receive the KV cache and output a sparse code including identification information and coefficients of some elements among a plurality of elements of the sparse dictionary; inputting a KV cache for an input token corresponding to the input data into the encoder model to output a sparse code; and compressing the KV cache based on the sparse code. Effects of the invention

[0022] According to the present invention as described above, a KV cache compression method and apparatus can be provided that receive a KV cache as input, train an encoder model that outputs a sparse code including identification information and coefficients of some elements among a plurality of elements of a sparse dictionary, and efficiently compress a KV cache corresponding to input data using the trained encoder model.

[0023] The present invention can provide a KV cache compression method and apparatus capable of generating sparse codes more precisely than a method of compressing the KV cache all at once by having an encoder model be performed repeatedly for a preset number of iterations and determining a preset number of selected elements at each iteration step.

[0024] The present invention can provide a KV cache compression method and apparatus capable of efficiently generating sparse codes while improving reconstruction accuracy for the KV cache by performing element determination of a subsequent iteration step using residual values ​​calculated in each iteration step.

[0025] The present invention can provide a KV cache compression method and apparatus capable of improving the reconstruction performance of a KV cache based on sparse codes by correcting coefficients for elements through least squares optimization on elements determined by an encoder model.

[0026] The present invention can provide a KV cache compression method and apparatus capable of reducing the storage capacity and computational load of the KV cache by excluding compression of low-importance input non-important tokens through the selection of input important tokens based on residual values ​​calculated at each iteration step or residual values ​​calculated during the OMP execution process for a sparse dictionary.

[0027] The present invention can provide a KV cache compression method and apparatus capable of reducing the influence of input non-important tokens in subsequent attention operations by replacing a key vector corresponding to an input non-important token with a fake key. Brief explanation of the drawing

[0028] FIG. 1 is a diagram illustrating a KV cache compression and restoration system according to one embodiment of the present invention. FIG. 2 is a flowchart illustrating a KV cache compression method according to an embodiment of the present invention. FIG. 3 is a diagram illustrating a sparse dictionary generation step and a compressed data generation step according to an embodiment of the present invention. FIG. 4 is a flowchart illustrating the process of selecting important input tokens using residual values ​​according to an embodiment of the present invention. FIG. 5 is a diagram illustrating the process of generating a sparse dictionary for a key vector and a sparse dictionary for a value vector through learning using a key vector and a value vector, respectively, according to an embodiment of the present invention. FIG. 6 is a diagram illustrating a method for compressing a KV cache by encoding generated key vectors and value vectors using a sparse dictionary when data is input into a large-scale language model according to one embodiment of the present invention. FIG. 7 is a diagram illustrating the process of generating a reconstruction vector by decoding a compressed KV cache using a sparse dictionary according to one embodiment of the present invention. FIG. 8 is a flowchart illustrating a KV cache compression method according to an embodiment of the present invention. FIG. 9 is a flowchart illustrating an encoder model-based iterative encoding process according to an embodiment of the present invention. FIG. 10 is a diagram illustrating an encoder model learning method according to an embodiment of the present invention. FIG. 11 is a diagram illustrating the overall processing flow of a KV cache compression method according to one embodiment of the present invention. FIG. 12 is a configuration diagram for explaining a computing device according to an embodiment of the present invention. Specific details for implementing the invention

[0029] In the following, embodiments related to the present invention are illustrated in the drawings and described in detail through the detailed description. However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms and should be understood to include all modifications, equivalents, and substitutions that fall within the spirit and scope of the present invention.

[0030] In describing the components of the present invention, terms such as first, second, A, B, (a), (b), etc., may be used. These terms are intended merely to distinguish the component from other components, and the essence, order, or sequence of the component is not limited by such terms. Furthermore, where it is stated in this specification that a component is "connected," "coupled," or "connected" to another component, it should be understood that the component may be directly connected or connected to the other component, but that another component may also be "connected," "coupled," or "connected" between each component. In the case of "connection," "coupled," or "connected," it may be understood not only as being physically "connected," "coupled," or "connected," but also, if necessary, electrically "connected," "coupled," or "connected."

[0031] Terms such as "~part (unit)," "~device," "~part," and "~module" as used in this specification refer to a unit that processes at least one function or operation, and may be implemented in hardware, software, or a combination of hardware and software. Furthermore, terms such as "include," "compose," or "have" as used in this specification, unless specifically stated otherwise, mean that the relevant component may be inherent; therefore, they should be interpreted as allowing for the inclusion of additional components rather than excluding other components.

[0032] Furthermore, it is intended to clarify that the classification of components in this specification is merely based on the primary function each component is responsible for. That is, two or more components described below may be combined into a single component, or a single component may be divided into two or more components based on more subdivided functions. Additionally, each component described below may additionally perform some or all of the functions of other components in addition to its primary function, and it is obvious that some of the primary functions of each component may be exclusively performed by other components.

[0033] Hereinafter, the present invention will be described with reference to the drawings related to embodiments of the present invention.

[0034] FIG. 1 is a diagram illustrating a KV cache compression and restoration system according to one embodiment of the present invention.

[0035] Referring to FIG. 1, the KV cache compression and restoration system may include a large-scale language model (20), a KV cache compression device (10), a storage unit (30), and a KV cache restoration device (40). The KV cache may refer to a set of key vectors and value vectors generated during the inference process of the large-scale language model. Additionally, the key vector may refer to a vector for calculating the degree of relevance with a query vector in an attention operation, and the value vector may refer to a vector containing information that is actually referenced according to the degree of relevance.

[0036] A large-scale language model (20) can generate token-specific hidden states, key vectors, and value vectors based on input data. The hidden states may be the result of operations of each transformer layer of the large-scale language model (20). The hidden states may be intermediate representation vectors prior to KV projection. The KV cache compression device (10) can distinguish between important tokens and non-important tokens based on information generated from the large-scale language model (20), and selectively compress the KV cache corresponding to the important tokens. At this time, important tokens may be tokens that are highly likely to be used relatively significantly in the subsequent inference process, and non-important tokens may be tokens with relatively low contribution. Accordingly, the KV cache compression device (10) can improve compression efficiency by performing compression processing on important tokens and removing or excluding non-important tokens from the compression target.

[0037] The storage unit (30) can store the compressed KV cache generated by the KV cache compression device (10). By storing the compressed KV cache instead of the original KV cache, the storage unit (30) can reduce storage space and reduce the input / output time required during the process of reading the compressed KV cache.

[0038] The KV cache recovery device (40) can generate a recovered KV cache by decoding the compressed KV cache stored in the storage unit (30). The KV cache recovery device (40) can provide the recovered KV cache to the large-scale language model (20). Accordingly, the KV cache stored in a compressed state can be recovered at the time of actual computation and used for attention computation or subsequent inference processes.

[0039] That is, the KV cache compression device (10) distinguishes between important tokens and non-important tokens and selectively compresses and restores the KV cache corresponding to the important tokens, thereby increasing the compression efficiency of the entire KV cache and reducing memory usage, storage capacity, and I / O burden.

[0041] FIG. 2 is a flowchart illustrating a KV cache compression method according to an embodiment of the present invention.

[0042] Each step illustrated in FIG. 2 can be performed by a KV cache compression device (10). The method of FIG. 2 may include the step of generating a sparse dictionary based on training data and the step of compressing a KV cache corresponding to input data using the generated sparse dictionary. In addition, the present invention can improve compression efficiency by distinguishing between important tokens and non-important tokens and selectively performing compression only on important tokens.

[0043] In step 210, the KV cache compression device (10) can select learning important tokens among a plurality of tokens included in the training data. Specifically, the KV cache compression device (10) divides the training data into token units and collects internal representation values ​​generated from one or more transformer layers of a large-scale language model (20) for each token. Then, the KV cache compression device (10) can calculate an important score indicating the likelihood that each token will be referenced in a subsequent inference process based on the internal representation values. At this time, the KV cache compression device (10) can select tokens with an important score greater than or equal to a reference value as learning important tokens. In one example, a surrogate model may be used when calculating the important score from the internal representation values.

[0044] Learning important tokens can be selected based on the prediction results of importance values ​​derived from internal representations, and accordingly, only tokens judged to be important, rather than all tokens included in the entire training data, can be used for subsequent sparse dictionary generation.

[0045] In step 220, the KV cache compression device (10) can collect key vectors and value vectors corresponding to learning important tokens. Specifically, the KV cache compression device (10) can extract and store key vectors and value vectors corresponding to learning important tokens selected in step 210. At this time, the key vectors and value vectors can be used as training data to generate a sparse dictionary in a subsequent step.

[0046] In one embodiment, the KV cache compression device (10) can collect key vectors and value vectors corresponding to learning important tokens separately. That is, the key vectors and value vectors can be collected separately so that a sparse dictionary for key vectors and a sparse dictionary for value vectors can be created separately from each other in a subsequent step.

[0047] In step 230, the KV cache compression device (10) can generate a sparse dictionary using the collected key vectors and value vectors. Specifically, the KV cache compression device (10) can generate a sparse dictionary using the key vectors and value vectors collected in step 220. In one embodiment, the KV cache compression device (10) can generate a sparse dictionary for key vectors using the key vectors of the learning important tokens, and can generate a sparse dictionary for value vectors using the value vectors of the learning important tokens.

[0048] The sparse dictionary generated in this way reflects the distribution of vectors corresponding to learning important tokens, and thus can better represent the characteristics of learning important tokens. Additionally, the sparse dictionary can be used to determine input important tokens in a subsequent step, and / or to encode key vectors and value vectors corresponding to input important tokens.

[0049] That is, in one embodiment, the sparse dictionary is not only used for encoding, but can also be utilized in the process of determining whether an input token is a significant token.

[0050] In step 240, the KV cache compression device (10) can select input important tokens among a plurality of input tokens corresponding to input data input to a large-scale language model (20). In one embodiment, the KV cache compression device (10) can calculate the importance value of each input token using an internal representation value generated in one or more transformer layers of the large-scale language model (20) for each input token included in the input data. Then, the KV cache compression device (10) can select input tokens whose importance value is greater than or equal to a reference value as input important tokens. That is, the selection of input important tokens can be performed based on internal representation values, just like the selection of learning important tokens.

[0051] In another embodiment, the KV cache compression device (10) can select input important tokens using the sparse dictionary generated in step 230. For example, the KV cache compression device (10) may attempt encoding using the relationship between the key vector or value vector corresponding to the input token and the atoms of the sparse dictionary, and determine whether the input token is an input important token based on the residual value calculated during the encoding process. In this case, input tokens with a residual value less than a threshold value may be selected as input important tokens, and input tokens with a residual value greater than or equal to the threshold value may be classified as input non-important tokens. A method for selecting input important tokens using residual values ​​is described in detail in FIG. 4.

[0052] In another embodiment, step 240 may be performed in such a manner that a method using internal representation values ​​and a method using residual values ​​are applied sequentially. For example, the KV cache compression device (10) may first select a primary important token among a plurality of input tokens based on internal representation values. Subsequently, the KV cache compression device (10) may select a secondary important token based on residual values ​​calculated during the encoding process using the sparse dictionary generated in step 230 for the primary important token. In this case, the internal representation values ​​may be used to primarily narrow down the candidate tokens among the input tokens, and the residual values ​​may be used to select the token among the candidates that is actually well represented by the sparse dictionary. Accordingly, step 240 may be performed in such a manner that uses only internal representation values, only residual values, or sequentially together using internal representation values ​​and residual values.

[0053] In step 250, the KV cache compression device (10) can compress the KV cache by encoding key vectors and value vectors corresponding to input important tokens using a sparse dictionary. Specifically, the KV cache compression device (10) can select one or more elements from the sparse dictionary generated in step 230. Then, the KV cache compression device (10) can compress and represent key vectors and value vectors corresponding to input important tokens using identification information and coefficients for the selected elements. That is, the encoding in step 250 may be a process of selecting one or more of the multiple elements of the sparse dictionary and representing the original key vectors and value vectors as data of a smaller size using identification information and coefficients of the selected elements.

[0054] In one embodiment, the KV cache compression device (10) may perform encoding by evaluating the similarity between the current encoding target vector and the reconstructed vector expressed using a sparse dictionary to calculate a residual value, and by repeatedly evaluating the similarity multiple times while reducing the residual value. Additionally, the encoding may include a process of repeatedly selecting elements a number of times corresponding to a preset sparsity. In this case, sparsity may refer to the number of elements selected or the number of repetitions (N, where N is a natural number) to express a key vector or value vector corresponding to a single input important token.

[0055] Additionally, in one embodiment, a residual value may be calculated even during the encoding process of step 250. In this case, if the residual value is greater than or equal to a threshold value, the corresponding input token may be determined as an input non-important token, and encoding for the corresponding input token may be stopped. On the other hand, if the residual value is less than the threshold value, the corresponding input token may be maintained as an input important token, and encoding may continue.

[0056] Therefore, the present invention can perform encoding only on important input tokens and can omit or stop encoding on non-important input tokens, thereby further improving compression efficiency.

[0057] Additionally, the compressed data generated in step 250 may include identification information and coefficients of selected elements. For example, the compressed data may include identification information and coefficients of elements for a key vector corresponding to an input important token, and identification information and coefficients of elements for a value vector.

[0059] FIG. 3 is a diagram illustrating a sparse dictionary generation step and a compressed data generation step according to an embodiment of the present invention.

[0060] Referring to FIG. 3, the KV cache compression device (10) can first generate a sparse dictionary and then generate a compressed KV cache (compressed data) by compressing the KV cache corresponding to the input data using the generated sparse dictionary. At this time, the KV cache compression device (10) can improve compression efficiency by distinguishing between important tokens and non-important tokens and performing compression centered on the KV cache corresponding to the important tokens.

[0061] 1. Sparse dictionary generation step

[0062] In the sparse dictionary generation step, the KV cache compression device (10) can determine a learning important token among a plurality of tokens included in the training data. In one embodiment, the KV cache compression device (10) can divide the training data into token units and obtain an internal representation value generated by one or more transformer layers of a large-scale language model (20) for each token. Then, the KV cache compression device (10) can calculate an importance value for each token based on the internal representation value and determine a token whose importance value is greater than or equal to a reference value as a learning important token. That is, the determination of the learning important token can be performed based on the internal representation value.

[0063] The KV cache compression device (10) can extract and collect key vectors and value vectors corresponding to the determined learning important tokens. Subsequently, the KV cache compression device (10) can generate a sparse dictionary using the collected key vectors and value vectors. In one embodiment, the KV cache compression device (10) can generate a sparse dictionary for key vectors using key vectors corresponding to the learning important tokens, and can generate a sparse dictionary for value vectors using value vectors corresponding to the learning important tokens. Thus, in the sparse dictionary generation step, instead of using all tokens included in the entire training data, a sparse dictionary can be generated based on vectors corresponding to the selected learning important tokens, thereby allowing for the construction of a sparse dictionary that better reflects the characteristics of the important tokens. The sparse dictionary can be commonly referenced in the input important token determination step and / or encoding step described later. In one example, the sparse dictionary can be generated based on K-SVD.

[0064] 2. Compressed data generation step

[0065] In the compressed data generation step, the KV cache compression device (10) can determine an input important token among a plurality of input tokens corresponding to input data input to a large-scale language model (20). In one embodiment, the KV cache compression device (10) can calculate an importance value for each input token using an internal representation value generated in one or more transformer layers of the large-scale language model (20) for each input token included in the input data. Then, an input token whose importance value is greater than or equal to a reference value can be determined as an input important token. That is, the determination of the input important token can also be performed based on the internal representation value.

[0066] Meanwhile, in another embodiment, the KV cache compression device (10) may determine an input important token using a sparse dictionary generated in the sparse dictionary generation step. For example, the KV cache compression device (10) may attempt encoding using the relationship between the key vector or value vector corresponding to the input token and the elements of the sparse dictionary, and determine whether the input token is an important token based on the residual value calculated in the process. In this case, the sparse dictionary may be used not only for encoding but also for determining the input important token.

[0067] In addition, in one embodiment, the determination of an input important token may be performed by sequentially applying a method using an internal representation value and a method using a residual value. For example, the KV cache compression device (10) may first determine a primary important token among a plurality of input tokens based on an internal representation value. Subsequently, the KV cache compression device (10) may determine a secondary important token based on a residual value calculated during the encoding process using the sparse dictionary for the primary important token. In this case, the internal representation value may be used to primarily select candidate important tokens, and the residual value may be used to select tokens among the candidates that are actually well represented by the sparse dictionary. Accordingly, the precision of determining the input important token can be increased, and unnecessary encoding operations can be reduced.

[0068] When an input important token is determined, the KV cache compression device (10) can encode a key vector and a value vector corresponding to the input important token using a sparse dictionary. At this time, the encoding may be a process of selecting one or more elements from a plurality of elements of the sparse dictionary and compressing and representing the original key vector and value vector using identification information and coefficients of the selected elements. Accordingly, the KV cache compression device (10) may selectively perform encoding only for the input important token and omit or stop encoding for the input non-important token. The compressed data generated in this way may be stored in a storage unit and may be used in a subsequent decoding or restoration process if necessary.

[0069] That is, according to the embodiment illustrated in FIG. 3, the KV cache compression device (10) can determine a learning important token in the sparse dictionary generation step and generate a sparse dictionary using a key vector and a value vector corresponding to the learning important token, and in the compression data generation step, determine an input important token based on an internal representation value and / or a residual value, and generate compressed data by encoding the key vector and the value vector corresponding to the input important token using the generated sparse dictionary. Accordingly, the present invention can compress the KV cache more efficiently compared to a method that processes the entire KV cache in batches without distinguishing between important tokens and non-important tokens.

[0070] In addition, in one embodiment, the present invention may be applied to a Retrieval-Augmented Generation (RAG) system. Specifically, the KV cache compression device (10) may generate a KV cache in advance using a large-scale language model (20) for a plurality of documents or corpus data. Then, the KV cache compression device (10) may generate compressed data by distinguishing important tokens and non-important tokens for the generated KV cache and encoding key vectors and value vectors corresponding to important tokens using a sparse dictionary. The generated compressed data may be stored in a storage unit (30).

[0071] Subsequently, when a user query is entered, the search device (not shown) can search for documents related to the query and transmit a compressed KV cache corresponding to the searched documents to the KV cache recovery device (40). Then, the KV cache recovery device (40) can decode the compressed KV cache to generate restored key vectors and value vectors, and provide the restored key vectors and value vectors to the large-scale language model (20). Accordingly, the large-scale language model (20) can perform inference using the pre-calculated and compressed KV cache without reprocessing the entire original text.

[0072] Accordingly, according to the present embodiment, since the document-specific KV cache can be stored in a compressed state in the RAG system, storage space can be reduced, and since only the compressed KV cache needs to be retrieved during a query, the I / O burden between the storage device and memory can be reduced. In addition, since a pre-calculated KV cache is used, the Time To First Token (TTFT) can be shortened.

[0073] Additionally, in another embodiment, the present invention may be applied to an online generation process. Specifically, when a large-scale language model (20) performs operations on each layer of input data, a KV cache corresponding to the input tokens in each layer may be generated. A KV cache compression device (10) may select important tokens from the KV cache generated in each layer and encode key vectors and value vectors corresponding to the selected important tokens using a sparse dictionary. Accordingly, after the operation of each layer is completed, the KV cache may be stored or maintained in a compressed state. Subsequently, if it is necessary to perform the next token generation or subsequent attention operation, a KV cache restoration device (40) may decode the compressed KV cache to generate restored key vectors and value vectors. The large-scale language model (20) may perform attention operations using the restored key vectors and value vectors. Then, when the corresponding layer or operation step is completed, the KV cache compression device (10) may again encode the KV cache corresponding to the important tokens and store it in a compressed state.

[0074] That is, according to the present embodiment, the KV cache of each layer can be used in a restored form at the time when the actual attention operation is performed, and can be maintained in a compressed form at other times. Accordingly, memory usage can be reduced compared to the case where the KV cache is always maintained at its original size across all layers.

[0076] FIG. 4 is a flowchart illustrating the process of selecting important input tokens using residual values ​​according to an embodiment of the present invention.

[0077] Referring to FIG. 4, the KV cache compression device (10) performs encoding using a sparse dictionary for a plurality of input tokens included in the input data, and can determine whether each input token is an important input token or an unimportant input token based on the residual value calculated during the encoding process. In this way, the present invention can select important tokens and unimportant tokens and improve compression efficiency by distinguishing between tokens that are well represented by the sparse dictionary and tokens that are not, rather than treating all input tokens equally.

[0078] In step 410, the KV cache compression device (10) can divide the input data into tokens. Specifically, the KV cache compression device (10) can divide the input data input to the large-scale language model (20) into multiple input tokens. At this time, the key vector and value vector corresponding to each input token can be the encoding targets of the subsequent step.

[0079] In step 420, the KV cache compression device (10) can select an element from the sparse dictionary that is most similar to the KV vector of the token and calculate the coefficient of the selected element. Specifically, the KV cache compression device (10) can evaluate the similarity between the KV vector corresponding to the input token and each element of the sparse dictionary, and can select one or more elements from the sparse dictionary that best approximate the KV vector. Additionally, the KV cache compression device (10) can calculate a coefficient for the selected element. The coefficient may be a value that allows the selected element to better represent the KV vector corresponding to the input token. That is, step 420 may be an encoding process for representing the KV vector corresponding to the input token as a combination of some elements of the sparse dictionary and coefficients. In one example, the selection of the element and the calculation of the coefficient may be performed using a sparse approximation method such as the Orthogonal Matching Pursuit (OMP) method.

[0080] In step 430, the KV cache compression device (10) can generate a reconstruction vector with selected elements and coefficients and calculate a residual value between the KV vector and the reconstruction vector. Specifically, the KV cache compression device (10) can generate a reconstruction vector based on the elements and coefficients selected in step 420 and calculate a residual value by calculating the difference between the reconstruction vector and the KV vector for the input token. The residual value may be a value indicating how accurately the KV vector for the input token can be represented using only the elements and coefficients selected so far. For example, the smaller the residual value, the better the KV vector for the input token is represented by the sparse dictionary, and the larger the residual value, the worse the KV vector for the input token is represented by the sparse dictionary.

[0081] In step 440, the KV cache compression device (10) determines whether the residual value is greater than a reference value, and if the residual value is greater than the reference value, it can determine the corresponding input token as an input non-important token. Specifically, the KV cache compression device (10) can determine whether the residual value calculated in step 430 is greater than a preset reference value. If the residual value is greater than the reference value, the KV cache compression device (10) can determine that the corresponding input token is not sufficiently well represented by the sparse dictionary. Accordingly, in step 470, the KV cache compression device (10) can determine the corresponding input token as an input non-important token. If the input token is determined to be an input non-important token, the KV cache compression device (10) can stop encoding for the corresponding input token. Therefore, the present invention can reduce unnecessary computational load by not performing additional encoding operations for input tokens that are not well approximated by the sparse dictionary.

[0082] In step 450, the KV cache compression device (10) can determine whether the number of iterations has reached a preset number. Specifically, the KV cache compression device (10) can determine whether the number of iterations for element selection, coefficient calculation, and residual value calculation has reached a preset number, for example, a number of iterations corresponding to the sparsity (N). If the number of iterations has not reached the preset number, the KV cache compression device (10) can proceed back to step 420 to perform additional element selection and coefficient calculation. If the number of iterations is 1, the KV vector is in a state where it is represented by one element, and if the number of iterations is 2, the KV vector is in a state where it is represented by two elements. Thus, if the number of iterations is N, the KV vector is in a state where it is represented by N elements. In one example, N can be set to a natural number greater than or equal to 1.

[0083] Steps 420 through 450 may be repeated to progressively approximate the KV vector corresponding to the input token with multiple elements of the sparse dictionary. The number of iterations may serve as a criterion for limiting the number of elements selected to represent the KV vector corresponding to a single input token.

[0084] In step 460, the KV cache compression device (10) can determine that the corresponding input token is an input important token if the number of iterations is N and the residual value is less than or equal to a reference value. Specifically, in step 440, if the residual value is not greater than the reference value, that is, if the residual value is less than or equal to the reference value, the KV cache compression device (10) can determine that the corresponding input token can be sufficiently well represented by a sparse dictionary. Accordingly, the KV cache compression device (10) can determine that the corresponding input token is an input important token. The KV vector corresponding to the input important token can be used in the subsequent compressed data generation step. For example, the KV cache compression device (10) can compress and represent the KV vector corresponding to the input important token using the identification information and coefficients of the selected element, and thereby generate compressed data.

[0085] That is, according to the embodiment illustrated in FIG. 4, the KV cache compression device (10) can repeatedly perform the process of dividing input data into token units and approximating the KV vector corresponding to each input token with elements and coefficients of a sparse dictionary. Then, based on the residual value calculated during the approximation process, it can determine whether the input token is an important input token or an unimportant input token. Accordingly, the present invention can perform selective KV cache compression that distinguishes between important tokens and unimportant tokens by not treating all input tokens equally, but by selecting tokens that are well represented by the sparse dictionary as important input tokens and excluding tokens that are not well represented by the sparse dictionary as unimportant input tokens.

[0087] FIG. 5 is a diagram illustrating the process of generating a sparse dictionary for a key vector and a sparse dictionary for a value vector through learning using a key vector and a value vector, respectively, according to an embodiment of the present invention.

[0088] Referring to FIG. 5, the KV cache compression device (10) can input a text corpus into a large-scale language model (20). The large-scale language model (20) can generate key vectors and value vectors corresponding to multiple tokens based on the text corpus. At this time, the key vectors are matrices It can be constructed in the form of, and the value vectors are matrices It can be composed in the form of.

[0089] Specifically, may be a training data set containing multiple key vectors in the form of column vectors or row vectors, and may be a training data set containing multiple value vectors in the form of column vectors or row vectors. As illustrated in FIG. 5, and Each can be a matrix containing multiple m-dimensional vectors. Here, p can represent the number of vectors used for training, and It can represent the dimension of each vector.

[0090] The KV cache compression device (10) is a set of key vectors Sparse dictionary for key vectors using Can learn. Sparse dictionary for key vectors is a plurality of elements It may include. In this case, each element may be a reference vector (or basis) for sparsely representing the key vector. That is, a sparse dictionary for key vectors may be a set of learned reference vectors such that the input key vector can be approximated by a combination of a small number of elements and coefficients.

[0091] Likewise, the KV cache compression device (10) is a set of value vectors Sparse dictionary for value vectors using Can learn. Sparse dictionary for value vectors is a plurality of elements It may include. In this case, each element may be a reference vector for sparsely representing the value vector. That is, a sparse dictionary for value vectors may be a set of learned reference vectors such that the input value vector can be approximated by a combination of a few elements and coefficients.

[0092] Accordingly, in the present invention, instead of treating key vectors and value vectors as a single common sparse dictionary, a sparse dictionary for key vectors and a sparse dictionary for value vectors can be generated independently, respectively. This is because key vectors and value vectors may have different distributional or representational characteristics. Therefore, a sparse dictionary for key vectors and sparse dictionary for value vectors By learning them separately, key vectors and value vectors can be represented more accurately and efficiently in a sparse manner.

[0093] In one embodiment, the key vector set and value vector set The vectors may correspond to selected important learning tokens among multiple tokens included in the training data. That is, the KV cache compression device (10) may first select important learning tokens among multiple tokens included in the training data, collect key vectors and value vectors corresponding to the important learning tokens, and then generate each sparse dictionary using the collected key vectors and value vectors. Accordingly, the sparse dictionaries may be configured to better reflect the characteristics of the important tokens.

[0094] In addition, a sparse dictionary for key vectors shown in Fig. 5 and sparse dictionary for value vectors It can be used in a subsequent encoding process. For example, when an input important token corresponding to the input data is determined, the KV cache compression device (10) determines the key vector corresponding to the input important token and the sparse dictionary for the key vector. It can be encoded using, and the value vectors corresponding to the input important tokens are a sparse dictionary for value vectors It can be encoded using . Therefore, key vectors and value vectors can be compressed independently by sparse dictionaries optimized for each.

[0095] That is, according to the embodiment illustrated in FIG. 5, the KV cache compression device (10) uses a set of key vectors and a set of value vectors generated using a large-scale language model (20) as training data, respectively, and a sparse dictionary for key vectors and sparse dictionary for value vectors By learning separately, each vector can be represented more efficiently in the subsequent KV cache compression process. Accordingly, the present invention can generate a sparse dictionary that reflects the characteristics of the key vector and the value vector, respectively, and thereby improve the accuracy and efficiency of KV cache compression.

[0097] FIG. 6 is a diagram illustrating a method for compressing a KV cache by encoding generated key vectors and value vectors using a sparse dictionary when data is input into a large-scale language model according to one embodiment of the present invention.

[0098] As illustrated in FIG. 6, when input data, for example, a retrieved document, is input into a large-scale language model (20), the large-scale language model (20) produces a key vector corresponding to the input data. and value vector Can generate key vector and value vector sparse dictionary for key vectors, respectively and sparse dictionary for value vectors It can be encoded using .

[0099] The KV cache compression device (10) is a key vector Regarding sparse dictionary for key vectors Encoding can be performed by referring to. Specifically, the KV cache compression device (10) is a key vector A sparse dictionary for key vectors containing one or more elements that best represent it It can be selected from. In addition, the KV cache compression device (10) can calculate a coefficient corresponding to each selected element. As a result, the key vector is the identification information of the selected element and coefficients corresponding to the elements It can be compressed and expressed in the form of. In the example of Fig. 6, the key vector Sparse dictionary for key vectors It can be expressed using the second, fifth, and eighth elements of, and the corresponding coefficients can be 0.2, 0.8, and 0.3, respectively.

[0100] Likewise, the KV cache compression device (10) is a value vector Regarding sparse dictionary for value vectors Encoding can be performed by referring to. Specifically, the KV cache compression device (10) is a value vector A sparse dictionary for value vectors containing one or more elements that best represent it You can select from, and calculate the coefficient corresponding to each selected element. As a result, the value vector is the identification information of the selected element and coefficients corresponding to the above elements It can be compressed and expressed in the form of. In the example of Fig. 6, the value vector A sparse dictionary for value vectors It can be expressed using the first, fourth, and seventh elements of, and the corresponding coefficients can be 0.9, 0.1, and 0.4, respectively.

[0101] In this way, the KV cache compression device (10) can generate compressed data by storing only the element identification information and coefficients of a selected sparse dictionary, instead of storing the entire original key vector and value vector as is. That is, the encoding process illustrated in FIG. 6 may be a process of approximating and representing the original high-dimensional key vector and value vector with a combination of a smaller number of elements and coefficients. Accordingly, the KV cache corresponding to the input data can be stored as compressed data of a smaller size than the original vector form.

[0102] In one embodiment, the encoding may be performed iteratively. For example, the KV cache compression device (10) may evaluate the similarity between the key vector or value vector currently being encoded and the elements of each sparse dictionary, and calculate coefficients while selecting the most appropriate elements one by one. Then, the input vector may be incrementally approximated based on the already selected elements and coefficients. Thus, a single key vector or value vector can be represented by only a small number of elements of the sparse dictionary.

[0103] In addition, a sparse dictionary for key vectors used in the embodiment of FIG. 6 and sparse dictionary for value vectors may be a pre-trained sparse dictionary. For example, sparse dictionaries may be generated using key vectors and value vectors corresponding to selected important learning tokens among multiple tokens included in the training data. In this case, the encoding process of FIG. 6 may be a process using a sparse dictionary generated by reflecting the characteristics of the important tokens. Therefore, the key vectors and value vectors corresponding to the input important tokens can be compressed more effectively by the sparse dictionary.

[0104] In addition, in one embodiment, the encoding shown in FIG. 6 may be performed only on the input important tokens. That is, the KV cache compression device (10) may perform encoding only on the key vector and value vector corresponding to the input important tokens determined to be important among the plurality of input tokens included in the input data. Accordingly, encoding may not be performed on the input non-important tokens, or the encoding process may be stopped. Therefore, the present invention can compress the KV cache more efficiently compared to a method of encoding all input tokens equally.

[0105] That is, according to the embodiment illustrated in FIG. 6, a large-scale language model (20) has a key vector corresponding to the input data. and value vector When generating, the KV cache compression device (10) generates a sparse dictionary corresponding to each and Key vectors and value vectors can be encoded using [this method]. The encoding result can be stored in the form of element identification information and coefficients. Accordingly, the present invention can represent key vectors and value vectors as compressed data of a smaller size, thereby reducing the storage capacity and memory usage of the KV cache.

[0107] FIG. 7 is a diagram illustrating the process of generating a reconstruction vector by decoding a compressed KV cache using a sparse dictionary according to one embodiment of the present invention.

[0108] Referring to FIG. 7, the KV cache recovery device (40) can recover the key vector and the value vector, respectively, using the identification information and coefficients of the elements included in the compressed data. Specifically, the compressed data is the identification information corresponding to the key vector. and coefficients , and identification information corresponding to the value vector and coefficients It may include.

[0109] In one embodiment, the KV cache recovery device (40) includes identification information contained in the compressed data for the key vector. Sparse dictionary for key vectors using Corresponding elements can be selected from. For example, as shown in FIG. 7, If [2, 5, 8], the KV cache restorer (40) is a sparse dictionary for key vectors The second element in , the fifth element , and the eighth element You can select it. Also, the KV cache recovery device (40) counts the selected elements. , for example, [0.2, 0.8, 0.3] can be applied respectively. Accordingly, the KV cache recovery device (40) , , and It can produce.

[0110] Subsequently, the KV cache recovery device (40) reconstructs the key vector by summing the elements to which the coefficients are applied. It can generate. That is, a reconstructed key vector It can be expressed by a linear combination of selected elements. Accordingly, even without directly storing the entire original key vector, the key vector can be restored using only the identification information and coefficients included in the sparse dictionary and compressed data.

[0111] Likewise, the KV cache recovery device (40) includes identification information contained in the compressed data for the value vector. Sparse dictionary for value vectors using You can select corresponding elements from. For example, If [1, 4, 7], the KV cache restorer (40) is a sparse dictionary for value vectors The first element in , fourth element , and the seventh element You can select it. In addition, the KV cache recovery device (40) counts the selected elements. , for example, [0.9, 0.1, 0.4] can be applied respectively. Accordingly, the KV cache recovery device (40) , , and It can produce.

[0112] Subsequently, the KV cache recovery device (40) reconstructs the value vector by summing the elements to which the coefficients are applied. It can generate. That is, a reconstructed value vector It can also be expressed by a linear combination of selected elements.

[0113] That is, according to the embodiment illustrated in FIG. 7, the compressed data may not include all elements of the sparse dictionary, but may include only identification information of some elements actually selected and coefficients of those elements. Then, the KV cache recovery device (40) can generate a reconstruction vector by selecting a corresponding element from the sparse dictionary based on the identification information and coefficients, applying coefficients to the elements, and summing them. Accordingly, the present invention can store the original key vector and value vector as compressed data of a smaller size, and can restore the key vector and value vector using the compressed data and the sparse dictionary when necessary.

[0114] In addition, in one embodiment, the reconstruction vector and It can be used in subsequent attention operations or inference processes of large-scale language models. For example, when compressed data stored in a storage unit is loaded, the KV cache recovery device (40) can decode the compressed data to generate reconstructed key vectors and value vectors, and can provide the generated reconstructed vectors to a large-scale language model. Accordingly, the present invention maintains the KV cache in a compressed form when stored, and when actual operations are required, it can generate and use reconstructed vectors through decoding.

[0115] That is, FIG. 7 may be a diagram illustrating a decoding process that reconstructs key vectors and value vectors, respectively, using compressed data expressed as element identification information and coefficients of a sparse dictionary. Through this, the present invention can support both compressed storage and restoration, and can improve both the storage efficiency and utilization efficiency of the KV cache.

[0117] FIG. 8 is a flowchart illustrating a KV cache compression method according to an embodiment of the present invention.

[0118] The KV cache compression device (10) receives a KV cache corresponding to a plurality of tokens included in training data, generates a sparse dictionary using the KV cache and trains an encoder model, and can compress the KV cache by converting the KV cache of an input token corresponding to the input data into a sparse code using the trained encoder model.

[0119] In step 810, the KV cache compression device (10) may receive KV caches corresponding to a plurality of tokens included in the training data. The training data may include a text corpus, a set of documents, a conversation history, document data for search augmentation generation (RAG), or any input sequence used for training or evaluating a large-scale language model. The KV cache compression device (10) may obtain a KV cache corresponding to each token by inputting the training data into the large-scale language model (20), or it may receive a pre-generated and stored KV cache from an external storage unit or another device. In this case, “receiving” is not limited to simply receiving via a network, but may include loading from memory or storage, directly obtaining the internal operation results of the large-scale language model (20), or reading a KV cache stored in file form. In one embodiment, the KV cache may include key vectors and value vectors generated for each token location, and, if necessary, only the KV cache corresponding to a specific layer or a specific attention head may be used. For example, the KV cache compression device (10) may receive KV caches for all of multiple layers, or may selectively receive only the KV caches of a specific layer.

[0120] In step 820, the KV cache compression device (10) can generate a sparse dictionary using the KV cache. Here, the sparse dictionary may be a set of elements learned to approximate the vectors constituting the KV cache with a small number of reference element combinations. In one embodiment, the KV cache compression device (10) can generate a sparse dictionary using a KV cache corresponding to all of the multiple tokens included in the training data. In another embodiment, the KV cache compression device (10) may generate a sparse dictionary by selecting only the KV caches corresponding to tokens judged to be of high importance. For example, the KV cache compression device (10) may generate a sparse dictionary by selecting important tokens based on internal representation values ​​or residual values, and then using the KV caches corresponding to the important tokens. Additionally, the sparse dictionary may be generated as a single common dictionary, or multiple dictionaries may be generated by layer, by head, or by data type as needed.

[0121] In step 830, the KV cache compression device (10) can receive a KV cache as input and train an encoder model to output a sparse code containing identification information and coefficients of some elements among a plurality of elements of a sparse dictionary. Here, the encoder model may be an artificial intelligence model configured to output information regarding which elements among a plurality of elements of a sparse dictionary to select for the input KV cache and which coefficients to assign to the selected elements. In one embodiment, the encoder model may have a multilayer perceptron (MLP) structure, but is not limited thereto and may be implemented as any neural network model capable of outputting element selection information and coefficient information from an input vector.

[0122] For example, if there are a total of 1,000 elements in a sparse dictionary, the encoder model can output coefficients corresponding to each element, along with identification information indicating that the 2nd, 5th, and 8th elements among the 1,000 elements must be selected. In this case, the sparse code can be expressed in the form of "set of indices of selected elements + set of coefficients corresponding to each index".

[0123] Additionally, in one embodiment, the encoder model training of step 830 may be performed in a two-stage training manner. Specifically, the KV cache compression device (10) may first perform a first method of training only the encoder model while the sparse dictionary is fixed. In this case, the elements of the sparse dictionary are not changed, and only the parameters of the encoder model may be updated. This first method may be a step to allow the encoder model to prioritize learning the element selection pattern for the current sparse dictionary. Subsequently, the KV cache compression device (10) may perform a second method of training the sparse dictionary and the encoder model together. In this case, based on the reconstruction error or residual value, not only the parameters of the encoder model but also the elements of the sparse dictionary may be updated together. That is, in the first method, the encoder model learns the element selection method based on the fixed sparse dictionary, and in the second method, the sparse dictionary and the encoder model may be jointly optimized to adapt to each other.

[0124] In step 840, the KV cache compression device (10) can input a KV cache for an input token corresponding to the input data into an encoder model to output a sparse code. Here, the input data may include a user query, a document, a conversation history, or any input sequence provided to a large-scale language model (20). The KV cache compression device (10) may acquire a KV cache for each of the plurality of input tokens included in the input data and input the KV cache into an encoder model. The encoder model may output a sparse code including some element identification information and coefficients of a sparse dictionary.

[0125] In one embodiment, the sparse code output process may be performed by repeating it a preset number of times. More specifically, the KV cache compression device (10) may determine a preset number of selected elements among a plurality of elements of the sparse dictionary at each iteration step. For example, if the entire sparse dictionary contains 1,000 elements and the sparse code is designed to consist of a total of 32 elements, the encoder model may select 4 elements at each iteration step and repeat the iteration step 8 times to finally determine 32 elements. Here, “number of iterations” may be a number indicating how many steps the entire encoding process is divided into to perform, and “number of selections” may be a number indicating how many elements are selected at each iteration step.

[0126] Additionally, in each iteration step, the residual value calculated in the previous iteration step may be input into the encoder model. Here, the residual value may represent the difference between the KV cache and the reconstructed vector based on the elements and coefficients selected so far. For example, in the first iteration step, the KV cache compression device (10) may determine some elements and coefficients using the KV cache or the initial residual value, then generate a reconstructed vector from the elements and coefficients, and then calculate the difference between the KV cache and the reconstructed vector as a new residual value. In the second iteration step, the KV cache compression device (10) may input the residual value back into the encoder model to determine additional elements to represent components that are not yet sufficiently explained. By repeating this process, the encoder model can output a sparse code by progressively approximating the input KV cache.

[0127] Additionally, in one embodiment, the KV cache compression device (10) can correct coefficients by performing least-squares optimization on the elements determined by the encoder model in each iteration step. Here, least-squares optimization may be a process of recalculating the coefficients of each element so that the error between the input KV cache and the reconstruction vector is reduced, given a set of elements that has already been selected. In other words, the encoder model performs element selection and initial coefficient prediction, and least-squares optimization can play a role in correcting the results more precisely. In one embodiment, since performing least-squares optimization in all iteration steps may increase computational cost or computational time, the KV cache compression device (10) may perform least-squares optimization only in the last iteration step.

[0128] Prior to or in parallel with step 840, the KV cache compression device (10) may select an input important token among a plurality of input tokens included in the input data. In one embodiment, the input important token may be selected based on a residual value calculated at each iteration step. That is, the KV cache compression device (10) may determine whether the token is an important token for subsequent inference by comparing the residual value calculated during the process of performing iterative encoding on the KV cache corresponding to a specific input token with a reference value. For example, if the residual value is greater than the reference value even after a certain number of iterations, the KV cache compression device (10) may determine the token as an unimportant token (input non-important token). Conversely, if the residual value is smaller than the reference value, the KV cache compression device (10) may determine the token as an important token (input important token). In this way, the KV cache compression device (10) selects input important tokens based on the residual value calculated in each iteration step, and the step of compressing the KV cache can be performed only on the input important tokens.

[0129] In another embodiment, the selection of important input tokens may be performed by performing OMP on a sparse dictionary a predetermined number of times and based on residual values ​​calculated during the OMP execution process. The KV cache compression device (10) may perform OMP for each input token a predetermined number of times and select the importance of each input token based on residual values ​​calculated during the OMP execution process. For example, the KV cache compression device (10) may select important input tokens based on residual values ​​obtained while performing OMP twice.

[0130] In step 850, the KV cache compression device (10) can compress the KV cache based on the sparse code. In this step, compression may mean reducing the data size by storing only the identification information and coefficients of some elements included in the sparse code, instead of storing the entire original input KV cache as is. Accordingly, storage space is reduced, and if necessary, the original KV cache can be restored using the sparse dictionary and the sparse code in the restoration step.

[0131] Additionally, in one embodiment, the KV cache compression device (10) may replace the key vector for the input non-important token with a fake key such that the inner product value with the query vector is less than or equal to a preset negative threshold. For example, the preset negative threshold may be set to -10,000 or an equivalent negative value. Accordingly, the result of the exponential operation during the softmax operation may be 0 or a value close to 0, and the attention weight corresponding to the input non-important token after the softmax operation may substantially converge to 0. Here, the input non-important token may refer to a token determined to have low importance in the aforementioned important token selection step. The fake key may be a replacement vector designed so that the corresponding token is hardly referenced in subsequent attention operations. That is, by replacing the key vector corresponding to the input non-important token with a fake key instead of removing it, the attention contribution can be significantly reduced while maintaining the same structure.

[0133] FIG. 9 is a flowchart illustrating an encoder model-based iterative encoding process according to an embodiment of the present invention.

[0134] A KV cache compression device (10) can generate a sparse code by inputting a KV cache corresponding to an input token into an encoder model, determining some elements among multiple elements of a sparse dictionary step by step, and then performing the process repeatedly until a preset number of iterations is reached while calculating a reconstruction vector and residual values ​​at each step.

[0135] In step 910, the KV cache compression device (10) can input the KV cache into an encoder model. The encoder model may be implemented in the form of a multilayer perceptron (MLP), but may also include any model capable of producing information necessary for generating a sparse code from the input KV cache.

[0136] In step 920, the KV cache compression device (10) may select a preset number of elements from a plurality of elements of a sparse dictionary. Here, the number of selections may refer to the number of elements selected at one time in each iteration step. For example, if the sparse dictionary contains a total of 1,000 elements and is set to select 4 elements in each iteration step, the KV cache compression device (10) may determine the top 4 elements based on the output value of the encoder model. Step 920 may be a process in which the encoder model determines which elements serve as the most appropriate approximation criteria for the input KV cache or residual value at the current step.

[0137] In step 930, the KV cache compression device (10) can calculate coefficients for the selected elements and generate a reconstruction vector. Here, the coefficients may be weights indicating how much each selected element reflects the input KV cache. That is, the coefficients may be values ​​that are multiplied by each selected element to form the final reconstruction vector. For example, if the second, fifth, and eighth elements are selected in a specific iteration step, the coefficients corresponding to each element may be determined to be 0.2, 0.8, and 0.3, respectively, and accordingly, the reconstruction vector may be generated in the form of a linear combination of the three elements. Here, the reconstruction vector may mean a vector that approximates the input KV cache using only the elements and coefficients selected so far. In other words, the reconstruction vector may be a result indicating to what extent the combination of elements selected up to the current step explains the input KV cache.

[0138] In step 940, the KV cache compression device (10) can calculate a residual value by calculating the difference between the input KV cache and the reconstruction vector. Here, the residual value may represent the difference between the input KV cache and the reconstruction vector. In one embodiment, the residual value may be used as a vector itself or as a scalar value, and may be used as an input to the encoder model in the next iteration step.

[0139] At step 950, the KV cache compression device (10) can determine whether a preset number of iterations has been reached. Here, the number of iterations may be a value indicating how many steps the entire encoding process will be divided into to perform. For example, if four elements are selected at each iteration step to form a final sparse code with a total of 32 elements, the number of iterations may be set to 8. If the preset number of iterations has not been reached, the KV cache compression device (10) may return to step 920 and perform the next iteration step. In this case, the residual value calculated at step 940 may be input back into the encoder model in the next iteration step. On the other hand, if the preset number of iterations has been reached, the iteration may be terminated and proceed to step 960.

[0140] In step 960, the KV cache compression device (10) can output a sparse code. Here, the sparse code may include identification information of elements determined in each iteration step and a set of coefficients corresponding to the elements. That is, the sparse code output in step 960 may not be merely the result of a single iteration step, but may be the combined result of elements and coefficients cumulatively determined in multiple iteration steps performed for a preset number of iterations. For example, if 4 elements are determined in the first iteration step and another 4 elements are determined in the second iteration step, and this process is repeated a total of 8 times, the final sparse code may include identification information of a total of 32 elements and corresponding coefficients. The sparse code thus generated may subsequently be stored as compressed data by the KV cache compression device (10) and, if necessary, may be used in conjunction with a sparse dictionary to restore the original KV cache. While the OMP method determines only one element at a time, the encoder model allows two or more elements to be determined at a time, thus enabling faster determination of elements.

[0141] As such, according to the embodiment illustrated in FIG. 9, the KV cache compression device (10) inputs the input KV cache into an encoder model, iteratively determines some elements among a plurality of elements of a sparse dictionary, and generates a final sparse code by performing coefficient calculation, reconstruction vector generation, residual value calculation, and determination of the number of iterations at each step. Due to this iterative structure, the encoder model can gradually complete the sparse representation by sequentially reducing the residual components that have not been explained so far at each iteration step, rather than compressing the entire input KV cache at once. Accordingly, higher reconstruction accuracy and stable sparse code generation can be achieved compared to a method in which the entire sparse code is determined at once in a single step.

[0143] FIG. 10 is a diagram illustrating an encoder model learning method according to an embodiment of the present invention.

[0144] Referring to FIG. 10, the KV cache compression device (10) can independently perform or combine the process of training only the encoder model with the sparse dictionary fixed in the first method and the process of training the sparse dictionary and the encoder model together in the second method.

[0145] The first method describes a method of training only an encoder model while keeping the sparse dictionary fixed. More specifically, the KV cache compression device (10) can use a KV cache corresponding to multiple tokens included in the training data as input and can provide the KV cache to the sparse dictionary and the encoder model. In this case, the sparse dictionary in the first method can be maintained in a fixed state that is not changed. That is, each element of the sparse dictionary is not updated during the training process and can function as a reference set that the encoder model references in its current state.

[0146] The encoder model can be trained to receive the KV cache as input, select some elements from a plurality of elements of a sparse dictionary, and output a sparse code containing coefficients for the selected elements. In the first method, the KV cache compression device (10) can reconstruct the KV cache based on the sparse code which is the output of the encoder model and calculate a residual value from the difference between the KV cache and the reconstructed vector. The KV cache compression device (10) can update the parameters of the encoder model based on the residual value.

[0147] The second method describes a method for training a sparse dictionary and an encoder model simultaneously. In the second method as well, the KV cache compression device (10) can use a KV cache corresponding to multiple tokens included in the training data as input, and can provide the KV cache to the sparse dictionary and the encoder model to output a sparse code. However, unlike the first method, in the second method, not only the parameters of the encoder model but also the elements of the sparse dictionary are the subjects of training.

[0148] The KV cache compression device (10) can generate a reconstruction vector based on the sparse code output by the encoder model for the KV cache in the second method, and can calculate a residual value from the difference between the reconstruction vector and the input KV cache. Then, along with updating the parameters of the encoder model using the residual value, each element of the sparse dictionary or selected elements can be updated. Here, updating elements may refer to a process of adjusting the orientation, size, distribution, or composition of the elements of the sparse dictionary itself to better suit the element pattern actually selected by the current encoder model. In the second method, the relationship in which the encoder model references the sparse dictionary is not maintained, but the sparse dictionary itself can also be adaptively updated to match the element selection method actually used by the encoder model.

[0149] In one embodiment, the KV cache compression device (10) may first perform a first method until a certain number of times or a certain condition is satisfied, and then switch to a second method. For example, the device may switch to a second method when the residual value in the first method decreases below a preset threshold or when the selected element pattern stabilizes above a certain level. In another embodiment, the point of transition between the first method and the second method may be determined based on a preset number of iterations or the magnitude of the residual value. Additionally, a learning rate reduction or a stabilization step may be performed after the second method.

[0151] FIG. 11 is a diagram illustrating the overall processing flow of a KV cache compression method according to one embodiment of the present invention.

[0152] Referring to FIG. 11, the KV cache compression device (10) can sequentially perform the following steps: 1) selecting important tokens among a plurality of tokens included in input data; 2) generating a sparse code using an encoder model for the KV cache corresponding to the selected important tokens; and 3) performing least squares optimization on the generated sparse code to output a final sparse code.

[0153] The KV cache compression device (10) may repeatedly perform OMP to determine the importance of tokens included in the input data. Here, OMP may be a sparse approximation algorithm that approximates an input vector by repeatedly selecting elements of a sparse dictionary. As illustrated in FIG. 11, the process of repeatedly performing OMP may include detailed steps, for example, element selection, coefficient calculation, and residual value calculation. More specifically, the KV cache compression device (10) may select an element of the sparse dictionary that is most similar to the KV cache corresponding to the token currently being processed, calculate a coefficient for the selected element, approximate the KV cache using the selected element and coefficient, and then calculate a residual value from the difference between the KV cache and the reconstruction result. This process may be repeated a predetermined number of times.

[0154] Important token selection can be performed using the results of the OMP iterations corresponding to each token, particularly the magnitude of the residuals. For example, tokens that are not sufficiently well represented by a sparse dictionary may have relatively large residuals, while conversely, tokens that are well represented by only a few elements may have relatively small residuals. In this way, residuals can be used as an indicator of how much important information the corresponding token contains in the subsequent inference process.

[0155] In one embodiment, the KV cache compression device (10) may perform OMP N times (N is a natural number). For example, the KV cache compression device (10) may perform OMP twice and determine the importance of each token based on the residual value calculated during the two executions.

[0156] The central area of ​​FIG. 11 represents a sparse code generation process using an encoder model. This process can be performed on important tokens selected in the OMP-based important token selection step. That is, the KV cache compression device (10) can use the KV cache corresponding to the important token as an input to the encoder model.

[0157] Iterative encoding may include a sequential process of inputting residual values, selecting elements, calculating coefficients, and calculating residual values. More specifically, the KV cache compression device (10) may input the KV cache into the encoder model in the first iteration step. Based on the input, the encoder model may select a preset number of elements from a plurality of elements of a sparse dictionary. For example, if it is set to select three elements at each iteration step, the encoder model may determine the three most appropriate elements for the current input. Subsequently, the KV cache compression device (10) may calculate coefficients for the selected elements. The KV cache compression device (10) may generate a reconstruction vector based on the selected elements and coefficients, and calculate a new residual value from the difference between the input KV cache and the reconstruction result to date.

[0158] The residual values ​​calculated in this way can be reused as input to the encoder model in the next iteration. In other words, rather than generating the entire sparse code in a single execution, the encoder model can form the sparse code by repeatedly receiving residual values ​​and progressively supplementing them.

[0159] In addition, the residual value is not only used as input for the next iteration step, but can also be used as an additional criterion for determining importance. That is, the KV cache compression device (10) can determine the importance of the token by checking the magnitude of the residual value at each iteration step.

[0160] The right area of ​​FIG. 11 illustrates the process of generating a final sparse code by least squares optimization. Specifically, the KV cache compression device (10) can generate a sparse code obtained through an iterative encoding process. The sparse code may include identification information of selected elements and initial coefficients calculated or predicted by an encoder model. Subsequently, the KV cache compression device (10) can calculate optimal coefficients that minimize reconstruction error while keeping the set of elements included in the sparse code fixed. Least squares optimization may be an operation that redefines the coefficients corresponding to each element so that the sum of squares of the reconstruction error is minimized. As a result, the KV cache compression device (10) can output a corrected final sparse code. Here, the final sparse code may be a form of sparse code in which the set of elements selected through iterative encoding is maintained, but the coefficients of each element are more finely adjusted. Also, in one example, the KV cache compression device (10) may apply least squares optimization only in the last iteration step instead of applying it to every iteration step.

[0161] As shown in the embodiment illustrated in FIG. 11, the KV cache compression device (10) first selects important tokens among all input tokens using OMP, then generates a sparse code by performing encoder model-based iterative encoding on the KV cache corresponding to the important tokens, and finally outputs a final sparse code with coefficients corrected through least squares optimization.

[0163] FIG. 12 is a configuration diagram for explaining a computing device according to an embodiment of the present invention.

[0164] Referring to FIG. 12, the computing device (1200) includes a processor (1210) and memory (1220). The computing device (1200) may represent a KV cache compression device (10) or a KV cache restoration device (40), etc. Additionally, the computing device (1200) may include all components included in a KV cache compression and restoration system.

[0165] The processor (1210) can process instructions of a computer program by performing basic arithmetic, logic, and input / output operations. Here, the instructions may be provided from memory (1220) or an external device.

[0166] For example, the functions performed by each module included in the processor (1210) may be performed by a single processor or by separate processors. The processor (1210) may perform operations or data processing regarding the control and / or communication of at least one other component.

[0167] The processor (1210) may be implemented as an array of multiple logic gates, or as a combination of a general-purpose microprocessor and memory storing a program that can be executed on the microprocessor. For example, the processor (1210) may include a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), a microprocessor, a digital signal processor (DSP), a controller, a microcontroller, a state machine, etc. In some environments, the processor (1210) may include an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), etc. For example, the processor (1210) may refer to a combination of processing devices such as a combination of a digital signal processor (DSP) and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors combined with a digital signal processor (DSP) core, or any other combination of such configurations.

[0168] The memory (1220) may include any non-transient computer-readable recording medium. As an example, the memory (1220) may include a non-perishable permanent mass storage device such as RAM (random access memory), ROM (read only memory), a disk drive, an SSD (solid state drive), or flash memory. As another example, a non-perishable permanent mass storage device such as ROM, an SSD, flash memory, or a disk drive may be a separate permanent storage device distinct from the memory. Additionally, the memory (1220) may store an operating system (OS) and at least one program code (e.g., code for the processor (1210) to execute).

[0169] These software components may be loaded from a computer-readable recording medium separate from memory (1220). This separate computer-readable recording medium may be a recording medium that can be directly connected to a computer and may include, for example, computer-readable recording media such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. Alternatively, the software components may be loaded into memory (1220) via a communication device other than a computer-readable recording medium. For example, at least one program may be loaded into memory (1220) based on a computer program installed by files provided through a communication device by developers or a file distribution system that distributes installation files for applications.

[0170] Meanwhile, the above-described method can be written as a program executable on a computer and can be implemented on a general-purpose digital computer that operates the program using a computer-readable recording medium. In addition, the structure of the data used in the above-described method can be recorded on a computer-readable recording medium through various means. The computer-readable recording medium includes storage media such as magnetic storage media (e.g., ROM, RAM, USB, floppy disk, hard disk, etc.) and optical reading media (e.g., CD-ROM, DVD, etc.).

[0171] A person skilled in the art to which the present invention pertains will understand that the present invention may be implemented in other specific forms without altering its technical concept or essential features. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims set forth below rather than by the detailed description above, and all modifications or variations derived from the meaning and scope of the claims and equivalent concepts should be interpreted as being included within the scope of the present invention. Explanation of the symbols

[0173] 10: KV Cache Compressor 20: Large-scale language models 30: Storage section 40: KV Cache Restorer

Claims

Claim 1 A method for compressing a KV cache of a large-scale language model, performed by a computing device comprising at least one processor, comprising: receiving a KV cache corresponding to a plurality of tokens included in training data; generating a sparse dictionary using the KV cache; training an encoder model to receive the KV cache and output a sparse code including identification information and coefficients of some elements among a plurality of elements of the sparse dictionary; inputting a KV cache for an input token corresponding to the input data into the encoder model to output a sparse code; and compressing the KV cache based on the sparse code. A KV cache compression method comprising: an encoder model being a multilayer perceptron model; a step of training the encoder model training the encoder model such that the reconstruction error between a vector reconstructed based on the sparse code and the KV cache is reduced; a step of outputting the sparse code being performed repeatedly for a preset number of iterations, wherein at each iteration step a preset number of selected elements among a plurality of elements of the sparse dictionary is determined, wherein the preset number of selected elements is a natural number greater than or equal to 2; further comprising a step of selecting input important tokens among a plurality of input tokens included in the input data, wherein the step of selecting input important tokens selects the input important tokens by determining the importance of each input token based on the residual value calculated at each iteration step, and a step of compressing the KV cache being performed only on the input important tokens. Claim 2 delete Claim 3 A KV cache compression method according to claim 1, wherein the step of outputting the sparse code is characterized in that, in each iteration step, the residual value calculated in the previous iteration step is input into the encoder model. Claim 4 A KV cache compression method according to claim 1, wherein the step of training the encoder model is characterized by using at least one of a first method of training the encoder model while the sparse dictionary is fixed and a second method of training the sparse dictionary and the encoder model together. Claim 5 delete Claim 6 A KV cache compression method according to claim 1, further comprising a step of selecting an input important token among a plurality of input tokens included in the input data, wherein the step of selecting the input important token comprises performing an Orthogonal Matching Pursuit (OMP) on the sparse dictionary a predetermined number of times and selecting the input important token based on residual values ​​calculated during the OMP execution process, and wherein the step of compressing the KV cache is performed only on the input important token. Claim 7 A KV cache compression method according to claim 1, wherein the step of outputting the sparse code further includes the step of correcting coefficients for the determined elements by performing least squares optimization on the elements determined by the encoder model. Claim 8 A KV cache compression method according to claim 7, characterized in that the least squares optimization is performed only in the last iteration step among each iteration step. Claim 9 A KV cache compression method according to claim 1, further comprising: a step of selecting an input important token among a plurality of input tokens included in the input data; and a step of replacing a key vector for an input non-important token with a fake key such that the inner product value with the query vector is less than or equal to a preset negative threshold. Claim 10 An apparatus for compressing a KV cache of a large-scale language model, comprising: a memory in which one or more instructions are stored; and at least one processor configured to execute one or more instructions stored in the memory. The method comprises, wherein the processor performs the steps of: receiving a KV cache corresponding to a plurality of tokens included in training data; generating a sparse dictionary using the KV cache; receiving the KV cache as input and training an encoder model to output a sparse code including identification information and coefficients of some elements among a plurality of elements of the sparse dictionary; inputting a KV cache for an input token corresponding to the input data into the encoder model to output a sparse code; and compressing the KV cache based on the sparse code. The encoder model is a multilayer perceptron model. In the step of training the encoder model, the processor trains the encoder model such that the reconstruction error between a vector reconstructed based on the sparse code and the KV cache is reduced. The processor performs the step of outputting the sparse code by repeating it a preset number of times, and in each iteration, determines a preset number of selected elements among a plurality of elements of the sparse dictionary, wherein the preset number of selected elements is a natural number greater than or equal to 2. The processor selects an input important token among a plurality of input tokens included in the input data. A KV cache compression device that performs a selection step, wherein the processor selects the input important tokens by determining the importance of each input token based on the residual value calculated in each iteration step in the selection step of the input important tokens, and compresses the KV cache only for the input important tokens in the step of compressing the KV cache.

Images