A method for decoding large language model based on computation and memory optimization and a decoding device implementing the same

Abstract

A decoding method according to some embodiments of the present invention is implemented by a computing device and is performed on a decoding model based on an attention mechanism in which a query, a key, and a value are utilized. The decoding method may include: a step of defining an input vector in a hidden state; a step of generating a compressed vector by compressing the input vector into a lower dimension so as to be associated with the query, the key, and the value; a step of generating a first query vector by processing the compressed vector associated with the query using a first processing method; a step of generating a first key vector and a value vector by processing the compressed vector associated with the key-value, which is retrieved from memory, using a second processing method; a step of generating an attention vector through the first query vector, the first key vector, and the value vector using an attention mechanism; and a step of generating an output token by converting the attention vector using a predetermined conversion method.

Description

Technology Field

[0001] The present invention relates to a method for decoding a large-scale language model for computation and memory optimization, and a decoding device for implementing the same. Background Technology

[0003] Generally, large-scale language models are implemented based on a transformer structure (see Fig. 1), and various attention mechanisms are utilized to calculate correlations between multiple tokens during the inference process.

[0004] As one attention method, Multi-Head Attention (MHA) calculates the similarity between all pairs of input tokens based on multiple differently defined heads to determine which other tokens to focus on when understanding a specific token. Since this method calculates attention weights for every pair of input tokens, it has the characteristic that computational complexity increases quadratically with input length. However, as the input length increases, the amount of computation increases rapidly, leading to a decrease in inference speed and limitations in its application to real-time applications.

[0005] Other attention methods include Grouped Query Attention (GQA) and Multi-Query Attention (MQA). These methods can reduce memory usage and improve computational efficiency by sharing keys and values ​​across multiple queries. However, query attention and multi-query attention suffer from the problem of reduced expressiveness and degraded model performance due to structural simplification. Prior art literature

[0007] Korean Published Patent Application No. 10-2025-0139141 (Published September 23, 2025) The problem to be solved

[0008] The present invention aims to solve the aforementioned problems by providing a decoding method for a large-scale language model and a decoding device for implementing the same, which can optimize computation and memory without degrading the performance of the model. means of solving the problem

[0010] A decoding method according to some embodiments of the present invention is implemented by a computing device and is performed on a decoding model based on an attention mechanism in which a query, a key, and a value are utilized. The decoding method may include: a step of defining an input vector in a hidden state; a step of generating a compressed vector by compressing the input vector into a lower dimension so as to be associated with the query, the key, and the value; a step of generating a first query vector by processing the compressed vector associated with the query using a first processing method; a step of generating a first key vector and a value vector by processing the compressed vector associated with the key-value, which is retrieved from memory, using a second processing method; a step of generating an attention vector through the first query vector, the first key vector, and the value vector using an attention mechanism; and a step of generating an output token by converting the attention vector using a predetermined conversion method.

[0011] Additionally, the step of generating the compression vector is to generate a key-value compression vector by compressing the input vector to a lower dimension so as to be associated with the key-value, and then updating the past key-value compression vector stored in memory to generate a final key-value compression vector, and the second processing method may be a method of generating the first key vector and the value vector by linearly restoring the final key-value compression vector, respectively.

[0012] In addition, the second processing method may be a method of generating the first key vector and the value vector by reflecting sequential correlations between input tokens in the final key-value compression vector and then linearly restoring each.

[0013] In addition, the second processing method may be a method of generating the first key vector and the value vector by applying a state-space model to the final key-value compression vector and then linearly restoring each.

[0014] Additionally, the method further includes the step of generating a second key vector related to the position of the key using the input vector; and the step of generating a key vector by stacking and merging the second key vector and the first key vector related to the content of the key; and the step of generating the attention vector may be the step of generating the attention vector by applying the key vector to the attention mechanism.

[0015] Additionally, the step of generating the second key vector may be a step of generating the second key vector by recalling from memory a position storage vector, which is a vector in which the input vector is embedded through a predetermined position embedding method, and reflecting the sequential correlation between input tokens.

[0016] Additionally, the step of generating the second key vector may be a step of generating the second key vector by reflecting the sequential correlation between input tokens in a vector in which the input vector is embedded through a predetermined position embedding method.

[0017] Additionally, the step of generating the compression vector is a step of generating a query compression vector by compressing the input vector into a lower dimension, and the first processing method may be a method of generating the first query vector by linearly restoring the query compression vector.

[0018] Additionally, the method further includes the step of generating a second query vector related to the location of the query using the input vector; and the step of generating a query vector by stacking and merging the second query vector and the first query vector related to the content of the query; and the step of generating the attention vector may be the step of generating the attention vector by applying the query vector to the attention mechanism.

[0019] Additionally, the step of generating the compression vector is a step of generating a query compression vector by compressing the input vector into a lower dimension, and the step of generating the second query vector may be a step of generating the query compression vector by embedding it through a predetermined position embedding method.

[0020] Additionally, the method further includes the step of generating a second query vector related to the location of the query using the input vector; and the step of generating a query vector by stacking and merging the second query vector and the first query vector related to the content of the query; and the attention mechanism may be an attention mechanism in which a vector formed by computing the value vector and the key vector is reflected in the query vector.

[0021] In addition, the above attention mechanism may be an attention method of the linear attention series.

[0022] Additionally, the second processing method comprises a second-1 processing method that generates the first key vector and the value vector by reflecting sequential correlations between input tokens in the final key-value compression vector and then linearly restoring each, and a second-2 processing method that generates the first key vector and the value vector by linearly restoring the final key-value compression vector, respectively, and the step of generating the first key vector and the value vector may be a step of selecting one of the second-1 processing method or the second-2 processing method depending on whether a predetermined condition is satisfied.

[0023] A decoding device according to some embodiments of the present invention comprises: a processor; a memory for loading a computer program executed by the processor; and a storage for storing the computer program, wherein the computer program may perform the following operations: receiving an input vector in a hidden state; compressing the input vector to a lower dimension to define a compressed vector; processing the compressed vector using a first processing method to generate a first query vector; processing the compressed vector using a second processing method to generate a first key vector and a value vector; generating an attention vector through an attention mechanism with the first query vector, the first key vector and the value vector; and converting the attention vector using a predetermined conversion method to generate an output token.

[0024] A computer program according to some embodiments of the present invention may be stored in a computer-readable storage medium for executing the decoding method in combination with a computer device which is hardware. Effects of the invention

[0026] The decoding method for a large-scale language model and the decoding device implementing the same according to the present invention can improve the computational efficiency of the language model.

[0027] In addition, the efficiency of computing resources, including memory, can be improved.

[0028] In addition, it can be expanded into various fields, ensuring scalability of applications.

[0029] However, the effects of the present invention are not limited to the effects described above, and unmentioned effects will be clearly understood by those skilled in the art from this specification and the attached drawings. Brief explanation of the drawing

[0031] FIG. 1 is a drawing illustrating the architecture of a conventional transformer of the present invention. FIG. 2 is a conceptual diagram of a decoding method according to an embodiment of the present invention. FIG. 3 is a flowchart of a decoding method according to an embodiment of the present invention. FIG. 4 is a conceptual diagram illustrating the process of calculating a query vector, a key vector, and a value vector in a decoding method according to an embodiment of the present invention. FIG. 5 is a diagram illustrating the step of generating a compression vector among a decoding method according to an embodiment of the present invention. FIG. 6 is a diagram illustrating the step of generating a query vector among a decoding method according to an embodiment of the present invention. FIG. 7 is a diagram illustrating the step of generating a key vector in a decoding method according to an embodiment of the present invention. FIG. 8 is a drawing illustrating the step of generating a value vector among a decoding method according to an embodiment of the present invention. FIG. 9 is a drawing illustrating an experimental example to explain the effect of a decoding method according to an embodiment of the present invention. FIG. 10 is a conceptual diagram illustrating the process of calculating a query vector, a key vector, and a value vector in a decoding method according to another embodiment of the present invention. FIG. 11 is a configuration diagram of a decoding device according to an embodiment of the present invention. Specific details for implementing the invention

[0032] Specific embodiments of the present invention will be described in detail below with reference to the drawings. However, the concept of the present invention is not limited to the presented embodiments, and those skilled in the art who understand the concept of the present invention may easily propose other inventions that are inferior or other embodiments included within the scope of the concept of the present invention by adding, changing, or deleting other components within the same scope of the concept, and such are also to be considered to be included within the scope of the concept of the present invention.

[0033] The terms first, second, A, B, etc., as used in this specification and claims may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0034] The terms used in this specification and claims are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" should be understood as not precluding the existence or addition of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification.

[0035] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0036] Furthermore, each component, process, procedure, or method included in each embodiment of the present invention may be shared within a scope that is not technically contradictory. Additionally, components with the same function within the scope of the same concept appearing in the drawings of each embodiment are described using the same reference numerals.

[0038] FIG. 2 is a conceptual diagram of a decoding method according to one embodiment of the present invention.

[0040] A decoding device according to some embodiments of the present invention may be a device that implements a decoding method based on an only decoding model in which only the decoder part is adopted and applied from the traditional transformer model illustrated in FIG. 1.

[0042] Referring to FIG. 2, input tokens are input into a decoding model, and an output token can be generated by utilizing techniques such as an embedding (not shown) of the input token, Root Mean Square Normalization (10) (60), residual connection (Add) (20, 30), attention (40), and a Feed-Forward Network (FFN) (50). RMS normalization can normalize the embedding vector of the input token so that the values ​​within the vector are maintained within a certain range. The embedding vector that has undergone RMS normalization can be applied to an attention mechanism to generate an attention vector. After applying RMS normalization again to the attention vector, a feed-forward network algorithm can be applied. The feed-forward network is a process of individually interpreting and refining the meaning of each token more deeply, and can usually be composed of two linear transformations and an activation function between them. "RMS normalization, attention, RMS normalization, feed-forward network" is a single transformer block, and a decoding method can be implemented by having multiple transformer blocks in sequence. Below, a detailed description of the decoder may be omitted within the scope of known technology.

[0044] A decoding device according to an embodiment of the present invention may receive an analysis request for any sentence / image / sound. To this end, the decoding device may receive input data and perform tokenization and embedding vectorization on the input data.

[0045] An input token can refer to the smallest unit input into a decoding model (decoding device).

[0046] For example, if the data to be analyzed is natural language, the input token can be a word; if the data to be analyzed is an image, the input token can be an image fragment; and if the data to be analyzed is sound, the input token can be a sound fragment.

[0047] The decoding model can generate the above-mentioned input token by performing tokenization on the input data, and a detailed explanation thereof may be omitted within the scope of the disclosed technology.

[0048] The input vector may refer to a vector generated per unit input token in the decoding model. Additionally, the input vector may be a vector generated by inputting an input token into Masked Multi-Headed Attention. This may be because if output tokens are generated using words that appear after the point of generation, the resulting output cannot be considered accurate. However, the present invention is not limited thereto. A detailed explanation regarding this may be omitted within the known technical scope of the Transformer Model Decoder. The input vector It may be a vector having a size of . For example, d may be 5120, but the present invention is not limited thereto. Multiple input vectors formed for each token may be applied to the decoding method in sequence, but the present invention is not limited thereto, and the input vectors may be applied to the decoding method in sequence. The input vector may be a hidden state as a vector processed by embedding the input token in the decoding model.

[0049] The input vector can be converted into a vector that performs the functions of query, key, and value in the attention mechanism described below.

[0050] A query can be an element that serves as a question regarding the process being handled. In other words, it can be an element corresponding to the goal or query condition being performed during the attention process. Consequently, it can serve as a criterion for exploring how related a given token is to other tokens during the attention process.

[0051] A key can refer to information about an address used to access a value in order to respond to a question called a query. A key can serve as an index to compare similarity with a query. It can be an element used to determine the position or characteristics of an input token within a sentence.

[0052] The value can refer to the actual meaning of the input token. That is, the value may be an element containing the information or meaning that the input token actually possesses. Weights are applied during the attention process, so it may contain a meaning that reflects the context.

[0053] Specific descriptions of queries, keys, and values ​​below may be omitted within the scope of the disclosed technology.

[0055] Hereinafter, a decoding method implemented by a decoder device according to some embodiments of the present invention will be described in detail.

[0057] FIG. 3 is a flowchart of a decoding method according to an embodiment of the present invention, and FIG. 4 is a conceptual diagram for explaining the process of calculating a query vector, a key vector, and a value vector in a decoding method according to an embodiment of the present invention.

[0059] Referring to FIGS. 3 and 4, a decoding method according to an embodiment of the present invention is implemented by a computing device and is performed on a decoding model based on an attention mechanism in which a query, a key, and a value are utilized. The decoding method may include: a step of defining an input vector in a hidden state (S100); a step of generating a compressed vector by compressing the input vector into a lower dimension so as to be associated with the query, the key, and the value (S200); a step of generating a first query vector by processing the compressed vector associated with the query using a first processing method (S310); a step of generating a first key vector and a value vector by processing the compressed vector associated with the key-value, which is retrieved from memory, using a second processing method (S410); a step of generating an attention vector through the first query vector, the first key vector, and the value vector using an attention mechanism (S500); and a step of generating an output token by converting the attention vector using a predetermined conversion method (S600). there is.

[0060] Additionally, a decoding method according to one embodiment of the present invention may further include the step of generating a second query vector related to the location of a query using the input vector (S320) and the step of generating a query vector by stacking and merging the second query vector and the first query vector related to the content of the query (S330).

[0061] Additionally, a decoding method according to one embodiment of the present invention may further include the step of generating a second key vector related to the position of a key using the input vector (S420) and the step of generating a key vector by stacking and merging the second key vector and the first key vector related to the content of the key (S430).

[0063] A decoding device according to one embodiment of the present invention can define an input vector of a hidden state (S100).

[0064] In step (S100), embedding is performed on the input token to obtain the input vector in the hidden state ( ) can be generated. For example, the input vector can be generated as the input token undergoes RMS normalization, residual addition, and Masked Multi-Headed Attention, but the present invention is not limited thereto. The input vector It may be a vector having a size of . Here, d may represent the dimension of the vector. For example, d may be 7168, but the present invention is not limited thereto.

[0065] The input vector can be defined as the input token is generated through embedding. However, the present invention is not limited thereto and can be applied to language models in which an encoder model and a decoder model exist in addition to a decoder model only, and the input vector generated by the encoder model can be utilized.

[0067] FIG. 5 is a diagram illustrating the step of generating a compression vector among a decoding method according to an embodiment of the present invention.

[0069] Referring to FIG. 5, a decoding device according to one embodiment of the present invention can compress the input vector into a lower dimension to generate a compressed vector and store the compressed vector in memory (S200).

[0070] The step of generating the above compression vector (S200) may include the step of generating a query compression vector by compressing the input vector to be related to a query in a lower dimension (S210) and the step of generating a key-value compression vector by compressing the input vector to be related to a key-value in a lower dimension (S220).

[0071] Additionally, the step of generating the compression vector (S200) may further include the step (S230) of generating a final key-value compression vector by merging the key-value compression vector with a past key-value compression vector stored in memory.

[0072] However, the present invention is not limited thereto, and the step of generating a query compression vector (S210) and the step of generating a key-value compression vector (S220) may be performed sequentially, but the two steps may also be performed simultaneously. Additionally, the step of generating a key-value compression vector (S220) may be performed first, followed by the step of generating a query compression vector (S220).

[0074] A decoding device according to one embodiment of the present invention compresses an input vector into a lower dimension to form a query compression vector ( ) can be generated (S210).

[0075] In step (S210), the query compression vector can be generated by applying the input vector and the query compression matrix. In other words, the query compression matrix and the input vector can be generated by performing a matrix-vector product (hereinafter referred to as matrix product). The detailed formula for generating the query compression vector may follow Equation 1-1. Here, the query compression matrix may be a query downward projection matrix that compresses the input vector to a specified low-dimensional vector size. The query compression matrix may perform the function of compressing the input vector into a much lower-dimensional latent space. For example, the query compression matrix It can be a matrix having a size. As a result, the above query compression vector is It can be a vector with a magnitude of . Here, can refer to the dimension of the query compression vector. For example, It may be 1536, but the present invention is not limited thereto.

[0077] (Mathematical Formula 1-1)

[0079] Here, is a query compression vector, and is the query compression matrix, and can mean an input vector.

[0080] The query compression vector is a processor corresponding to the generation of the key-value compression vector, which can increase the computational efficiency of the decoding method and minimize the possibility of errors.

[0082] A decoding device according to one embodiment of the present invention can generate a key-value compression vector by compressing an input vector into a lower dimension so as to be associated with a key-value (S220).

[0083] In step (S220), a key-value compression matrix is ​​applied to the input vector to obtain the key-value compression vector ( ) can be generated. In other words, the key-value compression matrix and the input vector can be matrix multiplied to generate the key-value compression vector. The detailed formula for generating the key-value compression vector may follow Equation 1-2. Here, the key-value compression matrix may be a key-value downward projection matrix that compresses the input vector to a specified lower-dimensional vector size. The key-value compression matrix may perform the function of compressing the input vector into a much lower-dimensional latent space. For example, the key-value compression matrix It can be a matrix having a size. As a result, the key-value compression vector is It can be a vector with a magnitude of . Here, can refer to the dimension of the key-value compression vector. Is It can be smaller. As a result, the above key-value compression vector may be smaller than the above query compression vector. This may be intended to increase memory efficiency by generating a smaller size for the key-value compression vector. For example, It may be 512, but the present invention is not limited thereto.

[0085] (Mathematical Formula 1-2)

[0087] Here, is a key-value compression vector, and is a key-value compression matrix, and can mean an input vector.

[0088] In step (S230), the key-value compression vector generated by the input vector ( ...can be stored in memory. By storing the above key-value compression vector side-by-side or aligned in a row together with the past key-value compression vector previously stored in memory, the key-value compression vector stored in memory ( ) is updated, and through this, the above final key-value compression vector can be generated.

[0089] In practice, the second processing method above allows the final key-value compression vector to be loaded from memory and used.

[0090] The final key-value compression vector can be newly updated and stored in memory by the aforementioned key-value compression vector, which is newly generated by the continuously inputting input vector. The final key-value compression vector can be stored in a memory cache and continuously utilized during a successive attention process. As a result, memory efficiency and computational efficiency can be maximized compared to storing the entire key-value set in the memory cache.

[0092] FIG. 6 is a diagram illustrating the step of generating a query vector among a decoding method according to an embodiment of the present invention.

[0094] Referring to FIG. 6, a decoding device according to one embodiment of the present invention can generate a first query vector by processing a compressed vector using a first processing method (S310).

[0095] Here, the first processing method may be a method of generating the first query vector by linearly restoring the query compression vector.

[0096] In step (S310), the first processing method linearly restores the query compression vector to the first query vector related to the content of the query ( It may be a processing method that generates ). Specifically, the first processing method may be a processing method that generates a first query vector by matrix multiplying the query content restoration matrix and the query compression vector together. The detailed formula for generating the first query vector may follow Equation 2-1. Here, the query content restoration matrix may be a query upward projection matrix, which is a matrix that linearly restores the query compression vector to a specified high-dimensional vector size. For example, the query content restoration matrix It can be a matrix with size. Here, can refer to the dimension of a query assigned to a single head, and can mean the number of heads. For example, It can be 128 days, can be 128. Here, the size of the query content restoration matrix may be based on a multi-head type attention. However, the present invention is not limited thereto and may also be applied to an attention having only one head. In such a case, Therefore, the query content restoration matrix is It can be a matrix with size.

[0098] (Mathematical Formula 2-1)

[0100] Here, is the first query vector, and is the query content restoration matrix, and can mean a query compression vector.

[0102] The first query vector may refer to a vector in which elements related to the content of the query are vectorized. This may be a vector in which only vectors related to the meaning of the query are extracted from the input vector by the query content restoration matrix. The first query vector It can be a vector having a size of . Here, if the attention mechanism is a multi-head series attention, each head has a first query vector ( ) can be divided by the number of heads and transmitted. In this case, the first query vector transmitted to each head of the attention mechanism ( )Is It can have the dimension of.

[0103] The dimension of the first query vector can be larger than the dimension of the input vector, which may be intended to analyze the relationships between input tokens more effectively in the attention mechanism. That is, The dimension can be larger than d dimension.

[0105] A decoding device according to one embodiment of the present invention can generate a second query vector related to the position of an input token by utilizing the input vector (S320).

[0106] That is, the step of generating the second query vector may be a step of generating the query compression vector by embedding it through a predetermined position embedding method.

[0107] In step (S320), the input vector is utilized to form a second query vector related to the location of the query ( ) can be generated. Specifically, the second query vector can be generated by embedding the query compression vector through a predetermined embedding method. The detailed formula for generating the second query vector may follow Equation 2-2.

[0109] (Mathematical Formula 2-2)

[0111] Here, represents the second query vector, and represents the query position matrix, and can mean a query compression vector.

[0113] A predetermined embedding method (first embedding method) may be a method for position embedding a vector generated by matrix multiplying the query position matrix and the query compression vector. The position embedding may be RoPE (Rotary Positional Embedding), but the present invention is not limited thereto, and other embedding methods may be utilized for position embedding. For example, Sinusoidal, AliBi, T5 Relative, etc. may be utilized for position embedding, but the present invention is not limited thereto.

[0114] The query position matrix may be a matrix for projecting the query compression vector onto a dimension capable of position embedding. Here, the query position matrix is It can be a matrix having the size of . Accordingly, the second query vector is It can be a vector having a size. Here, here, if the attention mechanism is a multi-head series attention, each head has a second query vector ( ) can be divided by the number of heads and transmitted. In this case, the second query vector transmitted to each head of the attention mechanism ( )Is It can have the dimension of. For example, The number may be 64, but the present invention is not limited thereto.

[0116] A decoding device according to one embodiment of the present invention can generate a query vector by stacking and merging the second query vector and the first query vector related to the content of the query (S330).

[0117] In step (S330), according to Equation 2-3, the first query vector and the second query vector may be stacked and merged to generate a single query vector. Here, stacking merging may refer to a method of simply merging the second query vector with the first query vector in the length direction (column direction). Since the second query vector is composed of a separate vector, the query vector can minimize distortion of meaning during the calculation process compared to a method where meaning and position are simply combined and calculated. The query vector It can have a size, and in the case of a multi-head attention, the query vector delivered to a single head( )Is Because of this, It can have a size.

[0119] (Mathematical Formula 2-3)

[0121] Here, refers to the query vector passed to a single head, and , represents the first query vector delivered to a single head, and can mean a second query vector delivered to a single head.

[0123] The following description is based on the assumption that a key vector and a value vector are generated based on a single input vector, but the present invention is not limited thereto.

[0125] FIG. 7 is a diagram illustrating the step of generating a key vector among a decoding method according to an embodiment of the present invention.

[0127] Referring to FIG. 7, a decoding device according to one embodiment of the present invention can generate a first key vector and a value vector by processing a compression vector using a second processing method (S410).

[0128] Here, the second processing method may be a method of generating the first key vector and the value vector by linearly restoring the key-value compression vectors, respectively.

[0129] In addition, the second processing method may be a method of generating the first key vector and the value vector by reflecting sequential correlations between input tokens in the key-value compression vector and then linearly restoring each.

[0131] In step (S410), the second processing method may be a processing method that generates the first key vector by linearly restoring the key-value compression vector after reflecting the sequential correlation between input tokens.

[0132] Here, reflecting sequential correlations between input tokens involves incorporating past key-value compression vectors generated during the iterative decoder process into the key-value compression vector at the time of output, thereby allowing the content of past key-value compression vectors to be reflected in future key-value compression vectors. Through this, the key-value compression vectors can be made to reflect the correlations (context) of the input tokens. For example, a State Space Model (SSM) may be utilized to reflect sequential correlations between input tokens, but the present invention is not limited thereto, and the means for reflecting sequential correlations between input tokens can be modified in various ways that are obvious to a person skilled in the art.

[0133] For example, the state-space model can perform operation procedures such as Equations 3-1 and 3-2.

[0135] (Mathematical Formula 3-1)

[0136] (Mathematical Formula 3-2)

[0138] Here, x is the input to the SSM It can mean, can represent an internal state of the state-space model, meaning a vector formed by accumulating and updating information up to time t, and can be a time-invariant transition matrix (State transition matrix), and and is a content-dependent matrix that can be dynamically adjusted by the manager's input. Additionally, D is a residual coefficient that can be used to compensate for loss by directly reflecting the input of x in the output, and It can refer to the vector output by the State-Space Model (SSM).

[0139] The second processing method may be a processing method that generates the first key vector related to the content of the key by linearly restoring the key-value compression vector reflecting the sequential correlation between input tokens. That is, the second processing method may be a method that generates the first key vector and the value vector by applying a state-space model to the key-value compression vector and then linearly restoring each. Specifically, the second processing method generates the first key vector ( It may be a processing method that generates ). The detailed formula for generating the first key vector may follow Equation 4-1. Here, the key content restoration matrix may be a key upward projection matrix, which is a matrix that linearly restores only the vectors related to the key in the key-value compression vector to a specified higher-order dimension vector size. For example, the key content restoration matrix It can have a size. Here, can refer to the dimension of a query assigned to a single head, and can mean the number of heads. For example, It can be 128 days, can be 128. Here, the size of the key content restoration matrix may be based on a multi-head type attention. However, the present invention is not limited thereto and may also be applied to an attention having only one head. In such a case, Therefore, the key content restoration matrix is It can be a matrix with size.

[0141] (Mathematical Formula 4-1)

[0143] Here, is the first key vector, and is the key content restoration matrix, and It can mean a key-value compression vector.

[0145] The first key vector may be a vector in which the content related to the key is vectorized. This may be a vector in which only the vectors related to the meaning of the key are extracted from the key-value compression vector by the key content restoration matrix. The first key vector ( )Is It can be a vector having a size of . Here, if the attention mechanism is a multi-head series attention, each head has a first key vector ( ) can be divided by the number of heads and transmitted. In this case, the first key vector transmitted to each head of the attention mechanism ( )Is It can have the dimension of.

[0146] The dimension of the first key vector can be larger than the dimension of the input vector, which may be intended to analyze the relationships between input tokens more effectively in the attention mechanism. That is, The dimension can be larger than d dimension.

[0148] Although the above description was based on calculating a first key vector based on a single input vector, when viewed as a whole attention mechanism, the second processing method may be a method of generating the first key vector by linearly restoring each of the final key-value compression vectors.

[0149] In the description of the second processing method above, the key-value compression vector above may be replaced with the final key-value compression vector, and a detailed explanation thereof may be omitted to the extent that it overlaps with the content described above.

[0150] For example, the second processing method may be a method of generating the first key vector and the value vector by linearly restoring the final key-value compression vectors. The second processing method may be a method of generating the first key vector by reflecting the sequential correlation between input tokens in the final key-value compression vector loaded from memory and then linearly restoring them. For example, the second processing method may be a processing method of generating the first key vector by matrix multiplying the key content restoration matrix and the vector reflecting the sequential correlation between input tokens in the final key-value compression vector. Here, the key content restoration matrix is ​​applied to each column, so that the first key vector at each time point is calculated and aligned side by side.

[0152] FIG. 8 is a diagram illustrating the step of generating a value vector among a decoding method according to an embodiment of the present invention.

[0154] Referring to FIG. 8, in step (S410), the second processing method may be a processing method that generates a value vector by linearly restoring the key-value compression vector after reflecting the sequential correlation between input tokens.

[0155] Here, a detailed explanation of the sequential correlation between input tokens may be omitted to the extent that it overlaps with the above.

[0156] The second processing method may be a processing method that generates a value vector by linearly restoring the key-value compression vector reflecting the sequential correlation between input tokens. Specifically, the second processing method may perform matrix multiplication of the value restoration matrix and a vector reflecting the sequential correlation between input tokens in the key-value compression vector to generate a value vector ( It may be a processing method that generates ). The detailed formula for generating the value vector may follow Equation 5-1. Here, the value restoration matrix is ​​a value upward projection matrix, and may be a matrix that linearly restores only the vectors related to the value in the key-value compression vector to a specified high-dimensional vector size. For example, the value restoration matrix is It can have a size. Here, can refer to the dimension of a query assigned to a single head, and can mean the number of heads. For example, It can be 128 days, can be 128. Here, the size of the value restoration matrix may be based on a multi-head type attention. However, the present invention is not limited thereto and may also be applied to an attention having only one head. In such a case, Therefore, the value restoration matrix is It can be a matrix with size.

[0158] (Mathematical Formula 5-1)

[0160] Here, is a value vector, and is the value restoration matrix, and It can mean a key-value compression vector.

[0162] The value vector may be a vectorized representation of values. This may be obtained by extracting only the vectors related to values ​​from the key-value compression vector using the aforementioned value restoration matrix. The value vector is It can be a vector with a size of . Here, . Here, if the attention mechanism is a multi-head series attention, each head has a value vector ( ) can be divided by the number of heads and transmitted. In this case, the value vector transmitted to each head of the attention mechanism ( )Is It can have the dimension of.

[0163] Although the above description was based on calculating a value vector based on a single input vector, when viewed as an attention mechanism as a whole, the second processing method may be a method of generating the value vector by linearly restoring each of the final key-value compression vectors.

[0164] In the description of the second processing method above, the key-value compression vector above may be replaced with the final key-value compression vector, and a detailed explanation thereof may be omitted to the extent that it overlaps with the content described above.

[0165] For example, the second processing method may be a processing method that generates a value vector by linearly restoring the final key-value compression vector after reflecting the sequential correlation between input tokens. The second processing method may be a processing method that generates the value vector by linearly restoring the final key-value compression vector reflecting the sequential correlation between input tokens. That is, the second processing method obtains a value vector ( It may be a processing method that generates ). Here, a value restoration matrix is ​​applied to each column, and a value vector for each time point is calculated and aligned side by side.

[0167] Again, referring to FIG. 7, a decoding device according to one embodiment of the present invention can generate a second key vector related to the position of a key by utilizing the input vector (S420).

[0168] That is, the step of generating the second key vector involves embedding the input vector through a predetermined position embedding method and reflecting the sequential correlation between input tokens to produce the second key vector ( It may be a step that generates ).

[0169] In step (S420), with reference to FIG. 7, the input vector can be utilized to generate a second key vector, which is a vector related to the position of the key. Specifically, the second key vector can be generated by reflecting the sequential correlation between input tokens using a vector calculated through a predetermined embedding method of the input vector. The detailed formula for generating the second key vector may follow Equation 4-2.

[0171] (Mathematical Formula 4-2)

[0173] Here, represents the second key vector, and represents the key-location matrix, and can mean an input vector.

[0175] A predetermined embedding method (second embedding method) may be a method for position embedding a vector generated by matrix multiplying the key position matrix and the input vector. A detailed description of the position embedding may be omitted to the extent that it overlaps with the above description.

[0176] In step (S420), the vector calculated through a predetermined embedding method is the position storage vector ( It can be defined as follows. The above location storage vector can be stored in memory paired with the above key-value compression vector. And, in order to calculate the above second key vector, the above location storage vector can be retrieved from memory. The above second key vector can be calculated by reflecting the sequential correlation between input tokens in the above location storage vector retrieved from memory.

[0177] The key-position matrix may be a matrix for projecting the above input vector into a dimension capable of position embedding. Here, the key-position matrix is It can be a matrix having the size of . Accordingly, the second key vector is It may be a vector having a size. Here, if the attention mechanism is a multi-head series attention, the second key vector may be provided to each head in the same way. Therefore, the same second key vector may be provided to a single head. This may be because the second key vector is a vector containing the location of the key, and maintaining the original vector size rather than splitting it minimizes data distortion regarding the key location.

[0179] A decoding device according to one embodiment of the present invention can generate a key vector by stacking and merging the second key vector and the first key vector related to the content of the key (S430).

[0180] In step (S430), according to Equation 4-3, the first key vector and the second key vector may be stacked and merged to generate a single key vector. A detailed explanation of the stacking and merging may be omitted to the extent that it overlaps with the above description. Since the second key vector is composed of a separate vector, the distortion of meaning during the operation process can be minimized compared to a method where meaning and position are simply combined and calculated. The second key vector may be stacked and merged for each first key vector applied to a single head. The key vector It can have a size, and the key vector transmitted to a single head is It can have a size.

[0182] (Mathematical Formula 4-3)

[0184] Here, represents the key vector transmitted to a single head, and represents the first key vector transmitted to a single head, and can mean a second key vector transmitted to a single head.

[0185] The key-value compression vector and the location storage vector calculated by the input vector can be aligned side by side (paired) and stored in memory. The final key-value compression vector and the location storage vectors for each of the key-value compression vectors constituting the final key-value compression vector can be stored in memory. Then, during the decoding process, the final key-value compression vector and the location storage vectors stored in memory can be reinforced through a state-space model, after which a first key vector, a second key vector, and a value vector can be generated.

[0187] Although the above description was based on calculating a second key vector based on a single input vector, when considering the attention mechanism as a whole, the second key vector can be generated by applying the final position storage vector described below to the state-space model. That is, the position storage vector can be described by substituting it with the final position storage vector, and a detailed explanation thereof may be omitted to the extent of redundancy.

[0188] Specifically, the step of generating the second key vector (S420) may comprise a step of generating a position storage vector by performing an operation on the input vector through a predetermined embedding method (S421), a step of generating a final position storage vector by merging the position storage vector with a past position storage vector stored in memory (S422), and a step of generating the second key vector by reflecting the sequential correlation between input tokens after loading the final position storage vector from memory (S423).

[0189] A detailed description of step (S421) may be omitted to the extent that it overlaps with the above description.

[0190] In step (S422), the location storage vector generated by the input vector is stored side-by-side or in a line alignment with the past location storage vector stored in the existing memory, thereby updating the location storage vector stored in memory and generating the final location storage vector. Each location storage vector can be stored in memory corresponding to the key-value compression vector.

[0191] In step (S423), the final location storage vector is loaded from memory and the state-space model is applied to generate the second key vector.

[0192] Through this, when the first key vector and the value vector are generated through the final key-value vector, a second key vector can be generated in correspondence with them.

[0194] According to one embodiment of the present invention, a decoding device can generate an attention vector through the first query vector, the second key vector, and the value vector and an attention mechanism (S500).

[0195] In step (S500), the query vector, the key vector, and the value vector may be applied to the attention mechanism to generate an attention vector. The attention mechanism may be an attention mechanism in which a vector formed by computing the value vector and the key vector is reflected in the query vector. The attention vector may be generated by performing operations on each vector for each head. The attention mechanism may be an attention method of the linear attention series. For example, the attention mechanism may be a multi-head linear attention, but the present invention is not limited thereto and may be applied to a single-head linear attention.

[0196] Linear attention may be a mechanism for calculating correlations through the query vector, the key vector, and the value vector. The query vector and the key vector are modified based on kernels and can be calculated by the following mathematical formulas 6-1, 6-2, and 6-3.

[0198] (Mathematical Formula 6-1)

[0199] (Mathematical Formula 6-2)

[0200] (Mathematical Formula 6-3)

[0202] Here, represents the query vector at time t, and represents the key vector at time point j, and is the value vector at time point j( It means the transpose vector of ), and , represents a non-linear kernel (feature map) transformation function, and represents the accumulated key-value cumulative correlation term up to time point t, and represents the accumulated key cumulative correlation term up to time point t, and may refer to the above attention vector. Here, t may refer to the time point at the time of analysis, and j may be an arbitrary time point. However, the above key-value cumulative correlation term and the above key cumulative correlation term may be calculated through the above key vector and the above value vector from the first operation to time point t due to the sigma operation. Here, the above key vector and the above value vector may be calculated through the above final key-value compression vector and the above final location storage vector. For example, and The final key-value compression vector and the location storage vector stored in memory can be linearly restored through a restoration matrix, but the present invention is not limited thereto. By storing and managing the KV values ​​in a compressed state in memory, memory efficiency can be maximized.

[0203] Linear attention can minimize computational load compared to self-attention (using Softmax), which computes the query vector and key vector first, by first computing the key vector and value vector and applying the result to the query vector. In other words, self-attention is Since it is calculated in the form of one head The operation of may be necessary. In contrast, linear attention is Since it is calculated in the form of a single head Since computation is required, computational efficiency can be excellent even for long sentences. However, although the reflection of the correlation between input tokens may be insufficient, by pre-reflecting the correlation between input tokens in the compression axis vector, it is possible to build a decoder model that improves memory efficiency without degrading decoder performance.

[0204] A detailed explanation of linear attention regarding this may be omitted within the scope of the disclosed technology.

[0205] The result vector generated by the attention mechanism can be generated for each head. The result vector is It may be a vector having a size of . In the case of an attention type attention mechanism, the attention vector may be calculated by vector-multiplying the output projection matrix and the vector generated by stacking and merging the result vector produced from a single head. The vector generated by stacking and merging the result vector is It may be a vector having dimensions (size). The method for calculating the above attention vector may follow Equation 7.

[0207] (Mathematical Formula 7)

[0209] Here, represents the attention vector, and represents the output projection matrix, and can mean the result vector to be produced by any head.

[0210] The output projection matrix may be a matrix for reducing the result vector generated by the attention mechanism to its original dimension. For example, the output projection matrix is It can have dimensions, and accordingly, the attention vector, like the input vector, It can have dimensions.

[0211] The attention vector is a vector that reflects the weights of the value vectors in the input vector, and it can be a vector that reflects the correlation between any input token and other surrounding input tokens.

[0213] A decoding device according to one embodiment of the present invention can generate an output token by converting the attention vector using a predetermined conversion method (S600).

[0214] In step (S600), the attention vector may be transformed using a predetermined transformation method to generate an output token. The predetermined transformation method may be a method of generating the output token by passing the output vector, which has undergone residual connection (Add), RMS normalization, and a feedforward network, through RMS normalization, unembedding, and a softmax function.

[0215] Specifically, through residual connection, the attention vector is combined with the input vector, and the combined vector passes through a feedforward network to generate an FFN output vector. Here, the RMS normalization-attention process (steps S200 to S500) and the RMS normalization-feedforward network illustrated in FIG. 2 can be executed multiple times as a single transformer block. For example, a transformer block may mean 60 iterations, but the present invention is not limited thereto. As the transformer block is repeated, the FFN output vector can evolve into a vector containing high-dimensional contextual information. The FFN output vector that has passed through all transformer blocks undergoes RMS normalization and is then projected onto the dimension of the entire vocabulary size through unembedding, thereby calculating a score for each word. Furthermore, the scores can be redefined as probabilities between 0 and 1 through Softmax. And, the output token may be generated as the word, image, or sound with the highest probability. Below, a detailed description of step (S600) may be omitted within the scope of the known technology.

[0217] FIG. 9 is a diagram illustrating an experimental example to explain the effect of a decoding method according to one embodiment of the present invention.

[0219] To verify the effects of the present invention, the present invention (MDLA) and existing attention modules (GQA, MHA, MLA) were compared based on FLOPs (in GFLOPs) at the prefill stage. The amount of computation was measured while varying the input sequence length to 1,024, 2,048, 4,096, and 16,384. As a result of the experiment, the MDLA module showed a significant performance improvement in terms of computational efficiency at the prefill stage compared to existing attention modules (GQA, MHA, MLA).

[0221] Figure 9(a) is a table comparing the amount of computation (in GBLOPs) per attention module according to the input sequence length. When the sequence length is 16,384, the present invention (MDLA) showed a computational reduction effect of approximately 63.6% compared to the MLA attention module, which confirms that the efficiency of the present invention becomes more pronounced as the sequence length increases.

[0222] FIG. 9(b) is a table comparing the token-unit computation efficiency of each attention module according to the input sequence length. As can be seen in the table, existing attention modules gradually increase the amount of computation per token as the sequence length increases, whereas the present invention (MDLA) is confirmed to remain constant regardless of the sequence length.

[0224] In this way, the present invention can provide stable and continuous computational efficiency while ensuring memory efficiency in a large-scale sequence processing environment.

[0226] Hereinafter, a decoding method according to another embodiment of the present invention will be described.

[0227] Further details may be omitted to the extent that they overlap with the above-mentioned content.

[0229] FIG. 10 is a conceptual diagram illustrating the process of calculating a query vector, a key vector, and a value vector in a decoding method according to another embodiment of the present invention.

[0231] Referring to FIG. 10, in a decoding method according to another embodiment of the present invention, the second processing method may comprise a second-1 processing method that generates the first key vector and the value vector by reflecting sequential correlations between input tokens in the final key-value compression vector and then linearly restoring each, and a second-2 processing method that generates the first key vector and the value vector by linearly restoring the final key-value compression vector.

[0232] Here, the step of generating the first key vector and value vector may be a step of selecting one of the processing method 2-1 or the processing method 2-2 depending on whether a predetermined condition is satisfied.

[0233] Additionally, the step of generating the second key vector may be a step of generating the second key vector through a first reflection method that reflects the sequential correlation between input tokens using a position storage vector, which is a vector in which the input vector is embedded through a predetermined position embedding method, or a step of generating the second key vector through a second reflection method that uses a position storage vector, which is a vector in which the input vector is embedded through a predetermined position embedding method.

[0234] Additionally, the step of generating the second key vector may be a step of selecting one of the first reflection method or the second reflection method according to predetermined conditions.

[0235] Additionally, the step of generating the attention vector may be a step of applying the query vector, the key vector, and the value vector by selecting one of an attention mechanism of the linear attention series or an attention mechanism of the self-attention series.

[0236] In addition, the step of generating the attention vector may be a method of selecting one of the attention mechanisms of the linear attention series or the attention mechanism of the self-attention series according to predetermined conditions.

[0238] Since the 2-1 processing method is identical to the 2nd processing method in the decoding method according to one embodiment of the present invention described above, a detailed explanation may be omitted within the scope of overlapping content.

[0239] The second-2 processing method (S1410) is the above key content restoration matrix ( It may be a method of generating the first key vector by matrix multiplying ) and the final key-value compression vector together. In addition, the second-2 processing method (S1410) is the value restoration matrix ( The method may be to generate the value vector by matrix multiplying the key content restoration matrix and the final key-value compression vector. A detailed description of the key content restoration matrix and the key-value compression vector may be omitted to the extent that it overlaps with the description above.

[0240] The step of generating the second key vector above, when based on the first reflection method, may comprise: a step of generating a position storage vector by performing calculations on the input vector through a predetermined embedding method; a step of generating a final position storage vector by merging the position storage vector with a past position storage vector stored in memory; and a step of generating the second key vector by reflecting the sequential correlation between input tokens after loading the final position storage vector from memory. Hereinafter, since the first reflection method is identical to the step of generating the second key vector according to an embodiment of the present invention, a detailed description may be omitted within the scope of overlapping content.

[0241] The above second reflection method (S1420) may be a method for calculating the second key vector using the position storage vector, which is a position embedding vector of a vector generated by matrix multiplying the key position matrix and the input vector. That is, it may be a method for calculating the second key vector using the final position storage vector, which is a position embedding vector of a vector generated by matrix multiplying the key position matrix and the input vector.

[0242] That is, the step of generating the second key vector may comprise, when based on the second reflection method, a step of generating a position storage vector by performing calculations on the input vector through a predetermined embedding method, a step of generating a final position storage vector by merging the position storage vector with a past position storage vector stored in memory, and a step of loading the final position storage vector from memory and utilizing it as the second key vector. Specifically, the position storage vector generated from a single input vector is stored in memory together with the key-value compression vector generated from a single vector, and is stored alongside the position storage vector and the previously stored key-value compression vector, thereby generating the final key-value compression vector and the final position storage vector. Furthermore, during the decoding process, the final position storage vector stored in memory can be utilized as is to be generated and utilized as the second key vector.

[0243] For example, an attention mechanism of the linear attention family may be a multi-head linear attention method (utilizing a kernel function), and an attention mechanism of the self-attention family may be a multi-head attention method (utilizing Softmax), but the present invention is not limited thereto.

[0245] A predetermined condition may be that the number of fields of input data consisting of input tokens is less than or equal to a predetermined number.

[0246] The judgment model can analyze the fields of the aforementioned input data. For example, if the input data is natural language, it can analyze the fields represented by the natural language included in the input data. Here, a field is a type of category and can serve as a criterion for classifying the input data. For example, fields can be classified into economics, science, medicine, art, etc.

[0247] The decision model can be trained by a separate computing or decoding device. For example, it can be trained through deep learning based on past input data and the labeled fields corresponding to that input data. For example, the decision model can estimate the field of the input data based on the grammar of words or sentences, or the expression method of the sentences. However, it is not limited to this, and the above decision model may be an open large language model (Gpt, Gemini, etc.).

[0248] Here, deep learning may utilize, but is not limited to, a back propagation algorithm, which is an algorithm that updates the weights of a neural network using labeled data from the output layer. Furthermore, since deep neural networks and back propagation algorithms are as previously known, a detailed description thereof may be omitted.

[0249] For example, the predetermined number may be three, but the present invention is not limited thereto, and the specific value of the predetermined number can be varied in various ways to the extent obvious to a person skilled in the art.

[0251] If a predetermined condition is satisfied, the step of generating the first key vector and the value vector may generate the first key vector and the value vector by selecting the processing method 2-1. Additionally, if the predetermined condition is satisfied, the step of generating the second key vector may generate the second key vector through the first reflection method. Additionally, if the predetermined condition is satisfied, the step of generating the attention vector may be a step of applying the query vector, the key vector, and the value vector by selecting an attention mechanism of the linear attention series.

[0252] In contrast, if the predetermined condition is not satisfied, the step of generating the first key vector and the value vector may generate the first key vector and the value vector by selecting the second-2 processing method. Additionally, if the predetermined condition is not satisfied, the step of generating the second key vector may generate the second key vector through the second reflection method. Additionally, if the predetermined condition is not satisfied, the step of generating the attention vector may be a step of applying the query vector, the key vector, and the value vector by selecting an attention mechanism of the self-attention series.

[0253] As such, when the input data contains many fields, the performance of the decoding model can be maintained by computationally processing the entire context in detail using the multi-head attention method and reflecting it in the attention vector, even if it increases the computational load somewhat. Conversely, when the input data contains few fields, the performance of the decoding model can be maintained even if a linear attention sequence is applied after correlating input tokens at the compression vector stage; therefore, a different method can be selected in terms of computational efficiency.

[0255] In this way, the decoding method according to another embodiment of the present invention can maximize the performance, computational efficiency, and memory efficiency of the decoding model by taking different methods depending on the state of the input data.

[0257] FIG. 11 is a configuration diagram of a decoding device according to one embodiment of the present invention.

[0259] Referring to FIG. 11, a decoding device (100) according to one embodiment of the present invention is a computing device and may include an input / output device (101, I / O), a processor (102, processor), a memory (103, memory), a storage (104, storage), an interface (105, interface), and a bus (106, bus).

[0260] In addition, the decoding device may further include a display unit (not shown) for displaying a user interface.

[0261] An input / output device (101), a processor (102), a memory (103), a storage (104), an interface (105), a display unit (not shown) and / or a communication unit (not shown) may be combined and connected to each other via a bus (106). Here, the bus (106) may correspond to a path through which data travels.

[0262] Specifically, the input / output device (101) may include at least one of a keypad, a keyboard, a touchscreen, and a display device.

[0263] Additionally, the processor (102) may include at least one of a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), GPU (Graphic Processing Unit), microprocessor, digital signal processor, microcontroller, application processor (AP), and logic elements capable of performing similar functions.

[0264] The memory (103) can load data and / or programs, etc. At this time, the memory (103) is an operational memory for enhancing the operation of the processor (101) and may include high-speed DRAM and / or SRAM, etc. The memory (103) may include one or more volatile memory devices such as DDR SDRAM (Double Data Rate Static DRAM) and SDR SDRAM (Single Data Rate SDRAM) and / or one or more non-volatile memory devices such as EEPROM (Electrical Erasable Programmable ROM) and flash memory.

[0265] Storage (104) (storage device) can store and retain data and / or programs, etc. Storage (104) may include one or more non-volatile memory devices such as a solid-state drive (SSD), a hard drive, and a flash memory. In the present invention, storage (104) can store a computer program consisting of instructions for performing a decoding method.

[0266] The interface (105) can perform the function of transmitting data to a communication network or receiving data from a communication network. The interface (105) may be in a wired or wireless form. For example, the interface (105) may include an antenna or a wired / wireless transceiver. The interface (150) may be configured to transmit and receive data with a device such as a server or other electronic device. The interface (105) may include a Bluetooth communication unit, a BLE (Bluetooth Low Energy) communication unit, a Near Field Communication unit, a WLAN (Wi-Fi) communication unit, a Zigbee communication unit, an infrared (IrDA, infrared Data Association) communication unit, a WFD (Wi-Fi Direct) communication unit, an UWB (ultra wideband) communication unit, an Ant+ communication unit, a mobile communication network, or a wired Ethernet network.

[0267] A display unit (not shown) may include any device capable of displaying an image, such as a display device, a screen device, or a beam projector. A display (e.g., a display) may include a panel, a holographic device, a projector, or a control circuit for controlling these. The panel may be implemented, for example, in a flexible, transparent, or wearable manner. The panel may be composed of a touch panel and one or more modules. According to one embodiment, the panel may include a pressure sensor (or force sensor) capable of measuring the intensity of pressure in response to a user's touch. The pressure sensor may be implemented integrally with the touch panel or as one or more sensors separate from the touch panel. A holographic device may display a three-dimensional image in mid-air using light interference. A projector may display an image by projecting light onto a screen. The screen may be located, for example, inside or outside an electronic device.

[0268] A server according to embodiments of the present invention may be a system formed by connecting a plurality of computing devices to each other through a network. In such cases, each module or combination of modules may be implemented as a computing device. However, the present embodiment is not limited thereto.

[0269] Additionally, the server may be implemented as at least one of a workstation, a data center, an internet data center (IDC), a direct attached storage (DAS) system, a storage area network (SAN) system, a network attached storage (NAS) system, a redundant array of inexpensive disks, or a redundant array of independent disks (RAID) system, and an Electronic Document Management (EDMS) system, but the present embodiment is not limited thereto.

[0270] In some embodiments of the present invention, when executed by the at least one processor, the program may perform the following operations: receiving an input vector in a hidden state; defining a compressed vector by compressing the input vector to a lower dimension; generating a first query vector by processing the compressed vector with a first processing method; generating a first key vector and a value vector by processing the compressed vector with a second processing method; generating an attention vector through an attention mechanism with the first query vector, the first key vector and the value vector; and generating an output token by converting the attention vector with a predetermined conversion method.

[0271] The decoding method according to the embodiment may also be implemented in the form of a computer-readable medium that stores instructions and data executable by a computer.

[0272] In this case, instructions and data may be stored in the form of program code, and when executed by a processor, they may generate a specific program module to perform a specific operation. Additionally, a computer-readable medium may be any available medium accessible by a computer, and includes both volatile and non-volatile media, as well as removable and inremovable media. Furthermore, a computer-readable medium may be a computer recording medium, which may include both volatile and non-volatile, removable and inremovable media implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. For example, a computer recording medium may be a magnetic storage medium such as HDDs and SSDs, an optical recording medium such as CDs, DVDs, and Blu-ray discs, or memory contained in a server accessible via a network.

[0273] In addition, the decoding method according to the embodiment may be implemented as a computer program (or computer program product) comprising instructions executable by a computer. The computer program includes programmable machine instructions processed by a processor and may be implemented in a high-level programming language, an object-oriented programming language, assembly language, or machine language, etc. Additionally, the computer program may be recorded on a tangible computer-readable recording medium (e.g., memory, hard disk, magnetic / optical medium, or SSD (Solid-State Drive), etc.).

[0274] Accordingly, the decoding method according to the embodiment can be implemented by executing a computer program as described above by a computing device. The computing device may include at least some of a processor, memory, storage, a high-speed interface connected to the memory and a high-speed expansion port, and a low-speed interface connected to the low-speed bus and storage. Each of these components is connected to one another using various buses and may be mounted on a common motherboard or mounted in other suitable ways.

[0275] Here, the processor can process instructions within the computing device, such as instructions stored in memory or storage devices to display graphic information for providing a Graphic User Interface (GUI) on external input and output devices, such as a display connected to a high-speed interface. In another embodiment, a plurality of processors and / or a plurality of buses may be utilized together with a plurality of memories and memory types. Additionally, the processor may be implemented as a chipset comprising chips including a plurality of independent analog and / or digital processors.

[0276] In addition, memory stores information within a computing device. For example, memory may consist of volatile memory units or a set thereof. As another example, memory may consist of non-volatile memory units or a set thereof. Furthermore, memory may be other forms of computer-readable media, such as magnetic or optical discs.

[0277] And storage can provide a large amount of storage space to computing devices. A storage device may be a computer-readable medium or a configuration containing such a medium, and may include, for example, devices or other configurations within a Storage Area Network (SAN), and may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, flash memory, or other similar semiconductor memory device or device array.

[0278] In addition, a decoding device according to some embodiments of the present invention includes a processor, a memory for loading a computer program executed by the processor, and a storage for storing the computer program, wherein the computer program may perform the operations of receiving an input vector in a hidden state, defining a compressed vector by compressing the input vector to a lower dimension, generating a first query vector by processing the compressed vector with a first processing method, generating a first key vector and a value vector by processing the compressed vector with a second processing method, generating an attention vector through an attention mechanism with the first query vector, the first key vector and the value vector, and generating an output token by converting the attention vector with a predetermined conversion method.

[0279] In addition, a computer program according to some embodiments of the present invention may be combined with a computer device which is hardware and stored in a computer-readable storage medium to execute the decoding method described above.

[0281] In some embodiments of the present invention, the decoding method and decoding device can be utilized in generative language models such as AI chatbots. The generative language model can be separately trained and operated based on internal documents provided to each customer. The generative language model is a customer domain-specific language model and can be operated in a closed network and multi-hybrid cloud environment for security purposes. In such an environment, some embodiments of the present invention propose a method and device that drastically reduce computational load and memory while maintaining or improving performance.

[0283] In order to more clearly express the technical concept of the present invention, the attached drawings briefly depict or omit configurations that are unrelated to or have little relevance to the technical concept of the present invention.

[0284] The above description is merely an illustrative explanation of the technical concept of the present embodiment, and a person skilled in the art to which the present embodiment belongs would be able to make various modifications and variations within the scope of the essential characteristics of the present embodiment. Accordingly, the present embodiments are intended to explain, not limit, the technical concept of the present embodiment, and the scope of the technical concept of the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment shall be interpreted by the claims below, and all technical concepts within an equivalent scope shall be interpreted as being included within the scope of rights of the present embodiment. Explanation of the symbols

[0286] 101: Input / Output Device 102: Processor 103 : Memory 104 : Storage 105 : Interface 106 : Bus

Claims

Claim 1 A decoding method performed on a decoding model based on an attention mechanism that utilizes a query, a key, and a value, implemented by a computing device, comprising: a step of defining an input vector in a hidden state; a step of generating a compressed vector by compressing the input vector into a lower dimension so as to be associated with the query, the key, and the value; a step of generating a first query vector by dividing the compressed vector associated with the query by the number of heads after processing it by a first processing method; a step of generating a first key vector and a value vector by processing the compressed vector associated with the key-value, which is retrieved from memory, by a second processing method and dividing it by the number of heads; a step of generating a second key vector associated with the location of the key and provided identically to one head by utilizing the input vector; a step of generating a key vector transmitted to one head by stacking and merging the second key vector for each first key vector associated with the content of the key and applied to one head; and a step of generating an attention vector through the attention mechanism with the first query vector, the key vector, and the value vector. The decoding method comprises the step of generating an output token by converting the attention vector using a predetermined conversion method; wherein the step of generating the compression vector is a step of generating a key-value compression vector by compressing the input vector to a lower dimension so as to be associated with the key-value, and then updating the past key-value compression vector stored in the memory to generate a final key-value compression vector; the second processing method is a method of generating the first key vector and the value vector by reflecting the sequential correlation between input tokens in the final key-value compression vector and then linearly restoring each; and the step of generating the second key vector is a step of generating the second key vector by reflecting the sequential correlation between input tokens in the position storage vector called from the memory, after the position storage vector, which is a vector in which the input vector is embedded through a predetermined position embedding method, is paired with the key-value compression vector and stored in memory. Claim 2 In claim 1, the second processing method is a decoding method that generates the first key vector by matrix multiplying the final key-value compression vector, in which sequential correlations between input tokens are reflected in the key content restoration matrix. Claim 3 In claim 1, the second processing method is a decoding method that generates the value vector by matrix multiplying the final key-value compression vector, in which sequential correlations between input tokens are reflected in the value restoration matrix. Claim 4 In paragraph 1, the sequential correlation between input tokens is identified by a decoding method utilizing a state-space model. Claim 5 In claim 1, the predetermined position embedding method is a decoding method that is a method of position embedding a vector generated by matrix multiplying a key position matrix and the input vector. Claim 6 In paragraph 5, the above position embedding is a decoding method that is a RoPE (Rotary Positional Embedding) embedding method. Claim 7 In claim 1, the step of generating the compression vector is a step of generating a query compression vector by compressing the input vector to a lower dimension, and the first processing method is a decoding method that generates the first query vector by linearly restoring the query compression vector. Claim 8 A decoding method according to claim 7, further comprising: a step of generating a second query vector related to the position of the query using the input vector; and a step of generating a query vector by stacking and merging the second query vector and the first query vector related to the content of the query; wherein the step of generating the attention vector is a step of generating an attention vector by applying the query vector to the attention mechanism. Claim 9 A decoding method according to claim 8, wherein the step of generating the compression vector is a step of generating a query compression vector by compressing the input vector into a lower dimension, and the step of generating the second query vector is a step of generating the query compression vector by embedding it through a predetermined position embedding method. Claim 10 A decoding method according to claim 1, further comprising: a step of generating a second query vector related to the location of the query using the input vector; and a step of generating a query vector by stacking and merging the second query vector and the first query vector related to the content of the query; wherein the attention mechanism is an attention mechanism in which a vector formed by computing the value vector and the key vector is reflected in the query vector. Claim 11 In paragraph 10, the above attention mechanism is a decoding method that is an attention method of the linear attention series. Claim 12 In claim 1, the second processing method comprises a second-1 processing method for generating the first key vector and the value vector by reflecting sequential correlations between input tokens in the final key-value compression vector and then linearly restoring each, and a second-2 processing method for generating the first key vector and the value vector by linearly restoring the final key-value compression vector, respectively, wherein the step of generating the first key vector and the value vector is a step of selecting one of the second-1 processing method or the second-2 processing method depending on whether a predetermined condition is satisfied. Claim 13 A processor; a memory for loading a computer program executed by the processor; and a storage for storing the computer program; wherein the computer program comprises: an operation of receiving an input vector in a hidden state; an operation of defining a compression vector by compressing the input vector into a lower dimension; an operation of generating a first query vector by processing the compression vector using a first processing method and then dividing it by the number of heads; an operation of generating a first key vector and a value vector by processing the compression vector using a second processing method and then dividing it by the number of heads; an operation of generating a second key vector that is related to the position of the key and is provided identically to one head by utilizing the input vector; an operation of generating a key vector transmitted to one head by stacking and merging the second key vector with each first key vector that is related to the content of the key and applied to one head; and an operation of generating an attention vector through an attention mechanism with the first query vector, the key vector, and the value vector. A decoding device comprising: executing an operation to generate an output token by converting the attention vector using a predetermined conversion method; wherein the operation to generate the compression vector is an operation to generate a key-value compression vector by compressing the input vector to a lower dimension so as to be associated with the key-value, and then updating the past key-value compression vector stored in the memory to generate a final key-value compression vector; wherein the second processing method is a method to generate the first key vector and the value vector by reflecting the sequential correlation between input tokens in the final key-value compression vector and then linearly restoring each; and wherein the operation to generate the second key vector is an operation in which a position storage vector, which is a vector in which the input vector is embedded through a predetermined position embedding method, is paired with the key-value compression vector and stored in memory, and then the sequential correlation between input tokens is reflected in the position storage vector called from the memory to generate the second key vector. Claim 14 A computer program stored in a computer-readable storage medium to execute a decoding method according to any one of claims 1 to 12, combined with a computer device which is hardware.

Images