Electronic device for enhancing position information of a word element and operating method thereof

Abstract

An electronic device and an operating method of the electronic device are provided. The method includes generating a plurality of pieces of enhanced position information for a first token by enhancing position information input to the first token to a large language model (LLM), generating a plurality of queries for the first token based on the plurality of pieces of enhanced position information, determining a plurality of first attention scores based on the plurality of queries, determining a first target attention score based on the plurality of first attention scores, and performing inference on an input based on the first target attention score.

Description

Cross-references to related applications

[0001] This application claims priority to Korean Patent Application No. 10-2024-0194542, filed on December 23, 2024, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference. Technical Field

[0002] The methods and apparatus consistent with the embodiments relate to an electronic device for enhancing the location information of tokens and a method of operating the same. Background Technology

[0003] Large Language Models (LLMs) are deep learning-based models trained using massive amounts of data. LLMs are specifically designed for understanding and generating text data. Recently, LLMs have revolutionized the field of natural language processing and become one of the core technologies enabling computers to understand and process human language. As an example, an LLM may include a Generative Pre-trained Transformer (GPT) and a Bidirectional Encoder Representation (BERT) based on the transformer. Summary of the Invention

[0004] One or more embodiments may at least solve the above-described problems and / or disadvantages, as well as other disadvantages not described above. Furthermore, embodiments do not need to overcome the above-described disadvantages, and embodiments may not overcome any of the above-described problems.

[0005] According to one aspect of this disclosure, an operating method for an electronic device is provided, the operating method comprising: generating multiple augmented positional information for a first lexical unit by augmenting positional information of a first lexical unit input to a language model (LM); generating multiple queries for the first lexical unit based on the multiple augmented positional information; determining multiple first attention scores based on the multiple queries; determining a first target attention score based on the multiple first attention scores; and performing inference on the input based on the first target attention score.

[0006] Multiple pieces of enhanced location information can be generated based on at least one of a plurality of first functions configured to enhance the location information of the first lexical unit.

[0007] The method may also include generating multiple keys for the second lexical by enhancing the positional information of the second lexical input to the LM, and the second lexical may be configured to perform an attention operation with the first lexical.

[0008] Determining multiple first attention scores may include: determining a first attention score among multiple first attention scores based on a pair of first queries and first keys among multiple queries and multiple keys, wherein the first queries and first keys include positional information enhanced by the same first function.

[0009] Determining the first target attention score may include: determining the first target attention score based on one of a plurality of second functions that determine the first target attention score from a plurality of first attention scores.

[0010] The method may also include: determining multiple second attention scores based on the keys of multiple queries and second tokens; and determining a second target attention score based on the multiple second attention scores.

[0011] Determining the second target attention score may include: determining the second target attention score based on one of a plurality of third functions that determine the second target attention score from a plurality of second attention scores.

[0012] The operation method may also include: removing the value and key of the second word from memory based on the second target attention score, which is less than or equal to a threshold.

[0013] The LM can include a multi-head attention layer, and can perform the generation of multiple enhanced positional information for the first word, the generation of multiple queries, the determination of multiple first attention scores, and the determination of the first target attention score for each head of the multi-head attention layer.

[0014] According to another aspect of this disclosure, an operating method for an electronic device is provided, the method comprising: inputting lexical units into a language model (LM) including a plurality of transformer blocks; and generating an output corresponding to the lexical units by the LM performing reasoning using the lexical units as input, wherein at least one of the plurality of transformer blocks is configured to: generate a plurality of enhanced positional information by enhancing the positional information of the lexical units; generate a plurality of queries and a plurality of keys for the lexical units based on the plurality of enhanced positional information; and perform reasoning corresponding to the lexical units based on a target attention score determined from the plurality of queries and the plurality of keys.

[0015] Multiple transformer blocks may include a first transformer block and a second transformer block, and the first transformer block and the second transformer block may be configured to enhance the position information through different functions.

[0016] According to another aspect of this disclosure, an electronic device is provided, comprising: a memory configured to store instructions; and at least one processor configured to execute the instructions, wherein the instructions, when executed individually or jointly by the at least one processor, cause the electronic device to: generate multiple enhanced positional information for a first lexical unit by enhancing positional information of the first lexical unit input to a language model (LM); generate multiple queries for the first lexical unit based on the multiple enhanced positional information; determine multiple first attention scores based on the multiple queries; determine a first target attention score based on the multiple first attention scores; and perform inference on the input based on the first target attention score.

[0017] When executed by at least one processor alone or together, the instructions enable an electronic device to generate multiple pieces of enhanced location information based on the location information, using at least one of a plurality of first functions configured to enhance the location information of the first lexical unit.

[0018] When executed by at least one processor alone or together, the instructions enable the electronic device to: generate multiple keys for a second lexical by enhancing the positional information of the second lexical input to the LM, and the second lexical can be configured to perform an attention operation with the first lexical.

[0019] When executed by at least one processor individually or jointly, the instructions enable an electronic device to: determine a first attention score among multiple first attention scores based on a pair of first queries and first keys among multiple queries and first keys, wherein the first queries and first keys include positional information enhanced by the same first function.

[0020] When executed by at least one processor individually or jointly, the instructions enable an electronic device to: determine a first target attention score based on one of a plurality of second functions that determine a first target attention score from a plurality of first attention scores.

[0021] When executed by at least one processor individually or jointly, the instructions enable the electronic device to: determine multiple second attention scores based on the keys of multiple queries and second terms; and determine a second target attention score based on the multiple second attention scores.

[0022] When executed by at least one processor individually or jointly, the instructions enable the electronic device to: determine a second target attention score based on one of a plurality of third functions that determine a second target attention score from a plurality of second attention scores.

[0023] When executed by at least one processor alone or together, the instruction can cause the electronic device to: remove the value and key of a second word from memory based on a second target attention score that is less than or equal to a threshold.

[0024] The LM may include a multi-head attention layer, and the instructions, when executed by at least one processor individually or jointly, enable the electronic device to: generate multiple enhanced positional information for a first term for each head of the multi-head attention layer, generate multiple queries, determine multiple first attention scores, and determine a first target attention score. Attached Figure Description

[0025] The above and / or other aspects will become more apparent from the description of specific embodiments in conjunction with the accompanying drawings, in which:

[0026] Figure 1 This is a diagram illustrating an electronic device according to an embodiment;

[0027] Figure 2 This is a diagram illustrating the architecture of a large language model (LLM) according to an embodiment;

[0028] Figure 3 It is a diagram showing attention calculations based on relevant technologies;

[0029] Figure 4 This is a diagram illustrating attention calculation using enhanced location information according to an embodiment;

[0030] Figure 5 This is a diagram illustrating an enhanced deletion key-value (KV) cache using location information according to an embodiment;

[0031] Figure 6 This is a diagram illustrating an LLM that enhances position information in some transformer blocks according to an embodiment; and

[0032] Figure 7 This is a flowchart illustrating an operation method of an electronic device according to an embodiment. Detailed Implementation

[0033] The detailed structural or functional descriptions below are provided as examples only, and various changes and modifications can be made to the embodiments. These embodiments should not be construed as limiting to this disclosure, and should be understood to include all changes, equivalents, and substitutions within the scope of the ideas and techniques of this disclosure.

[0034] In this document, terms such as first, second, etc., may be used to describe components. Each of these terms is not intended to define the nature, order, or sequence of the corresponding component, but only to distinguish the corresponding component from other components. For example, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component.

[0035] It should be noted that if a component is described as “connected,” “coupled,” or “joined” to another component, a third component can be “connected,” “coupled,” and “joined” between the first and second components, although the first component can be directly connected, coupled, or joined to the second component.

[0036] Unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. It will also be understood that the terms “comprising / containing” and / or “including / including” as used herein mean the presence of the said feature, integer, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or combinations thereof.

[0037] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. Terms (such as those defined in common dictionaries) shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art and shall not be interpreted as having an ideal or overly formal meaning unless expressly defined herein.

[0038] In the following description, embodiments will be illustrated with reference to the accompanying drawings. When describing embodiments with reference to the accompanying drawings, the same reference numerals refer to the same elements, and repeated descriptions related to them will be omitted.

[0039] Figure 1 This is a diagram illustrating an electronic device according to an embodiment.

[0040] refer to Figure 1 Electronic device 100 may include a host processor 110, memory 120, and accelerator 130. The host processor 110, memory 120, and accelerator 130 can communicate with each other via a bus, network on-chip (NoC), peripheral component interconnect (PCIe), etc. Figure 1 In the example shown, only components relevant to the embodiments described herein are illustrated in electronic device 100. Therefore, electronic device 100 may also include, in addition to... Figure 1 Other general-purpose components besides the components shown.

[0041] The host processor 110 can perform overall functions for controlling the electronic device 100. The host processor 110 can control the electronic device 100 as a whole by executing programs and / or instructions stored in the memory 120. The host processor 110 can be implemented as a central processing unit (CPU), graphics processing unit (GPU), application processor (AP), etc., included in the electronic device 100, but the embodiments are not limited thereto.

[0042] The memory 120 can be hardware used to store data to be processed in the electronic device 100. Additionally, the memory 120 can store applications, drivers, etc., to be driven by the electronic device 100. The memory 120 may include volatile memory (e.g., dynamic random access memory (DRAM)) and / or non-volatile memory.

[0043] Electronic device 100 may include an accelerator 130 for easy operation. Accelerator 130 can handle tasks that, due to their characteristics, are more efficiently handled by a separate dedicated processor (i.e., accelerator 130) than by a general-purpose host processor 110. For example, a large language model (LLM) can be executed by accelerator 130. Here, one or more processing elements (PEs) included in accelerator 130 can be utilized. Accelerator 130 may correspond to, for example, a neural processing unit (NPU), tensor processing unit (TPU), digital signal processor (DSP), GPU, neural engine, etc., that performs operations according to a neural network. Tasks that can be handled more efficiently by accelerator 130 are not necessarily handled by accelerator 130, but may be handled by host processor 110.

[0044] An LLM (Learning Model) is a deep learning-based model trained on extremely large datasets and can be a neural network. LLMs can be specifically designed for understanding and generating text data. Because LLMs should include many more parameters to improve performance, their size can be very large. For example, an LLM might include tens to hundreds of billions of parameters. Due to the large size of LLMs, the hardware resources of the electronic device 100 executing the LLM can be critical to its performance. For example, a significant amount of memory resources may be used for LLM execution (e.g., inference). Therefore, the inference speed of the LLM may be slowed down, and various problems may occur because the LLM consumes a large portion of the memory resources of the electronic device 100.

[0045] Therefore, making the most of the loaded data can be crucial. For example, even when using the same size key-value (KV) cache by effectively managing key-value pairs, a method to improve performance may be needed.

[0046] The architecture of a typical LLM will be described below.

[0047] Figure 2 This is a diagram illustrating the architecture of an LLM according to an embodiment.

[0048] refer to Figure 2 An LLM 200 may include an input layer (input embedding), one or more transformer blocks, and an output layer (prediction). For example, the input embedding may include converting tokens (e.g., words) into vector form in a way that the LLM 200 can understand. To enable the transformer blocks to process multiple consecutive pieces of information, positional information (e.g., relative order information) of the tokens may be required. In an LLM 200, positional embedding may be an operation that adds positional information corresponding to a word (or token) to a vector. An LLM 200 may include an operation that trains the input order of words using positional embeddings.

[0049] According to an embodiment, one or more transformer blocks in the LLM 200 may include a first transformer block 210, a second transformer block 220, and a third transformer block 230. The first transformer block 210 may receive tokens as input, wherein input embedding and position embedding are performed on the tokens. The first transformer block 210, the second transformer block 220, and the third transformer block 230 may be connected in series. For example, a transformer block may receive the output of a preceding transformer block as input.

[0050] According to an embodiment, each of the first converter block 210, the second converter block 220, and the third converter block 230 may include multiple layers. For example, the second converter block 220 may include, but is not limited to, a normalization layer (e.g., LayerNorm), a linear layer (e.g., Linear), a multi-head attention layer (e.g., Multi-head attention), and a feedforward layer (e.g., Feedforward). The first converter block 210 and the third converter block 230 may include the same layers as the second converter block 220. However, this disclosure is not limited thereto; thus, according to another embodiment, the first converter block 210 and the third converter block 230 may include one or more layers different from those of the second converter block 220.

[0051] Normalization layers normalize the output of the previous layer and stabilize training. Linear layers perform linear transformations on the input and / or output of a multi-head attention layer. For example, linear layers can be used to reconstruct the output of a multi-head attention layer and adjust its dimensions. Multi-head attention layers can identify relationships between input words using an attention mechanism. Feedforward layers can perform additional non-linear transformations after the attention mechanism terminates.

[0052] The tokens input to the multi-head attention layer can all be converted into queries, keys, and values. Query, key, and value can all be vectors.

[0053] A query can be information used to evaluate the relationship between the current lexical and other lexicals. For example, a query can determine which lexical should be followed by the current lexical. A key can be a vector in which each lexical represents information about that lexical. For example, the key of the current lexical could be a vector used to calculate the similarity between the current lexical and other lexicals compared to queries about other lexicals. A value can be a vector indicating the contextual information possessed by each lexical. Values ​​can be used to generate a final attention value by applying attention weights.

[0054] According to embodiments, positional embedding (e.g., positional encoding) can be performed when transforming lexical units into queries, keys, and values. For the LLM 200 to understand the context of natural language based on the transformer, it may require inputting positional information (e.g., sequence information) of the lexical units. For example, in the sentences "the cat chases the mouse" and "the mouse chases the cat," the meaning may be completely different when the positions of "cat" and "mouse" change. Therefore, positional embedding reflecting positional information can be performed to reflect the position of the lexical units. The LLM 200 can determine the order of the lexical units through positional embedding. Positional embedding can be very important because attention mechanisms can identify sentence structure and positional patterns and make more accurate predictions.

[0055] According to embodiments, position embedding can have absolute position encoding and relative position encoding. Absolute position encoding can be a method of adding the absolute position of a lexical to the position information. Relative position encoding can be a method of adding the relative distance from each lexical to its surrounding lexical to the position information. For example, relative position encoding can include methods of adding position information by using rotation transformations (e.g., Rotated Position Embedding (RoPE)) and methods of adding position information that is dynamically adjusted based on context (e.g., Context Position Encoding (CoPE)).

[0056] However, the structure of the converter block described above is merely an example, and therefore, this disclosure is not limited thereto. According to another example, the converter block may include, in addition to... Figure 2 One or more layers other than the layer shown, and one or more converter blocks can be defined as up-converter blocks. According to an embodiment, Figure 2 Two or more consecutive converter blocks can be defined as an upconverter block.

[0057] According to the embodiments, such as Figure 2 As shown, the output layer (prediction) in LLM 200 can generate the final prediction result based on the output of the last transformer block.

[0058] The performance of an LLM 200 can be correlated with the number of transformer blocks. As the number of transformer blocks increases, the LLM 200 can perform more in-depth training. For example, with an increased number of transformer blocks, the LLM 200 can train on more complex contextual information and the correlations between distant terms in long sentences. As the number of transformer blocks increases, the size of the LLM 200 may also increase. More hardware resources may be required for electronic devices to execute a larger-scale LLM 200. Increased hardware resources can translate to increased costs.

[0059] The following section will describe the calculation of attention scores in the attention calculation of related technologies.

[0060] Figure 3This is a diagram illustrating attention calculations based on relevant technologies.

[0061] The terms input to the LLM can all be converted into queries, keys, and values. Electronic devices can embed location information into the queries, keys, and values. For ease of description, the following description is based on a query 300 (e.g., a query) as the first term in the input term set and a key 301 (e.g., a key) as the second term in the input term set. Query 300 and key 301 can be in a state prior to the embedding of location information. The first term can refer to the term currently being targeted by the attention operation. However, the following description can also be applied to query 300 and key of the first term.

[0062] In blocks 310 and 320, the electronic device can embed location information. For illustration, the input “I have a meeting today” can be entered into the LLM. The electronic device can embed location information (e.g., “0”) for the query of “I” (the first lexical element) and can embed location information (e.g., “1”) for the key of “have” (the second lexical element).

[0063] Positional information can be information indexed from 0 to the position of the corresponding word in the input. However, this is just an example, and positional information can also indicate the relative distance from each word to its surrounding words.

[0064] The query 311 for the first word element (e.g., query ') may include the position information of the first word element. The key 312 for the second word element (e.g., key ') may include the position information of the second word element.

[0065] In block 330, the electronic device can perform attention operations on the first and second lexical units, determining an attention score based on query 311 and key 312. The attention score can indicate the similarity or relevance between the first and second lexical units. The attention score can be normalized by a normalization process (e.g., the SoftMax function) and multiplied by the value of the second lexical unit to determine the attention value.

[0066] In related technical methods, because LLM uses fixed positional information of lexical units based on the input, it may provide different responses depending on how the sentence is structured. For example, in the cases of "I ate rice today" and "I ate um... rice today," the sentences express the same content, but the attention score may change due to the addition of "um...", and the attention score may also change due to the change in the relative positions of the lexical units. Similarly, in the cases of "I ate rice today" and "Today I ate rice," the sentences express the same content, but the attention score may change because the order of the lexical units is different.

[0067] In other words, in attention operations where information is exchanged between lexical units, this exchange may occur differently depending on the user input. This can lead to distorted attention scores due to positional changes rather than the meaning or importance of the words. For example, in related technical methods, LLM struggles to find the correct answer when the user expects the output to be in the middle of the input. Thus, when using only lexical positional information based on a given input, LLM may not operate efficiently or accurately.

[0068] As a result, because LLM relies too heavily on positional information from the input sentence, it may fail to accurately reflect the meaning of the actual sentence and the various contexts that may arise in the transformed sentence. The following describes methods to enhance positional information to overcome these problems.

[0069] Figure 4 This is a diagram illustrating attention calculation using enhanced location information according to an embodiment.

[0070] refer to Figure 4 The lexical units input into the LLM can all be converted into queries, keys, and values. Electronic devices can embed location information into the queries, keys, and values. For ease of description, the following description is based on a query 400 (e.g., a query) as the first lexical unit in the input lexical units and a key 401 (e.g., a key) as the second lexical unit. Query 400 and key 401 can be in a state prior to the embedding of location information. For example, query 400 and key 401 can be information prior to the embedding of location information. The first lexical unit can refer to the lexical unit currently being targeted by the attention operation. However, the following description can also be applied to query 400 and key of the first lexical unit.

[0071] In block 410, the electronic device can enhance the location information. For example, the electronic device can generate multiple enhanced location information based on the location information.

[0072] For example, an electronic device can generate multiple enhanced location information based on location information according to at least one of a plurality of first functions.

[0073] For example, an electronic device can generate multiple enhanced location information based on at least one of a plurality of first functions by using the location information of lexical units as input. The plurality of first functions can include, but are not limited to, various functions such as logarithmic functions (e.g., f(x) = log(x)), exponential functions (e.g., f(x) = exp(x)), multiplicative functions (e.g., f(x) = 2x), division functions (e.g., f(x) = x / 2), addition functions (e.g., f(x) = x + n, where n is a real number), subtraction functions (e.g., f(x) = x^n, where n is a real number), and trigonometric functions. The plurality of first functions can include combinations of the above functions as well as various functions such as user-predefined functions (e.g., f(x) = 3log(x) + 0.1).

[0074] According to an embodiment, enhanced location information with added location information can be generated using a first function. In an example case where enhanced location information with added location information is generated using the first function, this can indicate that attention relationships are additionally considered when a given lexical unit is at a greater distance. According to an embodiment, enhanced location information with reduced location information can be generated using the first function. In an example case where enhanced location information with reduced location information is generated using the first function, this can indicate that attention relationships are additionally considered when a given lexical unit is at a closer distance.

[0075] In the example case where the position information of the first word is 20, multiple enhanced position information (e.g., "20", "10", and "5") can be generated by at least one of multiple first functions (e.g., f(x)=x, f(x)=x / 2, and f(x)=x / 4). In the example case where the position information of the second word is 16, multiple enhanced position information (e.g., "16", "8", and "4") can be generated by at least one of multiple first functions (e.g., f(x)=x, f(x)=x / 2, and f(x)=x / 4).

[0076] In blocks 420 and 430, the electronic device can embed location information into query 400 and key 401.

[0077] In block 420, the electronic device can generate multiple queries targeting the first term based on multiple pieces of enhanced location information. For example, the electronic device can generate query 411 (e.g., query ') with location information "20", query 412 (e.g., query '') with location information "10", and query 413 (e.g., query ''') with location information "5".

[0078] In block 430, the electronic device can generate multiple keys for the second lexical unit based on multiple pieces of enhanced positional information. For example, the electronic device can generate key 421 (e.g., key') with positional information "16", key 422 (e.g., key'') with positional information "8", and key 423 (e.g., key''') with positional information "4".

[0079] In blocks 440, 450, and 460, the electronic device can determine attention scores. The electronic device can determine multiple initial attention scores based on multiple queries and multiple keys.

[0080] An electronic device can determine a first attention score based on pairs of queries and keys enhanced by the same first function among multiple queries and multiple keys. For example, in block 440, the electronic device can determine a first attention score based on a pair of queries (e.g., query') 411 and key' 421 that include location information enhanced by the same first function (e.g., f(x) = x). For example, in block 450, the electronic device can determine a first attention score based on a pair of queries (e.g., query'') 412 and key'' 422 that include location information enhanced by the same first function (e.g., f(x) = x / 2). For example, in block 460, the electronic device can determine a first attention score based on a pair of queries (e.g., query''') 413 and key''' 423 that include location information enhanced by the same first function (e.g., f(x) = x / 4). The first attention score can be determined based on an operation that determines the similarity between queries and keys. For example, the first attention score can be determined based on an inner product operation between queries and keys.

[0081] In block 470, the electronic device can determine a first target attention score. For example, the electronic device can determine the first target attention score based on multiple first attention scores.

[0082] Electronic devices can determine a first target attention score based on one of a plurality of second functions that determine a first target attention score from a plurality of first attention scores.

[0083] According to an embodiment, the plurality of second functions may be functions that select representative values ​​from a plurality of first attention scores. For example, the plurality of second functions may include a function that determines the median from a plurality of first attention scores. For example, the plurality of second functions may include a function that determines the mode from a plurality of first attention scores. However, this is an example, and the present disclosure is not limited thereto, and various functions for determining representative values ​​may be applied.

[0084] According to an embodiment, the plurality of second functions may be functions that generate representative values ​​based on a plurality of first attention scores. The plurality of second functions may include a function that determines an average value from the plurality of first attention scores. The plurality of second functions may include a function that determines a weighted average value from the plurality of first attention scores. In an example case of determining a weighted average value based on a plurality of first attention scores, the weight of a first attention score generated based on unenhanced location information (e.g., location information enhanced by a first function of the form f(x) = x) may be determined to be higher than the weight of other first attention scores.

[0085] An LLM can include multiple transformer blocks, and these transformer blocks can include multi-head attention layers. A multi-head attention layer can perform multiple attention operations in parallel on a single input. Therefore, the generation of multiple enhanced location information, the generation of multiple queries, the determination of multiple first attention scores, and the determination of a first target attention score can be performed for each head of the multi-head attention layer.

[0086] LLM can perform inference on input based on a first-target attention score. According to embodiments, by enhancing the positional information given when inputting in various ways, LLM can understand the multiple layers of meaning contained in the input sentence and can understand various contextual meanings by reducing over-reliance on user-provided input.

[0087] The following describes methods for minimizing resource consumption by efficiently using preloaded data (e.g., lexical keys and values).

[0088] Figure 5 This is a diagram illustrating an enhanced deletion key-value (KV) cache using location information according to an embodiment.

[0089] Key-value (KV) caches can store the keys and values ​​of previous terms to efficiently perform attention operations in LLMs. Because the KV cache stores the key-value pairs of previous terms, it may not be necessary to recompile the keys and values ​​of previous terms during the attention operation of the current cache, thus improving efficiency. However, as the input sequence increases, the number of key-value pairs of previous terms that need to be stored in the KV cache can increase exponentially. In an example case of an LLM with 3 billion parameters operating at a sequence length of 1024 and a batch size of 128, 180 gigabytes (GB) of memory might be required. Methods for efficiently managing preloaded data (e.g., key-value pairs of previous terms) are described below.

[0090] According to an embodiment, all terms input to the LLM can be converted into queries, keys, and values. The electronic device can embed location information into the queries, keys, and values. For ease of description, the following description is based on a query 500 (e.g., a query) as the first term in the input term set and a key 501 (e.g., a key) as the second term in the input term set. Query 500 and key 501 can be in a state prior to the embedding of location information. For example, query 500 and key 501 can be information prior to the embedding of location information. The first term can refer to the term currently being targeted by the attention operation. However, the following description can also be applied to query 500 and key of the first term.

[0091] In block 510, electronic devices can enhance location information.

[0092] The electronic device can generate multiple pieces of enhanced location information based on location information, using at least one of a plurality of first functions. (Reference) Figure 4 The method for generating multiple enhanced location information is described, therefore its detailed description is omitted.

[0093] In blocks 520 and 530, the electronic device can embed location information into query 500 and key 501.

[0094] In block 520, the electronic device can generate multiple queries targeting the first term based on multiple pieces of enhanced location information. For example, the electronic device can generate query 511 (e.g., query ') with location information "20", query 512 (e.g., query '') with location information "10", and query 513 (e.g., query ''') with location information "5".

[0095] In block 530, the electronic device can generate a key for the second word element based on positional information. The electronic device can generate key 521 (e.g., key') by embedding the positional information of the second word element. Key 521 may include non-enhanced positional information of the second word element (e.g., the positional information of the second word element in the input).

[0096] In blocks 540, 550, and 560, the electronic device can determine multiple second attention scores based on multiple queries of the first lexical unit and the key 521 of the second lexical unit. The electronic device can determine the second attention score between each of the multiple queries and the key 521. For example, in block 540, the electronic device can determine the second attention score between query 511 (e.g., query ') and key 521. For example, in block 550, the electronic device can determine the second attention score between query 512 (e.g., query '') and key 521. For example, in block 560, the electronic device can determine the second attention score between query 513 (e.g., query ''') and key 521.

[0097] In block 570, the electronic device can determine a second target attention score. The electronic device can determine the second target attention score based on multiple second attention scores. This can be referenced above. Figure 4 The method for determining the first target attention score is applied in the same way as the method for determining the second target attention score, therefore its detailed description is omitted.

[0098] The electronic device can determine whether the second target attention score has an attention score less than or equal to a threshold. When the second target attention score has an attention score less than or equal to the threshold, the electronic device can remove the key 501 and lexical value of the second lexical from memory (e.g., a KV cache). The threshold can be set by the user or determined experimentally.

[0099] The fact that the second target attention score has an attention score less than or equal to the threshold indicates that, based on the enhancement of positional information, the second word has low relevance to the first word located at various positions. The fact that the second target attention score has an attention score exceeding the threshold indicates that, based on the enhancement of positional information, the second word has high relevance to the first word located at various positions. By storing only key-value pairs of second words that are highly relevant to the first word located at various positions in the KV cache, the consumption of memory resources can be minimized.

[0100] According to an embodiment, the LLM may include multiple transformer blocks, and these transformer blocks may include a multi-head attention layer. The multi-head attention layer can perform multiple attention operations in parallel on a single input. Therefore, the generation of multiple enhanced location information, the generation of multiple queries, the determination of multiple second attention scores, and the determination of a second target attention score, as described above, can be performed for each head of the multi-head attention layer.

[0101] The following text refers to the implementation of the above reference. Figure 4 and Figure 5 The location information is described using an enhanced LLM.

[0102] Figure 6 This is a diagram illustrating an LLM that enhances position information in some transformer blocks according to an embodiment.

[0103] refer to Figure 6 According to an embodiment, the LLM 600 may include multiple converter blocks. For example, the LLM 600 may include 10 converter blocks, but this disclosure is not limited thereto. Thus, according to another embodiment, the number of converter blocks may be different from 10.

[0104] According to an embodiment, the electronic device can input tokens into an LLM 600, which includes multiple transformer blocks. The LLM 600 can generate outputs corresponding to the tokens by performing reasoning using the tokens as input.

[0105] According to an embodiment, at least one of the plurality of transformer blocks can perform the above reference in a multi-head attention layer. Figure 4 and Figure 5 Enhanced location information description.

[0106] According to an embodiment, at least one of the plurality of transformer blocks can generate multiple pieces of enhanced positional information by enhancing the positional information of the lexical. According to an embodiment, at least one of the plurality of transformer blocks can generate multiple queries and multiple keys for the lexical based on the multiple pieces of enhanced positional information. According to an embodiment, at least one of the plurality of transformer blocks can perform inference corresponding to the lexical based on a target attention score determined according to the multiple queries and multiple keys. (See above reference...) Figure 4 and Figure 5 The operation of at least one of the multiple converter blocks is described, therefore its detailed description is omitted.

[0107] For example, transformer blocks 610, 620, 630 and 640 can generate multiple enhanced position information by enhancing the position information of the lexical, generate multiple queries and multiple keys for the lexical based on the multiple enhanced position information, and perform inference corresponding to the lexical based on the target attention score determined according to the multiple queries and multiple keys.

[0108] According to embodiments, the method for enhancing position information can vary for each transformer block. For example, transformer blocks 610 and 620 can enhance position information using the same first function. For example, the first function can be an exponential function. For example, transformer blocks 610, 630, and 640 can enhance position information using different functions. According to embodiments, transformer block 610 can enhance position information using a first function, transformer block 630 can enhance position information using a second function, and transformer block 640 can enhance position information using a third function. The first, second, and third functions can be different from each other. For example, transformer block 610 can enhance position information using an exponential function, transformer block 630 can enhance position information using a logarithmic function, and transformer block 640 can enhance position information using a multiplicative function. According to embodiments, position embedding can be performed again for each transformer block to enhance position information based on different functions.

[0109] Figure 7 This is a flowchart illustrating an operation method of an electronic device according to an embodiment.

[0110] according to Figure 7 The one or more embodiments shown can be performed sequentially, but this is not required. For example, the order of operations can be changed, and at least two operations can be performed in parallel. Figure 7 The operations shown can be performed by at least one component of the electronic device. For example, the electronic device can perform the following operations when instructions included in the memory are executed individually and / or jointly by at least one processor. The at least one processor may include at least one of a host processor and an accelerator.

[0111] In operation 710, the operation method may include: enhancing the positional information of the first word input into the LLM and generating multiple enhanced positional information. For example, an electronic device may enhance the positional information of the first word input into the LLM and generate multiple enhanced positional information for the first word.

[0112] In operation 720, the operation method may include: generating multiple queries for the first term based on multiple pieces of enhanced location information. For example, an electronic device may generate multiple queries for the first term based on multiple pieces of enhanced location information.

[0113] In operation 730, the operation method may include obtaining multiple first attention scores based on multiple queries. For example, an electronic device may obtain or determine multiple first attention scores based on multiple queries.

[0114] In operation 740, the operation method may include: obtaining a first target attention score based on multiple first attention scores. For example, an electronic device may obtain or determine the first target attention score based on multiple first attention scores.

[0115] In operation 750, the operation method may include: performing inference on the input based on a first target attention score. For example, an electronic device may perform inference on the input based on a first target attention score.

[0116] The above reference Figures 1 to 6 The detailed descriptions of operations 710 to 750 are described, therefore their detailed descriptions are omitted.

[0117] The embodiments described herein can be implemented using hardware components, software components, and / or combinations thereof. For example, components such as “block,” “module,” “unit,” etc., can be implemented using hardware components, software components, and / or combinations thereof. For instance, a processing device or hardware component can be implemented using one or more general-purpose or special-purpose computers, such as processors, controllers, and arithmetic logic units (ALUs), digital signal processors (DSPs), microcomputers, field-programmable gate arrays (FPGAs), programmable logic units (PLUs), microprocessors, or any other device capable of responding to and executing instructions in a defined manner. The processing device can run an operating system (OS) and one or more software applications running on the OS. The processing device can also access, store, manipulate, process, and create data in response to the execution of software. For simplicity, the description of the processing device is used as the singular; however, those skilled in the art will understand that a processing device can include multiple processing elements and various types of processing elements. For example, a processing device can include multiple processors, or a single processor and a single controller. Additionally, different processing configurations are possible, such as parallel processors.

[0118] Software can include computer programs, code segments, instructions, or some combination thereof, to independently or uniformly instruct or configure a processing device to operate as intended. Software and data can be stored in any type of machine, component, physical or virtual device, or computer storage medium, or device capable of providing instructions or data to or being interpreted by the processing device. Software can also be distributed across network-coupled computer systems, enabling the software to be stored and executed in a distributed manner. Software and data can be stored on one or more non-transitory computer-readable recording media.

[0119] The methods according to the above embodiments can be recorded in a non-transitory computer-readable medium including program instructions to implement the various operations of the above embodiments. The medium may also include data files, data structures, etc., alone or in combination with the program instructions. The program instructions recorded on the medium may be program instructions specifically designed and constructed for the purposes of the embodiments, or they may be program instructions well known and usable by those skilled in the art of computer software. Examples of non-transitory computer-readable media include: magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs, DVDs, and / or Blu-ray discs; magneto-optical media such as optical discs; and hardware devices specifically configured to store and execute program instructions, such as read-only memory (ROM), random access memory (RAM), flash memory (e.g., USB flash drives, memory cards, memory sticks, etc.). Examples of program instructions include both machine code generated by a compiler and files containing high-level code that can be executed by a computer using an interpreter.

[0120] The aforementioned hardware device can be configured to function as one or more software modules to perform the operations of the above embodiments, and vice versa.

[0121] As described above, although embodiments have been described with reference to the limited accompanying drawings, those skilled in the art can apply various technical modifications and variations thereon. For example, suitable results may be achieved if the described techniques are performed in a different order and / or if components in the described system, architecture, device, or circuit are combined in different ways and / or replaced or supplemented by other components or their equivalents.

[0122] Therefore, other implementations, other embodiments, and equivalents of the claims are also within the scope of the appended claims.

Claims

1. A method of operating an electronic device, the method comprising: Multiple enhanced positional information pieces are generated for the first word element by enhancing the positional information of the first word element input into the language model LM; Based on the multiple enhanced location information, generate multiple queries targeting the first word element; Multiple first attention scores are determined based on the multiple queries; A first target attention score is determined based on the multiple first attention scores; as well as Inference is performed on the input based on the first target attention score.

2. The operating method according to claim 1, wherein, The multiple enhanced location information is generated based on at least one of a plurality of first functions configured to enhance the location information of the first lexical unit.

3. The operating method according to claim 1 further includes: Multiple keys for the second word are generated by enhancing the positional information of the second word input to the LM. The second lexical unit is configured to perform an attention operation with the first lexical unit.

4. The operating method according to claim 3, wherein, Determining the plurality of first attention scores includes: determining a first attention score among the plurality of first attention scores based on a pair of a first query and a first key among the plurality of queries and keys, wherein the first query and the first key include positional information enhanced by the same first function.

5. The operating method according to claim 1, wherein, Determining the first target attention score includes: determining the first target attention score based on one of a plurality of second functions that determine the first target attention score from the plurality of first attention scores.

6. The operating method according to claim 1 further includes: Based on the keys of the multiple queries and second words, multiple second attention scores are determined; as well as The second target attention score is determined based on the multiple second attention scores.

7. The operating method according to claim 6, wherein, Determining the second target attention score includes: determining the second target attention score based on one of a plurality of third functions that determine the second target attention score from the plurality of second attention scores.

8. The operating method according to claim 6 further includes: The value and key of the second term are removed from memory based on the second target attention score, which is less than or equal to the threshold.

9. The operating method according to claim 1, wherein, The LM includes a multi-head attention layer. Specifically, for each head of the multi-head attention layer, the generation of multiple enhanced location information for the first lexical unit, the generation of multiple queries, the determination of multiple first attention scores, and the determination of the first target attention score are performed.

10. A method of operating an electronic device, the method comprising: Input the lexical units into the language model LM, which includes multiple transformer blocks; as well as The LM generates an output corresponding to the given word by using the word as input for reasoning. Wherein, at least one of the plurality of converter blocks is configured as follows: Multiple enhanced positional information messages are generated by enhancing the positional information of the lexical units. Based on the multiple enhanced location information, generate multiple queries and multiple keys for the term, and Inference corresponding to the lexical is performed based on the target attention score determined from the plurality of queries and the plurality of keys.

11. The operating method according to claim 10, wherein, The plurality of converter blocks include a first converter block and a second converter block, and The first transformer block and the second transformer block are configured to enhance the position information through different functions.

12. An electronic device, comprising: The memory is configured to store instructions; as well as At least one processor is configured to execute the instructions. Wherein, when the instructions are executed individually or jointly by the at least one processor, the electronic device: Multiple enhanced positional information pieces are generated for the first word element by enhancing the positional information of the first word element input into the language model LM; Based on the multiple enhanced location information, generate multiple queries targeting the first word element; Multiple first attention scores are determined based on the multiple queries; A first target attention score is determined based on the plurality of first attention scores; and Inference is performed on the input based on the first target attention score.

13. The electronic device according to claim 12, wherein, When the instructions are executed individually or jointly by the at least one processor, the electronic device: generates the plurality of enhanced location information based on the location information, using at least one of a plurality of first functions configured to enhance the location information of the first lexical.

14. The electronic device according to claim 12, wherein, When the instructions are executed individually or jointly by the at least one processor, the electronic device causes the electronic device to: generate multiple keys for the second lexical by enhancing the positional information of the second lexical input to the LM, and The second lexical unit is configured to perform an attention operation with the first lexical unit.

15. The electronic device according to claim 14, wherein, When the instructions are executed individually or jointly by the at least one processor, the electronic device causes the electronic device to: determine a first attention score among the plurality of first attention scores based on a pair of a first query and a first key among the plurality of queries and keys, wherein the first query and the first key include location information enhanced by the same first function.

16. The electronic device according to claim 12, wherein, When the instructions are executed individually or jointly by the at least one processor, the electronic device: determines the first target attention score based on one of a plurality of second functions that determine the first target attention score from the plurality of first attention scores.

17. The electronic device according to claim 12, wherein, When the instructions are executed individually or jointly by the at least one processor, the electronic device: Multiple second attention scores are determined based on the keys of the multiple queries and second words; as well as The second target attention score is determined based on the multiple second attention scores.

18. The electronic device according to claim 17, wherein, When the instructions are executed individually or jointly by the at least one processor, the electronic device determines the second target attention score based on one of a plurality of third functions that determine the second target attention score from the plurality of second attention scores.

19. The electronic device according to claim 17, wherein, When the instruction is executed by the at least one processor alone or together, the electronic device: removes the value and key of the second word from memory based on the second target attention score where the attention score is less than or equal to a threshold.

20. The electronic device according to claim 12, wherein, The LM includes a multi-head attention layer. When the instructions are executed individually or jointly by the at least one processor, the electronic device causes the electronic device to: generate the multiple enhanced location information for the first lexical unit for each head of the multi-head attention layer, generate the multiple queries, determine the multiple first attention scores, and determine the first target attention score.

Images