Autoregression method and autoregression system

Abstract

An autoregressive method for a large language model includes: receiving a hidden state associated with at least one tag; key data, first numerical data and query data are generated according to the received hidden state; performing position coding on the key data to generate key data of first position coding; performing position coding on the query data to generate position-coded query data; performing a first element-level dot product operation according to the key data of the first position code, the query data of the position code and the key data of the second position code to generate an attention score; performing a second element-level dot product operation according to the first numerical data, the attention score, and second numerical data to generate an attention output; and, adding the attention output and the hidden state to generate an updated hidden state. Correspondingly, the invention further provides an autoregression system, and the calculation amount can be reduced and the efficiency can be improved by adopting the autoregression method and the autoregression system.

Description

[Technical field]

[0001] The present invention generally relates to a language model, and more particularly, to an autoregressive method and an autoregressive system for a transformer-based large language model (LLM). [Background technology]

[0002] Against the backdrop of rapidly evolving technology, Large Language Models (LLMs) have shown great potential in a range of applications. LLMs, a form of artificial intelligence (AI), are able to process and generate human language. By training LLMs on large amounts of text data, they are able to identify patterns and rules of language, enabling them to perform a variety of tasks. For example, LLMs can generate text, translate languages, answer queries, and create various forms of creative text. In particular, the transformer decoder in an LLM requires information from past tokens to predict the next token. A common technique for speeding up transformer decoder inference during inference intervals is to introduce a cache. This avoids the need to recalculate numerical data and key data associated with previously calculated tokens.

[0003] Although cache is introduced to LLM, the cache data is fully read, processed, and written in every iteration cycle of the LLM model. This causes the size of the input and output dimensions of the LLM to increase, resulting in significant latency. In addition, cache output may be written and rewritten in the same memory address space. Therefore, repeatedly rewriting data in the same memory address space may introduce additional software overhead.

[0004] Therefore, designing an autoregressive system for LLM that can reduce the dimensionality of LLM input and output is a significant design challenge. [Summary of the invention]

[0005] In view of this, the present invention provides an autoregressive method and system, which can reduce the operation dimension and thus reduce the amount of calculation.

[0006] In a first aspect, the present invention provides an autoregressive method, which is applicable to a large language model based on a transformer. The autoregressive method includes: receiving a hidden state associated with at least one tag; generating key data, first numerical data and query data according to the received hidden state; position encoding the key data to generate the first position encoded key data; position encoding the query data to generate the position encoded query data; performing a first element-level dot product operation according to the first position encoded key data, the position encoded query data and the second position encoded key data to generate an attention score; performing a second element-level dot product operation according to the first numerical data, the attention score and the second numerical data to generate an attention output; and adding the attention output to the hidden state to generate an updated hidden state; wherein the second position encoded key data is obtained and cached before generating the first position encoded key data, and the second numerical data is obtained and cached before generating the first numerical data.

[0007] In some embodiments, performing a first element-wise dot product operation based on the key data encoded at the first position, the query data encoded at the position, and the key data encoded at the second position to generate an attention score includes: performing a first matrix dot product on the key data encoded at the first position and the query data encoded at the position to generate a first product output; performing a second matrix dot product on the key data encoded at the second position and the query data encoded at the position to generate a second product output; and, concatenating the first product output and the second product output to generate the attention score.

[0008] In some embodiments, performing a second element-wise dot product operation based on the first numerical data, the attention score, and the second numerical data to generate an attention output includes: performing a third matrix dot product based on the attention score and the second numerical data to generate a third product output; performing a fourth matrix dot product based on the attention score and the first numerical data to generate a fourth product output; and, adding the third product output and the fourth product output to generate the attention output.

[0009] In some embodiments, the autoregressive method further includes: performing a Softmax function on the attention score to generate an attention weight, wherein the second element-wise dot product operation is performed based on the first numerical data, the attention weight, and the second numerical data to generate the attention output.

[0010] In some embodiments, the autoregressive method further includes: normalizing the hidden state into a normalized hidden state, wherein the key data, the first numerical data, and the query data are generated based on the normalized hidden state; and normalizing the updated hidden state into the updated normalized hidden state.

[0011] In some embodiments, the autoregressive method also includes: cutting out a first portion of the storage segment from the circular buffer as a sliding window for use in a first numerical cache and / or a first key cache; and cutting out a second portion of the storage segment from the circular buffer for use in a second numerical cache and / or a second key cache; wherein the first numerical cache is configured to cache the first numerical data, the second numerical cache is configured to cache the second numerical data, the first key cache is configured to cache the key data encoded in the first position, and the second key cache is configured to cache the key data encoded in the second position.

[0012] In some embodiments, the autoregressive method further includes: after the first value cache and / or the first key cache caches the corresponding data, updating the second portion of the storage segment of the ring buffer by increasing the offset value of the memory address.

[0013] In some embodiments, the autoregressive method further includes: after the second portion of the storage segments is shifted to touch the end storage segment of the circular buffer, copying the second portion of the storage segments of the circular buffer according to the memory address initialized by the circular buffer.

[0014] In some embodiments, the signal shape format of the first numerical data and the key data encoded at the first position is expressed as (B, N, T, D), and the signal shape format of the second numerical data and the key data encoded at the second position is expressed as (B, N, C, D), wherein B is the batch size, T is the number of tags, N is the number of attention heads, D is the head dimension of each attention head, and C is a predetermined value predefined by the user.

[0015] In some embodiments, the autoregressive method further includes: replacing the null value of the attention score with a mask value to generate an updated attention score, wherein the attention weight is generated based on the updated attention score.

[0016] In a second aspect, the present invention provides an autoregressive system suitable for a Transformer-based Large Language Model (LLM). The autoregressive system includes: a layer input module configured to receive a hidden state associated with at least one tag processed by the LLM; a linear transformation module configured to generate key data, first numerical data and query data based on the received hidden state; a key position encoder configured to position-encode the key data to generate first position-encoded key data; a query position encoder configured to position-encode the query data to generate position-encoded query data; a first multiplication module configured to perform a first element-level dot product operation based on the first position-encoded key data, the position-encoded query data and the second position-encoded key data to generate an attention score; a second multiplication module configured to perform a second element-level dot product operation based on the first numerical data, the attention score and the second numerical data to generate an attention output; a first adder configured to add the attention output and the hidden state to generate an updated hidden state; wherein the second position-encoded key data is obtained and cached before the first position-encoded key data is generated, and the second numerical data is obtained and cached before the first numerical data is generated.

[0017] In some embodiments, the first multiplication module includes: a first batch matrix multiplication module, configured to perform a first matrix dot product on the key data encoded at the first position and the query data encoded at the position to generate a first product output; a second batch matrix multiplication module, configured to perform a second matrix dot product on the key data encoded at the second position and the query data encoded at the position to generate a second product output; and a cascade module, configured to cascade the first product output and the second product output to generate the attention score.

[0018] In some embodiments, the second multiplication module includes: a third batch matrix multiplication module, configured to perform a third matrix dot product based on the attention score and the second numerical data to generate a third product output; a fourth batch matrix multiplication module, configured to perform a fourth matrix dot product based on the attention score and the first numerical data to generate a fourth product output; and a second adder, configured to add the third product output and the fourth product output to generate the attention output.

[0019] In some embodiments, the autoregressive system also includes: a Softmax module configured to perform a Softmax function on the attention score to generate an attention weight, wherein the second element-wise dot product operation is performed based on the first numerical data, the attention weight and the second numerical data to generate the attention output.

[0020] In some embodiments, the autoregressive system also includes: an input layer normalization module, configured to normalize the hidden state into a normalized hidden state, wherein the key data, the first numerical data and the query data are generated based on the normalized hidden state; an output layer normalization module, configured to normalize the updated hidden state into an updated normalized hidden state.

[0021] In some embodiments, the autoregressive system also includes: a circular buffer, configured to cache the first numerical data and the second numerical data, and / or, to cache the key data encoded in the first position and the key data encoded in the second position; wherein, the first portion of the storage segments of the circular buffer is divided to generate a first numerical cache and / or a first key cache, the second portion of the storage segments of the circular buffer is divided to generate a second numerical cache and / or a second key cache, the first numerical cache is configured to cache the first numerical data, the second numerical cache is configured to cache the second numerical data, the first key cache is configured to cache the key data encoded in the first position, and the second key cache is configured to cache the key data encoded in the second position.

[0022] In some embodiments, after the first value cache and / or the first key cache caches the corresponding data, the second portion of the storage segment of the ring buffer is updated by increasing the offset value of the memory address.

[0023] In some embodiments, after the second portion of the storage segments is shifted to touch the end storage segment of the ring buffer, the second portion of the storage segments of the ring buffer are copied according to the memory address initialized by the ring buffer.

[0024] In some embodiments, the signal shape format of the first numerical data and the key data encoded at the first position is expressed as (B, N, T, D), and the signal shape format of the second numerical data and the key data encoded at the second position is expressed as (B, N, C, D), wherein B is the batch size, T is the number of tags, N is the number of attention heads, D is the head dimension of each attention head, and C is a predetermined value predefined by the user.

[0025] In some embodiments, the autoregressive system further includes: a mask value generator and a third adder configured to replace the null value of the attention score with the mask value to generate an updated attention score, wherein the attention weight is generated based on the updated attention score.

[0026] Those skilled in the art will no doubt understand these and other objects of the present invention after reading the following detailed description of the preferred embodiments shown in the accompanying drawings. The detailed description will be given in the following embodiments with reference to the accompanying drawings.

Brief Description of the Drawings

[0027] The present invention may be more fully understood by reading the following detailed description and by referring to the examples given in the accompanying drawings.

[0028] Figure 1 is a block diagram of a portion of the operation of one of the transformer layers of an autoregressive system (i.e., a self-attention mechanism within the transformer layer) according to an embodiment of the present invention.

[0029] Figure 2 yes Figure 1 Structure of the first multiplication module in the autoregressive system.

[0030] Figure 3 yes Figure 1 Structure of the second multiplication module in the autoregressive system.

[0031] Figure 4 yes Figure 1 Schematic diagram of the first state of the ring buffer of the autoregressive system.

[0032] Figure 5 yes Figure 1 Schematic diagram of the second state of the ring buffer of the autoregressive system.

[0033] Figure 6 yes Figure 1 Schematic diagram of the third state of the ring buffer of the autoregressive system.

[0034] Figure 7 yes Figure 1 Schematic diagram of the fourth state of the ring buffer of the autoregressive system.

[0035] Figure 8 is through Figure 1 Flowchart of the autoregressive method for implementing large transformer-based language models.

[0036] In the following detailed description, for the purpose of illustration, many specific details are set forth so that those skilled in the art can more thoroughly understand the embodiments of the present invention. However, it is apparent that one or more embodiments may be implemented without these specific details, and different embodiments may be combined as needed, and should not be limited to the embodiments listed in the accompanying drawings. [Specific implementation method]

[0037] The following description is a preferred embodiment of the present invention, which is only used to illustrate the technical features of the present invention, and is not used to limit the scope of the present invention. Certain words are used throughout the specification and claims to refer to specific components. It should be understood by those skilled in the art that manufacturers may use different names to refer to the same components. Therefore, this specification and claims do not use the difference in name as a way to distinguish components, but use the difference in function of the components as a basis for distinction. The terms "component", "system" and "device" used in the present invention can be entities related to computers, wherein the computer can be hardware, software, or a combination of hardware and software. The terms "including" and "comprising" mentioned in the following description and claims are open-ended terms, so they should be interpreted as "including, but not limited to...".

[0038] In autoregressive systems like Generative Pre-trained Transformer (GPT) and other Transformer-based architectures, at least one token in the input sequence may first be converted into a hidden state associated with the at least one token, and it is understood that the hidden state includes basic information about the at least one token. The hidden state is then processed through multiple transformer layers of the system. In the architecture of the disclosed system, each transformer layer includes an attention or self-attention mechanism for updating the hidden state (which is associated with the input token). This multi-layer processing ensures that the final output is based on a comprehensive and detailed understanding of the entire input sequence, thereby obtaining more accurate and contextually relevant results.

[0039] A portion of the operations of each transformer layer (which is related to the attention mechanism) can be shown as Figure 1 , to update the hidden state. Figure 11 is a schematic diagram of a portion of the operation of a transformer layer in an autoregressive system 100 according to an embodiment of the present invention, which can be used, for example, for a transformer-based large language model (LLM). LLM can be an artificial intelligence (AI) capable of processing and generating human language. For example, a neural network-based LLM can implement an attention mechanism or a self-attention mechanism. The autoregressive system 100 includes a layer input module 10, an input layer normalization module 11, a linear transformer module 111, a first value cache 12, a key positional encoder 13, a first key cache 14, a query positional encoder 15, a first multiplication module 16, a second multiplication module 17, a first adder 18, an output layer normalization module 19, a second key cache 14', a second value cache 12' and a Softmax module 171. It should be noted that Figure 1 Some modules in are optional, but for ease of explanation and understanding, Figure 1 All possible modules associated are shown, but the present invention is not limited to Figure 1The specific structure shown. The layer input module 10 is configured to receive a hidden state P10, which is a representation associated with at least one tag processed by the LLM. It is worth noting that the term "hidden state" generally refers to the initial hidden state or the hidden state of the intermediate layer output during the sequence processing. The hidden state can take various forms, such as vectors, matrices, etc. For ease of explanation and understanding, this embodiment is illustrated by an example in which the hidden state is in matrix form, but the present invention is not limited to this example. In an embodiment of the present invention, the hidden state received by the input layer module 10 can be described as an input feature matrix, wherein the input feature matrix is ​​a numerical matrix for representing the features of at least one tag. However, it should be noted that the present invention is not limited to this configuration. In this embodiment, at least one tag of the LLM may include words, symbols or phrases segmented from a text file by the LLM, etc. The signal format of the hidden state P10 may be a floating point number. The input layer normalization (or root mean square normalization) module 11 is configured to normalize the hidden state P10 to generate a normalized hidden state. After normalization, the normalized hidden state can be processed by the linear transformation module 111 to perform linear transformation on the same normalized hidden state according to different weight matrices to generate key data (key data, K), first numerical data (value data, V) and query data (query data, Q), wherein the key data (K), the first numerical data (V) and the query data (Q) can also be in matrix form. In another embodiment, the input layer normalization module 11 can be optional. For example, when the hidden state P10 inherently satisfies the power normalization constraint that its sum of squares is equal to a fixed value, the input layer normalization (or root mean square normalization) module 11 can be omitted. The key data (K), the first numerical data (V) and the query data (Q) are generated according to the hidden state P10 (for example, the hidden state P10 or the normalized hidden state derived therefrom). It should be noted that the derivation of the key data (K), the first numerical data (V) and the query data (Q) is well known to those of ordinary skill in the art. For example, they are generated by the fully connected layer of the neural network. The present invention focuses on the improvement of the multiplication operation involving the key data K and the query data Q and the multiplication operation involving the attention weight and the first numerical data V. Therefore, there is no restriction on the generation of the key data (K), the first numerical data (V) and the query data (Q). In particular, Figure 1In the example of , the first numerical data V, key data K and query data Q are feature representations generated according to the weight matrix and the normalized hidden state. They can be different representations derived from the normalized hidden state by performing a linear transformation for calculating the attention score and the weighted sum. For ease of illustration and understanding, the hidden state P10 can be described as having a shape of (B, T, H). B is the batch size. T is the number of tokens. For example, it can represent the number of at least one token associated with the hidden state P10. H is the hidden size. In addition, the first numerical data V, key data K and query data Q can be described as having a shape of (B, N, T, D). N is the number of attention heads. D is the head dimension for each attention head. Hidden size (H) = number of attention heads (N) × head dimension (D) for each attention head. It can be understood that the meanings of these parameters B, T, H, N, and D in the converter-based architecture are understandable to those of ordinary skill in the art.

[0040] The first value cache 12 is configured to cache the first value data V. Figures 4 to 7 In another embodiment shown, the first numerical cache 12 may be a memory segment that is part of a ring buffer. The ring buffer may be configured to further cache second numerical data that has been obtained and cached before the first numerical data V (also referred to as a "numerical state") is generated. To avoid ambiguity, the memory segment used to cache the second numerical data is referred to as the second numerical cache 12' hereinafter. The first numerical cache 12 is used to cache the currently calculated numerical state (i.e., the first numerical data V), which is one of the outputs of the autoregressive inference channel that executes the transformer architecture (in Figures 4 to 7, which are collectively referred to as "current cache outputs" in the text). These numerical states are cached to prevent redundant recalculation of the same numerical states in the autoregressive inference channel after the transformer layer. The numerical states cached in the past are reintroduced into the transformer through the second numerical cache 12'. The cache mechanism for storing and retrieving past numerical caches can be a ring buffer mechanism. The key position encoder 13 is configured to position encode the key data K (i.e., apply position encoding to the key data K) to generate first position encoded key data P13. The first key cache 14 is configured to cache the first position encoded key data P13. The query position encoder 15 is configured to position encode the query data Q (i.e., apply position encoding to the query data Q) to generate position encoded query data P15. In an embodiment of the present invention, the first position encoded key data P13 and the position encoded query data P15 can be described as having a shape of (B, N, T, D). The first multiplication module 16 is configured to perform element-wise dot product operations based on the first position-encoded key data P13, the position-encoded query data P15, and the second position-encoded key data cached in the second key cache 14' to generate an attention score P16 having a (B, N, T, C+T) shape. C is a predetermined value predefined by the user. For example, C is used to determine the (maximum) size of the cached past values / key information. Typically, C is much larger than T. Similarly, the first key cache 14 can be part of a circular buffer. The circular buffer can be configured to further cache the second position-encoded key data, which is obtained and cached before the first position-encoded key data P13 (also referred to as the "key state") is generated. To avoid ambiguity, the storage segment for caching the second position-encoded key data is referred to as the second key cache 14' hereafter. The first key cache 14 is used to cache the currently calculated key state (for example, the position-encoded key data P13, in Figures 4 to 7 The key states are collectively referred to as "current cache outputs" in the diagram, and are one of the outputs of the autoregressive inference channel that executes the transformer architecture. These key states are cached to prevent redundant recalculation of the same key states in the autoregressive inference channel after the transformer layer. The past cached key states are reintroduced into the transformer via the second key cache 14'. The cache mechanism for storing and retrieving the past key cache can be a ring buffer mechanism.

[0041] The second multiplication module 17 is configured to perform an element-wise dot product operation based on the attention score P16 (e.g., the attention score P16 or a parameter derived from the attention score P16, such as the updated attention score P20 or the attention weight P17a described later), the first numerical data V, and the second numerical data cached in the second numerical cache 12' to generate an attention output (attention output) P17, for example, the attention output P17 has a shape (B, T, H). Figure 1 In the structure shown, the Softmax module 171 can convert the attention score P16 into the attention weight P17a based on the Softmax function, wherein the attention weight is a representation of the probability distribution (e.g., the attention score P16 is used as the input of the Softmax function of the Softmax module 171 to generate the attention weight P17a). It can be understood that in another embodiment, the Softmax module 171 can be omitted. For example, when the attention score P16 representing the element-level probability is expressed as follows:

[0042]

[0043] Here, x i Represents an element of the attention score P16. R is the number of elements of the attention score P16.

[0044] The first adder 18 is configured to add the hidden state P10 and the attention output P17 to generate a weighted sum data P18 (which can be regarded as an updated hidden state), for example, having a shape (B, T, H). The output layer normalization (or root mean square normalization) module 19 is configured to normalize the weighted sum data P18 to generate an updated normalized hidden state P19. In another embodiment, the output layer normalization (or root mean square normalization) module 11 can be omitted. For example, when the weighted sum data P18 inherently satisfies the power normalization constraint, its square sum is equal to a fixed number (e.g., 1). As described above, the LLM can be composed of multiple transformer layers. Therefore, the autoregressive system 100 can be applied to a multi-stage cascade processing flow. For example, the updated hidden state or normalized hidden state P19 of the current stage can be processed and input to the layer input module 10 of the subsequent stage for further similar processing.

[0045] In the autoregressive system 100, the size of the first value cache 12 and the size of the first key cache 14 are determined according to the parameter T, which is the number of tags associated with the hidden state P10, which is much smaller than the parameter C. For example, the signal shape format of the first value data V cached in the first value cache 12 is represented as (B, N, T, D), and the signal shape format of the first position encoded key data P13 cached in the first key cache 14 is represented as (B, N, T, D). For a fixed batch size B, a fixed number of attention heads N, and a fixed head dimension D, the capacity requirement of the first key cache 12 and the capacity requirement of the first value cache 14 only depend on the number of tags T associated with the current state. In the autoregressive system 100, without introducing past key information, the current key information (e.g., key data, key data encoded at the first position) will not be directly cascaded with the past key information. Similarly, without introducing past value information, the current value information (e.g., first value data) will not be directly cascaded with the past value information. In traditional autoregressive systems, directly concatenating the current key information and the past key information will significantly increase the dimension, for example, (B, N, C + T, D), which will greatly increase the amount of computation. Directly concatenating the current value information and the past value information will significantly increase the dimension, for example, (B, N, C + T, D), which will result in a large number of storage write operations and a huge amount of computation, which is not conducive to the inference speed of LLM because LLM is inherently storage-limited due to its huge parameter size (usually billions of parameters). By using Figure 1In the architecture shown, in the autoregressive system 100, the cache size of the key data P13 encoded at the first position for output caching in the first key cache 14 and the first numerical data V cached in the first numerical cache 12 can be reduced, thereby speeding up the model inference speed. In addition, since the cache size of the key data P13 encoded at the first position for output caching in the first key cache 14 and the first numerical data V cached in the first numerical cache 12 are reduced, the temporary model storage occupancy can also be reduced. In addition, a mask value generator 20 can be introduced into the autoregressive system 100. The mask value generator 20 is configured to replace all null values ​​of the attention score P16 with the mask value generated by the mask value generator 20 to generate an updated attention score P20. For example, the mask value can be set to "-∞" to replace the null value "0" to update the attention score P16. In the autoregressive system 100, the size of the updated normalized hidden state P19 output from the output layer normalization (or root mean square normalization) module 19 depends on the number of tags T of the layer input module 10. For example, the signal shape format of the hidden state P10 is expressed as (B, T, H). The signal shape format of the updated normalized hidden state P19 is expressed as (B, T, H). B is the batch size. T is the number of tags. H is the hidden size. It should be understood that the hidden size (H) can be written as the number of attention heads (N) × the head dimension (D) of each attention head. Therefore, for a fixed batch size B and a fixed head size H, since the size of the updated normalized hidden state P19 output from the output layer normalization module 19 depends on the number of tags T, the autoregressive system 100 is suitable for application in a multi-stage cascade preprocessing flow. Below, the details of the paired (two) batch matrix multiplication modules applied to the first multiplication module 16 and the second multiplication module 17 will be described. It should be noted that Figure 1 The input layer normalization (or RMS normalization) module 11, the mask value generator 20, the adder 21, the Softmax module 171 and the output layer normalization (or RMS normalization) module 19 in are optional. In other words, at least one of these modules can be selected from Figure 1 Therefore, according to the description of the present disclosure, a person skilled in the art will be able to know the changes of the other variant embodiments. Figure 1 Similar autoregressive methods are obtained in its variant embodiments, and therefore, the present disclosure does not further describe similar descriptions in detail.

[0046] Figure 2It is the structure of the first multiplication module 16 of the autoregressive system 100. The first multiplication module 16 includes a first batch matrix multiplication module BMM1, a second batch matrix multiplication module BMM2 and a concatenation module 16b. The first batch matrix multiplication module BMM1 is configured to perform a first matrix dot product on the first position-coded key data P13 and the position-coded query data P15 to generate a first product output PBMM1, for example, having a shape (B, N, T, T). Here, the first position-coded key data P13 and the position-coded query data P15 may be in matrix form. The first product output PBMM1 may be a matrix generated by performing a matrix element-level product on the first position-coded key data P13 and the position-coded query data P15. The second key cache 14' is configured to cache the second position-coded key data P16a having a shape of (B, N, C, D). As described above, the second key cache 14' and the first key cache 14 may share a common cache space. The second batch matrix multiplication module BMM2 is configured to perform a second matrix dot product on the second position-coded key data P16a and the position-coded query data P15 to generate a second product output PBMM2 having a shape (B, N, T, C). Here, the second position-coded key data P16a and the position-coded query data P15 can be in matrix form. The second product output PBMM2 can be a matrix generated by performing a matrix element-level product on the second position-coded key data P16a and the position-coded query data P15. The cascade module 16b is configured to cascade the first product output PBMM1 and the second product output PBMM2 to generate an attention score P16 having a shape of (B, N, T, C+T). Here, the first product output PBMM1 and the second product output PBMM2 are in matrix form. Therefore, the dimension of the product output matrix can be expanded by cascading the first product output PBMM1 and the second product output PBMM2, while reducing the amount of calculation.

[0047] Figure 3 1 is the structure of the second multiplication module 17 of the autoregressive system 100. The second multiplication module 17 includes a third batch matrix multiplication module BMM3, a fourth batch matrix multiplication module BMM4 and a second adder 17c. Figure 1 As described above, the second multiplication module 17 can be connected to the Softmax module 171 to process the attention weight P17a output from the Softmax module 171. According to the definition, the Softmax function executed by the Softmax module 171 can convert M real values ​​into M normalized real values, where the sum of the M normalized real values ​​is equal to one. Figure 3, the second numerical cache 12' is configured to cache the second numerical data P17b. As described above, the second numerical cache 12' and the first numerical cache 12 can share a common cache space. The third batch matrix multiplication module BMM3 is configured to perform a third matrix dot product according to the attention weight P17a and the second numerical data P17b to generate a third product output PBMM3 having a (B, N, T, D) shape. Here, the attention weight P17a and the second numerical data P17b can be in matrix form. The third product output PBMM3 can be a matrix generated by performing a matrix element-level product of the attention weight P17a and the second numerical data P17b. The fourth batch matrix multiplication module BMM4 numerical cache is configured to perform a fourth matrix dot product according to the attention weight P17a and the first numerical data V to generate a fourth product output PBMM4 having a (B, N, T, D) shape. Here, the attention weight P17a and the first numerical data V can be in matrix form. The fourth product output PBMM4 may be a matrix generated by performing a matrix element-wise product of the attention weight P17a and the first numerical data V. The second adder 17c is configured to add the third product output PBMM3 and the fourth product output PBMM4. Finally, the second adder 17c outputs the attention output P17 having a (B, T, H) shape. Figure 2 and Figure 3 In the embodiment, it can be understood that the currently cached data (e.g., the first numerical data V, the position-encoded query data P15, and the first position-encoded key data P13) and the previously cached data (e.g., the second numerical data P17b and the second position-encoded key data P16a) are introduced into the autoregressive system 100 to improve efficiency. In order to further optimize the utilization of storage capacity, a ring buffer mechanism can be introduced to implement each cache of the autoregressive system 100, as shown in the following figure. For example, the first numerical cache 12 and the second numerical cache 12' are implemented by the first ring buffer, and the first key cache 14 and the second key cache 14' are implemented by the second ring buffer. For another example, the first numerical cache 12 and the second numerical cache 12', the first key cache 14 and the second key cache 14' are implemented by different parts of the same ring buffer, etc., and the present invention does not limit this. For the sake of convenience, the following example is given by taking the ring buffer to implement the first numerical cache 12 and the second numerical cache 12' as an example.

[0048] Figure 4 is a schematic diagram of a first state of the ring buffer RC of the autoregressive system 100 . Figure 5is a schematic diagram of the second state of the ring buffer RC of the autoregressive system 100. In the present embodiment, the ring buffer RC may be a virtual storage structure established by a software program. In another embodiment, the ring buffer RC may be a physical storage circuit of hardware. Any technical modifications belong to the scope of the present invention. The ring buffer RC includes a plurality of memory segments MS. The second value cache 12' and / or the second key cache 14' may be established by cutting out a portion of the memory segments from the ring buffer RC. For example, for ease of explanation and understanding, the embodiment of the present invention takes the ring buffer RC including 30 memory segments MS as an example, takes 20 memory segments MS in the ring buffer RC for the second value cache 12' as an example, and takes 1 memory segment in the ring buffer RC for the first value cache 12 as an example for illustration, but the present invention is not limited to this example. Therefore, the second value cache 12' (and / or the second key cache 14') may be established by virtually cutting out 20 memory segments from the ring buffer RC. The second value cache 12' (and / or the second key cache 14') is used to cache "previous" cache data (such as the second value data). In addition, the first value cache 12 (and / or the first key cache 14) can be established by cutting out at least one storage segment (one is used as an example in the figure) from the ring buffer RC, and the at least one storage segment is located after (or adjacent to) the 20 storage segments allocated as the second value cache 12' (and / or the second key cache 14'). To avoid ambiguity, the second value cache 12' (and / or the second key cache 14') will be referred to as cache 30 hereinafter. Figure 4 In the embodiment, a portion of the plurality of storage segments (30 storage segments) (20 storage segments) are allocated for establishing a cache 30 (e.g., a second numerical cache 12' and / or a second key cache 14'). In addition, another portion of the plurality of storage segments (10 storage segments) are allocated as a sliding window of length K (e.g., K=10, which indicates that the sliding window includes 10 storage segments). Here, K is a positive integer. K is a predetermined integer related to the overhead configuration for accessing data of the ring buffer RC. The sliding window of length K can be used to implement the first numerical cache 12 (and / or the first key cache 14) to cache "current" cache data (e.g., cache the first numerical data). For example, K is a user-defined integer. When K increases, the overhead of the ring buffer RC increases. When K decreases, the overhead of the ring buffer RC decreases.

[0049] As mentioned above, in the present embodiment, the second value cache 12' (and / or the second key cache 14') can be established by cutting out 20 storage segments from the ring buffer RC. The first value cache 12 and / or the first key cache 14 can be established by cutting out K storage segments from the ring buffer RC (for example, the first value cache 12 is at least one of the K storage segments), which is located after or adjacent to the 20 storage segments allocated as the second value cache 12' (and / or the second key cache 14'). In other words, the 20 slice storage segments of the ring buffer RC are used to cache the second value data P17b (or the key data P16a encoded in the second position). A storage segment in the sliding window of the ring buffer RC is used to cache the "current" first value data V (or the key data P13 encoded in the first position), hereinafter referred to as D1, D2, DK and D(K+1).

[0050] exist Figure 4 In , after the autoregressive system generates the current cache output D1 (such as the first numerical data K described above), the current cache output D1 is stored in an empty storage segment of the sliding window according to the kth memory address. For example, in the first iteration, after the current cache output D1 is generated, the current cache output D1 can be stored in an empty storage segment according to memory address #1. Memory address #1 corresponds to the first storage segment of the sliding window. Then, after the current cache output D1 is stored in the empty storage segment of the sliding window according to the kth memory address, the storage segment portion (20 storage segments) of the ring buffer RC used to establish the cache 30 can be moved by increasing the offset value to update the cache 30 and the second numerical data. Figure 5 , the 20 memory segments MS of the cache 30 can be moved by increasing the offset value of a memory segment. Therefore, the memory segment corresponding to the memory address #1 storing the current cache output D1 belongs to the cache 30, and thus, the second numerical data is updated. Similarly, when the autoregressive system generates the current cache output D2, the current cache output D2 is stored in an empty memory segment of the sliding window according to the (k+1)th memory address. For example, in the second iteration, after the current cache output D2 is generated, the current cache output D2 can be stored in an empty memory segment according to the memory address #2 (for example, which corresponds to the second memory segment in the sliding window).

[0051] Figure 6 is a schematic diagram of a third state of the ring buffer RC of the autoregressive system 100 . Figure 7 is a schematic diagram of the fourth state of the ring buffer RC of the autoregressive system 100. Figure 6 In , when the memory segments of the cache 30 are shifted to the end memory segments of the sliding window, the memory segments of the cache 30 are copied according to their initialization memory addresses. Figure 6 , the sliding window includes K storage segments corresponding to memory address #1 to memory address #K. After the current cache output DK is generated, the current cache output DK can be stored in an empty storage segment (e.g., the Kth storage segment in the sliding window) according to the memory address #K. Then, the storage segment MS of the cache 30 can be shifted accordingly. Specifically, when the storage segment of the cache 30 is shifted to the storage segment with memory address #K, it means that there are no additional storage segments in the sliding window available for caching the "next" cache output. Therefore, in Figure 7 In the example, the memory segments of the cache 30 are copied to the first 20 memory segments of the ring buffer RC according to the initialized memory address #Ini1 to the initialized memory address #Ini20. After the memory segments of the cache 30 are copied, when the current cache output D(K+1) is generated, the current cache output D(K+1) can be stored in a memory segment of the sliding window (for example, the first memory segment in the sliding window) according to the memory address #1. Therefore, due to the introduction of the ring buffer mechanism, the need for reallocation of the cache output address space can be reduced, thereby optimizing the utilization of the storage capacity.

[0052] Figures 4 to 7 The process of updating the key data and / or the second numerical data encoded in the second position cached in the ring buffer RC is shown. As previously described, the storage segment portion of the ring buffer RC corresponding to the initialized memory address #Ini1 to the initialized memory address #Ini20 can be updated by increasing the offset value of the memory address after the first numerical cache (and / or the first key cache) caches the current cache output (first numerical data). After the storage segment portion of the ring buffer is updated (for example, from the initialized memory address #Ini2 to the memory address #1), the storage segment portion may include the first numerical cache and / or the first key cache, as well as a portion of the key data encoded in the second position and / or a portion of the second numerical data. In other words, the second numerical data and / or the key data encoded in the second position can be updated after the storage segment portion is shifted.

[0053] In the autoregressive system 100, any hardware or technical modifications are within the scope of the present invention. For example, the first numerical cache 12 and the first key cache 14 may be pre-allocated in the memory of an accelerated processing unit (APU). The first numerical cache 12 and the first key cache 14 may be pre-allocated in the memory of a central processing unit (CPU). The first numerical cache 12 and the first key cache 14 may be pre-allocated in the memory of a neural processing unit (NPU). In another embodiment, the ring buffer RC may be pre-allocated in the APU storage or the memory of the CPU. The cache 30 may cache data in an array format. Alternatively, the cache 30 may cache data in a tuple format. Alternatively, the cache 30 may cache data in a tensor format.

[0054] Figure 8 1 is a flow chart of an autoregressive method for performing a transformer-based large language model (LLM) by an autoregressive system 100. The autoregressive method for transformer-based LLM includes steps S801 to S807. Any hardware or technical modifications are within the scope of the present invention. Steps S801 to S807 are as follows.

[0055] Step S801: Receive a hidden state associated with at least one tag.

[0056] Step S802: Generate key data, first numerical data and query data according to the received hidden state.

[0057] Step S803: Position-encode the key data to generate first position-encoded key data.

[0058] Step S804: Position-encode the query data to generate position-encoded query data.

[0059] Step S805: Perform a first element dot product operation based on the key data encoded in the first position, the query data encoded in the position, and the key data encoded in the second position to generate an attention score.

[0060] Step S806: Perform a second element-wise dot product operation based on the first numerical data, the attention score, and the second numerical data to generate an attention output.

[0061] Step S807: Add the attention output to the hidden state to generate an updated hidden state.

[0062] The detailed process of step S801 to step S807 has been described before. Therefore, it is omitted here. In the autoregressive system 100, the current key information is not directly cascaded with the past key information, and similarly, the current numerical information is not directly cascaded with the past numerical information. Therefore, the cache size for outputting the key data of the first position encoding cached in the first key cache and the first numerical data cached in the first numerical cache can be reduced. Thus, the model reasoning speed can be accelerated. Temporary model storage occupancy can also be reduced.

[0063] In summary, the present invention discloses an autoregressive system for LLM. The autoregressive system introduces a cache so that the numerical value and key information can be directly accessed after the current key / value projection calculation is completed. In addition, the cache size for outputting the key data of the first / second position encoding and the first / second numerical value data can be reduced, thereby accelerating model reasoning. Since the cache size can be reduced, the temporary model storage occupancy can also be reduced. Since the cache size can be reduced, the number of storage write accesses can also be reduced. In addition, the autoregressive system introduces a ring buffer mechanism. The implementation of the ring buffer mechanism reduces the need to reallocate the cache output address space, thereby optimizing the utilization of storage capacity.

[0064] Although the present invention has been described by way of example and in terms of preferred embodiments, it should be understood that the present invention is not limited to the disclosed embodiments. Instead, it is intended to cover various modifications and similar structures (as will be apparent to those skilled in the art), for example, combinations or replacements of different features in different embodiments. Therefore, the scope of the appended claims should be given the broadest interpretation to cover all such modifications and similar structures.

Claims

1. An autoregressive method, applicable to large transformer-based language models, comprising: receiving a hidden state associated with at least one tag; generating key data, first numerical data, and query data according to the received hidden state; Position-encoding the key data to generate first position-encoded key data; Position-encoding the query data to generate position-encoded query data; Performing a first element-wise dot product operation based on the key data encoded at the first position, the query data encoded at the position, and the key data encoded at the second position to generate an attention score; performing a second element-wise dot product operation based on the first numerical data, the attention score, and the second numerical data to generate an attention output; as well as Add the attention output to the hidden state to generate an updated hidden state; The key data of the second position code is obtained and cached before the key data of the first position code is generated, and the second numerical data is obtained and cached before the first numerical data is generated.

2. The method of claim 1, wherein: Performing a first element-wise dot product operation according to the key data of the first position encoding, the query data of the position encoding, and the key data of the second position encoding to generate an attention score includes: performing a first matrix dot product on the key data of the first position encoding and the query data of the position encoding to generate a first product output; performing a second matrix dot product on the second position-encoded key data and the position-encoded query data to generate a second product output; and The first product output and the second product output are concatenated to generate the attention score.

3. The method of claim 2, wherein: Performing a second element-wise dot product operation according to the first numerical data, the attention score, and the second numerical data to generate an attention output includes: performing a third matrix dot product based on the attention score and the second numerical data to generate a third product output; performing a fourth matrix dot product based on the attention score and the first numerical data to generate a fourth product output; and The third product output and the fourth product output are added to generate the attention output.

4. The method of claim 1, wherein: The autoregressive method also includes: A Softmax function is performed on the attention score to generate an attention weight, wherein the second element-wise dot product operation is performed based on the first numerical data, the attention weight, and the second numerical data to generate the attention output.

5. The method of claim 1, wherein: The autoregressive method also includes: normalizing the hidden state into a normalized hidden state, wherein the key data, the first numerical data, and the query data are generated according to the normalized hidden state; and Normalize the updated hidden state to the updated normalized hidden state.

6. The method of claim 1, wherein: The autoregressive method also includes: Cut out a first portion of storage segments from the ring buffer as a sliding window for use in a first value cache and / or a first key cache; and Cut out a second portion of storage segments from the ring buffer for use in a second value cache and / or a second key cache; Wherein, the first numerical cache is configured to cache the first numerical data, the second numerical cache is configured to cache the second numerical data, the first key cache is configured to cache the key data encoded in the first position, and the second key cache is configured to cache the key data encoded in the second position.

7. The method of claim 6, wherein: The autoregressive method also includes: After the first value cache and / or the first key cache caches the corresponding data, the second portion of the storage segment of the ring buffer is updated by increasing the offset value of the memory address.

8. The method of claim 7, wherein: The autoregressive method also includes: After the second portion of the storage segments is shifted to touch the end storage segment of the ring buffer, the second portion of the storage segments of the ring buffer are copied according to the memory address initialized by the ring buffer.

9. The method of claim 1, wherein: The signal shape format of the first numerical data and the key data encoded at the first position is expressed as (B, N, T, D), and the signal shape format of the second numerical data and the key data encoded at the second position is expressed as (B, N, C, D), wherein B is the batch size, T is the number of tags, N is the number of attention heads, D is the head dimension of each attention head, and C is a predetermined value predefined by the user.

10. The method of claim 4, wherein: The autoregressive method also includes: The null value of the attention score is replaced with the mask value to generate an updated attention score, wherein the attention weight is generated based on the updated attention score.

11. An autoregressive system adapted for a transformer-based large language model (LLM), the autoregressive system comprising: a layer input module configured to receive a hidden state associated with at least one token processed by the LLM; a linear transformation module configured to generate key data, first numerical data, and query data according to the received hidden state; a key position encoder configured to position encode the key data to generate first position-encoded key data; a query position encoder configured to position encode the query data to generate position-encoded query data; a first multiplication module configured to perform a first element-wise dot product operation based on the key data encoded at the first position, the query data encoded at the position, and the key data encoded at the second position to generate an attention score; a second multiplication module configured to perform a second element-wise dot product operation based on the first numerical data, the attention score, and the second numerical data to generate an attention output; a first adder configured to add the attention output and the hidden state to generate an updated hidden state; The key data of the second position code is obtained and cached before the key data of the first position code is generated, and the second numerical data is obtained and cached before the first numerical data is generated.

12. The autoregressive system of claim 11, wherein: The first multiplication module comprises: a first batch matrix multiplication module configured to perform a first matrix dot product on the key data of the first position encoding and the query data of the position encoding to generate a first product output; a second batch matrix multiplication module configured to perform a second matrix dot product on the second position-encoded key data and the position-encoded query data to generate a second product output; A cascade module is configured to cascade the first product output and the second product output to generate the attention score.

13. The autoregressive system of claim 11, wherein: The second multiplication module comprises: a third batch matrix multiplication module configured to perform a third matrix dot product based on the attention score and the second numerical data to generate a third product output; a fourth batch matrix multiplication module configured to perform a fourth matrix dot product based on the attention score and the first numerical data to generate a fourth product output; The second adder is configured to add the third product output and the fourth product output to generate the attention output.

14. The autoregressive system of claim 11, wherein: The autoregressive system also includes: A Softmax module is configured to perform a Softmax function on the attention score to generate an attention weight, wherein the second element-wise dot product operation is performed based on the first numerical data, the attention weight and the second numerical data to generate the attention output.

15. The autoregressive system of claim 11, wherein: The autoregressive system also includes: an input layer normalization module configured to normalize the hidden state into a normalized hidden state, wherein the key data, the first numerical data, and the query data are generated according to the normalized hidden state; The output layer normalization module is configured to normalize the updated hidden state into an updated normalized hidden state.

16. The autoregressive system of claim 11, wherein: The autoregressive system also includes: A ring buffer configured to cache the first numerical data and the second numerical data, and / or to cache the key data of the first position code and the key data of the second position code; Among them, the first part of the storage segments of the ring buffer is divided to generate a first numerical cache and / or a first key cache, the second part of the storage segments of the ring buffer is divided to generate a second numerical cache and / or a second key cache, the first numerical cache is configured to cache the first numerical data, the second numerical cache is configured to cache the second numerical data, the first key cache is configured to cache the key data encoded in the first position, and the second key cache is configured to cache the key data encoded in the second position.

17. The autoregressive system of claim 16, wherein: After the first value cache and / or the first key cache caches the corresponding data, the second portion of the storage segment of the ring buffer is updated by increasing the offset value of the memory address.

18. The autoregressive system of claim 17, wherein: After the second portion of the storage segments is shifted to touch the end storage segment of the ring buffer, the second portion of the storage segments of the ring buffer are copied according to the memory address initialized by the ring buffer.

19. The autoregressive system of claim 11, wherein: The signal shape format of the first numerical data and the key data encoded at the first position is expressed as (B, N, T, D), and the signal shape format of the second numerical data and the key data encoded at the second position is expressed as (B, N, C, D), wherein B is the batch size, T is the number of tags, N is the number of attention heads, D is the head dimension of each attention head, and C is a predetermined value predefined by the user.

20. The autoregressive system of claim 14, wherein: The autoregressive system also includes: The mask value generator and the third adder are configured to replace the null value of the attention score with the mask value to generate an updated attention score, wherein the attention weight is generated based on the updated attention score.

Images