Transformer acceleration device and operation method thereof

Abstract

To provide a transformer acceleration device for executing a transformer that generates output tokens based on a plurality of input tokens.SOLUTION: A transformer acceleration device for executing a transformer including a first plurality of decoder layers and a second plurality of decoder layers can include: a first memory bank array for storing a first plurality of weight matrices corresponding to the first plurality of decoder layers and a first plurality of key-value vector pairs corresponding to the second plurality of decoder layers; and a second memory bank array for storing a second plurality of weight matrices corresponding to the second plurality of decoder layers, and a second plurality of key-value vector pairs corresponding to the first plurality of decoder layers.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] The present disclosure relates to a Transformer accelerator that implements a Transformer that generates output tokens based on multiple input tokens, and more particularly, to a Transformer accelerator that implements a Transformer that generates output tokens in an auto-regressive manner, and a method of operating the same. [Background technology]

[0002] A Transformer accelerator generates an output token based on multiple input tokens. For example, a Transformer accelerator can execute a Transformer based on multiple input tokens to generate a first output token suitable for following the multiple input tokens.

[0003] The Transformer may operate in an autoregressive manner. For example, the Transformer may further use a first output token along with multiple input tokens to generate a second output token suitable to follow the multiple input tokens and the first output token. That is, the Transformer may operate sequentially over multiple iterations, with the results generated through each iteration being used as input for the next iteration.

[0004] Transformers can reuse various types of data across multiple iterations. For example, a Transformer can reuse data used in a previous iteration in the next iteration. However, if the method by which a Transformer stores and reads reused data is not optimized, the storage and reading of reused data can cause a bottleneck in the operation of the Transformer accelerator. Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure is directed to solving the above-mentioned technical problems, and more particularly, to providing a transformer accelerator device and method of operation that are implemented to store and retrieve data reused by the transformer in an optimized manner. [Means for solving the problem]

[0006] A transformer accelerator device for executing a transformer including a first plurality of decoder layers and a second plurality of decoder layers according to an embodiment of the present disclosure may include a first memory bank array for storing a first plurality of weight matrices corresponding to the first plurality of decoder layers and a first plurality of key-value vector pairs corresponding to the second plurality of decoder layers, and a second memory bank array for storing a second plurality of weight matrices corresponding to the second plurality of decoder layers and a second plurality of key-value vector pairs corresponding to the first plurality of decoder layers.

[0007] A transformer accelerator operating based on a first activation vector corresponding to a first input token according to an embodiment of the present disclosure may include a first memory bank array for storing a first plurality of weight matrices, a second memory bank array for storing a first plurality of preceding key-value vector pairs each corresponding to a first plurality of preceding tokens for the first input token, a first processing unit for generating a first key-value vector pair based on the first activation vector and the first plurality of weight matrices, and a second processing unit for performing a first attention operation based on the first key-value vector pair and the first plurality of preceding key-value vector pairs.

[0008] A method for operating a transformer accelerator device including a plurality of memory bank arrays according to an embodiment of the present disclosure may include the steps of: generating a first activation vector based on a first input token; reading a first plurality of weight matrices from a first memory bank array among the plurality of memory bank arrays; generating a first key-value vector pair based on the first plurality of weight matrices and the first activation vector; storing the first key-value vector pair in a second memory bank array among the plurality of memory banks, the second memory bank array being different from the first memory bank array; generating a second activation vector based on the first key-value vector pair; and generating a first output token corresponding to the first input token based on the second activation vector. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating a transformer accelerator according to an embodiment of the present disclosure.

[0010] [Figure 2] FIG. 2 is a block diagram showing the transformer accelerator of FIG. 1 in more detail.

[0011] [Figure 3] FIG. 2 is a block diagram illustrating the operation of the transformer of FIG. 1.

[0012] [Figure 4] 4 is a diagram showing the configuration of the transformer of FIG. 3 in more detail.

[0013] [Figure 5] 5 is a diagram illustrating the configuration of the decoder layer of FIG. 4 in more detail.

[0014] [Figure 6] 6 is a diagram illustrating the linear calculation of FIG. 5 in more detail.

[0015] [Figure 7] 6 is a diagram illustrating the attention score calculation of FIG. 5 in more detail.

[0016] [Figure 8] 8 is a diagram illustrating the attention vector calculation of FIG. 7 in more detail.

[0017] [Figure 9] 4 is a diagram illustrating the operation of the transformer of FIG. 3 in more detail.

[0018] [Figure 10] 10 is a diagram illustrating a memory bank for storing a weight matrix and key-value vector pairs according to one embodiment.

[0019] [Figure 11] 11 is a diagram illustrating a bank collision for the first memory bank of FIG. 10;

[0020] [Figure 12] 3 is a diagram illustrating the memory device of FIG. 2 according to an embodiment.

[0021] [Figure 13] 3 is a diagram illustrating the memory device of FIG. 2 according to an embodiment of the present disclosure.

[0022] [Figure 14] 14 is a block diagram illustrating the memory device of FIG. 13 implemented according to one embodiment.

[0023] [Figure 15] 2 is a flowchart illustrating a method of operating the transformer accelerator of FIG. 1 according to an embodiment of the present disclosure.

[0024] [Figure 16] 16 is a flowchart showing step S200 of FIG. 15 in more detail.

[0025] [Figure 17] 17 is a flowchart showing step S240 of FIG. 16 in more detail.

[0026] [Figure 18] FIG. 1 is a block diagram illustrating a transformer accelerator according to an embodiment.

[0027] [Figure 19] FIG. 19 is a block diagram showing the transformer accelerator of FIG. 18 in more detail.

[0028] [Figure 20] FIG. 20 is a block diagram illustrating the operation of the transformer of FIG. 18 according to one embodiment.

[0029] [Figure 21] 21 is a diagram showing the configuration of the transformer of FIG. 20 in more detail.

[0030] [Figure 22] 22 is a diagram illustrating in more detail the linear operation performed in one of the decoder layers of FIG. 21.

[0031] [Figure 23] 20 is a diagram illustrating the memory device of FIG. 19 according to an embodiment of the present disclosure.

[0032] [Figure 24] FIG. 24 is a block diagram illustrating the memory device of FIG. 23 implemented according to one embodiment.

[0033] [Figure 25] FIG. 4 is a block diagram showing the configuration of the transformer of FIG. 3 according to one embodiment.

[0034] [Figure 26] 26 is a diagram illustrating the configuration of the decoder layer of FIG. 25 in more detail.

[0035] [Figure 27] 26 is a diagram showing the full activation vector of FIG. 25 in more detail.

[0036] [Figure 28] 27 is a diagram showing the linear calculation of FIG. 26.

[0037] [Figure 29] 27 is a diagram showing the attention calculation of FIG. 26.

[0038] [Figure 30] 19 is a diagram illustrating the transformer accelerator of FIG. 1 or FIG. 18 implemented according to an embodiment.

[0039] [Figure 31] 19 is a diagram illustrating the transformer accelerator of FIG. 1 or FIG. 18 implemented according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0040] Hereinafter, embodiments of the present disclosure will be described clearly and in detail so that those skilled in the art can easily practice the present disclosure. Details such as detailed configurations and structures are provided solely for the purpose of providing a general understanding of the embodiments of the present disclosure. Therefore, modifications to the embodiments described herein may be made by those skilled in the art without departing from the technical spirit and scope of the present disclosure. Furthermore, descriptions of well-known functions and structures are omitted for clarity and conciseness. The components in the following drawings or detailed description may be connected to other components in addition to those illustrated in the drawings or described in the detailed description. The terms used in the present disclosure are defined in consideration of the functions of the present disclosure and are not limited to specific functions. The definitions of the terms can be determined based on the details described in the detailed description.

[0041] Components described with reference to terms such as driver or block in the detailed description may be implemented in the form of software, hardware, or a combination thereof. For example, software may be machine code, firmware, embedded code, and application software. For example, hardware may include electrical circuits, electronic circuits, processors, computers, integrated circuit cores, pressure sensors, inertial sensors, MEMS (Micro Electro Mechanical Systems), passive elements, or a combination thereof.

[0042] FIG. 1 is a block diagram illustrating a Transformer accelerator according to an embodiment of the present disclosure. Referring to FIG. 1, the Transformer accelerator 100 can receive one or more input tokens (TKin) and output an output token (TKout). For example, the Transformer accelerator 100 can execute a Transformer (TF). The Transformer (TF) can generate an output token (TKout) suitable for following the one or more input tokens (TKin) based on the one or more input tokens (TKin).

[0043] The Transformer (TF) may operate in an auto-regression manner. For example, the Transformer (TF) may generate a token subsequent to the generated output token (TKout) based on the generated output token (TKout). In this manner, the Transformer (TF) may generate one output token (TKout) each time the Transformer (TF) performs one iteration (i.e., one operation cycle). In this manner, the Transformer (TF) may sequentially perform multiple iterations to sequentially generate multiple output tokens (TKout). A specific operation method of the Transformer (TF) will be described in more detail below with reference to FIG. 3.

[0044] In one embodiment, the Transformer accelerator 100 can be used to implement a large language model (LLM). For example, the Transformer accelerator 100 can predict an output token (TKout) that follows multiple input tokens (TKin) based on multiple input tokens (TKin). However, the scope of the present disclosure is not limited to a specific model type in which the Transformer accelerator 100 is used. For example, the Transformer accelerator 100 can be used to implement any type of artificial intelligence model, such as an image generation model, a translation model, etc. However, for the sake of simplicity, the following description will representatively describe an embodiment in which the Transformer accelerator 100 is used to implement an LLM.

[0045] Figure 2 is a block diagram showing in more detail the transformer accelerator of Figure 1. Referring to Figures 1 and 2, the transformer accelerator 100 may include a first processing circuit 110, a second processing circuit 120, a work scheduler 130, an interface circuit 140, a memory controller 150, and a memory device 160. The first processing circuit 110, the second processing circuit 120, the work scheduler 130, the interface circuit 140, the memory controller 150, and the memory device 160 may be connected via a bus.

[0046] The first processing circuit 110 and the second processing circuit 120 can perform different types of operations. Each of the first processing circuit 110 and the second processing circuit 120 may be dedicated hardware optimized to perform different types of operations. For example, the first processing circuit 110 may be dedicated hardware for performing linear operations, and the second processing circuit 120 may be dedicated hardware for performing attention operations. The specific types of operations performed by each processing circuit will be described in more detail with reference to the following drawings.

[0047] In one embodiment, the first processing circuit 110 and the second processing circuit 120 may be processing cores included in different types of processing units, or different types of processing cores included in a single processing unit. However, the scope of the present disclosure is not limited thereto, and the first processing circuit 110 and the second processing circuit 120 may be processing cores of the same type. That is, the scope of the present disclosure is not limited to the specific implementation of the first processing circuit 110 and the second processing circuit 120.

[0048] The work scheduler 130 can control the operations of the transformer accelerator 100. For example, the work scheduler 130 can allocate the work or calculations required to drive the transformers (TF) to the first processing circuit 110 and the second processing circuit 120.

[0049] 2 illustrates work scheduler 130 as a separate component from first processing circuit 110 and second processing circuit 120 for ease of explanation, but the scope of the present disclosure is not limited in this respect. For example, work scheduler 130 may be configured as hardware separate from first processing circuit 110 and second processing circuit 120, or may be implemented as software running on first processing circuit 110, second processing circuit 120, or other processing circuit.

[0050] The interface circuit 140 can support interfacing with external devices. For example, the interface circuit 140 can receive an input token (TKin) from an external device and output an output token (TKout) to the external device.

[0051] In one embodiment, the interface circuit 140 can communicate with external devices based on various types of communication interfaces, such as PCIe (Peripheral Component Interconnect Express), DDR (Double Data Rate), etc. However, the scope of the present disclosure is not limited to the specific operation method of the interface circuit 140.

[0052] The memory controller 150 may control the memory device 160 in response to control from the first processing circuit 110, the second processing circuit 120, and the work scheduler 130. For example, the memory controller 150 may store data in the memory device 160 or read data stored in the memory device 160.

[0053] The memory device 160 can operate as an operating memory for the transformer accelerator 100. For example, the memory device 160 can store data that the transformer (TF) reuses across multiple iterations. More specifically, the memory device 160 can store data that is repeatedly used in the operations performed by the first processing circuit 110 and the second processing circuit 120.

[0054] In one embodiment, the memory device 160 may store weight matrices and key-value vector pairs used in a decoding layer included in a transformer (TF).

[0055] In one embodiment, memory device 160 may be a dynamic random access memory (DRAM) device, although the scope of this disclosure is not limited to any particular type of memory device 160.

[0056] The memory device 160 may include a first memory bank array (MBA1) and a second memory bank array (MBA2). Each of the first memory bank array (MBA1) and the second memory bank array (MBA2) may include one or more memory banks. For example, the first memory bank array (MBA1) may include a first memory bank (MB1), and the second memory bank array (MBA2) may include a second memory bank (MB2).

[0057] The memory banks included in the first memory bank array (MBA1) and the second memory bank array (MBA2) can operate independently of each other. For example, the first memory bank (MB1) and the second memory bank (MB2) can each include a sense amplifier circuit. Therefore, any memory cell row included in the first memory bank (MB1) can be activated simultaneously with any memory cell row included in the second memory bank (MB). In this case, while the first processing circuit 110 performs a read operation on the first memory bank (MB1), the second processing circuit 120 can perform a read operation on the second memory bank (MB2).

[0058] The memory device 160 can store data that is reused by the first processing circuit 110 and the second processing circuit 120 across multiple iterations of the Transformer (TF), thereby reducing the amount of computation required by the first processing circuit 110 and the second processing circuit 120 in subsequent iterations, thereby increasing the operating speed of the Transformer accelerator 100.

[0059] In one embodiment, memory device 160 can pre-store data that is reused across multiple iterations during the training step of the Transformer (TF) or during earlier iterations of the Transformer (TF).

[0060] When the first processing circuit 110 and the second processing circuit 120 simultaneously access a single memory bank, a bank collision may occur. For example, if the first processing circuit 110 and the second processing circuit 120 simultaneously access different rows of the first memory bank (MB1) (e.g., perform read operations on different rows), the read of data stored in the first memory bank (MB1) may be delayed. More specifically, for example, if the first processing circuit 110 activates the first row of the first memory bank (MB1) and then performs a read operation, and the second processing circuit 120 activates the second row of the first memory bank (MB1), the first processing circuit 110 cannot perform the read operation unless it reactivates the first row of the first memory bank (MB1). In this case, the first processing circuit 110's read of data from the first memory bank (MB1) may be delayed. Bank collisions are described in more detail below with reference to FIG. 11.

[0061] The work scheduler 130 can distribute and allocate data that may be accessed simultaneously by the first processing circuit 110 and the second processing circuit 120 to different memory bank arrays.

[0062] The memory device 160 may distribute and store data that may be accessed simultaneously by the first processing circuit 110 and the second processing circuit 120 in different memory bank arrays. For example, the memory device 160 may store some of the data that may be accessed simultaneously in the first memory bank array (MBA1) and the remaining data in the second memory bank array (MBA2). More specifically, for example, the memory device 160 may store some of the data that may be accessed simultaneously in the first memory bank (MB1) and the remaining data in the second memory bank (MB2). In this case, bank collisions that may occur due to simultaneous access by the first processing circuit 110 and the second processing circuit 120 may be prevented. A specific manner in which the memory device 160 distributes and stores data that may be accessed simultaneously in different memory bank arrays will be described in more detail with reference to the following drawings.

[0063] For the sake of simplicity, an example in which the memory device 160 stores data that may be accessed simultaneously in a distributed manner across a first memory bank (MB1) and a second memory bank (MB2) will be described below. However, the scope of the present disclosure is not limited thereto. The memory device 160 may store some of the data that may be accessed simultaneously in a distributed manner across multiple memory banks included in the first memory bank array (MBA1), and the remaining data in a distributed manner across multiple memory banks included in the second memory bank array (MBA2). The method by which the memory device 160 stores data in a distributed manner across multiple memory banks included in the first memory bank array (MBA1) and the second memory bank array (MBA2) will be described in more detail below with reference to FIG. 30.

[0064] In one embodiment, the memory device 160 may store a weight matrix and a key-value vector pair for a decoder layer included in a transformer (TF) in a distributed manner in different memory bank arrays. In this case, even if the weight matrix and the key-value vector pair are accessed simultaneously while the transformer (TF) is operating, a bank collision may not occur.

[0065] Figure 3 is a block diagram showing the operation of the transformer in Figure 1. Referring to Figures 1 to 3, the transformer (TF) can operate sequentially over multiple iterations. Below, the operation of the transformers (TF_IT1 to TF_IT3) performing successive first to third iterations will be representatively described.

[0066] For the sake of simplicity, it is assumed below that the n-th token (TKn) is generated through the summarization stage of the Transformer (TF), and that the first to third iterations are included in the generation stage. For example, it is assumed below that a token stream including the first to (n-1)-th tokens (TK1 to TKn-1) is provided from outside the Transformer accelerator 100, and that the n-th token (TKn) is generated by performing the 0-th iteration prior to the first iteration based on the token stream. However, the scope of the present disclosure is not limited thereto.

[0067] The Transformer (TF) can perform multiple iterations in an autoregressive manner. The Transformer (TF) can generate one output token (TKout) for each of the multiple iterations. For example, when performing one iteration, the Transformer (TF) can use the output token (TKout) generated through the previous iteration as the input token (TKin).

[0068] More specifically, for example, a Transformer (TF_IT1) performing a first iteration may use an nth token (TKn) generated through an iteration prior to the first iteration (e.g., the 0th iteration) as an input token (TKin) to generate an n+1th token (TKn+1). Similarly, a Transformer (TF_IT2) performing a second iteration may use the n+1st token (TKn+1) as an input token (TKin) to generate an n+2nd token (TKn+2), and a Transformer (TF_IT3) performing a third iteration may use the n+2nd token (TKn+2) as an input token (TKin) to generate an n+3rd token (TKn+3).

[0069] In this manner, the Transformer accelerator 100 can execute the Transformer (TF) for multiple iterations to sequentially generate multiple tokens. In this case, the multiple tokens generated by the Transformer (TF) can form a single token sequence together with a token stream (e.g., the first through (n-1)th tokens (TK1 through TKn-1)) provided from an external device. For example, the first through (n+2)th tokens (TK1 through TKn+2) can form a single token sequence.

[0070] In one embodiment, when the Transformer accelerator 100 is used to implement an LLM, the token sequence corresponds to one or more sentences, although the scope of the present disclosure is not limited in this respect.

[0071] In one embodiment, the maximum length of the token sequence may be predetermined, for example, by the number of tokens that the Transformer (TF) can process at one time.

[0072] In one embodiment, a token preceding any other token in a token sequence may be referred to as its preceding token. For example, the first through n-1st tokens (TK1 through TKn-1) may be referred to as preceding tokens for the nth token (TKn), and the first through n+1st tokens (TK1 through TKn+1) may be referred to as preceding tokens for the n+2nd token (TKn+2).

[0073] During one iteration, a Transformer (TF) can generate an output token (TKout) based on an input token (TKin) and the tokens preceding it. For example, a Transformer (TF_IT1) performing a first iteration can generate an (n+1)-th token (TKn+1) based on an n-th token (TKn) and the first through (n-1)-th tokens (TK1 through TKn-1). Similarly, a Transformer (TF_IT2) performing a second iteration can generate an (n+2)-th token (TKn+2) based on an n+1-th token (TKn+1) and the first through (n)-th tokens (TK1 through TKn).

[0074] As a result, when the Transformer (TF) performs any iteration, it repeatedly requires the results of operations on multiple tokens used in the previous iteration. Therefore, when the Transformer (TF) performs any iteration, it can reuse the results of operations on multiple tokens used in the previous iteration. For example, the Transformer (TF_IT1) performing the first iteration can reuse the results of operations on the first through n-1th tokens (TK1 through TKn-1), and the Transformer (TF_IT2) performing the second iteration can reuse the results of operations on the first through nth tokens (TK1 through TKn).

[0075] More specifically, the Transformer (TF) performing each iteration can store the operation result for the input token (TKin) in the memory device 160. In this case, the amount of operation when the Transformer (TF) performs a subsequent iteration can be minimized. The operation of the Transformer (TF) for each iteration will be described below.

[0076] In one embodiment, the results of the operations on the tokens may include key-value vector pairs, which are described in more detail with reference to Figures 4-9 below.

[0077] First, the transformer (TF) can store the operation results for the preceding tokens (i.e., the 1st to n-1th tokens (TK1 to TKn-1)) for the nth token (TKn) in the memory device 160 before performing the first iteration (e.g., while performing the 0th iteration).

[0078] Thereafter, the Transformer (TF_IT1) performing the first iteration may read the operation results for the first through (n-1)th tokens (TK1 through TKn-1) stored in the memory device 160 and generate the operation result for the nth token (TKn). In this case, the Transformer (TF_IT1) performing the first iteration may generate the (n+1)th token (TKn+1) without calculating the operation results for the first through (n-1)th tokens (TK1 through TKn-1). Meanwhile, the Transformer (TF_IT1) performing the first iteration may further store the operation result for the nth token (TKn) in the memory device 160.

[0079] Thereafter, the Transformer (TF_IT2) performing the second iteration can read the operation results for the first through nth tokens (TK1 through TKn) stored in the memory device 160 and generate the operation result for the (n+1)th token (TKn+1). In this case, the Transformer (TF_IT2) performing the second iteration can generate the (n+2)th token (TKn+2) without calculating the operation results for the first through nth tokens (TK1 through TKn). Meanwhile, the Transformer (TF_IT2) performing the second iteration can further store the operation result for the (n+1)th token (TKn+1) in the memory device 160.

[0080] In this manner, the transformer (TF_IT3) performing the third iteration can read the operation results for the first to n+1th tokens (TK1 to TKn+1) from the memory device 160 and store the operation results for the n+2nd token (TKn+2) in the memory device 160.

[0081] That is, according to an embodiment of the present disclosure, the Transformer (TF) during each iteration can read an operation result corresponding to a preceding token for the output token (TKout) of the previous iteration from the memory device 160. Then, the Transformer (TF) can generate an operation result corresponding to the output token (TKout) of the previous iteration and store it in the memory device 160.

[0082] Figure 4 is a diagram illustrating in more detail the configuration of the transformer in Figure 3. Hereinafter, the configuration and operation of the transformer (TF) performing the first iteration will be representatively described with reference to Figures 1 to 4. However, the scope of the present disclosure is not limited thereto, and transformers (TF) performing other iterations can also be realized in a similar manner.

[0083] The transformer (TF) may include a pre-hidden layer (HL_pre), first to m-th decoder layers (DL1 to DLm), and a post-hidden layer (HL_post).

[0084] The prehidden layer (HL_pre) can generate an activation vector corresponding to the input token (TKin) based on the input token (TKin). For example, the prehidden layer (HL_pre) can generate an activation vector (ACT1_TKn) for the nth token based on the nth token (TKn).

[0085] In one embodiment, the pre-hidden layer (HL_pre) can perform various types of token processing operations, such as a token embedding operation, a positional encoding operation, and the like.

[0086] The first to m-th decoder layers (DL1 to DLm) may sequentially convert activation vectors to generate activation vectors (ACTm+1_TKn). For example, the first decoder layer (DL1) may generate an activation vector (ACT2_TKn) based on the activation vector (ACT1_TKn), and the second decoder layer (DL2) may generate an activation vector (ACT3_TKn) based on the activation vector (ACT2_TKn). In this manner, the m-th decoder layer (DLm) may generate an activation vector (ACTm+1_TKn) based on the activation vector (ACTm_TKn).

[0087] In one embodiment, the activation vectors (ACT1_TKn) to (ACTm+1_TKn) may have the same dimensions, e.g., the activation vectors (ACT1_TKn) to (ACTm+1_TKn) may each include the same number of elements.

[0088] The first to m-th decoder layers (DL1 to DLm) can generate different operation results for the input token (TKin) (e.g., the n-th token (TKn)). For example, the first to m-th decoder layers (DL1 to DLm) can calculate different key-value vector pairs corresponding to the n-th token (TKn).

[0089] Each of the first to m-th decoder layers (DL1 to DLm) can generate an activation vector based on the generated key-value vector pair. For example, the first decoder layer (DL1) can generate an activation vector (ACT2_TKn) based on the generated key-value vector pair.

[0090] Each of the first to m-th decoder layers (DL1 to DLm) may store the generated key-value vector pair in the memory device 160. In this case, when the transformer (TF) performs a subsequent iteration (e.g., the second and third iterations of FIG. 3), it may not be able to calculate the key-value vector pair corresponding to the n-th token (TKn) of each of the first to m-th decoder layers (DL1 to DLm).

[0091] When the first iteration is performed, the first to m-th decoder layers (DL1 to DLm) can read key-value vector pairs corresponding to each preceding token for the input token (TKin) (e.g., the n-th token (TKn)) of the first to m-th decoder layers (DL1 to DLm) from the memory device 160. Therefore, each of the first to m-th decoder layers (DL1 to DLm) can generate an activation vector without calculating the key-value vector pairs for the first to n-1-th tokens (TK1 to TKn-1).

[0092] The configuration and operation of each of the first to m-th decoder layers (DL1 to DLm) will be described in more detail below with reference to FIGS. 5 to 9.

[0093] The post-hidden layer (HL_post) can calculate an output token (TKout) (e.g., the n+1th token (TKn+1)) based on the activation vector (ACTm+1_TKn).

[0094] In one embodiment, the post-hidden layer (HL_post) can perform various types of vector processing operations, such as layer normalizing operations, fully connected network calculation operations, and the like.

[0095] In one embodiment, the activation vector (ACTi_TKj) may refer to the activation vector provided to the (i)th decoder layer corresponding to the (j)th token (TKj), where (i) is an integer between 1 and m. In one embodiment, the activation vector (ACTm+1_TKj) may refer to the activation vector provided to the post-hidden layer (HL_post) corresponding to the j-th token (TKj).

[0096] Figure 5 is a diagram illustrating in more detail the configuration of the decoder layer in Figure 4. Hereinafter, the configuration and operation of the first decoder layer (DL1) will be described representatively with reference to Figures 1 to 5. However, the scope of the present disclosure is not limited thereto, and the second to mth decoder layers (DL2 to DLm) may also be realized in a similar manner.

[0097] The operation of one head that executes the first decoder layer (DL1) will be described below. The operation of the first decoder layer (DL1) executed in a multi-head manner will be described in more detail with reference to Figures 25 to 29 below. However, the scope of the present disclosure is not limited to the specific manner in which the first decoder layer (DL1) is executed.

[0098] The first decoder layer (DL1) may include linear calculation (LNC) and attention calculation (ATC). The linear calculation (LNC) may be performed by the first processing circuit 110. The attention calculation (ATC) may be performed by the second processing circuit 120.

[0099] The linear operation (LNC) may include a query vector generation (VG_QRY), a key vector generation (VG_KEY), and a value vector generation (VG_VAL). That is, the first processing circuit 110 may generate a query vector (QRY), a key vector (KEY), and a value vector (VAL).

[0100] In one embodiment, the first processing circuit 110 may calculate a query vector (QRY), a key vector (KEY), and a value vector (VAL) based on the result of multiplying activation vectors (e.g., activation vectors (ACT1_TKn)) provided to the first decoder layer (DL1) by different weight matrices. The manner in which the first processing circuit 110 calculates the query vector (QRY), key vector (KEY), and value vector (VAL) will be described in more detail below with reference to FIG. 6.

[0101] The attention operation (ATC) may include an attention score operation (CAL_ATS) and an attention vector operation (CAL_AV). That is, the second processing circuit 120 may perform an attention operation to calculate an attention score (ATS) and an attention vector (AV).

[0102] In one embodiment, the second processing circuit 120 can calculate multiple attention scores (ATS) based on the query vector (QRY) and the key vector (KEY). For example, the second processing circuit 120 can calculate multiple attention scores (ATS) for the key vector (KEY) corresponding to the input token (TKin) and the key vector corresponding to the preceding token for the input token (TKin) based on the query vector (QRY). The manner in which the second processing circuit 120 calculates the multiple attention scores (ATS) will be described in more detail with reference to FIG. 7 below.

[0103] In one embodiment, the second processing circuit 120 can calculate an attention vector (AV) based on a plurality of attention scores (ATS) and a value vector (VAL). For example, the second processing circuit 120 can calculate the attention vector (AV) by accumulating a value vector (VAL) corresponding to an input token (TKin) and value vectors corresponding to preceding tokens relative to the input token (TKin) based on a plurality of attention scores (ATS). The manner in which the second processing circuit 120 calculates the attention vector (AV) will be described in more detail with reference to FIG. 8 below.

[0104] Figure 6 is a diagram illustrating in more detail the linear operation of Figure 5. That is, hereinafter, with reference to Figures 1 to 6, an embodiment in which the first decoder layer (DL1) performs linear operation (LNC) corresponding to the n-th token (TKn) will be representatively described.

[0105] The first processing circuit 110 can read the first query weight matrix (WMQ_L1), the first key weight matrix (WMK_L1), and the first value weight matrix (WMV_L1) stored in the memory device 160.

[0106] In one embodiment, the weight matrices used by the transformer (TF) for the linear operations (LNC) of different decoder layers (DL) may be different. For the sake of simplicity, the (i)th query weight matrix (WMQ_Li) may refer to the weight matrix used for the query vector generation (VG_QRY) of the (i)th decoder layer (DLi), the (i)th key weight matrix (WMK_Li) may refer to the weight matrix used for the key vector generation (VG_KEY) of the (i)th decoder layer (DLi), and the (i)th value weight matrix (WMV_Li) may refer to the weight matrix used for the value vector generation (VG_VAL) of the (i)th decoder layer (DLi).

[0107] The first processing circuit 110 can generate a query vector (QRY1_TKn) based on the result of multiplying the activation vector (ACT1_TKn) by the first query weight matrix (WMQ_L1).

[0108] In one embodiment, the query vector (QRYi_TKj) may refer to the query vector corresponding to the (j)th token (TKj) generated in the (i)th decoder layer (DLi).

[0109] The first processing circuit 110 can generate a key vector (KEY1_TKn) based on the result of multiplying the activation vector (ACT1_TKn) by the first key weight matrix (WMK_L1).

[0110] In one embodiment, the key vector (KEYi_TKj) may refer to the key vector corresponding to the (j)th token (TKj) generated in the (i)th decoder layer (DLi).

[0111] The first processing circuit 110 can generate a value vector (VAL1_TKn) based on the result of multiplying the activation vector (ACT1_TKn) by the first value weight matrix (WMV_L1).

[0112] In one embodiment, the value vector (VALi_TKj) may refer to the value vector corresponding to the (j)th token (TKj) generated in the (i)th decoder layer (DLi).

[0113] The weight matrices used by the transformer (TF) for the linear computation (LNC) of the first decoder layer (DL1) in each of the multiple iterations may be the same. For example, the first query weight matrix (WMQ_L1), the first key weight matrix (WMK_L1), and the first value weight matrix (WMV_L1) may be reused across multiple iterations.

[0114] In one embodiment, the Transformer (TF) can perform linear computations (LNC) on multiple token streams in a batch scheme. For example, the Transformer (TF) can perform linear computations (LNC) on token streams provided by different users in a batch scheme. The method by which the Transformer (TF) performs linear computations (LNC) on multiple token streams in a batch scheme is described in more detail below with reference to Figures 18 to 24.

[0115] Figure 7 is a diagram illustrating in more detail the attention score calculation of Figure 5. That is, hereinafter, with reference to Figures 1 to 7, the operation of the first decoder layer (DL1) performing the attention score calculation (CAL_ATS) corresponding to the n-th token (TKn) will be representatively described.

[0116] The second processing circuit 120 can generate attention scores (ATS1_TKn_1 to ATS1_TKn_n) corresponding to the nth token (TKn) and its preceding tokens (e.g., the 1st to n-1th tokens (TKn-1)) based on the query vector (QRY1_TKn). For example, the second processing circuit 120 can calculate the attention score (ATS1_TKn_1) based on the result of dot-product of the query vector (QRY1_TKn) and the key vector (KEY1_TK1), and can calculate the attention score (ATS1_TKn_2) based on the result of dot-product of the query vector (QRY1_TKn) and the key vector (KEY1_TK2). In this manner, the second processing circuit 120 can calculate the attention score (ATS1_TKn_n) based on the result of the inner product of the query vector (QRY1_TKn) and the key vector (KEY1_TKn). In this case, the key vectors (KEY1_TK1 to KEY1_TKn-1) corresponding to the preceding tokens (hereinafter referred to as preceding key vectors) may be calculated in advance in the aforementioned iteration (e.g., the 0th iteration).

[0117] In one embodiment, the attention score (ATSi_TKj_k) may refer to the attention score corresponding to the (j)th token (TKj) and the (k)th token (TKk) generated in the (i)th decoder layer (DLi). For example, the attention score (ATSi_TKj_k) may indicate the relevance to the (j)th token (TKj) and the (k)th token (TKk).

[0118] That is, to perform the attention score calculation (CAL_ATS), the second processing circuit 120 requires not only the key vector (KEY) for the input token (TKin) but also the preceding key vector. Therefore, the second processing circuit 120 can store the calculated key vector in the memory device 160 and reuse it. For example, while the transformer (TF) is performing a first iteration, the first decoder layer (DL1) can store the key vector (KEY1_TKn) in the memory device 160. In this case, while the transformer (TF) is performing a second iteration, the second processing circuit 120 can reuse the key vector (KEY1_TKn) stored in the memory device 160 to perform the attention score calculation (CAL_ATS).

[0119] Figure 8 is a diagram illustrating in more detail the attention vector calculation of Figure 7. That is, hereinafter, with reference to Figures 1 to 8, the operation of the first decoder layer (DL1) performing attention vector calculation (CAL_AV) corresponding to the n-th token (TKn) will be representatively described.

[0120] The second processing circuit 120 can perform attention vector calculation (CAL_AV) for the n-th token (TKn) based on the attention scores (ATS) and value vectors (VAL) corresponding to the n-th token (TKn) and its preceding tokens (e.g., the 1st to (n-1st) tokens (TKn-1)). For example, the second processing circuit 120 can generate an attention vector (AV1_TKn) based on the result of accumulating the values ​​obtained by multiplying the value vectors (VAL1_TK1 to VAL1_TKn) and the attention scores (ATS1_TKn_1 to ATS1_TKn_n). In this case, the value vectors (VAL1_TK1 to VAL1_TKn-1) corresponding to the preceding tokens (hereinafter, referred to as preceding value vectors) may be calculated in advance by the aforementioned iteration (e.g., the 0th iteration).

[0121] That is, to perform the attention vector calculation (CAL_AV), the second processing circuit 120 requires not only the value vector (VAL) for the input token (TKin) but also the preceding value vector. Therefore, the second processing circuit 120 can reuse the calculated value vector after storing it in the memory device 160. For example, while the transformer (TF) is performing a first iteration, the first decoder layer (DL1) can store the value vector (VAL1_TKn) in the memory device 160. In this case, while the transformer (TF) is performing a second iteration, the second processing circuit 120 can reuse the value vector (VAL1_TKn) stored in the memory device 160 to perform the attention vector calculation (CAL_AV).

[0122] In one embodiment, the first decoder layer (DL1) can convert the attention vector (AV1_TKn) based on various types of vector processing algorithms such as residual calculation, feed forward network, etc. to generate an activation vector (ACT2_TKn).

[0123] Figure 9 illustrates in more detail the operation of the Transformer in Figure 3. For a simpler explanation, the key-value vector pairs that the Transformer (TF) stores in the memory device 160 during the first iteration will be described below with reference to Figures 1 to 9.

[0124] While the transformer (TF) performs the first iteration, the first through m-th decoder layers (DL1 through DLm) may store different key vectors in the memory device 160. For example, the first through m-th decoder layers (DL1 through DLm) may store key vectors (KEY1_TKn through KEYm_TKn) in the memory device 160, respectively.

[0125] Similarly, while the transformer (TF) performs a first iteration, the first through m-th decoder layers (DL1 through DLm) may store different value vectors in the memory device 160. For example, the first through m-th decoder layers (DL1 through DLm) may store value vectors (VAL1_TKn through VALm_TKn) in the memory device 160, respectively.

[0126] In other words, when one decoder layer (DL) is executed, the first processing circuit 110 can store the key vector (KEY) and value vector (VAL) corresponding to the input token (TKin) in the memory device 160. Hereinafter, for easier explanation, the key vector (KEY) and value vector (VAL) that the first processing circuit 110 stores in the memory device 160 while one decoder layer (DL) is executed will be referred to as a key-value vector pair (KVP). For example, the key vector (KEY1_TKn) and the value vector (VAL1_TKn) will be referred to as a key-value vector pair (KVP_L1_TKn), and the key vector (KEY2_TKn) and the value vector (VAL2_TKn) will be referred to as a key-value vector pair (KVP_L2_TKn).

[0127] In one embodiment, a key-value vector pair (KVP_Li_TKj) may refer to a key vector (KEY) and a value vector (VAL) for the (j)th token (TKj) generated in the (i)th decoder layer (DLi). That is, a key-value vector pair (KVP_Li_TKj) may refer to a key vector (KEY_Li_TKj) and a value vector (VAL_Li_TKj).

[0128] In this manner, each time the transformer (TF) performs one decoder layer (DL), one additional key-value vector pair (KVP) is stored in the memory device 160. Thus, each time the transformer (TF) performs one iteration, m additional key-value vector pairs (KVP) are stored in the memory device 160.

[0129] In one embodiment, a key-value vector pair corresponding to a preceding token for an input token (TKin) is referred to as a “preceding key-value vector pair.” For example, a preceding key-value vector pair may include a preceding key vector and a preceding value vector.

[0130] 10 is a diagram showing memory banks for storing weight matrices and key-value vector pairs according to one embodiment. Hereinafter, with reference to FIGS. 1 to 10, an embodiment in which a plurality of weight matrices and a plurality of key-value vector pairs (KVPs) required for the operation of a transformer (TF) are stored in one memory bank (e.g., the first memory bank (MB1)) will be described as a representative example.

[0131] The first memory bank (MB1) can include a first memory area (MA1) and a second memory area (MA2).

[0132] The first memory area (MA1) can store multiple weight matrices required for the operation of the transformer (TF). For example, the first memory area (MA1) can store at least a portion of the first through m-th query weight matrices (WMQ_L1 through WMQ_Lm), the first through m-th key weight matrices (WMK_L1 through WMK_Lm), and the first through m-th value weight matrices (WMV_L1 through WMV_Lm) used in the linear computation (LNC) of the first through m-th decoder layers (DL1 through DLm), respectively. In this case, the first memory area (MA1) can be accessed by the first processing circuit 110 that performs the linear computation (LNC).

[0133] The second memory area (MA2) may store a plurality of key-value vector pairs (KVP) required for the operation of the transformer (TF). For example, the second memory area (MA2) may store at least a portion of the plurality of key-value vector pairs (KVP_L1_TK1 to KVP_L1_TKn) for the first decoder layer (DL1). Similarly, the second memory area (MA2) may store at least a portion of the plurality of key-value vector pairs (KVP_L2_TK1 to KVP_L2_TKn) for the second decoder layer (DL2), and may store at least a portion of the plurality of key-value vector pairs (KVP_Lm_TK1 to KVP_Lm_TKn) for the mth decoder layer (DLm). In this case, the second memory area (MA2) may be accessed by the second processing circuit 120 that performs attention calculation (ATC).

[0134] That is, the first memory area (MA1) and the second memory area (MA2) can be accessed by different processing circuits. Accordingly, access requests to the first memory area (MA1) and the second memory area (MA2) can occur simultaneously. That is, a bank collision can occur in the first memory bank (MB1). For example, while the first processing circuit 110 is performing a read operation on the first memory cell row (R1) in the first memory area (MA1), the second processing circuit 120 can request a read operation on the second memory cell row (R2) in the second memory area (MA2). In this case, the read operation on the first memory cell row (R1) can be delayed. The manner in which the read operation on the first memory bank (MB1) is delayed when a bank collision occurs will be described in more detail below with reference to FIG. 11.

[0135] 11 is a diagram illustrating a bank collision for the first memory bank of FIG. 10. Referring to FIGS. 1 to 11, the first memory bank (MB1) may include a memory cell array (MCA) and a sense amplifier circuit (SA). The memory cell array (MCA) may include first and second memory cell rows (R1, R2).

[0136] The first processing circuit 110 can activate the first memory cell row R1. That is, the first processing circuit 110 can issue an active command for the first memory cell row R1. The first memory bank MB1 can store data stored in the first memory cell row R1 in the sense amplifier circuit SA in response to the active command for the first memory cell row R1. For example, data DTa_R1 to DTd_R1 for the first memory cell row are stored in the sense amplifier circuit SA.

[0137] Thereafter, the first processing circuit 110 can access the data of the first memory cell row (R1). That is, the first processing circuit 110 can issue multiple read commands to sequentially read the data (DTa_R1 to DTd_R1) for the first memory cell row.

[0138] However, while the first processing circuit 110 is accessing data in the first memory cell row R1, the second processing circuit 120 may issue an activation request for the second memory cell row R2. That is, the second processing circuit 120 may issue an activation command for the second memory cell row R2 while the first processing circuit 110 is unable to read some of the data stored in the sense amplifier circuit SA. For example, an activation command for the second memory cell row R2 may be provided to the first memory bank MB1 while the data DTc_R1 to DTd_R1 are not being read.

[0139] The first memory bank (MB1) can store data stored in the second memory cell row (R2) in the sense amplifier circuit (SA) in response to an active command for the second memory cell row (R2). In this case, the data (DTa_R1 to DTd_R4) for the first memory cell row stored in the sense amplifier circuit (SA) is erased, and the data (DTa_R2 to DTd_R2) for the second memory cell row is newly stored in the sense amplifier circuit (SA). Therefore, when the second processing circuit 120 issues an active command for the second memory cell row (R2), the first processing circuit 110 must re-issue an active command for the first memory cell row (R1) to read the data (DTc_R1 to DTd_R1).

[0140] That is, when a bank collision occurs, the throughput of the first memory bank (MB1) is reduced, and in this case, the time when data is read from the first memory bank (MB1) is delayed, thereby degrading the operational performance of the transformer accelerator 100.

[0141] 12 is a diagram illustrating the memory device of FIG. 2 according to one embodiment. Referring to FIGS. 1 to 9 and 12, the memory device 160 may include a first memory bank array (MBA1) and a second memory bank array (MBA2). The first memory bank array (MBA1) may include a first memory bank (MB1), and the second memory bank array (MBA2) may include a second memory bank (MB2).

[0142] The memory device 160 may store multiple weight matrices and multiple key-value vector pairs (KVPs) in different memory bank arrays. For example, the memory device 160 may store the first through m-th query weight matrices (WMQ_L1 through WMQ_Lm), the first through m-th key weight matrices (WMK_L1 through WMK_Lm), and the first through m-th value weight matrices (WMV_L1 through WMV_Lm) in a first memory bank array (MBA1). The memory device 160 may store multiple preceding key-value vector pairs in a second memory bank array (MBA2).

[0143] This allows multiple weight matrices and multiple key-value vector pairs (KVPs) to be stored in different memory banks. For example, the first through m-th query weight matrices (WMQ_L1 through WMQ_Lm), the first through m-th key weight matrices (WMK_L1 through WMK_Lm), and the first through m-th value weight matrices (WMV_L1 through WMV_Lm) can be stored in the first memory bank (MB1). The key-value vector pairs (KVP_L1_TK1 through KVP_Lm_TKn) for the input token (TKin) and its preceding tokens can be stored in the second memory bank (MB2).

[0144] In this case, it is possible to prevent simultaneous access to one memory bank by the first processing circuit 110 and the second processing circuit 120. For example, the first processing circuit 110 reads a plurality of weight matrices stored in the first memory bank (MB1), and the second processing circuit 120 reads a key-value vector pair (KVP) stored in the second memory bank (MB2), thereby preventing bank collisions.

[0145] In one embodiment, the total capacity of the weight matrices and the total capacity of the key-value vector pairs (KVPs) may be different from each other, which may result in an imbalance in the capacity stored in the first memory bank (MB1) and the second memory bank (MB2).

[0146] 13 is a diagram illustrating the memory device of FIG. 2 according to an embodiment of the present disclosure. Referring to FIGS. 1 to 9 and 13, the memory device 160 may include a first memory bank array (MBA1) and a second memory bank array (MBA2). The first memory bank array (MBA1) may include a first memory bank (MB1), and the second memory bank array (MBA2) may include a second memory bank (MB2). The first memory bank (MB1) may include first and second memory regions (MAa, MAb), and the second memory bank (MB2) may include a third and fourth memory region (MAc, MAd).

[0147] The memory device 160 may store a plurality of weight matrices and a plurality of key-value vector pairs (KVPs) for each decoder layer (DL) in different memory bank arrays. For example, the work scheduler 130 may classify the first through m-th decoder layers (DL1 through DLm) into first and second layer groups (LG1 and LG2). In this case, the memory device 160 may store a plurality of weight matrices and a plurality of key-value vector pairs (KVPs) for each layer group in different memory bank arrays.

[0148] In one embodiment, the number of decoder layers included in the first and second layer groups (LG1, LG2) may correspond to each other, for example, the number of decoder layers included in the first and second layer groups (LG1, LG2) may be the same.

[0149] The memory device 160 may store a plurality of weight matrices for the first layer group (LG1) in a first memory bank array (MBA1) and may store a plurality of key-value vector pairs (KVPs) for the first layer group (LG1) in a second memory bank array (MBA2). For example, the memory device 160 may store a plurality of weight matrices for the first layer group (LG1) in the first memory bank (MB1) and may store a plurality of key-value vector pairs (KVPs) for the first layer group (LG1) in the second memory bank (MB2).

[0150] More specifically, for example, the memory device 160 may store a plurality of query weight matrices (WMQ_LG1), a plurality of key weight matrices (WMK_LG1), and a plurality of value weight matrices (WMV_LG1) for the first layer group (LG1) in the second memory area (MAb). The memory device 160 may store a plurality of key-value vector pairs (KVP_LG1) for the first layer group (LG1) in the third memory area (MAc).

[0151] Conversely, the memory device 160 may store a plurality of weight matrices for the second layer group (LG2) in the second memory bank array (MBA2) and a plurality of key-value vector pairs (KVPs) for the second layer group (LG2) in the first memory bank array (MBA1). For example, the memory device 160 may store a plurality of weight matrices for the second layer group (LG2) in the second memory bank (MB2) and a plurality of key-value vector pairs (KVPs) for the second layer group (LG2) in the first memory bank (MB1).

[0152] More specifically, for example, the memory device 160 may store a plurality of query weight matrices (WMQ_LG2), a plurality of key weight matrices (WMK_LG2), and a plurality of value weight matrices (WMV_LG2) for the second layer group (LG2) in the fourth memory area (MAd). The memory device 160 may store a plurality of key-value vector pairs (KVP_LG2) for the second layer group (LG2) in the first memory area (MAa).

[0153] In one embodiment, the plurality of query weight matrices (WMQ_LG1) may include query weight matrices used for linear computation (LNC) of each of the plurality of decoder layers (DL) included in the first layer group (LG1). The plurality of query weight matrices (WMQ_LG2) may include query weight matrices used for linear computation (LNC) of each of the plurality of decoder layers (DL) included in the second layer group (LG2). For the sake of simplicity, detailed descriptions of the plurality of key weight matrices (WMK_LG1), the plurality of key weight matrices (WMK_LG2), the plurality of value weight matrices (WMV_LG1), and the plurality of value weight matrices (WMV_LG2), which are defined in a manner similar to the plurality of query weight matrices (WMQ_LG1) and the plurality of query weight matrices (WMQ_LG2), will be omitted.

[0154] Therefore, multiple weight matrices and multiple key-value vector pairs (KVPs) for each decoder layer are stored in different memory banks. In this case, simultaneous access by the first processing circuit 110 and the second processing circuit 120 to the same memory bank can be prevented. For example, when the Transformer (TF) executes a decoder layer included in the first layer group (LG1), the first processing circuit 110 can access the second memory area (MAb), and the second processing circuit 120 can access the third memory area (MAc). On the other hand, when the Transformer (TF) executes a decoder layer included in the second layer group (LG2), the first processing circuit 110 can access the fourth memory area (MAd), and the second processing circuit 120 can access the first memory area (MAa). Therefore, according to the embodiment of the present disclosure, bank collision can be prevented. This allows the transformer accelerator 100 according to the embodiments of the present disclosure to increase the efficiency and speed of generating multiple output tokens based on multiple input tokens.

[0155] Meanwhile, the decoder layers may have the same configuration. Accordingly, the total capacity of the weight matrices used in each decoder layer may be the same, and the total capacity of the key-value vector pairs (KVPs) used in each decoder layer may be the same. Therefore, if the first and second layer groups (LG1, LG2) include the same number of decoder layers, the capacities of the first memory area (MAa) and the third memory area (MAc) may be the same, and the capacities of the second memory area (MAb) and the fourth memory area (MAd) may be the same. In this case, the total capacity of the first memory area (MAa) and the second memory area (MAb) and the total capacity of the third memory area (MAc) and the fourth memory area (MAd) may be the same. As a result, according to an embodiment of the present disclosure, data can be stored uniformly in the first memory bank (MB1) and the second memory bank (MB2). As a result, the transformer accelerator 100 according to the embodiment of the present disclosure can increase the lifespan of the first memory bank (MB1) and / or the second memory bank (MB2).

[0156] FIG. 14 is a block diagram illustrating the memory device of FIG. 13 implemented according to an embodiment. Referring to FIGS. 1 to 9 and 13 to 14, the work scheduler 130 may classify the first through m-th decoder layers (DL1 to DLm) into an odd layer group and an even layer group (LGodd LGeven) according to the order of the first through m-th decoder layers (DL1 to DLm). For example, the work scheduler 130 may classify odd-numbered decoder layers (e.g., the first decoder layer (DL1), the third decoder layer (DL3), etc.) among the first through m-th decoder layers (DL1 to DLm) into the odd layer group (LGodd). Conversely, the work scheduler 130 may classify even-numbered decoder layers (e.g., the second decoder layer (DL2), the fourth decoder layer (DL4), etc.) among the first through m-th decoder layers (DL1 to DLm) into the even layer group (LGeven).

[0157] In this case, the memory device 160 can store multiple weight matrices for the odd layer group (LGodd) in a first memory bank array (MBA1) and multiple key-value vector pairs (KVPs) for the odd layer group (LGodd) in a second memory bank array (MBA2).

[0158] More specifically, for example, the memory device 160 may store a plurality of query weight matrices (WMQ_LGodd), a plurality of key weight matrices (WMK_LGodd), and a plurality of value weight matrices (WMV_LGodd) for the odd layer group (LGodd) in the second memory area (MAb). The memory device 160 may store a plurality of key-value vector pairs (KVP_LGodd) for the odd layer group (LGodd) in the third memory area (MAc).

[0159] Conversely, the memory device 160 can store multiple weight matrices for the even layer group (LGeven) in the second memory bank array (MBA2) and multiple key-value vector pairs (KVPs) for the even layer group (LGeven) in the first memory bank array (MBA1).

[0160] More specifically, for example, the memory device 160 may store a plurality of query weight matrices (WMQ_LGeven), a plurality of key weight matrices (WMK_LGeven), and a plurality of value weight matrices (WMV_LGeven) for the even layer group (LGeven) in the fourth memory area (MAd). The memory device 160 may store a plurality of key-value vector pairs (KVP_LGeven) for the even layer group (LGeven) in the first memory area (MAa).

[0161] In one embodiment, "m" may be an even number. In this case, the number of decoder layers included in the odd layer group (LGodd) may be the same as the number of decoder layers included in the even layer group (LGeven). In this case, the first memory bank array (MBA1) and the second memory bank array (MBA2) may store the same amount of data. However, the scope of the present disclosure is not limited thereto. For example, according to an embodiment of the present disclosure, even when "m" is an odd number, the first memory bank array (MBA1) and the second memory bank array (MBA2) may store very similar amounts of data.

[0162] 15 is a flowchart illustrating a method of operation of the transformer acceleration device of FIG. 1 according to an embodiment of the present disclosure. Hereinafter, with reference to FIGS. 1 to 9 and 13 to 15, a method of accelerating one iteration of a transformer (TF) by the transformer acceleration device 100 will be described.

[0163] First, in step S100, the transformer accelerator 100 may execute a prehidden layer (HL_pre) based on an input token (TKin) to generate an activation vector. For example, when the transformer accelerator 100 executes a transformer (TF) performing a first iteration, the transformer accelerator 100 may generate an activation vector (ACT1_TKn).

[0164] In step S200, the transformer accelerator 100 may execute the decoder layer (DL) based on the activation vector (ACT) to generate the next activation vector. For example, when the transformer accelerator 100 executes the first decoder layer (DL1), the transformer accelerator 100 may generate an activation vector (ACT2_TKn) based on the activation vector (ACT1_TKn).

[0165] In step S300, it may be determined whether all decoder layers (DL) have been executed. For example, the transformer accelerator 100 may determine whether activation vectors (ACTm+1_TKn) have been generated through all decoder layers (DL) included in the transformer (TF).

[0166] If it is determined in step S300 that there is a decoder layer (DL) that has not yet been executed, the above-described step S200 can be repeated. If it is determined that all decoder layers (DL) have been executed, the following step S400 can be performed. In this manner, the transformer accelerator 100 can generate an activation vector (ACTm+1_TKn) by gradually transforming the activation vector (ACT1_TKn) based on the first through m-th decoder layers (DL1 through DLm). In one embodiment, the activation vector (ACTm+1_TKn) can be referred to as a last activation vector.

[0167] In step S400, the transformer accelerator 100 may execute a post-hidden layer (HL_post) based on the final activation vector to generate an output token (TKout).

[0168] In one embodiment, the transformer accelerator 100 can use the output token (TKout) generated through step S400 as the input token (TKin) of the subsequent iteration of the transformer (TF). In this case, the input token (TKin) used in step S100 becomes the predecessor token for the input token (TKin) of the subsequent iteration.

[0169] Figure 16 is a flowchart illustrating step S200 of Figure 15 in more detail. For a simpler explanation, a method in which the transformer acceleration device 100 accelerates the first decoder layer (DL1) included in the transformer (TF) will be representatively described below with reference to Figures 1 to 9 and 13 to 16. However, the scope of the present disclosure is not limited thereto, and the transformer acceleration device 100 may accelerate the second through m-th decoder layers (DL2 to DLm) included in the transformer (TF) in a similar manner.

[0170] Step S200 may include the following steps S210 to S240.

[0171] In step S210, the transformer accelerator 100 may read multiple weight matrices corresponding to the executed decoder layer (DL) from one memory bank array (MBA). For example, the transformer accelerator 100 may read multiple weight matrices (e.g., a first query weight matrix (WMQ_L1), a first key weight matrix (WMK_L1), and a first value weight matrix (WMV_L1)) for the first decoder layer (DL1) from a first memory bank array (MBA1).

[0172] In step S220, the transformer accelerator 100 may generate a key-value vector pair (KVP) based on the activation vector and a plurality of weight matrices. For example, the first processing circuit 110 may generate a key-value vector pair (KVP_L1_TKn) based on the results of multiplying the activation vector (ACT1_TKn) by the first key-weight matrix (WMK_L1) and the first value-weight matrix (WMV_L1), respectively.

[0173] In step S230, the transformer accelerator 100 may store the key-value vector pairs (KVPs) in another memory bank array (MBA). That is, the transformer accelerator 100 may store the generated key-value vector pairs (KVPs) in a memory bank array different from the memory bank array in which the weight matrices for the currently executing decoder layer (DL) are stored. For example, the first processing circuit 110 may store the key-value vector pairs (KVP_L1_TKn) in the second memory bank array (MBA2). In this case, multiple weight matrices and key-value vector pairs (KVPs) accessed during execution of the first decoder layer (DL1) are stored in different memory banks.

[0174] In step S240, the transformer accelerator 100 may generate a next activation vector based on the key-value vector pair (KVP). For example, the second processing circuit 120 may perform an attention calculation (ATC) based on the generated key-value vector pair (KVP), and the transformer accelerator 100 may generate a next activation vector based on the result of the attention calculation (ATC). More specifically, for example, the second processing circuit 120 may perform an attention calculation (ATC) based on the key-value vector pair (KVP_L1_TKn) to generate an attention vector (AV1_TKn). In this case, the transformer accelerator 100 may generate an activation vector (ACT2_TKn) based on the attention vector (AV1_TKn). Step S240 will be described in more detail with reference to FIG. 17 below.

[0175] Figure 17 is a flowchart showing in more detail step S240 of Figure 16. With reference to Figures 1 to 9 and 13 to 17, step S240 may include the following steps S241 to S242.

[0176] In operation S241, the transformer accelerator 100 may read multiple previous key-value vector pairs from another memory bank array (MBA). For example, the second processing circuit 120 may read multiple previous key-value vector pairs stored through a previous iteration from the second memory bank array (MBA2). In this case, each previous key-value vector pair read by the second processing circuit 120 from the second memory bank array (MBA2) may correspond to the same decoder layer but may correspond to different tokens. More specifically, for example, the second processing circuit 120 may read key-value vector pairs (KVP_L1_TK1 to KVP_L1_TKn-1) from the second memory bank array (MBA2).

[0177] In step S242, the transformer accelerator 100 may perform an attention calculation (ATC) based on the key-value vector pair (KVP) and a plurality of previous key-value vector pairs to generate a next activation vector. For example, the second processing circuit 120 may perform an attention calculation (ATC) based on a plurality of previous key-value vector pairs (i.e., key-value vector pairs (KVP_L1_TK1 to KVP_L1_TKn-1)) and the key-value vector pair (KVP_L1_TKn). The specific manner in which the second processing circuit 120 performs the attention calculation has been described above with reference to Figures 7 to 8, and therefore a detailed description thereof will be omitted.

[0178] 18 is a block diagram illustrating a transformer accelerator according to an embodiment. Referring to FIG. 18, the transformer accelerator 200 can collectively receive input tokens (TKin) for each of a plurality of token streams and collectively output output tokens (TKout) for each of the plurality of token streams.

[0179] For the sake of simplicity, the following description will be directed to an embodiment in which Transformer accelerator 200 simultaneously generates output tokens (TKout) corresponding to the first and second token streams. However, the scope of the present disclosure is not limited to the number of token streams that Transformer accelerator 200 processes simultaneously.

[0180] For the sake of simplicity, the following description will be directed to an example in which the transformer accelerator 200 generates output tokens (TKout) for the first and second token streams in a batch scheme. Therefore, the first token stream will be referred to as the first batch (B1) and the second token stream will be referred to as the second batch (B2).

[0181] The transformer accelerator 200 may execute a transformer (TF). The transformer (TF) may generate an output token (TKout) suitable for following the input token (TKin) for each batch based on one or more input tokens (TKin) for each batch. For example, the transformer accelerator 200 may receive one or more input tokens (TKin_B1) for a first batch and output an output token (TKout_B1) for the first batch. The transformer accelerator 200 may receive one or more input tokens (TKin_B2) for a second batch and output an output token (TKout_B2) for the second batch.

[0182] That is, unlike the transformer accelerator 100 described with reference to Figures 1 to 17, the transformer accelerator 200 can perform operations on multiple token streams in a batch. For the sake of simplicity, the following description will mainly focus on the differences between the transformer accelerator 100 and the transformer accelerator 200.

[0183] Figure 19 is a block diagram showing in more detail the transformer accelerator of Figure 18. Referring to Figures 18 and 19, the transformer accelerator 200 may include a first processing circuit 210, a second processing circuit 220, a work scheduler 230, an interface circuit 240, a memory controller 250, and a memory device 260. For the sake of brevity, detailed descriptions of the first processing circuit 210, the second processing circuit 220, the work scheduler 230, the interface circuit 240, the memory controller 250, and the memory device 260 described above will be omitted.

[0184] The memory device 260 may include a first memory bank array (MBA1) and a second memory bank array (MBA2). The first memory bank array (MBA1) may include a first memory bank (MB1), and the second memory bank array (MBA2) may include a second memory bank (MB2).

[0185] The interface circuit 240 can receive a batch-specific input token (TKin) from an external device and can output a batch-specific output token (TKout) to the external device. For example, the interface circuit 240 can receive an input token (TKin_B1) for a first batch and an input token (TKin_B2) for a second batch from the external device and can output an output token (TKout_B1) for the first batch and an output token (TKout_B2) for the second batch to the external device.

[0186] The memory device 260 can store data that is repeatedly used in the operations performed by the first processing circuit 210 and the second processing circuit 220. For example, the memory device 260 can store a weight matrix and a key-value vector pair.

[0187] The work scheduler 230 can distribute and allocate data that may be accessed simultaneously by the first processing circuit 210 and the second processing circuit 220 to different memory bank arrays. In response to the control of the work scheduler 230, the memory device 260 can distribute and store data that may be accessed simultaneously by the first processing circuit 210 and the second processing circuit 220 to different memory bank arrays.

[0188] Figure 20 is a block diagram illustrating the operation of the Transformer of Figure 18 according to one embodiment. Referring to Figures 18 to 20, the Transformer (TF) can operate sequentially over multiple iterations. The Transformer (TF) can generate multiple output tokens (TKout) corresponding to multiple token streams, respectively, for each iteration.

[0189] The Transformer (TF) can perform multiple iterations for each batch using an auto-regression method. The Transformer (TF) can generate one output token (TKout) for each batch during each iteration. For example, the Transformer (TF_IT1) performing the first iteration can generate the p+1th token (TKp+1_B1) for the first batch based on the pth token (TKp_B1) for the first batch, and the q+1th token (TKq+1_B2) for the second batch based on the qth token (TKq_B2) for the second batch.

[0190] Then, the Transformer performing the second iteration (TF_IT2) can generate the p+2 token for the first batch (TKp+2_B1) based on the p+1 token for the first batch (TKp+1_B1), and the q+2 token for the second batch (TKq+2_B2) based on the q+1 token for the second batch (TKq+1_B2).

[0191] In this manner, the transformer accelerator 100 can execute a transformer (TF) multiple iterations to sequentially generate multiple tokens for each batch. In this case, multiple tokens corresponding to a first batch can form a first token sequence, and multiple tokens corresponding to a second batch can form a second token sequence. For example, the first through p+3 tokens (TK1_B1 through TKp+3_B1) for the first batch can form the first token sequence, and the first through q+3 tokens (TK1_B2 through TKq+3_B2) for the second batch can form the second token sequence.

[0192] In one embodiment, when the Transformer accelerator 100 is used to implement an LLM, different token sequences can correspond to sentences output to different users, although the scope of this disclosure is not limited in this respect.

[0193] In one embodiment, "p" and "q" may be the same or different integers, i.e., the transformer accelerator 100 can perform each iteration without regard to the relative lengths of the first and second token sequences.

[0194] In one embodiment, the tokens preceding any given token in a token sequence are referred to as its preceding tokens. For example, the first through p-1st tokens (TK1_B1 through TKp-1_B1) for the first batch are referred to as preceding tokens for the pth token (TKp_B1) for the first batch, and the first through pth tokens (TK1_B1 through TKp_B1) for the first batch are referred to as preceding tokens for the p+1th token (TKp+1_B1) for the first batch. Similarly, the first through q-1st tokens for the second batch (TK1_B2 through TKq-1_B2) are referred to as preceding tokens for the qth token for the second batch (TKq_B2), and the first through qth tokens for the second batch (TK1_B2 through TKq_B2) are referred to as preceding tokens for the q+1st token for the second batch (TKq+1_B2).

[0195] During one iteration, the Transformer (TF) can generate an output token (TKout) based on the input token (TKin) for each batch and its preceding tokens. For example, the Transformer (TF_IT1) performing the first iteration can generate the p+1st token (TKp+1_B1) for the first batch based on the pth token (TKp_B1) for the first batch and the 1st through the p-1st tokens (TK1_B1 through TKp-1_B1) for the first batch, and can generate the q+1st token (TKq+1_B2) for the second batch based on the qth token (TKq_B2) for the second batch and the 1st through the q-1st tokens (TK1_B2 through TKq-1_B2) for the second batch.

[0196] That is, during any iteration, the Transformer (TF) can repeatedly request operation results (e.g., key-value vector pairs (KVPs)) corresponding to the preceding tokens for the first batch and the second batch used in the preceding iteration. Thus, the Transformer (TF) stores the operation results for the input tokens for the first batch and the input tokens for the second batch at each iteration and then reuses them in subsequent iterations. The method by which the Transformer (TF) generates an output token for one batch during one iteration is similar to that described above with reference to FIGS. 1 to 17, and therefore a detailed description thereof will be omitted.

[0197] Figure 21 is a diagram showing the configuration of the transformer in Figure 20 in more detail. Hereinafter, the configuration and operation of the transformer (TF) performing the first iteration will be representatively described with reference to Figures 18 to 21. However, the scope of the present disclosure is not limited thereto, and the configuration and operation of the transformer (TF) can be realized in a similar manner even when performing other iterations.

[0198] The transformer (TF) may include a pre-hidden layer (HL_pre), first to m-th decoder layers (DL1 to DLm), and a post-hidden layer (HL_post).

[0199] The prehidden layer (HL_pre) can generate a corresponding activation vector (ACT1_TKp_B1) based on an input token (TKin_B1) for the first batch (e.g., the pth token (TKp_B1) for the first batch), and can generate a corresponding activation vector (ACT1_TKq_B2) based on an input token (TKin_B2) for the second batch (e.g., the qth token (TKq_B2) for the second batch).

[0200] The first to m-th decoder layers (DL1 to DLm) may sequentially convert each batch activation vector to generate an activation vector (ACTm+1_TKp_B1) for the first batch and an activation vector (ACTm+1_TKq_B2) for the second batch. The operation of each of the first to m-th decoder layers (DL1 to DLm) for one batch is similar to that described above with reference to FIG. 4, so detailed description will be omitted.

[0201] In one embodiment, the dimensions of activation vectors for different batches provided to a single decoder layer (DL) may be the same, for example, activation vector (ACT1_TKp_B1) and activation vector (ACT1_TKq_B2) may contain the same number of elements.

[0202] The first through m-th decoder layers (DL1 through DLm) may collectively generate calculation results for the input token (TKin) for each batch. For example, the first through m-th decoder layers (DL1 through DLm) may collectively calculate key-value vector pairs for the first batch and key-value vector pairs for the second batch. The manner in which the first through m-th decoder layers (DL1 through DLm) collectively calculate key-value vector pairs for different batches will be described in more detail below with reference to FIG. 22.

[0203] Each of the first to m-th decoder layers (DL1 to DLm) may store the generated key-value vector pairs in the memory device 260. Each of the first to m-th decoder layers (DL1 to DLm) may generate a batch-specific activation vector based on the generated batch-specific key-value vector pairs.

[0204] The post-hidden layer (HL_post) can calculate an output token (TKout_B1) for the first batch (e.g., the p+1th token (TKp+1_B1) for the first batch) based on the activation vector for the first batch (ACTm+1_TKp_B1), and can calculate an output token (TKout_B2) for the second batch (e.g., the q+1th token (TKq+1_B2) for the second batch) based on the activation vector for the second batch (ACTm+1_TKq_B2).

[0205] In one embodiment, the activation vector (ACTi_TKj_Bb) may refer to the activation vector provided to the (i)th decoder layer corresponding to the (j)th token (TKj_Bb) for the bth batch, where (i) is an integer between 1 and m. In one embodiment, the activation vector (ACTm+1_TKj_Bb) corresponds to the jth token (TKj) for the bth batch and may refer to the activation vector provided to the post-hidden layer (HL_post).

[0206] Figure 22 is a diagram illustrating in more detail the linear operation performed in one of the decoder layers of Figure 21. Hereinafter, with reference to Figures 18 to 22, an example in which the first decoder layer (DL1) performs linear operation (LNC) on the pth token (TKp_B1) for the first batch and the qth token (TKq_B2) for the second batch will be described as a representative example.

[0207] The first processing circuit 210 may read the first query weight matrix (WMQ_L1), the first key weight matrix (WMK_L1), and the first value weight matrix (WMV_L1) stored in the memory device 260.

[0208] The first processing circuit 210 can perform linear computation (LNC) based on the results of multiplying activation vectors for different batches by the same weight matrix, i.e., the first decoder layer (DL1) can compute query vectors for the first batch and query vectors for the second batch based on one query weight matrix, compute key vectors for the first batch and key vectors for the second batch based on one key weight matrix, and compute value vectors for the first batch and value vectors for the second batch based on one value weight matrix.

[0209] More specifically, the first processing circuit 210 can generate a query vector (QRY1_TKp_B1) and a query vector (QRY1_TKq_B2) based on the result of multiplying the activation vector (ACT1_TKp_B1) and the activation vector (ACT1_TKq_B2), respectively, by the first query weight matrix (WMQ_L1).

[0210] For example, the first processing circuit 210 may multiply a matrix generated by merging the activation vector (ACT1_TKp_B1) and the activation vector (ACT1_TKq_B2) by the first query weight matrix (WMQ_L1) to generate a matrix including the query vector (QRY1_TKp_B1) and the query vector (QRY1_TKq_B2). In this case, the first processing circuit 210 may read the first query weight matrix (WMQ_L1) from the memory device 260 once and collectively calculate multiple query vectors for different batches, thereby minimizing the number of accesses to the memory device 260 by the first processing circuit 210. Therefore, according to an embodiment of the present disclosure, the operating speed of the transformer accelerator 200 may be improved.

[0211] Similarly, the first processing circuit 210 can collectively calculate the key vector (KEY1_TKp_B1) and the key vector (KEY1_TKq_B2), and collectively calculate the value vector (VAL1_TKp_B1) and the value vector (VAL1_TKq_B2). In this case, the number of times the first processing circuit 210 accesses the memory device 260 to read the first key weight matrix (WMK_L1) and the first value weight matrix (WMV_L1) is minimized, thereby improving the operating speed of the transformer accelerator 200.

[0212] The key vector (KEY1_TKp_B1) and the value vector (VAL1_TKp_B1) are designated as the key-value vector pair (KVP1_TKp_B1) for the first batch. The key vector (KEY1_TKq_B2) and the value vector (VAL1_TKq_B2) are designated as the key-value vector pair (KVP1_TKq_B2) for the second batch. The first processing circuit 110 can store the key-value vector pair (KVP1_TKp_B1) and the key-value vector pair (KVP1_TKq_B2) in the memory device 260.

[0213] In this case, similar to what was previously described with reference to Figures 7 and 8, the key-value vector pair (KVP1_TKp_B1) can be reused when performing an attention calculation (ATC) for the first batch of the transformer (TF) performing a subsequent iteration, and the key-value vector pair (KVP1_TKq_B2) can be reused when performing an attention calculation (ATC) for the second batch of the transformer (TF) performing a subsequent iteration.

[0214] 23 is a diagram illustrating the memory device of FIG. 19 according to an embodiment of the present disclosure. Referring to FIGS. 18 to 23, the memory device 260 may include a first memory bank array (MBA1) and a second memory bank array (MBA2). The first memory bank array (MBA1) may include a first memory bank (MB1), and the second memory bank array (MBA2) may include a second memory bank (MB2). The first memory bank (MB1) may include first and second memory regions (MAa, MAb), and the second memory bank (MB2) may include a third and fourth memory region (MAc, MAd).

[0215] The memory device 260 may store a plurality of weight matrices and a plurality of key-value vector pairs (KVPs) for each decoder layer (DL) in different memory bank arrays. For example, the work scheduler 230 may classify the first through m-th decoder layers (DL1 through DLm) into first and second layer groups (LG1 and LG2). In this case, the memory device 260 may store a plurality of weight matrices and a plurality of key-value vector pairs (KVPs) for each layer group in different memory bank arrays.

[0216] That is, the memory device 260 can store a plurality of weight matrices for the first layer group (LG1) in the first memory bank array (MBA1), and can store a plurality of key-value vector pairs (KVPs) for the first layer group (LG1) of the first batch and the second batch in the second memory bank array (MBA2). For example, the memory device 260 can store a plurality of weight matrices for the first layer group (LG1) in the first memory bank (MB1), and can store a plurality of key-value vector pairs (KVPs) for the first layer group (LG1) of the first batch and the second batch in the second memory bank (MB2).

[0217] More specifically, for example, the memory device 260 may store a plurality of query weight matrices (WMQ_LG1), a plurality of key weight matrices (WMK_LG1), and a plurality of value weight matrices (WMV_LG1) for the first layer group (LG1) in the second memory area (MAb). The memory device 260 may store a plurality of key-value vector pairs (KVP_LG1_B1) for the first batch of the first layer group (LG1) and a plurality of key-value vector pairs (KVP_LG1_B2) for the second batch in the third memory area (MAc).

[0218] Conversely, the memory device 260 can store a plurality of weight matrices for the second layer group (LG2) in the second memory bank array (MBA2), and can store a plurality of key-value vector pairs (KVPs) for the second layer group (LG2) of the first batch and the second batch in the first memory bank array (MBA1). For example, the memory device 260 can store a plurality of weight matrices for the second layer group (LG2) in the second memory bank (MB2), and can store a plurality of key-value vector pairs (KVPs) for the second layer group (LG2) of the first batch and the second batch in the first memory bank (MB1).

[0219] More specifically, for example, the memory device 260 may store a plurality of query weight matrices (WMQ_LG2), a plurality of key weight matrices (WMK_LG2), and a plurality of value weight matrices (WMV_LG2) for the second layer group (LG2) in the fourth memory area (MAd). The memory device 260 may store a plurality of key-value vector pairs (KVP_LG2_B1) for the first batch of the second layer group (LG2) and a plurality of key-value vector pairs (KVP_LG2_B2) for the second batch in the first memory area (MAa).

[0220] That is, the memory device 260 can store key-value vector pairs for different batches included in the same layer group in the same memory bank array.

[0221] That is, multiple weight matrices and multiple key-value vector pairs (KVPs) for each decoder layer can be stored in different memory banks. In this case, simultaneous access to the same memory bank by the first processing circuit 210 and the second processing circuit 220 can be prevented. Therefore, according to the embodiment of the present disclosure, bank collision can be prevented.

[0222] In one embodiment, the first layer group (LG1) and the second layer group (LG2) may have the same number of decoder layers (DL). In this case, during one iteration, the transformer (TF) may generate the same number of key-value vector pairs (KVP_LG1_B1) for the first batch of the first layer group (LG1) as the same number of key-value vector pairs (KVP_LG2_B1) for the first batch of the second layer group (LG2). Similarly, during one iteration, the transformer (TF) may generate the same number of key-value vector pairs (KVP_LG1_B2) for the second batch of the first layer group (LG1) as the same number of key-value vector pairs (KVP_LG2_B2) for the second batch of the second layer group (LG2). Therefore, according to an embodiment of the present disclosure, the amount of data stored in the first memory bank array (MBA1) may be the same as the amount of data stored in the second memory bank array (MBA2).

[0223] 24 is a block diagram illustrating the memory device of FIG. 23 implemented according to an embodiment. Referring to FIGS. 18 to 24, the work scheduler 230 may classify the first through m-th decoder layers (DL1 through DLm) into an odd layer group and an even layer group (LGoddLGeven) according to the order of the first through m-th decoder layers (DL1 through DLm). In this case, the memory device 260 may store a plurality of weight matrices for the odd layer group (LGodd) in the first memory bank array (MBA1) and a plurality of key-value vector pairs (KVPs) for the odd layer group (LGodd) in the second memory bank array (MBA1).

[0224] More specifically, for example, the memory device 260 may store a plurality of query weight matrices (WMQ_LGodd), a plurality of key weight matrices (WMK_LGodd), and a plurality of value weight matrices (WMV_LGodd) for the odd layer group (LGodd) in the second memory area (MAb). The memory device 260 may store a plurality of key-value vector pairs (KVP_LGodd_B1) for the first batch of the odd layer group (LGodd) and a plurality of key-value vector pairs (KVP_LGodd_B2) for the second batch in the third memory area (MAc).

[0225] Conversely, the memory device 260 can store multiple weight matrices for the even layer group (LGeven) in the second memory bank array (MBA2) and multiple key-value vector pairs (KVPs) for the even layer group (LGeven) in the first memory bank array (MBA1).

[0226] More specifically, for example, the memory device 260 may store a plurality of query weight matrices (WMQ_LGeven), a plurality of key weight matrices (WMK_LGeven), and a plurality of value weight matrices (WMV_LGeven) for the even layer group (LGeven) in the fourth memory area (MAd). The memory device 260 may store a plurality of key-value vector pairs (KVP_LGeven_B1) for the first batch of the even layer group (LGeven) and a plurality of key-value vector pairs (KVP_LGeven_B2) for the second batch of the even layer group (LGeven) in the first memory area (MAa).

[0227] In one embodiment, “m” may be an even number. In this case, the number of decoder layers (DL) included in the odd layer group (LGodd) and the even layer group (LGeven) may be the same. In this case, the capacity of data stored in the first memory bank array (MBA1) may be the same as the capacity of data stored in the second memory bank array (MBA2).

[0228] Figure 25 is a block diagram showing the configuration of the Transformer of Figure 3 according to one embodiment. Below, the configuration and operation of a Transformer (TF) that operates in a multi-headed manner based on a single token stream will be described with reference to Figures 1 to 17 and 25. However, the scope of the present disclosure is not limited thereto, and the Transformer (TF) can operate in a similar manner for multiple token streams.

[0229] The transformer (TF) may include a pre-hidden layer (HL_pre), first to m-th decoder layers (DL1 to DLm), and a post-hidden layer (HL_post).

[0230] The prehidden layer (HL_pre) can generate a full activation vector (FACT) corresponding to the input token (TKin) based on the input token (TKin). For example, the prehidden layer (HL_pre) can generate a full activation vector (FACT1_TKn) based on the n-th token (TKn).

[0231] The first through m-th decoder layers (DL1 through DLm) may sequentially convert the full activation vectors (FACT1_TKn) to generate full activation vectors (FACTm+1_TKn). For example, the first decoder layer (DL1) may generate a full activation vector (FACT2_TKn) based on the full activation vector (FACT1_TKn), and the second decoder layer (DL2) may generate a full activation vector (FACT3_TKn) based on the full activation vector (FACT2_TKn). In this manner, the m-th decoder layer (DLm) may generate a full activation vector (FACTm+1_TKn) based on the full activation vector (FACTm_TKn).

[0232] In one embodiment, the full activation vectors (FACT1_TKn to FACTm+1_TKn) may have dimensions that are integer multiples of the dimensions of the activation vectors (ACT1_TKn to ACTm+1_TKn) previously described with reference to FIGS.

[0233] The specific configuration and operation of each of the first to m-th decoder layers (DL1 to DLm) will be described in more detail below with reference to FIGS. 26 to 29.

[0234] The post-hidden layer (HL_post) can calculate an output token (TKout) (e.g., the n+1th token (TKn+1)) based on the full activation vector (FACTm+1_TKn).

[0235] In one embodiment, the full activation vector (FACTi_TKj) may refer to the full activation vector provided to the (i)th decoder layer (DLi) corresponding to the (j)th token (TKj), where (i) is an integer between 1 and m.

[0236] In one embodiment, the full activation vector (FACTm+1_TKj) may refer to the full activation vector provided to the post-hidden layer (HL_post) corresponding to the jth token (TKj).

[0237] Figure 26 is a diagram illustrating in more detail the configuration of the decoder layer in Figure 25. Hereinafter, the configuration and operation of the first decoder layer (DL1) operating in a multi-head manner will be representatively described with reference to Figures 1 to 17 and Figures 25 to 26. However, the scope of the present disclosure is not limited thereto, and the second to mth decoder layers (DL2 to DLm) may also be realized in a similar manner.

[0238] The first decoder layer (DL1) may include linear calculation (LNC) and attention calculation (ATC).

[0239] The first decoder layer (DL1) can distribute linear computation (LNC) and attention computation (ATC) based on multiple heads. For easier explanation, it is assumed that the Transformer (TF) performs multi-head attention operations based on N heads. That is, it is assumed that the number of heads is N in the following.

[0240] The first decoder layer (DL1) may include linear operations for the first head (LNC_H1) through the Nth head (LNC_HN). Each of the linear operations for the first head (LNC_H1) through the Nth head (LNC_HN) may include the query vector generation (VG_QRY), key vector generation (VG_KEY), and value vector generation (VG_VAL) previously described with reference to FIGS. 1 through 17. Hereinafter, for easier explanation, the query vector generation (VG_QRY), key vector generation (VG_KEY), and value vector generation (VG_VAL) included in the linear operation for the i-th head (LNC_Hi) will be referred to as query vector generation for the i-th head (VG_QRY_Hi), key vector generation for the i-th head (VG_KEY_Hi), and value vector generation for the i-th head (VG_VAL_Hi), respectively.

[0241] In one embodiment, the linear calculation (LNC) can be performed by the first processing circuit 110. For example, the linear calculations for the first head (LNC_H1) through the Nth head (LNC_HN) can be performed by different processing cores included in the first processing circuit 110. However, the scope of the present disclosure is not limited thereto, and one or more processing cores included in the first processing circuit 110 can sequentially perform two or more of the linear calculations for the first head (LNC_H1) through the Nth head (LNC_HN).

[0242] The first decoder layer (DL1) can include attention calculations for the first head (ATC_H1) through the Nth head (ATC_HN). Each of the attention calculations for the first head (ATC_H1) through the Nth head (ATC_HN) can include the attention score calculation (CAL_ATS) and attention vector calculation (CAL_AV) previously described with reference to Figures 1 through 17. Hereinafter, for easier explanation, the attention score calculation (CAL_ATS) and attention vector calculation (CAL_AV) included in the attention calculation for the (i)th head (ATC_Hi) will be referred to as the attention score calculation for the (i)th head (CAL_ATS_Hi) and the attention vector calculation (CAL_AV_Hi), respectively.

[0243] In one embodiment, the attention calculation (ATC) can be performed by the second processing circuit 120. For example, the attention calculation for the first head (ATC_H1) through the attention calculation for the Nth head (ATC_HN) can be performed by different processing cores included in the second processing circuit 120. However, the scope of the present disclosure is not limited thereto, and one or more processing cores included in the second processing circuit 120 can sequentially perform two or more of the attention calculation for the first head (ATC_H1) through the attention calculation for the Nth head (ATC_HN).

[0244] FIG. 27 is a diagram showing the full activation vector of FIG. 25 in more detail. Referring to FIGS. 1 to 17 and 25 to 27, the full activation vector (FACT1_TKn) can be divided into multiple activation vectors corresponding to multiple heads. For the sake of simplicity, the dimension of each head is assumed to be "4." In other words, the dimension of a head is assumed to be "4" hereinafter. However, the scope of the present disclosure is not limited thereto.

[0245] The full activation vector (FACT1_TKn) can contain multiple elements. The full activation vector (FACT1_TKn) is divided into activation vectors, one for each element corresponding to the dimension of the head (e.g., 4). For example, the full activation vector (FACT1_TKn) is divided into activation vectors (ACT1_TKn_H1) to (ACT1_TKn_HN).

[0246] In one embodiment, the activation vector (ACTi_TKj_Hh) may refer to the activation vector provided to the (i)th decoder layer corresponding to the hth head and the (j)th token (TKj).

[0247] In one embodiment, the activation vector (ACT1_TKn) previously described with reference to FIGS. 1 to 17 may correspond to one of the activation vectors (ACT1_TKn_H1) to (ACT1_TKn_HN).

[0248] For the sake of simplicity, the configuration of the full activation vector (FACT1_TKn) is representatively illustrated in Fig. 27, but the scope of the present disclosure is not limited to this. For example, the full activation vectors (FACT2_TKn to FACTm+1_TKn) may also be divided and calculated for multiple heads in a similar manner.

[0249] Fig. 28 is a diagram showing the linear calculation of Fig. 26. Referring to Figs. 1 to 17 and 25 to 28, the full activation vector (FACT1_TKn) is divided into activation vectors (ACT1_TKn_H1) to (ACT1_TKn_HN).

[0250] When the first processing circuit 110 performs query vector generation (VG_QRY_H1) for the first head, the first processing circuit 110 may read the first query weight matrix (WMQ_L1_H1) stored in the memory device 160. In this case, the first query weight matrix (WMQ_L1_H1) for the first head may correspond to the first to fourth rows of the first full-query weight matrix (FWMQ_L1).

[0251] The first processing circuit 110 can generate a query vector (QRY1_TKn_H1) based on the result of multiplying the activation vector (ACT1_TKn_H1) by the first query weight matrix (WMQ_L1_H1) for the first head.

[0252] Similarly, when the first processing circuit 110 performs query vector generation (VG_QRY_H2) for the second head, the first processing circuit 110 may read the first query weight matrix (WMQ_L1_H2) for the second head stored in the memory device 160. In this case, the first query weight matrix (WMQ_L1_H2) for the second head may correspond to the fifth to eighth rows of the first full-query weight matrix (FWMQ_L1).

[0253] The first processing circuit 110 can generate a query vector (QRY1_TKn_H2) based on the result of multiplying the activation vector (ACT1_TKn_H2) by the first query weight matrix (WMQ_L1_H2) for the second head.

[0254] In this manner, the first processing circuit 110 can independently perform the query vector generation for the first head (VG_QRY_H1) through the query vector generation for the Nth head (VG_QRY_HN). For example, the first processing circuit 110 can perform the query vector generation for the first head (VG_QRY_H1) through the query vector generation for the Nth head (VG_QRY_HN) based on different processing cores or at different times.

[0255] In one embodiment, the query vector (QRY1_TKn) previously described with reference to FIGS. 1 to 17 may correspond to one of the query vectors (QRY1_TKn_H1) to (QRY1_TKn_H1).

[0256] In one embodiment, the product of the full activation vector (FACT1_TKn) and the first full-query weight matrix (FWMQ_L1) may correspond to a first full-query vector (hereinafter referred to as “FQRY1_TKn”). In this case, the first full-query vector (FQRY1_TKn) may include query vectors (QRY1_TKn_H1) through (QRY1_TKn_HN).

[0257] In one embodiment, the query vector (QRYi_TKj_Hh) may refer to the query vector corresponding to the (j)th token (TKj) generated at the hth head for the (i)th decoder layer (DLi).

[0258] For the sake of simplicity, FIG. 28 exemplarily illustrates query vector generation (VG_QRY) for full activation vector (FACT1_TKn). However, the scope of the present disclosure is not limited thereto, and the transformer (TF) can also perform key vector generation (VG_KEY) and value vector generation (VG_VAL) for full activation vector (FACT1_TKn) in a similar manner. For example, the first processing circuit 110 can independently generate key vectors (KEY1_TKn_H1) through (KEY1_TKn_HN) and independently generate value vectors (VAL1_TKn_H1) through (VAL1_TKn_HN).

[0259] In one embodiment, the key vector (KEYi_TKj_Hh) may refer to the key vector corresponding to the (j)th token (TKj) generated at the hth head for the (i)th decoder layer (DLi).

[0260] In one embodiment, the value vector (VALi_TKj_Hh) may refer to the value vector corresponding to the (j)th token (TKj) generated at the hth head for the (i)th decoder layer (DLi).

[0261] Figure 29 is a diagram showing the attention calculation of Figure 26. Referring to Figures 1 to 17 and 25 to 29, the second processing circuit 120 can perform an attention calculation (ATC_H1) for the first head based on the query vector (QRY1_TKn_H1), the key vector (KEY1_TKn_H1), and the value vector (VAL1_TKn_H1). That is, the second processing circuit 120 can generate an attention vector (AV1_TKn_H1) based on the query vector (QRY1_TKn_H1), the key vector (KEY1_TKn_H1), and the value vector (VAL1_TKn_H1).

[0262] More specifically, the second processing circuit 120 may calculate a plurality of attention scores corresponding to the key vector (KEY1_TKn_H1) based on the key vector (KEY1_TKn_H1) and its corresponding preceding key vector (e.g., key vectors (KEY1_TK1_H1 to KEY1_TKn-1_H1)). Then, the second processing circuit 120 may generate an attention vector (AV1_TKn_H1) based on the result of accumulating the value vector (VAL1_TKn_H1) and its corresponding preceding value vector (e.g., value vectors (VAL1_TK1_H1 to VAL1_TKn-1_H1)) based on the plurality of attention scores. The specific method of performing the attention calculation (ATC_H1) for the first head is similar to the attention calculation (ATC) previously described with reference to FIGS. 1 to 17, and therefore a detailed description thereof will be omitted.

[0263] In this manner, the second processing circuit 120 can independently perform attention calculations ATC_H1 to ATC_HN for the first to Nth heads. For example, the second processing circuit 120 can calculate attention vectors AV1_TKn_H1 to AV1_TKn_HN based on different processing cores or at different times.

[0264] The Transformer (TF) can then concatenate the attention vectors (AV1_TKn_H1) through (AV1_TKn_HN). For example, the Transformer (TF) can concatenate the attention vectors (AV1_TKn_H1) through (AV1_TKn_HN) to generate a full attention vector (FAV1_TKn). The Transformer (TF) can then generate a full activation vector (FACT2_TKn) based on the full attention vector (FAV1_TKn).

[0265] 30 is a diagram showing the transformer accelerator of FIG. 1 or 18 implemented according to an embodiment. Referring to FIGS. 1 to 30, the transformer accelerator 100 or the transformer accelerator 200 can be implemented as the transformer accelerator 300 of FIG.

[0266] The transformer accelerator 300 may include a first processing circuit 310, a second processing circuit 320, a work scheduler 330, an interface circuit 340, a memory controller 350, and a memory device 360. The configurations and operations of the first processing circuit 310, the second processing circuit 320, the work scheduler 330, the interface circuit 340, the memory controller 350, and the memory device 360 ​​are similar to those previously described with reference to FIGS. 1 to 29, and therefore detailed descriptions thereof will be omitted.

[0267] The memory device 360 ​​may include a first memory bank array (MBA1) and a second memory bank array (MBA2). Each of the first memory bank array (MBA1) and the second memory bank array (MBA2) may include a plurality of memory banks. Each of the plurality of memory banks included in the first memory bank array (MBA1) and the second memory bank array (MBA2) may operate independently of each other.

[0268] In one embodiment, the plurality of memory banks included in the first memory bank array (MBA1) and the plurality of memory banks included in the second memory bank array (MBA2) do not overlap with each other.

[0269] The work scheduler 130 can distribute and allocate data that may be accessed simultaneously by the first processing circuit 310 and the second processing circuit 320 to different memory bank arrays.

[0270] The memory device 360 ​​can store data that may be simultaneously accessed by the first processing circuit 310 and the second processing circuit 320 in different memory bank arrays in a distributed manner. That is, the first memory bank array (MBA1) and the second memory bank array (MBA2) can store a plurality of weight matrices and a plurality of key-value vector pairs in a distributed manner.

[0271] 1 to 29, the first memory bank array (MBA1) can store a plurality of weight matrices for the first layer group (LG1), and the second memory bank array (MBA2) can store a plurality of key-value vector pairs (KVPs) for the first layer group (LG1). The second memory bank array (MBA2) can store a plurality of weight matrices for the second layer group (LG2), and the first memory bank array (MBA1) can store a plurality of key-value vector pairs (KVPs) for the second layer group (LG2). In this case, simultaneous access to one memory bank by the first processing circuit 310 and the second processing circuit 320 is prevented in advance, thereby improving the operating efficiency of the transformer accelerator 300.

[0272] The memory controller 350 may distribute and store a plurality of key-value vector pairs (KVPs) for the second layer group (LG2) in the first memory bank array (MBA1). For example, the memory controller 350 may distribute and store a plurality of key-value vector pairs (KVPs) for the second layer group (LG2) in a plurality of memory banks included in the first memory bank array (MBA1) in an interleaved manner.

[0273] The memory controller 350 may distribute and store a plurality of key-value vector pairs (KVPs) for the first layer group (LG1) in the second memory bank array (MBA2). For example, the memory controller 350 may distribute and store a plurality of key-value vector pairs (KVPs) for the first layer group (LG1) in a plurality of memory banks included in the second memory bank array (MBA2) in an interleaved manner.

[0274] 31 is a diagram showing the transformer accelerator of FIG. 1 or 18 implemented according to an embodiment. Referring to FIGS. 1 to 29 and 31, the transformer accelerator 100 or the transformer accelerator 200 can be implemented as the transformer accelerator 400 of FIG.

[0275] The transformer accelerator 400 may include a first processing circuit 410, a second processing circuit 420, a work scheduler 430, an interface circuit 440, a memory controller 450, and a plurality of memory devices 460. The configurations and operations of the first processing circuit 410, the second processing circuit 420, the work scheduler 430, the interface circuit 440, the memory controller 450, and the memory devices 460 are similar to those previously described with reference to FIGS. 1 to 29, and therefore detailed description thereof will be omitted.

[0276] The memory controller 450 may control multiple memory devices 460. For example, the memory controller 450 may perform read and write operations on multiple memory devices 460.

[0277] Each of the plurality of memory devices 460 may include multiple memory banks.

[0278] In one embodiment, the work scheduler 430 may classify memory banks included in the plurality of memory devices 460 into a first memory bank array (MBA1) and a second memory bank array (MBA2). The memory controller 450 may distribute and store a plurality of weight matrices and a plurality of key-value vector pairs based on the first memory bank array (MBA1) and the second memory bank array (MBA2). However, the scope of the present disclosure is not limited to the specific manner in which the work scheduler 130 classifies the memory banks included in the plurality of memory devices 460.

[0279] In one embodiment, each of the plurality of memory devices 460 may store data necessary for the operation of a decoder layer (DL) for a different head. For example, the plurality of memory devices 460 may be configured to store weight matrices for the first through Nth heads, respectively, and may be configured to store key-value vector pairs for the first through Nth heads. In this case, each memory device 460 may include at least two or more memory bank arrays, and the weight matrices and key-value vector pairs for each decoder layer may be distributed and stored in at least two or more memory bank arrays.

[0280] The above-described content is a specific example for implementing the present disclosure. The present disclosure includes not only the above-described examples, but also examples that can be simply modified or easily changed. The present disclosure also includes techniques that can be easily implemented by using the examples. Therefore, the scope of the present disclosure should not be limited to the above-described examples, but should be determined not only by the claims below but also by equivalents of the claims of the present disclosure. [Explanation of symbols]

[0281] 100 Transformers Accelerator 110 First Processing Circuit 120 Second Processing Circuit 160 Memory Device TF Transformers DL decoder layer MBA Memory Bank Array

Claims

1. 1. A transformer accelerator for executing a transformer including a first plurality of decoder layers and a second plurality of decoder layers, a first memory bank array for storing a first plurality of weight matrices corresponding to the first plurality of decoder layers and a first plurality of key-value vector pairs corresponding to the second plurality of decoder layers; and a second memory bank array configured to store a second plurality of weight matrices corresponding to the second plurality of decoder layers and a second plurality of key-value vector pairs corresponding to the first plurality of decoder layers;

2. 2. The transformer accelerator of claim 1, further comprising a first processing circuit that computes the first plurality of key-value vector pairs based on the first plurality of weight matrices and computes the second plurality of key-value vector pairs based on the second plurality of weight matrices.

3. 3. The transformer acceleration device of claim 2, further comprising: a second processing circuit that performs an attention operation for the second plurality of decoder layers based on the first plurality of key-value vector pairs, and that performs an attention operation for the first plurality of decoder layers based on the second plurality of key-value vector pairs.

4. The transformer accelerator of claim 1 , wherein the number of the first plurality of decoder layers and the number of the second plurality of decoder layers correspond to each other.

5. The transformer accelerator of claim 3 , wherein the first and second decoder layers are alternately ordered.

6. 2. The transformer accelerator of claim 1, wherein a sum of the capacities of the first plurality of weight matrices and the first plurality of key-value vector pairs corresponds to a sum of the capacities of the second plurality of weight matrices and the second plurality of key-value vector pairs.

7. The first plurality of key-value vector pairs comprises: a first plurality of key vectors corresponding to a first token stream provided by an external device; and a first plurality of value vectors corresponding to the first token stream; The second plurality of key-value vector pairs comprises: a second plurality of key vectors corresponding to the first token stream; and 2. The transformer accelerator of claim 1, further comprising a second plurality of value vectors corresponding to said first token stream.

8. The first plurality of key-value vector pairs comprises: a third plurality of key vectors corresponding to the second token stream provided by the external device; and further comprising a third plurality of value vectors corresponding to the second token stream; The second plurality of key-value vector pairs comprises: a fourth plurality of key vectors corresponding to the second token stream; and 8. The transformer accelerator of claim 7, further comprising a fourth plurality of value vectors corresponding to the second token stream.

9. 8. The transformer acceleration device of claim 7, wherein the capacity of each of the first plurality of key vectors, the first plurality of value vectors, the second plurality of key vectors, and the second plurality of value vectors is smaller than the capacity of one memory cell row of a memory bank included in the first memory bank array and the second memory bank array.

10. 1. A transformer accelerator operating on a first activation vector corresponding to a first input token, comprising: a first memory bank array for storing a first plurality of weight matrices; a second memory bank array for storing a first plurality of preceding key-value vector pairs respectively corresponding to a first plurality of preceding tokens for the first input token; a first processing unit that generates a first key-value vector pair based on the first activation vector and the first plurality of weight matrices; and a second processing unit configured to perform a first attention operation based on the first key-value vector pair and the first plurality of preceding key-value vector pairs;

11. the first key-value vector pair includes a first key vector and a first value vector; the first plurality of preceding key-value vector pairs includes a first plurality of preceding key-vectors and a first plurality of preceding value vectors; The second processing unit generating a first plurality of attention scores corresponding to the first key vector and the first plurality of preceding key vectors, respectively; The transformer acceleration device of claim 10 , further comprising: accumulating the first value vector and the first plurality of preceding value vectors to generate a first attention vector based on the first plurality of attention scores.

12. The first processing unit 12. The transformer accelerator of claim 11, configured to store the first key vector and the first value vector in the second memory bank array.

13. the transformer accelerator further operates based on a second activation vector generated based on the first attention vector; the second memory bank array further stores a second plurality of weight matrices; the first memory bank array further stores a second plurality of preceding key-value vector pairs respectively corresponding to the first plurality of preceding tokens; the first processing unit further generates a second key-value vector pair based on the second activation vector and the second plurality of weight matrices; The transformer accelerator of claim 12 , wherein the second processing unit further performs a second attention operation based on the second key-value vector pair and the second plurality of preceding key-value vector pairs.

14. The second processing unit 14. The transformer accelerator of claim 13, configured to store the second key-value vector pair in the first memory bank array.

15. 12. The transformer accelerator of claim 11, further configured to generate a first output token corresponding to the first input token and the first plurality of preceding tokens based on the first attention vector.

16. 1. A method of operating a transformer accelerator device including a plurality of memory bank arrays, comprising: generating a first activation vector based on the first input token; reading a first plurality of weight matrices from a first memory bank array that is one of the plurality of memory bank arrays; generating a first key-value vector pair based on the first plurality of weight matrices and the first activation vector; storing the first key-value vector pair in a second memory bank array, the second memory bank array being one of the plurality of memory bank arrays and different from the first memory bank array; generating a second activation vector based on the first key-value vector pair; and A method of operation comprising generating a first output token corresponding to the first input token based on the second activation vector.

17. The step of generating the second activation vector comprises: reading from the second memory bank array a plurality of preceding key-value vector pairs respectively corresponding to a plurality of preceding tokens for the first input token; and 17. The method of claim 16, further comprising performing a first attention operation based on the plurality of preceding key-value vector pairs and the first key-value vector pair to generate the second activation vector.

18. generating a third activation vector based on the first output token; reading the first plurality of weight matrices from the first memory bank array; generating a third key-value vector pair based on the first plurality of weight matrices and the third activation vector; storing the third key-value vector pair in the second memory bank array; generating a fourth activation vector based on the third key-value vector pair; and 20. The method of claim 17, further comprising generating a second output token corresponding to the first output token based on the fourth activation vector.

19. The step of generating the fourth activation vector comprises: reading the preceding key-value vector pairs and the first key-value vector pair from the second memory bank array; and 20. The method of claim 18, further comprising: performing a second attention operation based on the plurality of preceding key-value vector pairs, the first key-value vector pair, and the third key-value vector pair to generate the fourth activation vector.

20. generating the first output token comprises: reading a second plurality of weight matrices from the second memory bank array; generating a second key-value vector pair based on the second plurality of weight matrices and the second activation vector; storing the second key-value vector pair in the first memory bank array; generating a fifth activation vector based on the second key-value vector pair; and 17. The method of claim 16, further comprising generating the first output token based on the fifth activation vector.

Images