Transformer accelerating apparatus

Abstract

To disclose a transformer accelerating apparatus for reading a key value vector.SOLUTION: A transformer accelerating apparatus according to the present invention has a memory device including first and second memory blocks for storing a plurality of cache vectors corresponding to a plurality of tokens and a memory striding circuit for accessing to the first and second memory blocks in response to a first striding request. The memory striding circuit has a memory block address management circuit for storing first and second memory block base addresses for the first and second memory blocks respectively, a target address generating circuit for calculating first and second target addresses based on the memory block base addresses and sub block offset in response to the striding request, and a command issuing circuit configured so as to issue a plurality of first and second memory access commands for the target sub block positioned at the target address.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] The present disclosure relates to a transformer accelerator that executes a transformer that generates an output token based on a plurality of input tokens. More particularly, the present disclosure relates to a transformer accelerator that includes a memory striding circuit configured to sequentially read a key-value vector cached for execution of the transformer. [Background technology]

[0002] A Transformer accelerator generates an output token based on multiple input tokens. For example, the Transformer accelerator may execute a Transformer based on the multiple input tokens to generate a first output token suitable for being followed by the multiple input tokens.

[0003] The Transformer may operate in an auto-regression manner. For example, the Transformer may further use a first output token along with multiple input tokens to generate a second output token suitable to be followed by the multiple input tokens and the first output token. That is, the Transformer may operate sequentially over multiple iterations, with the tokens generated by each iteration being used as input for the next iteration.

[0004] A transformer can reuse key-value vectors generated across multiple iterations. For example, a transformer can recycle a key-value vector calculated in a previous iteration in the next iteration. However, if the method by which a transformer stores and reads key-value vectors is not optimized, the storage and reading of the key-value vectors can reduce the operating efficiency of the transformer accelerator. Summary of the Invention [Problem to be solved by the invention]

[0005] The present disclosure is directed to solving the above-mentioned technical problems. More specifically, an object of the present disclosure is to provide a transformer accelerator including a memory striding circuit configured to read a key-value vector in an n-optimized manner. [Means for solving the problem]

[0006] A transformer accelerator according to an embodiment of the present disclosure may include a memory device including a first memory block storing a first plurality of cache vectors for a first plurality of tokens and a second memory block storing a second plurality of cache vectors for a second plurality of tokens; and a memory striding circuit configured to access the first and second memory blocks in response to a first striding request provided from an external device. In this case, the memory striding circuit may include: a memory block address management circuit storing a first memory block base address for the first memory block and a second memory block base address for the second memory block; a target address generation circuit configured to calculate a first target address included in the first memory block based on the first memory block base address and a first sub-block offset and to calculate a second target address included in the second memory block based on the second memory block base address and the first sub-block offset in response to the first striding request; and a command issuance circuit configured to issue a first plurality of memory access commands for a first target sub-block located at the first target address and a second plurality of memory access commands for a second target sub-block located at the second target address.

[0007] According to one embodiment of the present disclosure, a transformer accelerator device for executing multiple decoder layers, each including a multi-head attention operation performed based on multiple heads, may include: a first memory block including a first sub-block that stores a first plurality of cache vectors generated for a first plurality of tokens based on a first head that is one of the multiple heads and a first decoder layer that is one of the multiple decoder layers; a second memory block including a second sub-block that stores a second plurality of cache vectors generated for a second plurality of tokens based on the first head and the first decoder layer; a memory striding circuit that sequentially accesses the first sub-block and the second sub-block in response to a first striding request provided from an external device to read the first plurality of cache vectors and the second plurality of cache vectors; and an operation circuit that performs a first attention operation for the first head and the first decoder layer based on the first plurality of cache vectors and the second plurality of cache vectors.

[0008] A transformer accelerator device including a plurality of sub-blocks according to an embodiment of the present disclosure may include: a memory device including a plurality of memory blocks; a memory striding circuit that sequentially accesses the plurality of sub-blocks in response to a first striding request provided from an external device; and an arithmetic circuit that performs a first attention operation based on a first plurality of sub-blocks among the plurality of sub-blocks accessed by the memory striding circuit during a first time interval, and performs a second attention operation based on a second plurality of sub-blocks among the plurality of sub-blocks accessed by the memory striding circuit during a second time interval after the first time interval. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating a transformer accelerator according to an embodiment of the present disclosure. [Figure 2]FIG. 2 is a block diagram showing the transformer accelerator of FIG. 1 in more detail. [Figure 3] FIG. 2 is a block diagram illustrating the operation of the transformer of FIG. 1. [Figure 4] FIG. 4 is a block diagram illustrating the configuration of the transformer of FIG. 3 according to one embodiment. [Figure 5] FIG. 5 shows the decoder layer configuration of FIG. 4 in more detail. [Figure 6] FIG. 6 illustrates the linear calculation of FIG. 5 in more detail. [Figure 7] FIG. 6 illustrates the multi-head attention calculation of FIG. 5 in more detail. [Figure 8] FIG. 6 illustrates the multi-head attention calculation of FIG. 5 in more detail. [Figure 9] FIG. 8 is a diagram illustrating the attention score calculation of FIG. 7 in more detail. [Figure 10] FIG. 8 is a diagram illustrating the attention vector calculation of FIG. 7 in more detail. [Figure 11] FIG. 4 is a diagram illustrating the overall operation of the transformer of FIG. 3. [Figure 12] FIG. 2 is a block diagram illustrating the operation of a memory striding circuit in more detail, according to an embodiment of the present disclosure. [Figure 13] FIG. 3 is a block diagram illustrating the memory striding circuit of FIG. 2 in more detail. [Figure 14] FIG. 14 is a diagram showing a memory block base address table of FIG. 13. [Figure 15] FIG. 13 shows the memory block of FIG. 12 in more detail. [Figure 16] FIG. 16 is a diagram showing in more detail the configuration of the sub-blocks shown in FIG. 15. [Figure 17] FIG. 16 shows the sub-block group of FIG. 15 in more detail. [Figure 18] FIG. 16 shows the sub-block group of FIG. 15 in more detail. [Figure 19] FIG. 1 illustrates the operation of a memory striding circuit according to one embodiment. [Figure 20]FIG. 20 illustrates the operation of the memory striding circuit of FIG. 19 in more detail. [Figure 21] 20 is a flowchart showing the operation of the memory striding circuit of FIG. 19. [Figure 22] 22 is a flowchart showing step S120 of FIG. 21 in more detail. [Figure 23] FIG. 1 illustrates the operation of a memory striding circuit according to one embodiment. [Figure 24] FIG. 21 illustrates the operation of the memory striding circuit of FIG. 20 in more detail. [Figure 25] 25 is a flowchart showing the operation of the memory striding circuit of FIG. 24. [Figure 26] 26 is a flowchart showing step S220 of FIG. 25 in more detail. [Figure 27] FIG. 1 illustrates the operation of a memory striding circuit according to one embodiment. [Figure 28] FIG. 28 illustrates the operation of the memory striding circuit of FIG. 27 in more detail. [Figure 29] 28 is a flowchart showing the operation of the memory striding circuit of FIG. 27. [Figure 30] 30 is a flowchart showing step S320 of FIG. 29 in more detail. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present disclosure will be described in detail and clearly so that those skilled in the art can easily implement the present disclosure. Details such as detailed configurations and structures are provided solely for a general understanding of the embodiments of the present disclosure. Therefore, modifications of the embodiments described herein may be implemented by those skilled in the art without departing from the technical spirit and scope of the present disclosure. Furthermore, descriptions of well-known functions and structures are omitted for clarity and conciseness. Components in the following drawings or detailed description may be connected to other components in addition to those shown in the drawings or described in the detailed description. Terms used in this specification are defined in consideration of the functions of the present disclosure and are not limited to specific functions. The definitions of the terms can be determined based on the details described in the detailed description.

[0011] Components described with reference to terms such as "driver" or "block" in the detailed description may be embodied in the form of software, hardware, or a combination thereof. For example, software may be machine code, firmware, embedded code, and application software. For example, hardware may include electrical circuits, electronic circuits, processors, computers, integrated circuit cores, pressure sensors, inertial sensors, MEMS (Micro Electro Mechanical Systems), manual elements, or a combination thereof.

[0012] FIG. 1 is a block diagram illustrating a Transformer accelerator according to an embodiment of the present disclosure. Referring to FIG. 1, the Transformer accelerator 100 can receive one or more input tokens TKin and output an output token TKout. For example, the Transformer accelerator 100 can execute a Transformer TF. Based on the one or more input tokens TKin, the Transformer TF can generate an output token TKout suitable for being followed by the one or more input tokens TKin.

[0013] The Transformer TF may operate in an auto-regression manner. For example, the Transformer TF may generate a token following the output token TKout based on the generated output token TKout. In this manner, the Transformer TF may generate one output token TKout each time the Transformer TF performs one iteration (i.e., one operation cycle). In this manner, the Transformer TF may perform multiple iterations sequentially to sequentially generate multiple output tokens TKout. A specific operation manner of the Transformer TF will be described in more detail below with reference to FIG. 3.

[0014] In one embodiment, the Transformer accelerator 100 can be used to implement a large language model (LLM). For example, the Transformer accelerator 100 can predict an output token TKout that follows multiple input tokens TKin based on multiple input tokens TKin. However, the scope of the present disclosure is not limited to the specific type of model in which the Transformer accelerator 100 is used. For example, the Transformer accelerator 100 can be used to implement any type of artificial intelligence model, such as an image generation model, a translation model, etc. However, for the sake of simplicity, the following description will representatively describe an embodiment in which the Transformer accelerator 100 is used to implement an LLM.

[0015] Figure 2 is a block diagram showing in more detail the transformer accelerator of Figure 1. Referring to Figures 1 and 2, the transformer accelerator 100 may include a processing circuit 110, an arithmetic circuit 120, a memory striding circuit 130, a host interface circuit 140, a memory controller 150, and a memory device 160. The processing circuit 110, the arithmetic circuit 120, the memory striding circuit 130, the host interface circuit 140, and the memory controller 150 may be connected via a bus.

[0016] The processing circuit 110 can control the overall operation of the transformer accelerator 100. The processing circuit 110 can schedule tasks or operations required to drive the transformer TF. That is, the processing circuit 110 can assign tasks or operations required to drive the transformer TF to the operation circuit 120, the memory striding circuit 130, and the memory controller 150. For example, to assign tasks to the memory striding circuit 130, the processing circuit 110 can transmit a striding request REQ_STRD to the memory striding circuit 130.

[0017] The calculation circuitry 120 may perform various types of calculations, such as linear calculations, attention calculations, etc. For example, the calculation circuitry 120 may include dedicated hardware optimized for linear calculations and / or attention calculations.

[0018] In one embodiment, the computing circuitry 120 may include one or more processing cores included in various types of processing units, such as a graphics processing unit (GPU), a central processing unit (CPU), etc. That is, the scope of the present disclosure is not limited to the specific implementation of the computing circuitry 120.

[0019] 2 illustrates the processing circuitry 110 as a separate component from the arithmetic circuitry 120 for ease of explanation, but the scope of the present disclosure is not limited thereto. For example, the processing circuitry 110 and the arithmetic circuitry 120 may be configured as separate hardware components or may be implemented as a single piece of hardware.

[0020] The memory striding circuit 130 may repeatedly access the memory device 160 in response to control by the processing circuit 110. For example, in response to a striding request REQ_STRD, the memory striding circuit 130 may sequentially access data stored in non-consecutive addresses in the memory device 160. That is, in response to the striding request REQ_STRD, the memory striding circuit 130 may sequentially issue a plurality of memory access commands CMD_MA to the memory device 160 via the memory controller 150. The configuration and operation of the memory striding circuit 130 will be described in more detail with reference to the following drawings.

[0021] In one embodiment, each of the plurality of memory access commands may be a read command or an activate command.

[0022] In one embodiment, the memory striding circuit 130 may be implemented as an intellectual property (IP) circuit configured to sequentially access non-consecutive addresses of the memory device 160 in response to a striding request REQ_STRD. However, the scope of the present disclosure is not limited to a specific implementation of the memory striding circuit 130.

[0023] Host interface circuit 140 may support interfacing with a host device (e.g., a central processing unit) external to transformer accelerator 100. For example, host interface circuit 140 may receive an input token TKin from the host device and output an output token TKout to the host device.

[0024] In one embodiment, the host interface circuit 140 can communicate with a host device based on various types of communication interfaces, such as PCIe (peripheral component interconnect express), DDR (double data rate), etc. However, the scope of the present disclosure is not limited to the specific operation method of the host interface circuit 140.

[0025] The memory controller 150 may control the memory device 160 in response to requests from the processing circuit 110, the arithmetic circuit 120, and the memory striding circuit 130. For example, the memory controller 150 may store data in the memory device 160 or read data stored in the memory device 160.

[0026] The memory device 160 can be used as an operating memory for the transformer accelerator 100. For example, the memory device 160 can store data that the transformer TF reuses across multiple iterations. More specifically, the memory device 160 can cache (e.g., store) key vectors KEY and value vectors VAL that are reused by the arithmetic circuit 120 across multiple iterations of the transformer TF. In this case, the arithmetic circuit 120 may not re-calculate the key vectors KEY and value vectors VAL stored in the memory device 160 during subsequent iterations, thereby improving the operating speed of the transformer accelerator 100.

[0027] Hereinafter, for the sake of simplicity, the key vector KEY, the value vector VAL, and any combination thereof stored in memory device 160 may be referred to as a key-value vector, a key-value cache, a cache vector, or a key-value cache vector. That is, each of the terms "key-value vector," "key-value cache," "cache vector," and "key-value cache vector" may refer to one or more key vectors KEY, one or more value vectors VAL, or one or more pairs of key vectors KEY and value vectors VAL. However, the scope of the present disclosure is not limited to such terms.

[0028] In one embodiment, the memory device 160 can pre-store multiple key vectors KEY and multiple value vectors VAL that are reused across multiple iterations during the training phase of the transformer TF.

[0029] In one embodiment, the memory device 160 can pre-store multiple key vectors KEY and multiple value vectors VAL that are reused in subsequent iterations during each iteration of the transformer TF.

[0030] In one embodiment, the memory device 160 may further store a weight matrix that is repeatedly reused in a decoder layer included in the transformer TF.

[0031] In one embodiment, memory device 160 may be a dynamic random access memory (DRAM) device, although the scope of the present disclosure is not limited to any particular type of memory device 160.

[0032] In one embodiment, the memory device 160 may include multiple memory blocks having different addresses, and multiple key vectors KEY and multiple value vectors VAL may be stored in a distributed manner across the multiple memory blocks.

[0033] In one embodiment, the memory striding circuit 130 may sequentially access multiple memory blocks in response to a single striding request REQ_STRD. That is, the memory striding circuit 130 may sequentially issue multiple memory access commands CMD_MA to multiple memory blocks having different addresses in response to a single striding request REQ_STRD. In this case, data stored in the multiple memory blocks may be sequentially read without the processing circuit 110 directly issuing multiple memory access commands CMD_MA to the different addresses. Therefore, according to an embodiment of the present disclosure, the processing load of the processing circuit 110 may be minimized.

[0034] For the sake of simplicity, FIG. 2 shows an example in which the striding request REQ_STRD is issued from the processing circuit 110, but the scope of the present disclosure is not limited thereto. For example, the striding request REQ_STRD may be issued from a host device external to the transformer accelerator 100. In this case, the striding request REQ_STRD may be transmitted to the memory striding circuit 130 via the processing circuit 110, or may be transmitted directly from the host interface circuit 140 to the memory striding circuit 130, unlike the example shown in FIG. 2.

[0035] Figure 3 is a block diagram showing the operation of the transformer in Figure 1. Referring to Figures 1 to 3, the transformer TF can operate sequentially over multiple iterations. Hereinafter, the operation of the transformers TF_IT1 to TF_IT3, which perform the first to third iterations consecutively, will be representatively described.

[0036] For the sake of simplicity, it is assumed below that the n-th token TKn is generated through the summarization stage of the Transformer TF, and that the first to third iterations are included in the generation stage. For example, it is assumed below that a token stream including the first to (n-1)-th tokens TK1 to TKn-1 is provided from outside the Transformer accelerator 100 (e.g., as a prompt), and that the n-th token TKn is generated by performing the 0-th iteration prior to the first iteration based on the token stream. However, the scope of the present disclosure is not limited thereto.

[0037] The Transformer TF can perform multiple iterations in an auto-regression manner. The Transformer TF can generate one output token TKout for each of multiple iterations. For example, when performing one iteration, the Transformer TF can use the output token TKout generated through the previous iteration as the input token TKin.

[0038] For example, a Transformer TF_IT1 performing a first iteration may use the nth token TKn generated through an iteration prior to the first iteration (e.g., the 0th iteration) as an input token TKin to generate the n+1th token TKn+1. Similarly, a Transformer TF_IT2 performing a second iteration may use the n+1st token TKn+1 as an input token TKin to generate the n+2nd token TKn+2; a Transformer TF_IT3 performing a third iteration may use the n+2nd token TKn+2 as an input token TKin to generate the n+3rd token TKn+3.

[0039] In this manner, the Transformer accelerator 100 can execute the Transformer TF for multiple iterations to sequentially generate multiple tokens. In this case, the multiple tokens generated by the Transformer TF can form a single token sequence together with a token stream (e.g., the first through (n-1)th tokens TK1 through TKn-1) provided from an external device. For example, the first through (n+2)th tokens TK1 through TKn+2 can form a single token sequence.

[0040] In one embodiment, when the Transformer accelerator 100 is used to implement an LLM, a token sequence can correspond to one or more sentences, although the scope of the present disclosure is not limited in this respect.

[0041] In one embodiment, the maximum length of the token sequence may be predetermined. For example, the maximum length of the token sequence may be determined by the number of tokens that the Transformer TF can process at one time. However, the scope of the present disclosure is not limited in this respect, and the maximum length of the token sequence may be determined by other components of the Transformer accelerator 100 or a user of the Transformer accelerator 100.

[0042] In one embodiment, tokens preceding any token in a token sequence may be referred to as its preceding tokens. For example, the first through (n-1)th tokens TK1 through TKn-1 may be referred to as preceding tokens for the nth token TKn, and the first through (n+1)th tokens TK1 through TKn+1 may be referred to as preceding tokens for the n+2th token TKn+2.

[0043] During one iteration, the Transformer TF may generate an output token TKout based on the input token TKin and the preceding tokens for the input token TKin. For example, the Transformer TF_IT1 performing the first iteration may generate the (n+1)th token TKn+1 based on the nth token TKn and the first through (n-1)th tokens TK1 through TKn-1. Similarly, the Transformer TF_IT2 performing the second iteration may generate the (n+2)th token TKn+2 based on the n+1st token TKn+1 and the first through nth tokens TK1 through TKn.

[0044] Therefore, when the transformer TF performs an arbitrary iteration, the calculation results for the multiple tokens used in the previous iteration may be repeatedly requested. For example, when the transformer TF performs an arbitrary iteration, the multiple key vectors KEY and multiple value vectors VAL corresponding to the multiple tokens used in the previous iteration may be repeatedly requested.

[0045] Therefore, when the Transformer TF performs any iteration, the Transformer TF can reuse the multiple key vectors KEY and multiple value vectors VAL calculated during the previous iteration. For example, the Transformer TF_IT1 performing the first iteration can reuse the multiple key vectors KEY and multiple value vectors VAL corresponding to the first through n-th tokens TK1 through TKn-1, and the Transformer TF_IT2 performing the second iteration can reuse the multiple key vectors KEY and multiple value vectors VAL corresponding to the first through n-th tokens TK1 through TKn.

[0046] More specifically, after each iteration, the Transformer TF may cache (e.g., store) a plurality of key vectors KEY and a plurality of value vectors VAL corresponding to the input token TKin in the memory device 160. In this case, the amount of calculations required by the Transformer TF to perform subsequent iterations may be minimized. The operation of the Transformer TF for each iteration will be described in more detail below.

[0047] First, before performing the first iteration (e.g., while performing the 0th iteration corresponding to the summarization stage), the transformer TF can store the preceding tokens for the nth token TKn (i.e., the key-value vectors for the first to (n-1)th tokens TK1 to TKn-1) in the memory device 160.

[0048] Thereafter, the Transformer TF_IT1 performing the first iteration may read the key-value vectors for the first through (n-1)th tokens TK1 through TKn-1 stored in the memory device 160 and generate the key-value vectors for the nth token TKn. In this case, the Transformer TF_IT1 performing the first iteration may generate the (n+1)th token TKn+1 without calculating the key-value vectors for the first through (n-1)th tokens TK1 through TKn-1. Meanwhile, the Transformer TF_IT1 performing the first iteration may further store the key-value vectors for the nth token TKn in the memory device 160.

[0049] Thereafter, the Transformer TF_IT2 performing the second iteration may read the key-value vectors for the first through n-th tokens TK1 through TKn stored in the memory device 160 and generate key-value vectors for the (n+1)th token TKn+1. In this case, the Transformer TF_IT2 performing the second iteration may generate the (n+2)th token TKn+2 without calculating the key-value vectors for the first through n-th tokens TK1 through TKn. Meanwhile, the Transformer TF_IT2 performing the second iteration may further store the key-value vectors for the (n+1)th token TKn+1 in the memory device 160.

[0050] In this manner, the transformer TF_IT3 performing the third iteration can read key-value vectors for the first to n+1th tokens TK1 to TKn+1 from the memory device 160 and store key-value vectors for the n+2nd token TKn+2 in the memory device 160.

[0051] That is, according to an embodiment of the present disclosure, during each iteration, the transformer TF can read key-value vectors corresponding to the preceding tokens for the output token TKout of the previous iteration from the memory device 160. Then, the transformer TF can generate key-value vectors corresponding to the output token TKout of the previous iteration and store them in the memory device 160.

[0052] 4 is a block diagram illustrating the configuration of the Transformer TF of FIG. 3 according to an embodiment. Hereinafter, the configuration and operation of the Transformer TF performing the first iteration will be described representatively with reference to FIGS. 1 to 4. However, the scope of the present disclosure is not limited thereto, and Transformer TFs performing other iterations may be implemented in a similar manner.

[0053] For the sake of simplicity, the following description will focus on the configuration and operation of a Transformer TF that processes one token stream. However, the scope of the present disclosure is not limited to this, and the Transformer TF may also be implemented to process multiple token streams simultaneously in a batching manner.

[0054] The transformer TF may include a pre-hidden layer HL_pre, first to m-th decoder layers DL1 to DLm, and a post-hidden layer HL_post.

[0055] The pre-hidden layer HL_pre may generate a full activation vector FACT corresponding to the input token TKin based on the input token TKin. For example, the pre-hidden layer HL_pre may generate a full activation vector FACT_L1_TKn based on the n-th token TKn.

[0056] In one embodiment, the pre-hidden layer HL_pre can perform various types of token processing operations, such as token embedding operations, positional encoding operations, and the like.

[0057] The first through m-th decoder layers DL1 through DLm may sequentially convert the full activation vector FACT_L1_TKn to generate a full activation vector FACT_Lm+1_TKn. More specifically, the (k)th decoder layer DLk may generate a full activation vector FACT_Lk+1_TKn based on the full activation vector FACT_Lk_TKn (where "k" is an integer between 1 and m). For example, the first decoder layer DL1 may generate a full activation vector FACT_L2_TKn based on the full activation vector FACT_L1_TKn, and the second decoder layer DL2 may generate a full activation vector FACT_L3_TKn based on the full activation vector FACT_L2_TKn. In this manner, the m-th decoder layer DLm can generate a full activation vector FACT_Lm+1_TKn based on the full activation vector FACT_Lm_TKn.

[0058] In one embodiment, the dimensions of the full activation vectors FACT_L1_TKn through FACT_Lm+1_TKn may be the same as each other, e.g., the full activation vectors FACT_L1_TKn through FACT_Lm+1_TKn may each include the same number of elements as each other.

[0059] In one embodiment, each of the first through m-th decoder layers DL1 through DLm may generate a plurality of key vectors KEY and a plurality of value vectors VAL for an input token TKin (e.g., the n-th token TKn) based on the provided full activation vector FACT. Each of the first through m-th decoder layers DL1 through DLm may store the generated plurality of key vectors KEY and a plurality of value vectors VAL in the memory device 160. In this case, the transformer TF may not calculate the plurality of key vectors KEY and a plurality of value vectors VAL corresponding to the n-th token TKn when performing subsequent iterations (e.g., the second and third iterations of FIG. 3). The specific configurations and operations of each of the first through m-th decoder layers DL1 through DLm will be described in more detail below with reference to FIGS. 5 through 11.

[0060] The post hidden layer HL_post can compute an output token TKout (eg, the n+1th token TKn+1) based on the full activation vector FACT_Lm+1_TKn.

[0061] The post hidden layer HL_post in one embodiment can perform various types of vector processing operations such as layer normalizing operations, fully connected network computation operations, and the like.

[0062] Figure 5 is a diagram illustrating in more detail the configuration of the decoder layer of Figure 4. Hereinafter, the configuration and operation of the first decoder layer DL1 will be representatively described with reference to Figures 1 to 5. However, the scope of the present disclosure is not limited thereto, and the second to mth decoder layers DL2 to DLm may also be embodied in a similar manner.

[0063] The first decoder layer DL1 may include a linear calculation (LNC) and a multi-head attention calculation (MHATC). The linear calculation LNC and the multi-head attention calculation MHATC can be performed by the calculation circuit 120.

[0064] Linear operation LNC can refer to the operation of multiplying the full activation vector FACT by various kinds of weight matrices. Linear operation LNC will be explained in more detail with reference to FIG. 6 below.

[0065] The multi-head attention computation MHATC can be referred to as a computation that reflects the interrelationship between an input token TKin (e.g., the nth token TKn) and its preceding tokens based on multiple heads. For a simpler explanation, it is assumed below that the transformer TF performs the multi-head attention computation MHATC based on N heads. That is, the number of heads in the following description is assumed to be N. The multi-head attention computation MHATC will be described in more detail with reference to FIGS. 7 to 10 below.

[0066] In the following, for the sake of concise explanation, the linear operation LNC performed in the (k)th decoder layer DLk may be referred to as the linear operation LNC_Lk, and the multi-head attention operation MHATC performed in the (k)th decoder layer DLk may be referred to as the multi-head attention operation MHATC_Lk.

[0067] 6 is a diagram illustrating the linear calculation of FIG. 5 in more detail. Referring to FIGS. 1 to 6, the calculation circuit 120 can perform the linear calculation LNC_L1. For example, the calculation circuit 120 can multiply the full activation vector FACT_L1_TKn by the query weight matrix WM_QRY to generate a full query vector FQRY_L1_TKn; multiply the full activation vector FACT_L1_TKn by the key weight matrix WM_KEY to generate a full key vector FKEY_L1_TKn; and multiply the full activation vector FACT_L1_TKn by the value weight matrix WM_VAL to generate a full value vector FVAL_L1_TKn.

[0068] In one embodiment, the computation circuitry 120 can read the query weight matrix WM_QRY, the key weight matrix WM_KEY, and the value weight matrix WM_VAL from the memory device 160.

[0069] In one embodiment, the weight matrices used by the transformers TF for the linear operations LNC included in different decoder layers DL may be different from each other. For example, the key weight matrix WM_KEY used for the linear operation LNC_L1 may be different from the key weight matrix WM_KEY used for the linear operation LNC_L2.

[0070] In one embodiment, the weight matrices used by the transformer TF to perform linear computation LNC of the same decoder layer DL included in each of the multiple iterations may be the same. For example, the transformer TF may perform linear computation LNC of the first decoder layer DL1 included in each of the multiple iterations based on the same weight matrix.

[0071] Each of the full query vector FQRY_L1_TKn, the full key vector FKEY_L1_TKn, and the full value vector FVAL_L1_TKn may be divided into a plurality of vectors corresponding to a plurality of heads. For a simpler explanation, the following representatively describes an embodiment in which each of the full query vector FQRY_L1_TKn, the full key vector FKEY_L1_TKn, and the full value vector FVAL_L1_TKn is divided into a plurality of vectors each having four elements. That is, the following assumes that the dimension of the head is "4." However, the scope of the present disclosure is not limited thereto.

[0072] The full query vector FQRY_L1_TKn may be split into query vectors QRY_L1_TKn_H1 through QRY_L1_TKn_HN; the full key vector FKEY_L1_TKn may be split into key vectors KEY_L1_TKn_H1 through KEY_L1_TKn_HN; and the full value vector FVAL_L1_TKn may be split into value vectors VAL_L1_TKn_H1 through VAL_L1_TKn_HN, where each of the query vectors QRY_L1_TKn_H1 through QRY_L1_TKn_HN, key vectors KEY_L1_TKn_H1 through KEY_L1_TKn_HN, and value vectors VAL_L1_TKn_H1 through VAL_L1_TKn_HN may contain four elements.

[0073] In one embodiment, the full query vector FQRY_L1_TKn may have a dimension N times that of each of the query vectors QRY_L1_TKn_H1 to QRY_L1_TKn_HN.

[0074] In one embodiment, the full key vector FKEY_L1_TKn may have dimensions N times larger than those of each of the key vectors KEY_L1_TKn_H1 to KEY_L1_TKn_HN.

[0075] In one embodiment, the full value vector FVAL_L1_TKn may have dimensions N times larger than those of each of the value vectors VAL_L1_TKn_H1 to VAL_L1_TKn_HN.

[0076] 7-8 show the multi-head attention computation of FIG. 5 in more detail.

[0077] 1 to 7, the first decoder layer DL1 may include N heads. The calculation circuit 120 may perform a multi-head attention calculation MHATC_L1 for the first decoder layer DL1 based on the N heads. For example, the multi-head attention calculation MHATC_L1 may include a plurality of attention calculations ATC_L1_H1 to ATC_L1_HN corresponding to the first to Nth heads (hereinafter referred to as H1 to HN, respectively). The calculation circuit 120 may independently perform the plurality of attention calculations ATC_L1_H1 to ATC_L1_HN.

[0078] The attention calculation ATC_L1_H1 may include an attention score calculation CAL_ATS_L1_H1 and an attention vector calculation CAL_AV_L1_H1. For example, the calculation circuit 120 may perform the attention score calculation CAL_ATS_L1_H1 to calculate a plurality of attention scores; and may perform the attention vector calculation CAL_AV_L1_H1 to generate an attention vector based on the calculated plurality of attention scores. The attention score calculation CAL_ATS_L1_H1 will be described in more detail with reference to FIG. 9 below, and the attention vector calculation CAL_AV_L1_H1 will be described in more detail with reference to FIG. 10 below.

[0079] For the sake of simplicity, the configuration of the attention calculation ATC_L1_H1 corresponding to the first head has been described as a representative example, but the scope of the present disclosure is not limited to this, and the attention calculations ATC_L1_H2 to ATC_L1_HN may also be configured in a similar manner. That is, the calculation circuit 120 can perform the attention score calculation CAL_ATS and the attention vector calculation CAL_AV for each of the multiple heads.

[0080] 1 to 8, the calculation circuit 120 can perform a plurality of attention calculations ATC_L1_H1 to ATC_L1_HN based on the first to N-th heads H1 to HN.

[0081] The arithmetic circuit 120 can perform a plurality of attention calculations ATC_L1_H1 to ATC_L1_HN based on the query vectors QRY_L1_TKn_H1 to QRY_L1_TKn_HN, the key vectors KEY_L1_TKn_H1 to KEY_L1_TKn_HN, and the value vectors VAL_L1_TKn_H1 to VAL_L1_TKn_HN, which were previously described with reference to Fig. 6. For example, the arithmetic circuit 120 can perform the attention calculation ATC_L1_H1 corresponding to the first head H1 of the first decoder layer DL1 based on the query vector QRY_L1_TKn_H1, the key vector KEY_L1_TKn_H1, and the value vector VAL_L1_TKn_H1 to generate the attention vector AV_L1_TKn_H1. The manner in which the arithmetic circuit 120 performs the attention calculation ATC_L1_H1 is described in more detail below with reference to FIGS.

[0082] Similarly, the calculation circuit 120 can perform attention calculation ATC_L1_H2 corresponding to the second head H2 of the first decoder layer DL1 based on the query vector QRY_L1_TKn_H2, the key vector KEY_L1_TKn_H2, and the value vector VAL_L1_TKn_H2 to generate an attention vector AV_L1_TKn_H2.

[0083] In this manner, the calculation circuit 120 can independently perform the attention calculations ATC_L1_H1 to ATC_L1_HN. For example, the calculation circuit 120 can calculate the attention vectors AV_L1_TKn_H1 to AV_L1_TKn_HN based on different processing cores or at different times.

[0084] The Transformer TF can then concatenate the attention vectors AV_L1_TKn_H1 through AV_L1_TKn_HN. For example, the Transformer TF can concatenate the attention vectors AV_L1_TKn_H1 through AV_L1_TKn_H to generate a full attention vector FAV_L1_TKn. The Transformer TF can then perform one or more of various types of vector processing algorithms, such as a residual layer, a fully-connected layer, etc., based on the full attention vector FAV_L1_TKn to generate a full activation vector FACT_L2_TKn.

[0085] In one embodiment, attention calculation ATC performed based on one head H can be referred to as single-head attention. For example, attention calculation ATC_L1_H1 performed based on the first head H1 can be referred to as single-head attention for the first head H1. However, the scope of the present disclosure is not limited to such terminology.

[0086] Figure 9 is a diagram illustrating in more detail the attention score calculation of Figure 7. That is, hereinafter, with reference to Figures 1 to 9, the attention score calculation CAL_ATS_L1_H1 performed in the first head H1 of the first decoder layer DL1 based on the n-th token TKn will be representatively described.

[0087] The calculation circuit 120 can generate attention scores ATS1_L1_TKn_H1 to ATSn_L1_TKn_H1 corresponding to the n-th token TKn and its preceding tokens (e.g., the 1st to n-1th tokens TK1 to TKn-1) based on the query vector QRY_L1_TKn_H1. For example, the calculation circuit 120 can calculate the attention score ATS1_L1_TKn_H1 based on the result of inner product of the query vector QRY_L1_TKn_H1 and the key vector KEY_L1_TK1_H1; The calculation circuit 120 may calculate an attention score ATS2_L1_TKn_H1 based on the result of dot-multiplying the query vector QRY_L1_TKn_H1 and the key vector KEY_L1_TK2_H1. In this manner, the calculation circuit 120 may calculate an attention score ATSn_L1_TKn_H1 based on the result of dot-multiplying the query vector QRY_L1_TKn_H1 and the key vector KEY_L1_TKn_H1. In this case, the key vectors KEY_L1_TK1_H1 to KEY_L1_TKn-1_H1 corresponding to the preceding tokens (hereinafter referred to as preceding key vectors) may have been calculated in advance through a previous iteration (e.g., the 0th iteration).

[0088] That is, to perform the attention score calculation CAL_ATS_L1_H1, the calculation circuit 120 may repeatedly require not only the key vector KEY for the input token TKin but also the preceding key vectors. Accordingly, the calculation circuit 120 can store the calculated key vector KEY in the memory device 160 and then reuse it. For example, while the Transformer TF is performing a first iteration, the first head H1 of the first decoder layer DL1 can store the key vector KEY_L1_TKn_H1 in the memory device 160. In this case, while the Transformer TF is performing a second iteration, the calculation circuit 120 can recycle the key vector KEY_L1_TKn_H1 stored in the memory device 160 to perform the attention score calculation CAL_ATS_L1_H1.

[0089] Figure 10 is a diagram illustrating in more detail the attention vector calculation of Figure 7. That is, hereinafter, with reference to Figures 1 to 10, the attention vector calculation CAL_AV_L1_H1 performed on the first head H1 of the first decoder layer DL1 based on the n-th token TKn will be representatively described.

[0090] The calculation circuit 120 can perform an attention vector calculation CAL_AV for the nth token TKn based on the attention score ATS and value vector VAL corresponding to the nth token TKn and its preceding tokens (e.g., the 1st to (n-1st) tokens TK1 to TKn-1). For example, the calculation circuit 120 can generate an attention vector AV_L1_TKn_H1 based on the result of accumulating the values ​​obtained by multiplying the value vectors VAL_L1_TK1_H1 to VAL_L1_TKn_H1 and the attention scores ATS1_L1_TKn_H1 to ATSn_L1_TKn_H1. In this case, the value vectors VAL_L1_TK1_H1 to VAL_L1_TKn-1_H1 corresponding to the preceding tokens (hereinafter referred to as preceding value vectors) may be in a state of having been calculated in advance through a previous iteration (e.g., the 0th iteration).

[0091] That is, to perform the attention vector calculation CAL_AV_L1_H1, the calculation circuit 120 may need not only the value vector VAL for the input token TKin but also the preceding value vectors, so that the calculation circuit 120 can store the calculated value vector VAL in the memory device 160 and then reuse it.

[0092] For example, while the Transformer TF performs a first iteration, the first head H1 of the first decoder layer DL1 can store the value vector VAL_L1_TKn_H1 in the memory device 160. In this case, while the Transformer TF performs a second iteration, the calculation circuit 120 can recycle the value vector VAL_L1_TKn_H1 stored in the memory device 160 to perform the attention vector calculation CAL_AV_L1_H1.

[0093] Figure 11 is a diagram illustrating the overall operation of the transformer of Figure 3. For a simpler explanation, the key vectors KEY and value vectors VAL that the transformer TF caches (e.g., stores) in the memory device 160 during the first iteration will be representatively described below with reference to Figures 1 to 11.

[0094] While the transformer TF performs the first iteration, each of the first through m-th decoder layers DL1 through DLm may store a plurality of key vectors KEY for the first through N-th heads H1 through HN in the memory device 160. For example, the first decoder layer DL1 may store key vectors KEY_L1_TKn_H1 through KEY_L1_TKn_HN in the memory device 160;

[0095] Similarly, while the transformer TF performs a first iteration, each of the first through m-th decoder layers DL1 through DLm may store a plurality of value vectors VAL for the first through N-th heads H1 through HN in the memory device 160. For example, the first decoder layer DL1 may store value vectors VAL_L1_TKn_H1 through VAL_L1_TKn_HN in the memory device 160;

[0096] In other words, when one decoder layer DL is executed, the arithmetic circuit 120 can store in the memory device 160 a plurality of key vectors KEY and a plurality of value vectors VAL corresponding to the input token TKin.

[0097] In this manner, each time the transformer TF performs one decoder layer DL, N additional "key vector KEY-value vector VAL" pairs may be stored in the memory device 160. Therefore, each time the transformer TF performs one iteration, (m×N) additional "key vector KEY-value vector VAL" pairs may be stored in the memory device 160.

[0098] In one embodiment, each of the multiple key vectors KEY and multiple value vectors VAL cached in memory device 160 may also be referred to as a cache vector.

[0099] 12 is a block diagram illustrating in more detail the operation of the memory striding circuit according to an embodiment of the present disclosure. Referring to FIGS. 1 to 12, the memory striding circuit 130 may receive a striding request REQ_STRD. In response to the striding request REQ_STRD, the memory striding circuit 130 may sequentially access multiple memory regions included in the memory device 160.

[0100] The memory device 160 may include a plurality of memory blocks BLK, for example, first to qth memory blocks BLK1 to BLKq.

[0101] Each of the memory blocks BLK can store key vectors (KEY) and value vectors (VAL) for multiple tokens. For example, the first memory block BLK1 can include a memory area for storing key vectors (KEY) and value vectors (VAL) for the first through pth tokens TK1 through TKp; the second memory block BLK2 can include a memory area for storing key vectors (KEY) and value vectors (VAL) for the (p+1)th through (2p)th tokens TKp+1 through TK2p. Similarly, the qth memory block BLKq can include a memory area for storing key vectors (KEY) and value vectors (VAL) for the (qp+1)th through ((q+1)p)th tokens TKqp+1 through TK(q+1)p.

[0102] More specifically, each memory block BLK may store key vectors KEY and value vectors VAL for a plurality of decoder layers DL included in the transformer TF. For example, a first memory block BLK1 may store key vectors KEY and value vectors VAL generated from the first to mth decoder layers DL1 to DLm for the first to pth tokens TK1 to TKp.

[0103] For a simpler explanation, an embodiment in which each memory block BLK is implemented to store both key vectors KEY and value vectors VAL will be described below. However, the scope of the present disclosure is not limited thereto, and each memory block BLK may be implemented to store only key vectors KEY or only value vectors VAL.

[0104] The capacity of the multiple memory blocks BLK may be the same. For example, the multiple memory blocks BLK may occupy the same memory area. In this case, the number of cache vectors (e.g., key vectors KEY or value vectors VAL) that can be stored in the multiple memory blocks BLK may be the same.

[0105] In one embodiment, the number of cache vectors that the memory blocks BLK can store may be determined based on the type of artificial intelligence model executed by the transformer accelerator 100. That is, the scope of the present disclosure is not limited to the size of "p."

[0106] In one embodiment, each of the plurality of memory blocks BLK may occupy a contiguous memory area. For example, each memory block BLK may occupy a memory area corresponding to a plurality of memory cells whose row addresses and / or bank addresses are adjacent to each other. However, the scope of the present disclosure is not limited in this respect.

[0107] In one embodiment, the memory regions occupied by the memory blocks BLK may be the same. For example, the row address ranges occupied by the memory blocks BLK may be the same, and the bank address ranges occupied by the memory blocks BLK may be the same. However, the scope of the present disclosure is not limited thereto.

[0108] Meanwhile, the processing circuit 110 may be implemented to sequentially allocate multiple memory blocks BLK in the memory device 160 according to the length of the token sequence generated through the operation of the auto-regressive transformer TF described above with reference to Figures 3 to 11. For example, the processing circuit 110 may determine whether to allocate a new memory block BLK based on whether space remains in the memory block BLK allocated in the memory device 160 to store the key vectors KEY and value vectors VAL for the newly generated tokens.

[0109] For example, if the transformer TF generates the p-th token TKp, the total length of the token sequence may be "p." In this case, the first memory block BLK1 has free space in which the key vector KEY and value vector VAL for the p-th token TKp may be stored. Therefore, the processing circuit 110 may not allocate memory space in the memory device 160 for the new memory block BLK.

[0110] On the other hand, if the transformer TF generates the (p+1)th token TKp+1, the total length of the token sequence may be p+1. In this case, the first memory block BLK1 may store the key vectors KEY and value vectors VAL for the first through pth tokens TK1 through TKp. That is, the first memory block BLK1 does not have any free space to store the key vectors KEY and value vectors VAL for the (p+1)th token TKp+1. Therefore, the processing circuit 110 may newly allocate a memory area for the second memory block BLK2 in the memory device 160.

[0111] In this manner, as the length of the token sequence increases, the processing circuit 120 can allocate new memory areas for new memory blocks BLK in the memory device 160. In this case, the memory blocks BLK are allocated at different times. Therefore, if additional data is stored in the memory device 160 between the times the memory blocks BLK are allocated, the memory blocks BLK may be allocated to non-contiguous addresses. For example, if the transformer accelerator 100 caches key-value vectors for other token sequences (not shown) generated through a batch operation together in the memory device 160, the memory blocks BLK may be allocated to non-contiguous addresses.

[0112] 1 to 11, when the arithmetic circuit 120 performs attention calculation ATC, the arithmetic circuit 120 may require all of the key vectors KEY and value vectors VAL for the preceding tokens. For example, when the arithmetic circuit 120 performs attention calculation ATC to generate the (qp+2)th token TKqp+2, the arithmetic circuit 120 may require all of the key vectors KEY and value vectors VAL for the first to (qp+1)th tokens TK1 to TKqp+1. That is, the arithmetic circuit 120 may require all of the key vectors KEY and value vectors VAL distributed across multiple memory blocks BLK to perform attention calculation ATC.

[0113] The memory striding circuit 130 may sequentially access a plurality of memory blocks BLK in response to a striding request REQ_STRD. For example, the memory striding circuit 130 may sequentially read a plurality of key vectors KEY and a plurality of value vectors VAL distributed across the first through qth memory blocks BLK1 through BLKq.

[0114] In one embodiment, the processing circuit 110 may be implemented to directly issue memory access commands CMD_MA for a plurality of key vectors KEY and a plurality of value vectors VAL distributed across a plurality of memory blocks BLK. In this case, the processing circuit 110 must directly manage addresses where the plurality of key vectors KEY and a plurality of value vectors VAL are stored, which may increase the processing load of the processing circuit 110.

[0115] In one embodiment, the memory striding circuit 130 may be implemented to issue memory access commands CMD_MA for the plurality of key vectors KEY and the plurality of value vectors VAL in response to the striding request REQ_STRD on behalf of the processing circuit 110. In this case, the processing circuit 110 may not directly manage addresses where the plurality of key vectors KEY and the plurality of value vectors VAL are stored, and the frequency at which the processing circuit 110 issues the memory access commands CMD_MA may be minimized, thereby minimizing the processing load on the processing circuit 110.

[0116] Figure 13 is a block diagram showing in more detail the memory striding circuit of Figure 2. Referring to Figures 1 to 13, the memory striding circuit 130 may include a memory block address management circuit 131, a target address generation circuit 132, and a command issuance circuit 133.

[0117] The memory block address management circuit 131 may include a memory block base address table MBBAT. The memory block base address table MBBAT may include the memory block base addresses of each of the multiple memory blocks BLK.

[0118] In one embodiment, the memory block base address may be an address included in the memory block BLK. That is, the memory block base address may be an address that represents the memory block BLK. More specifically, for example, the memory block base address may refer to the smallest address among the addresses of the memory area occupied by the memory block BLK. However, the scope of the present disclosure is not limited thereto.

[0119] In one embodiment, the memory block address management circuit 131 may receive the memory block base address table MBBAT from the processing circuit 110. For example, the memory block address management circuit 131 may receive a striding request REQ_STRD including the memory block base address table MBBAT. However, the scope of the present disclosure is not limited thereto, and the memory block address management circuit 131 may also receive the memory block base address table MBBAT from a host device (e.g., a central processing unit) external to the transformer accelerator 100 via the host interface circuit 140.

[0120] The target address generation circuit 132 can generate a target address (hereinafter referred to as "TG") in which a key vector KEY or a value vector VAL to be read from the memory device 160 is stored. For example, the target address generation circuit 132 can generate the target address based on a memory block base address stored in a memory block base address table MBBAT.

[0121] In one embodiment, the target address generation circuit 132 can generate the target address TG based on a relative address of the memory area to be accessed with respect to the memory block base address (hereinafter referred to as a sub-block offset OFST). For example, the target address generation circuit 132 can generate the target address TG by adding the sub-block offset OFST to the memory block base address BASE.

[0122] The command issuing circuit 133 can issue a memory access command CMD_MA for the target address TG. For example, the command issuing circuit 133 can issue an activate command and a read command for the target address TG.

[0123] Figure 14 is a diagram showing the memory block base address table of Figure 13. Referring to Figures 1 to 14, the memory block base address table MBBAT can store a plurality of memory block base addresses BASE corresponding to a plurality of memory blocks BLK, respectively. For example, the memory block base address table MBBAT can store first to q-th memory block base addresses BASE1 to BASEq corresponding to the first to q-th memory blocks BLK1 to BLKq, respectively.

[0124] The first to qth memory block base addresses BASE1 to BASEq may be different from each other. For example, the first memory block base address BASE1 may be "0x11101", the second memory block base address BASE2 may be "0x21101", the third memory block base address BASE3 may be "0x31101", and the qth memory block base address BASEq may be "0xq1101".

[0125] For the sake of simplicity, the present disclosure typically illustrates an example in which each of the plurality of memory block base addresses BASE is 20 bits long, but the scope of the present disclosure is not limited thereto. For example, each of the plurality of memory block base addresses BASE may be any number of bits corresponding to a bank group address, a bank address, a row address, a column address, and / or any combination thereof.

[0126] 15 is a diagram illustrating the memory block of FIG. 12 in more detail. The configuration of the first memory block BLK1 will be described below with reference to FIGS. 1 to 15. However, the scope of the present disclosure is not limited thereto, and each of the multiple memory blocks BLK included in the memory device 160 may be configured in a similar manner.

[0127] The first memory block BLK1 may include a plurality of sub-blocks SB. For example, the first memory block BLK1 may include a plurality of sub-blocks SB referred to as "SB_LkHj (where k is an integer greater than or equal to 1 and less than or equal to m, and j is an integer greater than or equal to 1 and less than or equal to N)." For example, the first memory block BLK1 may include sub-blocks SB_L1H1 to SB_LmHN.

[0128] The plurality of sub-blocks SB may correspond to different combinations of the plurality of decoder layers DL and the plurality of heads H. Each of the plurality of sub-blocks SB may correspond to one decoder layer DL and one head H. For example, sub-block SB_LkHj may correspond to the kth decoder layer DLk and the jth head Hj. More specifically, for example, sub-block SB_L1H1 may correspond to the first decoder layer DL1 and the first head H1; sub-block SB_L1H2 may correspond to the first decoder layer DL1 and the second head H2; and sub-block SB_L2H1 may correspond to the second decoder layer DL2 and the first head H1.

[0129] In one embodiment, the number of sub-blocks included in one memory block BLK may be determined based on the product of the number of decoder layers (e.g., m) and the number of heads (e.g., N).

[0130] Each of the plurality of sub-blocks SB may store a plurality of key vectors KEY and / or a plurality of value vectors VAL cached in the memory device 160 for the attention calculation ATC for the corresponding decoder layer DL and head H. For example, sub-block SB_LkHj may store a plurality of key vectors KEY and / or a plurality of value vectors VAL cached for the attention calculation ATC_Lk_Hj. That is, sub-block SB_LkHj included in the first memory block BLK1 may store key vectors KEY_Lk_TK1_Hj to KEY_Lk_TKp_Hj and / or value vectors VAL_Lk_TK1_Hj to VAL_Lk_TKp_Hj for the first to p-th tokens TK1 to TKp cached for the attention calculation ATC_Lk_Hj. More specifically, for example, the sub-block SB_L1H1 included in the first memory block BLK1 can store key vectors KEY_L1_TK1_H1 to KEY_L1_TKp_H1 and / or value vectors VAL_L1_TK1_H1 to VAL_L1_TKp_H1 for the first to p-th tokens TK1 to TKp; the sub-block SB_L1H2 included in the first memory block BLK1 can store key vectors KEY_L1_TK1_H1 to KEY_L1_TKp_H1 and / or value vectors VAL_L1_TK1_H1 to VAL_L1_TKp_H1 for the first to p-th tokens TK1 to TKp; The sub-block SB_L2H1 included in the first memory block BLK1 can store key vectors KEY_L2_TK1_H1 to KEY_L2_TKp_H1 and / or value vectors VAL_L2_1_H1 to VAL_L2_TKp_H1 for the first to pth tokens TK1 to TKp.

[0131] That is, according to an embodiment of the present disclosure, each of the sub-blocks SB may store a plurality of key vectors KEY and / or a plurality of value vectors VAL. However, for the sake of simplicity, addresses at which the key vectors KEY are stored within the sub-blocks SB will be representatively described below. Accordingly, an embodiment in which the memory striding circuit 130 accesses a plurality of key vectors KEY stored in the memory device 160 will be representatively described below. However, the scope of the present disclosure is not limited thereto, and one sub-block SB may be implemented to store a plurality of value vectors VAL, or to store both the plurality of key vectors KEY and the plurality of value vectors VAL. In this case, the memory striding circuit 130 may access a plurality of value vectors VAL stored in the memory device 160, or may access all of the plurality of key vectors KEY and the plurality of value vectors VAL stored in the memory device 160.

[0132] Among the plurality of sub-blocks SB, sub-blocks corresponding to the k-th decoder layer DLk may form a sub-block group SBG_Lk. For example, sub-blocks SB_L1H1 to SB_L1HN may form a sub-block group SBG_L1.

[0133] Among the plurality of sub-blocks SB, sub-blocks corresponding to the j-th head Hj may form a sub-block group SBG_Hj. For example, sub-blocks SB_L1H1 to SB_LmH1 may form a sub-block group SBG_H1.

[0134] Each of the sub-blocks included in one sub-block group SBG may have the same address interval. For example, the address intervals between sub-blocks SB_L1H1 to SB_L1HN may be the same, and the address intervals between sub-blocks SB_L1H1 to SB_LmH1 may be the same. Therefore, the target address generation circuit 132 can identify sub-block base addresses corresponding to all of the sub-blocks SB included in the first memory block BLK1 by sequentially adding address intervals to the first memory block base address BASE1. The address intervals between the sub-blocks SB will be described below with reference to FIGS. 17 and 18, and the operation of the target address generation circuit 132 will be described in more detail below with reference to FIGS. 19 to 30.

[0135] Figure 16 is a diagram showing in more detail the configuration of the sub-block shown in Figure 15. The configuration of the sub-block SB_L1H1 will be described below as a representative example with reference to Figures 1 to 16. However, the scope of the present disclosure is not limited thereto, and the other sub-blocks SB included in the first memory block BLK1 may also be configured in a similar manner.

[0136] The sub-block SB_L1H1 may store key vectors KEY_L1_TK1_H1 to KEY_L1_TKp_H1. The key vectors KEY_L1_TK1_H1 to KEY_L1_TKp_H1 may be stored in consecutive addresses within the memory device 160. For example, the key vector KEY_L1_TK1_H1 may be stored at address 0x01101, the key vector KEY_L1_TK2_H1 may be stored at address 0x01102, and the key vector KEY_L1_TK3_H1 may be stored at address 0x01103. Similarly, the key vector KEY_L1_TKp_H1 may be stored at address 0x0110p. That is, the sub-block SB_L1H1 can occupy the memory area corresponding to addresses "0x01101" to "0x0110p" in the memory device 160.

[0137] In one embodiment, the size of the sub-block SB may be the same as a reading size, which is a unit of consecutive reads from the memory striding circuit 130. For example, if the sub-block SB_L1H1 occupies a memory area corresponding to p addresses that increase by 0x00001 from the address 0x01101, the memory striding circuit 130 may consecutively read data stored at p addresses that increase by 0x00001 from the target address. The relationship between the reading size and the size of the sub-block SB will be described in more detail with reference to the following drawings.

[0138] 16 shows an example in which the sub-block SB_L1H1 occupies a memory area corresponding to p addresses that increase by 0x00001 from the address "0x01101" for a simpler explanation, but the scope of the present disclosure is not limited to this. For example, the sub-block SB_L1H1 may occupy a memory area corresponding to p addresses that increase by any amount from the first memory block base address BASE1.

[0139] In one embodiment, sub-block SB_L1H1 may correspond to one page of memory device 160. For example, the memory area occupied by sub-block SB_L1H1 may correspond to one memory bank and one row address included in memory device 160. However, the scope of the present disclosure is not limited in this respect.

[0140] In one embodiment, sub-block SB_L1H1 may correspond to two or more pages of memory device 160. In this case, sub-block SB_L1H1 may occupy a memory region corresponding to two or more row addresses in one memory bank included in memory device 160, or a memory region corresponding to a single row address in two or more memory banks included in memory device 160. However, the scope of the present disclosure is not limited in this respect.

[0141] Figures 17 and 18 are diagrams showing in more detail the sub-block groups of Figure 15. Hereinafter, the configuration of the sub-block group SBG_H1 will be described with reference to Figure 17, and the configuration of the sub-block group SBG_L1 will be described with reference to Figure 18.

[0142] 1 to 17, the sub-block group SBG_H1 may include a plurality of sub-blocks SB. For example, the sub-block group SBG_H1 may include sub-blocks SB_L1H1 to SB_L3H1.

[0143] Similar to what was described above with reference to Figure 16, sub-block SB_L1H1 can store key vectors KEY_L1_TK1_H1 to KEY_L1_TKp_H1, sub-block SB_L2H1 can store key vectors KEY_L2_TK1_H1 to KEY_L2_TKp_H1, and sub-block SB_L3H1 can store key vectors KEY_L3_TK1_H1 to KEY_L3_TKp_H1.

[0144] The address intervals between the sub-blocks SB included in the sub-block group SBG_H1 may be the same. For example, the address interval between the sub-block SB_L1H1 and the sub-block SB_L2H1, and the address interval between the sub-block SB_L2H1 and the sub-block SB_L3H1 may be the layer address interval INTV_layer. In the following, for easier explanation, it is assumed that the layer address interval INTV_layer is "0x01000".

[0145] More specifically, for example, the difference between the addresses where the key vectors KEY_L1_TK1_H1 to KEY_L1_TKp_H1 are stored and the addresses where the key vectors KEY_L2_TK1_H1 to KEY_L2_TKp_H1 are stored may be the layer address interval INTV_layer; the difference between the addresses where the key vectors KEY_L2_TK1_H1 to KEY_L2_TKp_H1 are stored and the addresses where the key vectors KEY_L3_TK1_H1 to KEY_L3_TKp_H1 are stored may be the layer address interval INTV_layer.

[0146] That is, addresses occupied by each sub-block SB included in the sub-block group SBG_H1 may correspond to values ​​obtained by adding an integer multiple of the layer address interval INTV_layer to the addresses occupied by the sub-block SB_L1H1. For example, addresses occupied by the sub-block SB_L2H1 may correspond to values ​​obtained by adding the layer address interval INTV_layer to the addresses occupied by the sub-block SB_L1H1; addresses occupied by the sub-block SB_L3H1 may correspond to values ​​obtained by adding twice the layer address interval INTV_layer to the addresses occupied by the sub-block SB_L1H1.

[0147] Therefore, according to one embodiment of the present disclosure, the target address generation circuit 132 can sequentially add the layer address interval INTV_layer to the first memory block base address BASE1 to calculate the sub-block base address for each of the sub-blocks SB included in the sub-block group SBG_H1. In more detail, for example, the target address generation circuit 132 can calculate the subblock base address "0x02101" for the subblock SB_L2H1 by adding the layer address interval INTV_layer "0x01000" to the subblock base address "0x01101" for the subblock SB_L1H1 (i.e., the first memory block base address BASE1); or can calculate the subblock base address "0x03101" for the subblock SB_L3H1 by adding the layer address interval INTV_layer "0x01000" to the subblock base address "0x02101" for the subblock SB_L2H1. In this case, the subblock base address for each of the multiple subblocks SB can be calculated even if the memory striding circuit 130 does not directly manage the subblock base address for each of the multiple subblocks SB.

[0148] For the sake of simplicity, the configuration of the sub-block group SBG_H1 is representatively illustrated in FIG. 17, but the scope of the present disclosure is not limited thereto. For example, the sub-block groups SBG_H2 to SBG_Hm may also be implemented similarly to the sub-block group SBG_H1. More specifically, the address interval between the sub-blocks included in the sub-block groups SBG_H2 to SBG_Hm may be a layer address interval INTV_layer.

[0149] 1 to 16 and 18, the sub-block group SBG_H1 may include a plurality of sub-blocks SB. For example, the sub-block group SBG_H1 may include sub-blocks SB_L1H1 to SB_L1H3.

[0150] Similar to what was described above with reference to Figure 16, sub-block SB_L1H1 can store key vectors KEY_L1_TK1_H1 to KEY_L1_TKp_H1, sub-block SB_L1H2 can store key vectors KEY_L1_TK1_H2 to KEY_L1_TKp_H2, and sub-block SB_L1H3 can store key vectors KEY_L1_TK1_H3 to KEY_L1_TKp_H3.

[0151] The address intervals between the sub-blocks SB included in the sub-block group SBG_L1 may be the same. For example, the address interval between the sub-block SB_L1H1 and the sub-block SB_L1H2 and the address interval between the sub-block SB_L1H2 and the sub-block SB_L1H3 may be the head address interval INTV_head. In the following, for easier explanation, it is assumed that the head address interval INTV_head is "0x00100".

[0152] More specifically, for example, the difference between the addresses where the key vectors KEY_L1_TK1_H1 to KEY_L1_TKp_H1 are stored and the addresses where the key vectors KEY_L1_TK1_H2 to KEY_L1_TKp_H2 are stored may be the head address interval INTV_head; the difference between the addresses where the key vectors KEY_L1_TK1_H2 to KEY_L1_TKp_H2 are stored and the addresses where the key vectors KEY_L1_TK1_H3 to KEY_L1_TKp_H3 are stored may be the head address interval INTV_head.

[0153] That is, addresses occupied by each sub-block SB included in the sub-block group SBG_L1 may correspond to values ​​obtained by adding an integer multiple of the head address interval INTV_head to the addresses occupied by the sub-block SB_L1H1. For example, addresses occupied by the sub-block SB_L1H2 may correspond to values ​​obtained by adding the head address interval INTV_head to the addresses occupied by the sub-block SB_L1H1; addresses occupied by the sub-block SB_L1H3 may correspond to values ​​obtained by adding twice the head address interval INTV_head to the addresses occupied by the sub-block SB_L1H1.

[0154] Therefore, according to one embodiment of the present disclosure, the target address generation circuit 132 can sequentially add the head address interval INTV_head to the first memory block base address BASE1 to identify the subblock base address for each of the multiple subblocks SB included in the subblock group SBG_L1. In more detail, for example, the target address generation circuit 132 can calculate the subblock base address "0x01201" for the subblock SB_L1H2 by adding the head address interval INTV_head "0x00100" to the subblock base address "0x01101" for the subblock SB_L1H1 (i.e., the first memory block base address BASE1); or can calculate the subblock base address "0x01301" for the subblock SB_L1H3 by adding the head address interval INTV_head "0x00100" to the subblock base address "0x01201" for the subblock SB_L1H2. In this case, the subblock base addresses for multiple subblocks SB can be calculated even if the memory striding circuit 130 does not directly manage the subblock base addresses for each of the multiple subblocks SB.

[0155] For the sake of simplicity, the configuration of the sub-block group SBG_L1 is representatively illustrated in FIG. 18, but the scope of the present disclosure is not limited thereto. For example, the sub-block groups SBG_L2 to SBG_Lm may also be implemented similarly to the sub-block group SBG_L1. More specifically, the address interval between the sub-blocks included in the sub-block groups SBG_L2 to SBG_Lm may be a head address interval INTV_head.

[0156] 1 through 19, the processing circuit 110 can issue a head striding request REQ_STRD_head. The memory striding circuit 130 can sequentially access a plurality of sub-blocks SB included in a plurality of memory blocks BLK in response to the head striding request REQ_STRD_head.

[0157] For the sake of simplicity, the following description will be directed to an example in which the transformer accelerator 100 generates the (q(p+1)+1)th token TKq(p+1)+1 based on cache vectors (e.g., key vector KEY) stored in the first through qth memory blocks BLK1 through BLKq. In this case, as described above with reference to FIGS. 1 through 11, the arithmetic circuit 120 requires cache vectors for the first through q(p+1)th tokens TK1 through TKq(p+1). Therefore, the following description will be directed to an example in which the memory striding circuit 130 reads key vectors KEY for the first through (q(p+1))th tokens TK1 through TKq(p+1) from the first through qth memory blocks BLK1 through BLKq.

[0158] The head striding request REQ_STRD_head may include a total number of blocks (NUM_block), a memory block base address table MBBAT, a subblock offset (OFST), and a reading size (RS).

[0159] The total number of blocks NUM_block may indicate the total number of memory blocks BLK to be accessed by the memory striding circuit 130. For example, the total number of blocks NUM_block may indicate the number of memory blocks BLK (e.g., “q”) allocated in the memory device 160.

[0160] The memory block base address table MBBAT may include a memory block base address BASE for each of the multiple memory blocks BLK. The configuration of the memory block base address table MBBAT has been described above with reference to FIG. 14, so a detailed description thereof will be omitted.

[0161] The subblock offset OFST can indicate the relative position within the plurality of memory blocks BLK of the subblock SB accessed by the memory striding circuit 130. For example, the subblock offset OFST can indicate the difference between the subblock base address Θ and the plurality of memory block base addresses BASE for the subblock SB accessed by the memory striding circuit 130.

[0162] The leading size RS may indicate the size of each of the sub-blocks SB accessed by the memory striding circuit 130. For example, the leading size RS may indicate the size of the memory area continuously read from the memory striding circuit 130 for one memory block BLK.

[0163] In response to the head striding request REQ_STRD_head, the memory striding circuit 130 may repeatedly issue commands to read sub-blocks SB corresponding to one decoder layer DL and one head H. For example, in response to the head striding request REQ_STRD_head, the memory striding circuit 130 may repeatedly issue commands to access sub-blocks SB corresponding to a specific decoder layer DL and a specific head H among the sub-blocks SB included in the plurality of memory blocks BLK.

[0164] That is, the memory striding circuit 130 can issue multiple memory access commands CMD_MA to read one sub-block SB from the first memory block BLK1, and can issue multiple memory access commands CMD_MA to read one sub-block SB from the second memory block BLK2. In this manner, the memory striding circuit 130 can also issue multiple memory access commands CMD_MA to read one sub-block SB from the q-th memory block BLKq.

[0165] More specifically, the target address generation circuit 132 can generate multiple target addresses (hereinafter referred to as "TG") by adding a sub-block offset OFST to each of multiple memory block base addresses BASE. The command issuing circuit 133 can read a memory area corresponding to the leading size RS from each of the multiple target addresses TG. The operation of the memory striding circuit 130 based on the sub-block offset OFST and the leading size RS will be described in more detail below with reference to FIG. 20.

[0166] In one embodiment, a memory area corresponding to a leading size RS located at a target address TG may correspond to one sub-block SB. That is, the memory striding circuit 130 may read cache vectors stored in one sub-block corresponding to the target address TG. In this case, the one sub-block corresponding to the target address TG may be referred to as a "target sub-block." However, the scope of the present disclosure is not limited to such terminology.

[0167] Figure 20 is a diagram illustrating in more detail the operation of the memory striding circuit of Figure 19. Hereinafter, with reference to Figures 1 to 20, a representative embodiment will be described in which the memory striding circuit 130 issues a memory access command CMD_MA to read sub-blocks SB_L2H2 corresponding to the second decoder layer DL2 and second head H2 from each of the first to qth memory blocks BLK1 to BLKq in response to a head striding request REQ_STRD_head. However, the scope of the present disclosure is not limited thereto.

[0168] The target address generation circuit 132 can generate the first to q-th target addresses TG1 to TGq by adding the sub-block offset OFST to each of the first to q-th memory block base addresses BASE1 to BASEq. In this case, the first to q-th target addresses TG1 to TGq can be sub-block base addresses for the sub-blocks SB_L2H2 included in the first to q-th memory blocks BLK1 to BLKq, respectively.

[0169] In one embodiment, the sub-block offset OFST may correspond to an integer multiple of the layer address interval INTV_layer and an integer multiple of the head address interval INTV_head. For example, the sub-block offset OFST corresponding to the sub-block SB_L2H2 may be "0x01100".

[0170] In one embodiment, the sub-block offset OFST included in the striding request REQ_STRD may have the format of an address size. For example, the sub-block offset OFST included in the striding request REQ_STRD may be "0x01100." However, the scope of the present disclosure is not limited thereto, and the sub-block offset OFST included in the striding request REQ_STRD may have the format of an ordered pair of integers that are multiplied by the layer address interval INTV_layer and the head address interval INTV_head, respectively.

[0171] The command issuing circuit 133 can issue multiple memory access commands CMD_MA for reading memory areas of only the leading size RS from each of the first to qth target addresses TG1 to TGq. For example, the command issuing circuit 133 can issue multiple memory access commands CMD_MA for reading memory areas of only the leading size RS from the first target address TG1; and can issue multiple memory access commands CMD_MA for reading memory areas of only the leading size RS from the second target address TG2. In this manner, the command issuing circuit 133 can issue multiple memory access commands CMD_MA for reading memory areas of only the leading size RS from the qth target address TGq.

[0172] The memory device 160 can output a key vector KEY stored in each sub-block (SM_L2H2; shown in gray) of the first to qth memory blocks BLK1 to BLKq in response to a plurality of memory access commands CMD_MA provided from the command issuing circuit 133. For example, the memory device 160 may provide to the arithmetic circuit 120 a plurality of key vectors KEY_L2_TK1_H2 to KEY_L2_TKp_H2 stored in the sub-block SM_L2H2 included in the first memory block BLK1; and may provide to the arithmetic circuit 120 a plurality of key vectors KEY_L2_TKp+1_H2 to KEY_L2_TK2p_H2 stored in the sub-block SM_L2H2 included in the second memory block BLK2; and in this manner, may provide to the arithmetic circuit 120 a plurality of key vectors KEY_L2_TKqp+1_H2 to KEY_L2_TKq(p+1)_H2 stored in the sub-block SM_L2H2 included in the qth memory block BLKq. In this case, the calculation circuit 120 should be able to perform attention calculation ATC_L2_H2 for the second decoder layer DL2 and the second head H2 based on the provided key vectors KEY_L2_TK1_H2 to KEY_L2_TKq(p+1)_H2.

[0173] Figure 21 is a flowchart showing the operation of the memory striding circuit of Figure 19. Referring to Figures 1 to 21, in step S110, a variable (i) may be set to 1. The variable (i) is used only to describe the repeating operation of the memory striding circuit 130 and does not limit the scope of the present disclosure.

[0174] In step S120, the memory striding circuit 130 may issue commands to read sub-blocks included in the i-th memory block BLKi. For example, the memory striding circuit 130 may sequentially issue a plurality of memory access commands CMD_MA to sequentially read a plurality of key vectors KEY stored in one sub-block included in the i-th memory block BLKi. Step S120 will be described in more detail with reference to FIG. 22 below.

[0175] In step S130, the memory striding circuit 130 determines whether the variable (i) is equal to the total number of memory blocks NUM_block. For example, the memory striding circuit 130 determines whether one sub-block SB is read from each of all memory blocks BLK.

[0176] In step S130, if the variable (i) is different from the total number of memory blocks NUM_block, the memory striding circuit 130 may perform the following step S140.

[0177] In step S140, the memory striding circuit 130 can increment the variable (i) by "1". Thereafter, the memory striding circuit 130 can repeatedly perform step S120 described above. In this manner, the memory striding circuit 130 can read one sub-block SB from each of the plurality of memory blocks BLK. In more detail, for example, the memory striding circuit 130 can read "p" key vectors KEY from each of the plurality of memory blocks BLK.

[0178] In step S130, if the variable (i) is equal to the total number of memory blocks NUM_block, the operation of the memory striding circuit 130 may be terminated. In this case, the arithmetic circuit 120 should be able to perform attention calculation ATC based on the read key vector KEY.

[0179] Figure 22 is a flowchart showing in more detail step S120 of Figure 21. Referring to Figures 1 to 22, step S120 may include the following steps S121 to S123.

[0180] In step S121, the memory striding circuit 130 may identify the memory block base address BASE of the (i)th memory block BLKi. For example, the target address generation circuit 132 may identify the (i)th memory block base address BASEi from the memory block base address table MBBAT.

[0181] In step S122, the memory striding circuit 130 may generate a target address TG by adding a memory block base address BASE and a sub-block offset OFST. For example, the target address generation circuit 132 may generate an (i)th target address TGi by adding an (i)th memory block base address BASEi and a sub-block offset OFST.

[0182] In step S123, the memory striding circuit 130 may issue commands to read a memory area of ​​leading size RS from a target address TG. For example, the command issuing circuit 133 may provide a plurality of memory access commands CMD_MA to the memory device 160 via the memory controller 150 to read a memory area of ​​leading size RS from an (i) target address TGi.

[0183] 23 illustrates the operation of a memory striding circuit according to an embodiment. Referring to FIGS. 1 through 18 and 23, the processing circuit 110 may issue a layer striding request REQ_STRD_layer. In response to the layer striding request REQ_STRD_layer, the memory striding circuit 130 may sequentially access a plurality of sub-blocks SB included in each of a plurality of memory blocks BLK.

[0184] The layer striding request REQ_STRD_layer includes the total number of blocks NUM_block, the memory block base address table MBBAT, the leading size RS, the head address interval INTV_head, and the total number of heads (NUM_head; total number of heads).

[0185] The total number of blocks NUM_block, the memory block base address table MBBAT, and the leading size RS are similar to those explained above with reference to FIG. 19, and therefore detailed explanations will be omitted.

[0186] The head address interval INTV_head may indicate the address interval between a plurality of sub-blocks SB corresponding to one decoder layer DL among a plurality of sub-blocks SB included in one memory block BLK.

[0187] The total number of heads NUM_head may indicate the number of sub-blocks SB that the memory striding circuit 130 should access from each memory block BLK. For example, the total number of heads NUM_head may indicate the number of heads H (e.g., “N”) used in the multi-head attention computation MHATC of the transformer TF executed by the transformer accelerator 100.

[0188] The memory striding circuit 130 may repeatedly issue commands to read a plurality of sub-blocks SB corresponding to one decoder layer DL in response to the layer striding request REQ_STRD_layer. For example, the memory striding circuit 130 may sequentially issue commands to access a plurality of sub-blocks SB corresponding to a specific decoder layer DL among the sub-blocks SB included in the plurality of memory blocks BLK in response to the layer striding request REQ_STRD_layer.

[0189] That is, the memory striding circuit 130 can issue a plurality of memory access commands CMD_MA for reading one of the plurality of sub-blocks SB included in one sub-block group SBG_L from each of the first to q-th memory blocks BLK1 to BLKq, and can issue a plurality of memory access commands CMD_MA for reading another of the plurality of sub-blocks SB included in one sub-block group SBG_L from each of the first to q-th memory blocks BLK1 to BLKq. In this manner, the memory striding circuit 130 can sequentially read the plurality of sub-blocks SB included in one sub-block group SBG_L from each of the first to q-th memory blocks BLK1 to BLKq.

[0190] More specifically, the target address generation circuit 132 can calculate an integer multiple of the head address interval INTV_head to calculate a sub-block offset OFST. The target address generation circuit 132 can generate multiple target addresses TG by adding the sub-block offset OFST to each of multiple memory block base addresses BASE for multiple memory blocks BLK. In this case, the command issue circuit 133 can issue multiple memory access commands CMD_MA to read memory areas corresponding to the leading size RS from each of the multiple target addresses.

[0191] The target address generation circuit 132 can then adjust the multiple of the head address interval INTV_head to change the sub-block offset OFST. In this case, multiple target addresses TG may be changed. The command issuance circuit 133 can issue multiple memory access commands CMD_MA to read memory areas corresponding to the leading size RS from each of the multiple target addresses.

[0192] In this manner, the memory striding circuit 130 sequentially changes the sub-block offset OFST to read a number of sub-blocks SB corresponding to the total number of heads NUM_head from each of the multiple memory blocks BLK. The operation of the memory striding circuit 130 based on the head address interval INTV_head and the total number of heads NUM_head will be described in more detail below with reference to FIG. 24.

[0193] Figure 24 is a diagram illustrating in more detail the operation of the memory striding circuit of Figure 20. Hereinafter, with reference to Figures 1 to 18 and Figures 23 to 24, an example will be described in which the memory striding circuit 130 issues a memory access command CMD_MA to read a plurality of sub-blocks SB included in the sub-block group SBG_L1 from each of the first to qth memory blocks BLK1 to BLKq in response to a layer striding request REQ_STRD_layer. However, the scope of the present disclosure is not limited thereto.

[0194] First, the memory striding circuit 130 can access sub-blocks SB_L1H1 included in each of the first through q-th memory blocks BLK1 through BLKq based on the first through q-th memory block base addresses BASE1 through BASEq, respectively. For example, the command issuing circuit 133 can issue multiple memory access commands CMD_MA to read memory areas of the leading size RS from the first through q-th memory block base addresses BASE1 through BASEq, respectively. In this case, the calculation circuit 120 can perform attention calculation ATC_L1_H1 for the first decoder layer DL1 and the first head H1 based on the key vectors KEY_L1_TK1_H1 through KEY_L1_TKq(p+1)_H1 read from the memory device 160.

[0195] Thereafter, the memory striding circuit 130 can access the sub-blocks SB_L1H2 included in each of the first through q-th memory blocks BLK1 through BLKq based on the first through q-th memory block base addresses BASE1 through BASEq and the head address interval INTV_head. For example, the target address generation circuit 132 can generate the first through q-th target addresses TG1 through TGq by adding "1" times the head address interval INTV_head (i.e., the sub-level offset OFST) to each of the first through q-th memory block base addresses BASE1 through BASEq; the command issuing circuit 133 can issue a plurality of memory access commands CMD_MA for reading memory areas of the leading size RS from the first through q-th target addresses TG1 through TGq. In this case, the calculation circuit 120 should be able to perform attention calculation ATC_L1_H2 for the first decoder layer DL1 and the second head H1 based on the key vectors KEY_L1_TK1_H2 to KEY_L1_TKq(p+1)_H2 read from the memory device 160.

[0196] In this manner, the memory striding circuit 130 can sequentially access the sub-blocks SB_L1H1 through SB_L1HN included in each of the first through qth memory blocks BLK1 through BLKq. For example, the target address generation circuit 132 can change the sub-level offset OFST by sequentially increasing the magnitude of the integer multiplied by the head address interval INTV_head, and the command issuing circuit 133 can issue multiple memory access commands CMD_MA to read multiple sub-blocks SB from the target address TG corresponding to the changed sub-block offset OFST. In this case, the calculation circuit 120 can perform attention calculations ATC_L1_H1 through ATC_L1_HN (i.e., multi-head attention calculation MHATC) for all the heads included in the first decoder layer DL1 based on the key vectors KEY read from the memory device 160.

[0197] That is, the memory striding circuit 130 can traverse a plurality of sub-blocks SB corresponding to one decoder layer DL for each memory block BLK. The method by which the memory striding circuit 130 accesses one sub-block (e.g., sub-block SB_L1H1) included in each of the first to qth memory blocks BLK1 to BLKq is similar to the method described above with reference to FIG. 20, and therefore a detailed description thereof will be omitted.

[0198] Fig. 25 is a flowchart showing the operation of the memory striding circuit of Fig. 24. Hereinafter, the operation of the memory striding circuit 130 for reading KEY from the key vector for performing the multi-head attention calculation MHATC_L1 will be representatively described with reference to Figs. 1 to 18 and 23 to 25. However, the scope of the present disclosure is not limited thereto.

[0199] In step S210, variables (i) and (j) may be set to 1. The variables (i) and (j) are only used to describe the iterative operation of the memory striding circuit 130 and do not limit the scope of the present disclosure.

[0200] In step S220, the memory striding circuit 130 may issue commands to read one sub-block for the (j)th head Hj included in the (i)th memory block BLKi. For example, the memory striding circuit 130 may sequentially issue multiple memory access commands CMD_MA to sequentially read multiple key vectors KEY stored in the sub-block SB_L1Hj included in the (i)th memory block BLKi. Step S220 will be described in more detail with reference to FIG. 26 below.

[0201] In step S230, the memory striding circuit 130 determines whether the variable (i) is equal to the total number of memory blocks NUM_block. For example, the memory striding circuit 130 determines whether the sub-blocks SB_L1Hj have been read from all the memory blocks BLK.

[0202] In step S230, if the variable (i) is different from the total number of memory blocks NUM_block, the memory striding circuit 130 may perform the following step S240.

[0203] In step S240, the memory striding circuit 130 can increment the variable (i) by 1. Thereafter, the memory striding circuit 130 can repeat step S220. In this manner, the memory striding circuit 130 can read the sub-block SB_L1Hj from each of the multiple memory blocks BLK. In this case, the calculation circuit 120 can perform the attention calculation ATC_L1_Hj for the (j)th head Hj based on the read key vector KEY.

[0204] In step S230, if the variable (i) is equal to the total number of memory blocks NUM_block, the memory striding circuit 130 may perform the following step S250.

[0205] In step S250, the memory striding circuit 130 can determine whether the variable (j) is equal to the total number of heads NUM_head. For example, the memory striding circuit 130 can determine whether all sub-blocks SB included in the sub-block group SBG_L1 have been read from all memory blocks BLK. In other words, the memory striding circuit 130 can determine whether the same number of sub-blocks SB as the total number of heads NUM_head have been read from all memory blocks BLK.

[0206] In step S250, if the variable (j) is different from the total number of heads NUM_head, the memory striding circuit 130 may perform the following step S260.

[0207] In step S260, the memory striding circuit 130 may decrease the variable (i) to "1" and increase the variable (j) by "1". Thereafter, the memory striding circuit 130 may repeat step S220. In this manner, the memory striding circuit 130 may read all sub-blocks SB included in the sub-block group SBG_L1 from each of the plurality of memory blocks BLK.

[0208] In step S250, if the variable (j) is equal to the total number of heads NUM_head, the operation of the memory striding circuit 130 may be terminated. In this case, the calculation circuit 120 should be able to perform the multi-head attention calculation MHATC_L1 based on the read key vector KEY.

[0209] Figure 26 is a flowchart showing in more detail step S220 of Figure 25. Referring to Figures 1 to 18 and 23 to 26, step S220 may include the following steps S221 to S224.

[0210] In step S221, the memory striding circuit 130 can identify the memory block base address BASE of the (i)th memory block BLKi. For example, the target address generation circuit 132 can identify the (i)th memory block base address BASEi from the memory block base address table MBBAT.

[0211] In step S222, the memory striding circuit 130 may calculate the sub-block offset OFST by multiplying the head address interval INTV_head by (j-1). For example, the target address generation circuit 132 may calculate the sub-block offset OFST by multiplying the head address interval INTV_head by (j-1).

[0212] In step S223, the memory striding circuit 130 may generate a target address TG by adding a memory block base address BASE and a sub-block offset OFST. In step S224, the memory striding circuit 130 may issue commands to read a leading size RS from the target address TG. Steps S223 to S224 are similar to steps S122 to S123 described with reference to FIG. 22, and therefore will not be described in detail.

[0213] 27 illustrates the operation of a memory striding circuit according to an embodiment. Referring to FIGS. 1 through 18 and 27, the processing circuit 110 can issue an iteration striding request REQ_STRD_iteration. In response to the iteration striding request REQ_STRD_iteration, the memory striding circuit 130 can sequentially access a plurality of sub-blocks SB included in each of a plurality of memory blocks BLK.

[0214] The iteration striding request REQ_STRD_iteration may include the total number of blocks NUM_block, the memory block base address table MBBAT, the leading size RS, the head address interval INTV_head, the total number of heads NUM_head, the layer address interval INTV_layer, and the total number of layers NUM_layer.

[0215] The total number of blocks NUM_block, the memory block base address table MBBAT, the leading size RS, and the head address interval INTV_head are similar to those explained above with reference to FIG. 23, and therefore detailed explanations will be omitted.

[0216] The layer address interval INTV_layer can indicate the address interval between a plurality of sub-blocks SB corresponding to one head H among a plurality of sub-blocks SB included in one memory block BLK.

[0217] The total number of heads NUM_head and the total number of layers NUM_layer may indicate the number of sub-blocks SB that the memory striding circuit 130 should access from each memory block BLK. For example, the total number of heads NUM_head may indicate the number of heads H (e.g., “N”) used in the multi-head attention computation MHATC of the Transformer TF executed by the Transformer accelerator 100; the total number of layers NUM_layer may indicate the number of decoder layers DL (e.g., “m”) included in the Transformer TF executed by the Transformer accelerator 100.

[0218] The memory striding circuit 130 may repeatedly issue commands to read a plurality of sub-blocks SB corresponding to one iteration of the transformer TF in response to the iteration striding request REQ_STRD_iteration. For example, the memory striding circuit 130 may sequentially issue commands to sequentially access all of the sub-blocks SB included in a plurality of memory blocks BLK in response to the iteration striding request REQ_STRD_iteration.

[0219] More specifically, the target address generation circuit 132 can calculate a sub-block offset OFST by adding an integer multiple of the head address interval INTV_head and an integer multiple of the layer address interval INTV_layer. The target address generation circuit 132 can generate multiple target addresses TG by adding the sub-block offset OFST to each of multiple memory block base addresses BASE for multiple memory blocks BLK. In this case, the command issue circuit 133 can issue multiple memory access commands CMD_MA to read memory areas corresponding to the leading size RS from each of the multiple target addresses.

[0220] The target address generation circuit 132 can then adjust the multiples of the head address interval INTV_head and the layer address interval INTV_layer to change the sub-block offset OFST. In this case, multiple target addresses TG may be changed. The command issuing circuit 133 can issue multiple memory access commands CMD_MA to read memory areas corresponding to the leading size RS from each of the multiple target addresses TG.

[0221] In this manner, the memory striding circuit 130 sequentially changes the sub-block offset OFST to read sub-blocks SB, the number of which corresponds to the product of the total number of heads NUM_head and the total number of layers NUM_layer, from each of the memory blocks BLK. The operation of the memory striding circuit 130 based on the head address interval INTV_head, the layer address interval INTV_layer, the total number of heads NUM_head, and the total number of layers NUM_layer will be described in more detail below with reference to Figures 28 to 30.

[0222] Figure 28 is a diagram illustrating in more detail the operation of the memory striding circuit of Figure 27. Referring to Figures 1 to 18 and Figures 27 to 28, in response to an iteration striding request REQ_STRD_iteration, the memory striding circuit 130 can issue a memory access command CMD_MA to read all sub-blocks SB included in each of the first to qth memory blocks BLK1 to BLKq.

[0223] First, the memory striding circuit 130 can access the sub-block group SBG_L1 included in each of the first to q-th memory blocks BLK1 to BLKq. For example, the memory striding circuit 130 can sequentially read the sub-blocks SB_L1H1 to SB_L1HN from each of the first to q-th memory blocks BLK1 to BLKq. The method by which the memory striding circuit 130 reads the sub-blocks SB_L1H1 to SB_L1HN is similar to that described above with reference to FIG. 24, and therefore a detailed description thereof will be omitted. In this case, the calculation circuit 120 should be able to perform the multi-head attention calculation MHATC_L1 based on the key vector KEY read from the memory device 160.

[0224] Thereafter, the memory striding circuit 130 can access the sub-block group SBG_L2 included in each of the first through q-th memory blocks BLK1 through BLKq. For example, the memory striding circuit 130 can sequentially read the sub-blocks SB_L2H1 through SB_L2HN from each of the first through q-th memory blocks BLK1 through BLKq. In this case, the calculation circuit 120 can perform the multi-head attention calculation MHATC_L2 based on the key vector KEY read from the memory device 160.

[0225] In this manner, the memory striding circuit 130 can sequentially read all of the sub-blocks SB included in each of the first to q-th memory blocks BLK1 to BLKq. That is, the memory striding circuit 130 sequentially accesses a plurality of sub-blocks corresponding to one decoder layer DL, and then sequentially accesses a plurality of sub-blocks corresponding to another decoder layer DL, thereby sequentially reading all of the sub-blocks SB included in each of the first to q-th memory blocks BLK1 to BLKq.

[0226] More specifically, the target address generation circuit 132 may change the sub-level offset OFST by sequentially increasing multiples of the head address interval INTV_head and the layer address interval INTV_layer. The command issuing circuit 133 may issue multiple memory access commands CMD_MA to read multiple sub-blocks SB from the target address TG corresponding to the changed sub-block offset OFST. In this case, the transformer accelerator 100 may sequentially execute the first through m-th decoder layers DL1 through DLm included in the transformer TF based on the key vector KEY read from the memory device 160 to generate an output token TKout (e.g., the (q(p+1)+1)-th token TK(q(p+1)+1)).

[0227] Figure 29 is a flowchart showing the operation of the memory striding circuit of Figure 27. Referring to Figures 1 to 18 and Figures 27 to 29, variables (i), (j), and (k) may be set to '1' in step S310. The variables (i), (j), and (k) are used only to describe the iterative operation of the memory striding circuit 130 and do not limit the scope of the present disclosure.

[0228] In step S320, the memory striding circuit 130 may issue commands to read one sub-block SB_LkHj corresponding to the (k)th decoder layer DLk and the (j)th head Hj included in the (i)th memory block BLKi. For example, the memory striding circuit 130 may sequentially issue a plurality of memory access commands CMD_MA to sequentially read a plurality of key vectors KEY stored in the sub-block SB_LkHj included in the (i)th memory block BLKi. Step S320 will be described in more detail with reference to FIG. 30 below.

[0229] The memory striding circuit 130 in step S330 can determine whether the variable (i) is equal to the total number of memory blocks NUM_block.

[0230] If the variable (i) is different from the total number of memory blocks NUM_block in step S330, the memory striding circuit 130 may perform the following step S340. In step S340, the memory striding circuit 130 may increment the variable (i) by 1. Thereafter, the memory striding circuit 130 may repeatedly perform step S320.

[0231] In step S330, if the variable (i) is equal to the total number of memory blocks NUM_block, the memory striding circuit 130 may perform the following step S350.

[0232] The memory striding circuit 130 in step S350 can determine whether the variable (j) is the same as the total number of heads NUM_head.

[0233] In step S350, if the variable (j) is different from the total number of heads NUM_head, the memory striding circuit 130 can perform the following step S360.

[0234] In step S360, the memory striding circuit 130 may decrease the variable (i) to 1 and increase the variable (j) by 1. Thereafter, the memory striding circuit 130 may repeat step S320.

[0235] The operation of the memory striding circuit 130 in steps S330 to S360 is similar to steps S230 to S260 described above with reference to FIG. 25, and therefore a detailed description thereof will be omitted.

[0236] In step S350, if the variable (j) is the total number of heads NUM_head, the memory striding circuit 130 can perform the following step S370.

[0237] The memory striding circuit 130 in step S370 can determine whether the variable (k) is the same as the total number of layers NUM_layer.

[0238] In step S370, if the variable (k) is different from the total number of layers NUM_layer, the memory striding circuit 130 may perform the following step S380.

[0239] In step S380, the memory striding circuit 130 may decrease the variables (i) and (j) to '1' and increase the variable (k) by '1'. Thereafter, the memory striding circuit 130 may repeat step S320. In this manner, the memory striding circuit 130 may read all sub-blocks SB included in the plurality of memory blocks BLK.

[0240] In step S370, if the variable (k) is equal to the total number of layers NUM_layer, the operation of the memory striding circuit 130 may be terminated. In this case, the calculation circuit 120 can execute the multi-head attention calculation MHATC included in all decoder layers DL of the Transformer TF based on the key vector KEY read from the memory device 160, and the Transformer accelerator 100 should be able to generate the output token TKout.

[0241] Figure 30 is a flowchart showing in more detail step S320 of Figure 29. Referring to Figures 1 to 18 and 27 to 30, step S320 may include the following steps S321 to S324.

[0242] In step S321, the memory striding circuit 130 may identify the memory block base address BASE of the (i)th memory block BLKi. For example, the target address generation circuit 132 may identify the (i)th memory block base address BASEi from the memory block base address table MBBAT.

[0243] In step S322, the memory striding circuit 130 may calculate the sub-block offset OFST by adding the product of the head address interval INTV_head and (j-1) and the product of the layer address interval INTV_layer and (k-1). For example, the target address generation circuit 132 may calculate the sub-block offset OFST by adding the product of the head address interval INTV_head and (j-1) and the product of the layer address interval INTV_layer and (k-1).

[0244] In step S323, the memory striding circuit 130 may generate a target address TG by adding a memory block base address BASE and a sub-block offset OFST. In step S324, the memory striding circuit 130 may issue commands to read a leading size RS from the target address TG. Steps S323 to S324 are similar to steps S122 to S123 described with reference to FIG. 22, and therefore, detailed description thereof will be omitted.

[0245] The above-described content is a specific example for implementing the present disclosure. The present disclosure is intended to include not only the above-described examples, but also examples that can be easily modified or changed by simple design changes. The present disclosure is also intended to include techniques that can be easily implemented by modifying the examples. Therefore, the scope of the present disclosure should not be limited to the above-described examples, but should be defined not only by the claims below, but also by equivalents of the claims of the present disclosure. [Explanation of symbols]

[0246] 100: Transformer Accelerator 110: Processing circuit 120: Arithmetic circuit 130: Memory striding circuit 160: Memory device TF:Transformers DL: Decoder Layer H: Head

Claims

1. 1. A transformer accelerator comprising: a memory device including a first memory block storing a first plurality of cache vectors for a first plurality of tokens and a second memory block storing a second plurality of cache vectors for a second plurality of tokens; a memory striding circuit configured to access the first and second memory blocks in response to an externally provided first striding request; Including, The memory striding circuit includes: a memory block address management circuit for storing a first memory block base address for the first memory block and a second memory block base address for the second memory block; a target address generation circuit that, in response to the first striding request, calculates a first target address included in the first memory block based on the first memory block base address and a first sub-block offset, and calculates a second target address included in the second memory block based on the second memory block base address and the first sub-block offset; and a command issuing circuit configured to issue a first plurality of memory access commands to a first target sub-block located at the first target address and a second plurality of memory access commands to a second target sub-block located at the second target address.

2. 2. The transformer accelerator of claim 1, wherein the size of the first and second target sub-blocks is a first leading size.

3. The first striding request is 3. The transformer accelerator of claim 2, comprising the first memory block base address, the second memory block base address, the first sub-block offset, and the first leading size.

4. The target address generation circuit in response to the first striding request, to calculate a third target address included in the first memory block based on the first memory block base address and a second sub-block offset, and to calculate a fourth target address included in the second memory block based on the second memory block base address and the second sub-block offset; The command issuing circuit 3. The transformer accelerator of claim 2, further configured to issue a third plurality of memory access commands to a third target sub-block located at the third target address and a fourth plurality of memory access commands to a fourth target sub-block located at the fourth target address.

5. The first striding request is 5. The transformer accelerator of claim 4, comprising the first memory block base address, the second memory block base address, a head address interval, a layer address interval, and the first leading size.

6. The target address generation circuit 6. The transformer accelerator of claim 5, further configured to calculate the first and second sub-block offsets based on the head address interval and the layer address interval.

7. 7. The transformer accelerator of claim 6, wherein each of the first and second sub-block offsets corresponds to a sum of an integer multiple of the head address interval and an integer multiple of the layer address interval.

8. The command issuing circuit configured to read the first and second plurality of memory access commands during a first time interval; 6. The transformer accelerator of claim 5, configured to read the third and fourth pluralities of memory access commands during a second time interval after the first time interval.

9. Among the first plurality of cache vectors, cache vectors included in the first target sub-block are stored at addresses adjacent to each other, 2. The transformer accelerator of claim 1, wherein the cache vectors included in the second target sub-block among the second plurality of cache vectors are stored in adjacent addresses.

10. 1. A transformer accelerator comprising: Execute multiple decoder layers, each of which includes a multi-head attention operation performed based on multiple heads; a first memory block including a first sub-block for storing a first plurality of cache vectors generated for a first plurality of tokens based on a first head that is one of the plurality of heads and a first decoder layer that is one of the plurality of decoder layers; a second memory block including a second sub-block for storing a second plurality of cache vectors generated for a second plurality of tokens based on the first head and the first decoder layer; a memory striding circuit that sequentially accesses the first sub-block and the second sub-block in response to an externally provided first striding request to read the first plurality of cache vectors and the second plurality of cache vectors; and a calculation circuit that performs a first attention calculation for the first head and the first decoder layer based on the first plurality of cache vectors and the second plurality of cache vectors.

Images