Neural network architecture for transformer-based multi-head attention

Abstract

The invention relates to a neural network architecture for transformer-based multi-head attention. A neural network architecture includes a plurality of in-memory processing (PIM) devices configured to perform multi-head attention on a transformer model. Each of the plurality of PIM devices includes: a plurality of memory banks configured to store key vectors and value vectors serving as input data for multi-head attention; and a plurality of processing units, corresponding to the plurality of memory banks, configured to perform multi-head attention via a transformer model using the key vector and the value vector. The memory banks are configured to store the key vector in a first manner and store the value vector in a second manner different from the first manner.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Korean Application No. 10-2024-0036096 filed on March 14, 2024, in the Korean Intellectual Property Office, which is incorporated herein by reference in its entirety. Technical Field

[0003] Various embodiments of the present teachings relate to neural network architectures, and more particularly to neural network architectures for Transformer-based multi-head attention. Background Art

[0004] The field of natural language processing (NLP) has recently been revolutionized by a new generation of deep neural networks utilizing the Transformer architecture. Transformers are deep neural networks that forgo recurrent connections in favor of layered "attention head" circuits that facilitate self-supervised training and inference on large corpora of real-world text. The Transformer architecture represents the meaning of words as numeric vectors in a high-dimensional "embedding" space, where closely related words are grouped together. Unlike previous generation word embeddings, which assigned each word a single static meaning (i.e., a meaning independent of context), Transformers assign contextual meaning to each word through an embedding process that incorporates information about word order. The attention head, the core circuit motif of the Transformer, integrates a weighted sum of the information exposed by different words, with relative weights assigning more attention to some words than others. Within a Transformer, attention heads at each layer work in parallel to update their contextual embeddings, allowing for complex representations of linguistic structure. Summary of the Invention

[0005] According to an embodiment of the present disclosure, a neural network architecture may include multiple in-memory processing (PIM) devices configured to perform multi-head attention on a transformer model. Each of the multiple PIM devices may include: multiple memory banks configured to store key vectors and value vectors used as input data for the multi-head attention; and multiple processing units corresponding to the multiple memory banks and configured to perform multi-head attention via the transformer model using the key vectors and value vectors. The multiple memory banks may be configured to store the key vectors in a first manner and to store the value vectors in a second manner different from the first manner.

[0006] According to an embodiment of the present disclosure, a neural network architecture may include: a plurality of processing-in-memory (PIM) devices configured to perform multi-head attention on a transformer model; a PIM controller configured to control the plurality of PIM devices; a local memory coupled to the PIM controller; and a local processing unit coupled to the PIM controller and the local memory. Each of the plurality of PIM devices may include: a plurality of memory banks configured to store a key vector used in a first matrix-vector multiplication of the multi-head attention and a value vector used in a second matrix-vector multiplication of the multi-head attention; a global buffer configured to store a query vector used in the first matrix-vector multiplication of the multi-head attention and an attention distribution vector used in the second matrix-vector multiplication; and a plurality of processing units corresponding to the plurality of memory banks and configured to perform a first matrix-vector multiplication on the key vector and the query vector, and a second matrix-vector multiplication on the value vector and the attention distribution vector.

[0007] According to one embodiment of the present disclosure, a method for performing a transformer-based multi-head attention operation on a neural network architecture may include: generating multiple query vectors (Q), multiple key vectors (K), and multiple value vectors (V) in at least one PIM device; writing the multiple query vectors, multiple key vectors, and multiple value vectors from the at least one PIM device to at least one local memory; writing the multiple key vectors to multiple storage banks of the at least one PIM device in a first manner; writing the multiple value vectors to multiple storage banks of the at least one PIM device in a second manner; writing the multiple query vectors to multiple global buffers of the at least one PIM device; performing a first matrix multiplication of operands Q·KT in multiple processing units of the at least one PIM device to generate an attention score vector, where KT is the transpose of K; applying a softmax operation in multiple processing units of the at least one PIM device to generate multiple attention distribution vectors; writing the multiple attention distribution vectors to multiple global buffers of the at least one PIM device; and performing a second matrix-vector multiplication on the multiple attention distribution vectors and the multiple value vectors to generate at least one multi-head attention value vector. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A block diagram illustrating an example of a neural network architecture for transformer-based multi-head attention according to an embodiment of the present disclosure.

[0009] Figure 2 To show that Figure 1 Block diagram of an example of a processing-in-memory (PIM) device included in a neural network architecture.

[0010] Figure 3 To show Figure 2 A circuit diagram of an example of a processing unit included in a PIM device.

[0011] Figure 4 To illustrate the embodiment of the present disclosure Figure 1 Figure 2. Neural network architecture processing diagram of the transformer model.

[0012] Figure 5 To illustrate the embodiment of the present disclosure Figure 4 Diagram of the multi-head attention of the Transformer model.

[0013] Figure 6 A diagram illustrating how query vectors, key vectors, and value vectors are generated during one sequence of multi-head attention according to an embodiment of the present disclosure.

[0014] Figure 7 To show Figure 5 Diagram of the first matrix-vector multiplication process in the first, second, and third heads in the first sequence of multi-head attention.

[0015] Figure 8 To show Figure 5 Diagram of the first matrix-vector multiplication process in the first, second, and third heads in the second sequence of multi-head attention.

[0016] Figure 9 To show Figure 5 Diagram of the first matrix-vector multiplication process in the first, second, and third heads in the third sequence of multi-head attention.

[0017] Figure 10 To show Figure 5 Diagram of the first matrix-vector multiplication process in the first, second, and third heads in the fourth sequence of multi-head attention.

[0018] Figure 11 To show Figure 5 Diagram of the second matrix-vector multiplication process in the first, second, and third heads in the first sequence of multi-head attention.

[0019] Figure 12 To show Figure 5 Diagram of the second matrix-vector multiplication process in the first, second, and third heads in the second sequence of multi-head attention.

[0020] Figure 13 To show Figure 5 Diagram of the second matrix-vector multiplication process in the first, second, and third heads in the third sequence of multi-head attention.

[0021] Figure 14 To show Figure 5 Diagram of the second matrix-vector multiplication process in the first, second, and third heads in the fourth sequence of multi-head attention.

[0022] Figure 15 Flowchart showing a process of performing multi-head attention in a neural network architecture according to an embodiment of the present disclosure.

[0023] Figure 16 is a diagram illustrating an example of a key matrix used as an operand in a first matrix-vector multiplication according to an embodiment of the present disclosure.

[0024] Figure 17 A diagram illustrating a method of writing first to eighth key vectors generated in a first sequence of multi-head attention and a first matrix-vector multiplication performed on the first head according to an embodiment of the present disclosure.

[0025] Figure 18 A diagram illustrating a first matrix-vector multiplication performed on a second head when first to eighth key vectors generated in a first sequence of multi-head attention are written according to an embodiment of the present disclosure.

[0026] Figure 19 A diagram illustrating a first matrix-vector multiplication performed on the eighth head when the first to eighth key vectors generated in the first sequence of multi-head attention are written according to an embodiment of the present disclosure.

[0027] Figure 20 A diagram illustrating a method of writing first to eighth key vectors generated in a second sequence of multi-head attention and a first matrix-vector multiplication process performed on the first head according to an embodiment of the present disclosure.

[0028] Figure 21 A diagram illustrating a method of writing the first to eighth key vectors generated in the sixteenth sequence of multi-head attention and a first matrix-vector multiplication process performed on the first head according to an embodiment of the present disclosure.

[0029] Figure 22 A diagram illustrating a method of writing the first to eighth key vectors generated in the seventeenth sequence of multi-head attention and a first matrix-vector multiplication process performed on the first head according to an embodiment of the present disclosure.

[0030] Figure 23 A diagram illustrating a method of writing the first to eighth key vectors generated in the eighteenth sequence of multi-head attention and a first matrix-vector multiplication process performed on the first head according to an embodiment of the present disclosure.

[0031] Figure 24 A diagram illustrating a method of writing the first to eighth key vectors generated in the thirty-second sequence of multi-head attention and a first matrix-vector multiplication process performed on the first head according to an embodiment of the present disclosure.

[0032] Figure 25A diagram illustrating a method of writing the first to eighth key vectors generated in the 241st to 256th sequences of multi-head attention and a first matrix-vector multiplication process performed on the first head of the 256th sequence according to an embodiment of the present disclosure.

[0033] Figure 26 A diagram illustrating a method of writing the first to eighth key vectors generated in the 257th sequence of multi-head attention and a first process of the first matrix-vector multiplication performed on the first head according to an embodiment of the present disclosure.

[0034] Figure 27 A diagram illustrating a second process of the first matrix-vector multiplication performed on the first head when the first to eighth key vectors generated in the 257th sequence of the multi-head attention are written according to an embodiment of the present disclosure.

[0035] Figure 28 2 is a diagram illustrating another example of a method of writing the first to eighth key vectors generated in the first to 257th sequences of multi-head attention into a memory bank of a PIM device according to an embodiment of the present disclosure.

[0036] Figure 29 2 is a diagram illustrating another example of a method of writing the first to eighth key vectors generated in the first to 257th sequences of multi-head attention into a memory bank of a PIM device according to an embodiment of the present disclosure.

[0037] Figure 30 To illustrate the embodiment of the present disclosure Figure 5 Figure 2 shows an example of a value vector generated in a sequence of multi-head attention and a value matrix used as an operand in the second matrix-vector multiplication.

[0038] Figure 31 FIG2 is a diagram illustrating how the first value vector generated in the first sequence of multi-head attention is distributed and stored in the first group of PIM devices according to an embodiment of the present disclosure.

[0039] Figure 32 A diagram illustrating a process according to an embodiment of the present disclosure, in which a first value vector generated in a first sequence of multi-head attention is distributed element-wise and stored in a memory bank of a first set of PIM devices, and in which a second matrix-vector multiplication is performed on the first head.

[0040] Figure 33 2 is a diagram illustrating how the second value vectors generated in the first sequence of multi-head attention are distributed and stored in the second group of PIM devices according to an embodiment of the present disclosure.

[0041] Figure 34A diagram illustrating a process according to an embodiment of the present disclosure, in which a second value vector generated in a first sequence of multi-head attention is distributed element-wise and stored in a memory bank of a second set of PIM devices, and in which a second matrix-vector multiplication is performed on the second head.

[0042] Figure 35 2 is a diagram illustrating how the third value vector generated in the first sequence of multi-head attention is distributed and stored in the first group of PIM devices according to an embodiment of the present disclosure.

[0043] Figure 36 A diagram illustrating a process according to an embodiment of the present disclosure, in which a third value vector generated in a first sequence of multi-head attention is distributed element-wise and stored in a memory bank of a first set of PIM devices, and a second matrix-vector multiplication is performed on a third head in the process.

[0044] Figure 37 2 is a diagram illustrating how the fourth value vector generated in the first sequence of multi-head attention is distributed and stored in the second group of PIM devices according to an embodiment of the present disclosure.

[0045] Figure 38 A diagram illustrating a process according to an embodiment of the present disclosure, in which a fourth value vector generated in a first sequence of multi-head attention is distributed element-wise and stored in a memory bank of a second set of PIM devices, and in which a second matrix-vector multiplication is performed on the fourth head.

[0046] Figure 39 A diagram illustrating a process according to an embodiment of the present disclosure, in which a fifth value vector generated in a first sequence of multi-head attention is distributed element-wise and stored in a memory bank of a first set of PIM devices, and in which a second matrix-vector multiplication is performed on the fifth head.

[0047] Figure 40 A diagram illustrating a process according to an embodiment of the present disclosure, in which a sixth value vector generated in a first sequence of multi-head attention is distributed element-wise and stored in a memory bank of a second set of PIM devices, and in which a second matrix-vector multiplication is performed on the sixth head.

[0048] Figure 41 A diagram illustrating a process according to an embodiment of the present disclosure, in which a seventh value vector generated in a first sequence of multi-head attention is distributed element-wise and stored in a memory bank of a first set of PIM devices, and in which a second matrix-vector multiplication is performed on the seventh head.

[0049] Figure 42A diagram illustrating a process according to an embodiment of the present disclosure, in which an eighth value vector generated in a first sequence of multi-head attention is distributed element-wise and stored in a memory bank of a second set of PIM devices, and in which a second matrix-vector multiplication is performed on the eighth head.

[0050] Figure 43 A diagram illustrating a process according to an embodiment of the present disclosure, in which a first value vector, a third value vector, a fifth value vector, and a seventh value vector generated in a second sequence of multi-head attention are distributed element-wise and stored in a memory bank of a first group of PIM devices, and in which a second matrix-vector multiplication is performed on the seventh head.

[0051] Figure 44 A diagram illustrating a process according to an embodiment of the present disclosure, in which a second value vector, a fourth value vector, a sixth value vector, and an eighth value vector generated in a second sequence of multi-head attention are distributed element-wise and stored in a memory bank of a second group of PIM devices, and in which a second matrix-vector multiplication is performed on the eighth head.

[0052] Figure 45 A diagram illustrating a process according to an embodiment of the present disclosure, in which the first value vector, third value vector, fifth value vector, and seventh value vector generated in the 257th sequence of multi-head attention are distributed element-by-element and stored in the memory bank of the first group of PIM devices, and in which the second matrix-vector multiplication is performed on the seventh head.

[0053] Figure 46 A diagram illustrating a process according to an embodiment of the present disclosure, in which the second value vector, fourth value vector, sixth value vector, and eighth value vector generated in the 257th sequence of multi-head attention are distributed element-by-element and stored in the memory banks of the second group of PIM devices, and in which a second matrix-vector multiplication is performed on the eighth head.

[0054] Figures 47 to 49 A diagram illustrating writing the value vectors generated in the first sequence into a memory bank in a second manner in a neural network architecture according to an embodiment of the present disclosure.

[0055] Figures 50 to 52 A diagram illustrating an example of writing a value vector generated in a second sequence into a memory bank in a second manner in a neural network architecture according to an embodiment of the present disclosure.

[0056] Figures 53 to 55 A diagram illustrating an example of writing a value vector generated in a second sequence into a memory bank in a second manner in a neural network architecture according to an embodiment of the present disclosure.

[0057] Figure 56A diagram illustrating another example of writing a first sequence of value vectors into a memory bank in a second manner in a neural network architecture according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0058] In the following description of the embodiments, it should be understood that the terms "first" and "second" are intended to identify elements, but are not intended to limit the specific number or order of elements. In addition, when an element is referred to as being "above," "on," "above," "below," or "below" another element, it is intended to represent a relative positional relationship, but is not intended to limit the element to directly contact another element, or to certain situations where there is at least one intermediate element between two elements. Therefore, terms such as "above," "above," "above," "below," "below," "below," etc. used herein are only used to describe the purpose of specific embodiments and are not intended to limit the scope of this disclosure.

[0059] Furthermore, when an element is referred to as being "connected" or "coupled" to another element, the element may be directly electrically or mechanically connected or coupled to the other element, or may be indirectly electrically or mechanically connected or coupled to the other element via one or more additional elements between the two elements. Furthermore, when a parameter is referred to as being "predetermined," it may be intended to mean that the value of the parameter is determined before the parameter is used in a process or algorithm. The value of the parameter may be set at the start of the process or algorithm, or may be set during the period that the process or algorithm is executed.

[0060] A logic "high" level and a logic "low" level can be used to describe the logic levels of electrical signals. A signal with a logic "high" level can be distinguished from a signal with a logic "low" level. For example, when a signal with a first voltage corresponds to a signal with a logic "high" level, a signal with a second voltage can correspond to a signal with a logic "low" level. In one embodiment, the logic "high" level can be set to a voltage level that is higher than the voltage level of the logic "low" level. The logic levels of a signal can be set to different or opposite levels depending on the embodiment. For example, a signal with a logic "high" level in one embodiment can be set to a logic "low" level in another embodiment.

[0061] Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. However, the embodiments described herein are only for illustrative purposes and are not intended to limit the scope of the present disclosure.

[0062] Figure 1 A block diagram illustrating an example of a neural network architecture for transformer-based multi-head attention according to an embodiment of the present disclosure.

[0063] Reference Figure 1, the neural network architecture 10 includes a plurality of in-memory processing PIM devices, a PIM controller 120, a local memory 130, and a local processing unit 140. Hereinafter, the neural network architecture 10 including the first to sixteenth PIM devices (PIM0 to PIM15) 110(0) to 110(15) will be described as an example.

[0064] Each of the first to sixteenth PIM devices 110(0), 110(1), ..., and 110(15) may include a data storage circuit (e.g., a memory bank and a global buffer) and a processing unit that performs calculations. In one example, the first to sixteenth PIM devices 110(0) to 110(15) may perform a read operation, a write operation, and a calculation operation. The first to sixteenth PIM devices 110(0) to 110(15) may perform a read operation to provide data stored in the memory bank to the PIM controller 120. The first to sixteenth PIM devices 110(0) to 110(15) may perform a write operation to store data provided from the PIM controller 120 in the memory bank. The first to sixteenth PIM devices 110(0) to 110(15) may perform a calculation operation to perform fully connected layer calculations and matrix-vector multiplications included in the transformer-based multi-head attention.

[0065] In one example, the first to sixteenth PIM devices 110 (0) to 110 (15) can generate query data (or query vector), key data (or key vector) and value data (or value vector) for the input token through a fully connected layer calculation. In one example, the first to sixteenth PIM devices 110 (0) to 110 (15) can perform a first matrix-vector multiplication and a second matrix-vector multiplication of multi-head attention. Here, the first matrix-vector multiplication can be a matrix-vector multiplication of the query data and the transposed key data. The first to sixteenth PIM devices 110 (0) to 110 (15) can generate an attention score (or an attention score vector) through the first matrix-vector multiplication. The second matrix-vector multiplication can be a matrix-vector multiplication of the value data and the attention distribution (or attention distribution vector) generated by applying the softmax function to the attention score obtained from the first matrix-vector multiplication. The first to sixteenth PIM devices 110 (0) to 110 (15) can generate a multi-head attention value (or a multi-head attention value vector) through the second matrix-vector multiplication.

[0066] The first to sixteenth PIM devices 110(0) to 110(15) are coupled to the PIM controller 120 via the first to sixteenth channels CH0 to CH15, respectively. For example, the first PIM device 110(0) is coupled to the PIM controller 120 via the first channel CH0. Therefore, the first PIM device 110(0) communicates with the PIM controller 120 via the first channel CH0. The second PIM device 110(1) is coupled to the PIM controller 120 via the second channel CH1. Therefore, the second PIM device 110(1) communicates with the PIM controller 120 via the second channel CH1. Similarly, the sixteenth PIM device 110(15) is coupled to the PIM controller 120 via the sixteenth channel CH15. Therefore, the sixteenth PIM device 110(15) communicates with the PIM controller 120 via the sixteenth channel CH15.

[0067] The PIM controller 120 is coupled to the first to sixteenth PIM devices 110(0) to 110(15), the local memory 130, the local processing unit 140, and the host. The PIM controller 120 can receive instructions from the host. The PIM controller 120 can control the first to sixteenth PIM devices 110(0) to 110(15), the local memory 130, and the local processing unit 140 based on the instructions from the host. Specifically, the PIM controller 120 controls read operations, write operations, and computation operations of the first to sixteenth PIM devices 110(0) to 110(15). The PIM controller 120 controls read operations and local write operations of the local memory 130. The PIM controller 120 controls local computation operations of the local processing unit 140.

[0068] In one example, the PIM controller 120 may read query data, key data, and value data from the first to sixteenth PIM devices 110(0) to 110(15) to store the query data, key data, and value data in the local memory 130. The PIM controller 120 may read the query data, key data, and value data stored in the local memory 130 to provide the query data, key data, and value data to at least one of the first to sixteenth PIM devices 110(0) to 110(15). The PIM controller 120 may read attention scores or multi-head attention values ​​from the first to sixteenth PIM devices 110(0) to 110(15) to store the attention scores or multi-head attention values ​​in the local memory 130, or provide the attention scores or multi-head attention values ​​to the local processing unit 140. The PIM controller 120 may control the local memory 130 so that the attention scores or multi-head attention values ​​stored in the local memory 130 are provided to the local processing unit 140. The PIM controller 120 may allow the attention distribution generated in the local processing unit 140 to be stored in the local memory 130. The PIM controller 120 may read the attention distribution stored in the local memory 130 to provide the attention distribution to at least one of the first to sixteenth PIM devices 110(0) to 110(15). The PIM controller 120 may receive the attention distribution directly from the local processing unit 140 to provide the attention distribution to at least one of the first to sixteenth PIM devices 110(0) to 110(15).

[0069] The local memory 130 is coupled to the PIM controller 120 and the local processing unit 140. The local memory 130 performs local write operations and local read operations based on control signals from the PIM controller 120. The local memory 130 can perform local write operations to store data provided from the PIM controller 120 or the local processing unit 140. In one example, the local memory 130 can perform local write operations to store query data, key data, and value data provided from the PIM controller 120. In one example, the local memory 130 can perform local write operations to store an attention score provided from the PIM controller 120. In one example, the local memory 130 can perform local write operations to store an attention distribution provided from the local processing unit 140. In one example, the local memory 130 can perform local write operations to store a multi-head attention value provided from the PIM controller 120.

[0070] The local memory 130 may provide the stored data to the PIM controller 120 or the local processing unit 140 through a local read operation. In one example, the local memory 130 may perform a local read operation to provide query data, key data, and vector data to the PIM controller 120. In one example, the local memory 130 may perform a local read operation to provide an attention score to the local processing unit 140. In one example, the local memory 130 may perform a local read operation to provide an attention distribution to the PIM controller 120. Figure 1 In the embodiment, the local memory 130 directly exchanges data with the local processing unit 140 , but in other examples, the local memory 130 may exchange data with the local processing unit 140 through the PIM controller 120 .

[0071] The local processing unit 140 performs local computing operations based on control signals from the PIM controller 120. The local processing unit 140 may perform softmax function processing based on the transformer's multi-head attention. To this end, the local processing unit 140 may receive attention scores required for the softmax function processing from the PIM controller 120 or the local memory 130. In addition, the local processing unit 140 may transmit local computing result data generated by performing the local computing operations (e.g., attention distribution generated by the softmax function processing) to the PIM controller 120 or the local memory 130.

[0072] Figure 2 It shows Figure 1 Block diagram of an example of a processing-in-memory (PIM) device included in a neural network architecture.

[0073] Reference Figure 2 , the PIM device 200 includes a plurality of memory banks BK, a plurality of processing units (PUs), a global buffer (GB), and a plurality of data input / output circuits DQ. Figure 2 In the example, the PIM device 200 includes first to sixteenth memory banks BK0 to BK15, first to sixteenth processing units PU0 to PU15, and first to sixteenth data input / output circuits DQ0 to DQ15. The following description of the PIM device 200 can be applied to Figure 1 Any one of the first to sixteenth PIM devices 110(0) to 110(15) included in the neural network architecture 10.

[0074] The first to sixteenth storage banks BK0 to BK15 can store the first to sixteenth groups of first data D1(1) to D1(16), respectively. The global buffer GB can store the second data D2. The first to sixteenth storage banks BK0 to BK15 are coupled to the first to sixteenth processing units PU0 to PU15, respectively. On the other hand, the global buffer GB is commonly coupled to the first to sixteenth processing units PU0 to PU15. Therefore, the first to sixteenth processing units PU0 to PU15 are respectively provided with the first to sixteenth groups of first data D1(1) to D1(16) for calculation from the first to sixteenth storage banks BK0 to BK15. In addition, the first to sixteenth processing units PU0 to PU15 are commonly provided with the second data D2 for calculation from the global buffer GB. For example, the first processing unit PU0 is provided with the first group of first data D1(1) from the first storage bank BK0 and the second data D2 from the global buffer GB. The second processing unit PU1 is supplied with the second set of first data D1(2) from the second memory bank BK1 and the second data D2 from the global buffer GB. In the same manner, the third to sixteenth processing units PU2 to PU15 are supplied with the third to sixteenth sets of first data D1(3) to D1(16) from the third to sixteenth memory banks BK2 to BK15, respectively, and the second data D2 from the global buffer GB.

[0075] In one example, the first to sixteenth groups of first data D1(1) to D1(16) may be weight data for each layer of the fully connected layer calculation constituting the multi-head attention. The second data D2 may be vector data for each layer of the fully connected layer calculation constituting the multi-head attention. In one example, the first to sixteenth groups of first data D1(1) to D1(16) may be key data used as operands in the first matrix-vector multiplication of the multi-head attention, and the second data D2 may be query data used as an operand in the first matrix-vector multiplication of the multi-head attention. In one example, the first to sixteenth groups of first data D1(1) to D1(16) may be value data used as operands in the second matrix-vector multiplication of the multi-head attention, and the second data D2 may be attention distribution used as an operand in the second matrix-vector multiplication of the multi-head attention.

[0076] The first to sixteenth processing units PU0 to PU15 receive the first to sixteenth groups of first data D1(1) to D1(16) from the first to sixteenth storage banks BK0 to BK15, respectively. The first to sixteenth processing units PU0 to PU15 collectively receive the second data D2 from the global buffer GB. The first to sixteenth processing units PU0 to PU15 perform calculations using the first to sixteenth groups of first data D1(1) to D1(16) and the second data D2 as operands. In one example, the first to sixteenth processing units PU0 to PU15 can perform fully connected layer calculations of multi-head attention, first matrix-vector multiplications, and second matrix-vector multiplications.

[0077] The first to sixteenth data input / output circuits DQ0 to DQ15 are connected to the PIM device 200 and the PIM controller ( Figure 1 In one example, the first to sixteenth data input / output circuits DQ0 to DQ15 can be connected to the PIM controller ( Figure 1 120) receives the second data D2 and receives the first to sixteenth groups of first data D1(1) to D1(16) respectively. The first to sixteenth data input / output circuits DQ0 to DQ15 transmit the first to sixteenth groups of first data D1(1) to D1(16) and the second data D2 to the first to sixteenth memory banks BK0 to BK15 and the global buffer GB of the PIM device 200, respectively. In addition, the first to sixteenth data input / output circuits DQ0 to DQ15 can receive read data from the first to sixteenth memory banks BK0 to BK15, respectively. The first to sixteenth data input / output circuits DQ0 to DQ15 can also receive calculation result data from the first to sixteenth processing units PU0 to PU15, respectively. The first to sixteenth data input / output circuits DQ0 to DQ15 can transmit the read data and the calculation result data to the PIM controller ( Figure 1 120). The first to sixteenth data input / output circuits DQ0 to DQ15 can exchange data with the first to sixteenth processing units PU0 to PU15 and the global buffer GB through global input / output (hereinafter referred to as "GIO") lines. Figure 2 Although not shown, the first to sixteenth data input / output circuits DQ0 to DQ15 may also exchange data with the first to sixteenth memory banks BK0 to BK15, respectively, through GIO lines.

[0078] Figure 3 To show Figure 2 The following description of the processing unit PU can also be applied to the PIM device. Figure 2 The PIM device 200 includes first to sixteenth processing units PU0 to PU15.

[0079] Reference Figure 3 , the processing unit PU includes a multiplication circuit 310, an addition circuit 320, an accumulation circuit 330 and an output circuit 340. In the following, as an example, Figure 3 The processing unit PU is shown to perform a multiplication and accumulation (MAC) operation on the weight data W and the vector data V. The following description of the MAC operation of the processing unit PU can also be applied to the processing unit PU that performs the fully connected layer calculation of the multi-head attention, the first matrix-vector multiplication, and the second matrix-vector multiplication. Figure 3 In FIG. 1 , it is assumed that the amount of data that the processing unit PU can process in one calculation operation is 16 pieces of weight data and 16 pieces of vector data, but this is merely an example and the embodiments are not limited to these amounts.

[0080] The multiplication circuit 310 includes first to sixteenth multipliers MUL0 to MUL15. The first to sixteenth multipliers MUL0 to MUL15 receive first to sixteenth weight data W1 to W16 and first to sixteenth vector data V1 to V16, respectively. The first to sixteenth weight data W1 to W16 correspond to the above reference Figure 2 One of the first to sixteenth groups of first data D1(1) to D1(16) described above. In addition, the first to sixteenth vector data V1 to V16 correspond to the above reference Figure 2 Described second data D2. The first to sixteenth weight data W1 to W16 can be provided from a memory bank. The first to sixteenth vector data V1 to V16 can be provided from a global buffer. The first to sixteenth multipliers MUL0 to MUL15 perform matrix-vector multiplication on the first to sixteenth weight data W1 to W16 and the first to sixteenth vector data V1 to V16 to generate and output first to sixteenth multiplication data DM1 to DM16, respectively. For example, the first multiplier MUL0 performs matrix-vector multiplication on the first weight data W1 and the first vector data V1 to output first multiplication data DM1. The second multiplier MUL1 performs matrix-vector multiplication on the second weight data W2 and the second vector data V2 to generate and output second multiplication data DM2. In the same manner, the remaining multipliers MUL2 to MUL15 output the third to sixteenth multiplication data DM3 to DM16, respectively. The first to sixteenth multiplication data DM1 to DM16 output from the multipliers MUL0 to MUL15 are transferred to the adding circuit 320 .

[0081] The adding circuit 320 can be configured by arranging a plurality of adders ADDER in a hierarchical structure, for example, a tree structure having stages. In this example, the adding circuit 320 includes half adders. However, this is merely an example, and the adding circuit 320 may include full adders. In this example, eight adders ADD11, ..., and ADD18 are provided at the topmost first stage of the adding circuit 320. Four adders ADD21, ..., and ADD24 are provided at the next second stage of the adding circuit 320. Although Figure 3 Although not shown in FIG. 3 , two adders are provided at the next third stage of the adding circuit 320. In addition, a single adder ADD41 is provided at the fourth stage, ie, the lowest stage of the adding circuit 320.

[0082] Each of the first-stage adders ADD11 to ADD18 receives two pieces of multiplication data DM from two of the first to sixteenth multipliers MUL0 to MUL15 of the multiplication circuit 310. Each of the first-stage adders ADD11 to ADD18 performs an addition operation on the input multiplication data DM to generate and output addition data. For example, the first-stage adder ADD11 receives first multiplication data DM1 and second multiplication data DM2 from the first multiplier MUL0 and the second multiplier MUL1, respectively, and adds the first multiplication data DM1 and the second multiplication data DM2 to output addition result data. Similarly, the first-stage adder ADD18 receives fifteenth multiplication data DM15 and sixteenth multiplication data DM16 from the fifteenth multiplier MUL14 and the sixteenth multiplier MUL15, respectively, and adds the fifteenth multiplication data DM15 and the sixteenth multiplication data DM16 to output addition result data.

[0083] Each of the adders ADD21, ..., and ADD24 in the second stage receives addition result data from two of the adders ADD11 to ADD18 in the first stage and performs an addition operation on the addition result data to output result data. For example, the adder ADD21 in the second stage receives addition result data from the adders ADD11 and ADD12 in the first stage. Adder ADD21 adds the input addition result data to output result data. The addition result data output from adder ADD21 has a value that is the sum of the first to fourth multiplication data DM1 to DM4. In this manner, the adder ADD41 in the fourth stage performs an addition operation on the addition result data input from the two adders in the third stage to generate and output multiplication addition data DADD, which is the output data of the addition circuit 320. The multiplication addition data DADD output from the addition circuit 320 is transmitted to the accumulation circuit 330.

[0084] The accumulation circuit 330 performs cumulative addition on the latched data DLAT and the multiplication-addition data DADD transmitted from the addition circuit 320 to generate accumulated data DACC. The accumulation circuit 330 latches the accumulated data DACC to output the accumulated data DACC as latched data DLAT. In one example, the accumulation circuit 330 includes an accumulator (ACC_ADD) 331 and a latch circuit (FF) 332. The accumulator 331 receives the multiplication-addition data DADD input from the addition circuit 320. The accumulator 331 receives the latched data DLAT generated by the previous MAC operation. The accumulator 331 performs cumulative addition on the multiplication-addition data DADD and the latched data DLAT to generate and output accumulated data DACC. The accumulated data DACC output from the accumulator 331 is transmitted to the input terminal of the latch circuit 332. The latch circuit 332 latches and outputs the accumulated data DACC transmitted from the accumulator 331 in synchronization with the clock signal CK_L. The accumulated data DACC output from the latch circuit 332 can be provided to the accumulator 331 as latch data DLAT in the next MAC operation. The accumulated data DACC output from the latch circuit 332 is also transmitted to the output circuit 340.

[0085] The output circuit 340 outputs the accumulated data DACC transmitted from the latch circuit 332 of the accumulator circuit 330 in response to, for example, a logic "high" result read signal RD_RES. In one example, when all scheduled MAC operations are executed in the processing unit PU, the accumulated data DACC transmitted from the latch circuit 332 of the accumulator circuit 330 during the last MAC operation constitutes the MAC result data RESULT. As all scheduled MAC operations are executed, a logic "high" result read signal RD_RES is transmitted to the output circuit 340, for example, from the PIM controller. The output circuit 340 outputs the accumulated data DACC as the MAC result data RESULT in response to the logic "high" result read signal RD_RES. On the other hand, when all scheduled MAC operations in the processing unit PU are not completed, the accumulated data DACC transmitted from the latch circuit 332 of the accumulator circuit 330 is not output from the output circuit 340 as the MAC result data RESULT. In other words, if not all scheduled MAC operations are completed, the result read signal RD_RES sent from the PIM controller remains at a logic "low" level. When the result read signal RD_RES is at a logic "low" level, the output circuit 340 does not output the accumulated data DACC as the MAC result data RESULT. The output circuit 340 may include an activation function circuit (AF) 341 that applies an activation function to the accumulated data DACC, so that the output circuit 340 outputs data obtained by performing the activation function processing on the accumulated data DACC transmitted from the accumulation circuit 330 as the MAC result data RESULT.

[0086] Figure 4 The embodiment of the present disclosure is shown in FIG. Figure 1 The neural network architecture of the transformer model is processed in Figure 2. In addition, Figure 5 This is a diagram showing an embodiment of the present disclosure. Figure 4 Diagram of the multi-head attention of the Transformer model.

[0087] First refer to Figure 4 , the transformer model includes a position embedding block 401, a plurality of transformer blocks 40, and a language model (LM) header block 402. The plurality of transformer blocks 40 are sequentially arranged between the position embedding block 401 and the language model header block 402. Figure 4Only the first transformer block 400 of the multiple transformer blocks 40 is shown in FIG, but each of the remaining transformer blocks is configured identically to the first transformer block 400. The first transformer block 400 receives a position embedding vector as input data from the position embedding block 401 and performs a first transformer calculation to generate output data. The output data generated by the first transformer calculation is input to the second transformer block (not shown). The output data generated by the last transformer block in the multiple transformer blocks 400 is transmitted to the language model header block 402. The data output from the language model header block 402 in the current sequence is used as an input token for the position embedding block 401 in the next sequence. For example, in the first sequence, the position embedding block 401 receives the first input token (i.e., the first input word). In another example, in the first sequence, the position embedding block 401 may receive multiple input tokens. The position embedding block 401 receives the data output from the language model header block 402 as the input token for the second sequence. The position embedding block 401 generates an embedding vector based on the input token. The position embedding block 401 generates a position embedding vector to which the position information vector is added. The position embedding block 401 transmits the position embedding vector to the first transformer block 400.

[0088] The first transformer block (hereinafter referred to as the "transformer block") 400 includes a first layer normalization block 410, a multi-head attention block 420, a first residual connection adder 430, a second layer normalization block 440, a feedforward network block 450, and a second residual connection adder 460. In another example of adopting post-layer normalization, the first layer normalization block 410 can be located after the first residual connection adder 430, and the second layer normalization block 440 can be located after the second residual connection adder 460. In another example, when a different normalization scheme, such as root mean square normalization (RMSNorm), is adopted instead of layer normalization, the first layer normalization block 410 and the second layer normalization block 440 can be replaced by root mean square normalization (RMSNorm) blocks, respectively. In yet another example, the normalization blocks within the transformer block 400 can be omitted.

[0089] The first layer normalization block 410 of the transformer block 400 receives the position embedding vector output from the position embedding block 401. The first layer normalization block 410 performs pre-layer normalization on the position embedding vector to output a first layer normalized vector. The first layer normalized vector has the same dimension as the position embedding vector. Specifically, the first layer normalization block 410 performs normalization of the mean and standard deviation of the sample-based data of the position embedding vector to generate a normalized vector. The first layer normalization block 410 multiplies the normalized vector by the β vector and adds the γ vector to the multiplication result to generate a first layer normalized vector. Here, the β vector and the γ vector are parameters that can be learned. The first layer normalization block 410 transmits the first layer normalized vector as a multi-head attention input vector to the multi-head attention block 420. The multi-headed attention block 420 of the transformer block 400 performs multi-headed attention on the multi-headed attention input vector (i.e., the first-layer normalized vector transmitted from the first-layer normalization block 410) and outputs the result of the multi-headed attention as a multi-headed attention output vector.

[0090] Multi-head attention block 420 reference Figure 5 The multi-head attention block 420 includes a first fully connected layer 421, multiple multi-head attention layers 422, and a second fully connected layer 423. Each multi-head attention layer 422 includes a first matrix-vector multiplication block 422-1, a softmax function block 422-2, and a second matrix-vector multiplication block 422-3. The number n_head of the multi-head attention layers 422 is equal to the number of heads in the multi-head attention.

[0091] The first fully connected layer 421 linearly transforms the first-layer normalized vectors transmitted from the first-layer normalization block 410 to generate the same number of query vectors Q, key vectors K, and value vectors V as the number of heads n_head of the multi-head attention. The query vector Q is responsible for determining how an element (i.e., a word) is related to other elements and is therefore used to calculate the similarity with all other elements in the input sequence. The key vector K, which is the vector used to calculate the similarity with the query vector Q, measures the similarity between the element and other elements. The value vector V is a vector representing the importance of the element and is used to assign weights. The query vector Q can be generated by matrix multiplication of the first-layer normalized vector and the query weight matrix. The key vector K can be generated by matrix multiplication of the first-layer normalized vector and the key weight matrix. The value vector V can be generated by matrix multiplication of the first-layer normalized vector and the value weight matrix. The query vector Q, key vector K, and value vector V generated during this linear transformation process can each have a dimension smaller than the dimension of the first-layer normalized vector. The query vector Q, key vector K, and value vector V are transmitted to the multi-headed attention layer 422. For example, the first query vector, the first key vector, and the first value vector are transmitted to the first multi-headed attention layer. The second query vector, the second key vector, and the second value vector are transmitted to the second multi-headed attention layer. Similarly, the "n_head"th query vector, the "n_head"th key vector, and the "n_head"th value vector are transmitted to the "n_head"th multi-headed attention layer. However, this is just an example. In another example, when grouped query attention is used, the multi-headed attention layer can be arranged in a group format, where one key vector and one value vector share multiple query vectors. As a result, the query vector of each group can also be transmitted to the multi-headed attention layer in a manner that the query vectors share the key vector and value vector of the group.

[0092] Each multi-head attention layer 422 sequentially performs a first matrix-vector multiplication, a softmax function application, and a second matrix-vector multiplication using the query vector Q, the key vector K, and the value vector V passed from the first fully connected layer 421. Specifically, the first matrix-vector multiplication block 422-1 of each multi-head attention layer 422 performs a matrix-vector multiplication on the query vector Q and the transposed key vector K. T Perform the first matrix-vector multiplication Q·K T, to generate a matrix-vector multiplication attention score vector. The softmax function block 422-2 of each multi-head attention layer 422 performs scaling on the attention score vector output from the first matrix-vector multiplication block 422-1. Scaling the attention score vector can be performed by dividing the attention score vector by the square root of the dimension of the query vector (and key vector). Scaling the attention score vector results in a scaled attention score vector. The softmax function block 422-2 applies the softmax function to the scaled attention score vector to generate an attention distribution vector, which is a weighted average of the scaled attention score vector. The second matrix-vector multiplication block 422-3 of each multi-head attention layer 422 performs a second matrix-vector multiplication (SV) on the attention distribution vector and the value vector to generate a multi-head attention value vector. In this way, the “n_head” multi-head attention value vectors generated by the “n_head” multi-head attention layers 422 are all concatenated to generate a “concatenated multi-head attention value vector”. The concatenated multi-head attention value vector generated in the multi-head attention layer 422 is transmitted to the second fully connected layer 423. The second fully connected layer 423 performs a linear transformation operation of multiplying the concatenated multi-head attention value vector transmitted from the multi-head attention layer 422 by a weight matrix to generate a multi-head attention output vector of the same dimension as the embedding vector.

[0093] Refer again Figure 4 The first residual connection adder 430 performs element-by-element addition on the position embedding vector output from the position embedding block 401 and the multi-head attention output vector generated in the second fully connected layer 423 of the multi-head attention block 420. The second layer normalization block 440 performs a second pre-layer normalization on the data generated as a result of the element-by-element addition in the first residual connection adder 430 to output a second layer normalized vector. The feedforward network block 450 performs calculations and applies activation functions to the second layer normalized vector in two fully connected layers to generate and output an output vector. The second residual connection adder 460 performs element-by-element addition on the output vector output from the feedforward network block 450 and the data generated as a result of the element-by-element addition in the first residual connection adder 430 to output a final encoded vector. The final encoded vector output from the second residual connection adder 460 passes through all remaining transformer blocks of the plurality of transformer blocks 40. The final encoded vector output from the last transformer block is then transmitted to the language model head block 402.

[0094] Figure 6 To illustrate the embodiment of the present disclosure Figure 5 A diagram showing how query, key, and value vectors are generated during a sequence of multi-head attention. Figure 6 As an example, the number of heads of multi-head attention ( Figure 5 The n_head) is three, but the embodiment is not limited to having only three heads.

[0095] Reference Figure 6 , the first fully connected layer 421 of the multi-head attention block 420 receives the normalization block ( Figure 4 410) transmits the first layer normalized vector LN. Figure 4 As described above, the first layer normalized vector LN is generated by performing first layer normalization on the position embedding vector generated by adding the position information vector to the embedding vector of the input token. As a result, the first layer normalized vector LN has the same dimension as the embedding vector. That is, the first layer normalized vector LN has a dimension of d_model×1. Here, “d_model” represents the dimension of the embedding vector. The first fully connected layer 421 of the multi-head attention block 420 performs a weight matrix W Q 、W K and W V (for query, key, and value) and the first layer normalized vector LN perform matrix-vector multiplication. The matrix-vector multiplication produces a query / key / value vector having the first query vector Q1, the second query vector Q2, the third query vector Q3, the first key vector K1, the second key vector K2, the third key vector K3, the first value vector V1, the second value vector V2, and the third value vector V3 as elements. The weight matrix W for query, key, and value Q 、W K and W V has a dimension of 3·d_model×d_model, and the first layer normalized vector LN has a dimension of d_model×1, so the resulting query / key / value vector has a dimension of 3·d_model×1. However, this is just an example, and when the heads are arranged in a group format, the query / key / value vectors can have different dimensions. In this embodiment, the first query vector Q1, the second query vector Q2, the third query vector Q3, the first key vector K1, the second key vector K2, the third key vector K3, the first value vector V1, the second value vector V2, and the third value vector V3 all have the same dimension d_head×1. Here, "d_head", which represents the dimension of one head, has a size equal to the value obtained by dividing d_model by 3 (the number of heads). This process of generating query / key / value vectors in the first fully connected layer 421 of the multi-head attention block 420 is repeated every time an input token is input to the transformer model, that is, for each sequence in all sequences.

[0096] Figures 7 to 10 To show Figure 5 Graph of the first matrix-vector multiplication process of multi-head attention. Specifically, Figure 7 It shows Figure 5Graph of the first matrix-vector multiplication process in the first sequence of multi-head attention, Figure 8 It shows Figure 5 Graph of the first matrix-vector multiplication process in the second sequence of multi-head attention, Figure 9 It shows Figure 5 A diagram of the first matrix-vector multiplication process in the third sequence of multi-head attention, and Figure 10 It shows Figure 5 Graph of the first matrix-vector multiplication process in the fourth sequence of multi-head attention. Figures 7 to 10 In , the number of heads n_head of the multi-head attention is 3. In addition, the key vector in this paper has a matrix format, where the key vector generated in the first fully connected layer is transposed.

[0097] First refer to Figure 7 , the first matrix-vector multiplication is performed on the first query vector Q1(1), the second query vector Q2(1), the third query vector Q3(1), the first key vector K1(1), the second key vector K2(1) and the third key vector K3(1) generated in the first sequence n_seq=1 in which the first input token is input. As mentioned above with reference to Figure 4 As described, in the first sequence n_seq=1, multiple input tokens can be input simultaneously. In the first head HEAD_1, a first matrix-vector multiplication is performed on the first key vector (or first key matrix) K1(1) and the first query vector Q1(1). In the second head HEAD_2, a first matrix-vector multiplication is performed on the second key vector (or second key matrix) K2(1) and the second query vector Q2(1). In the third head HEAD_3, a first matrix-vector multiplication is performed on the third key vector (or third key matrix) K3(1) and the third query vector Q3(1). In the first head HEAD_1 of the first sequence n_seq=1, as a result of the first matrix-vector multiplication of the first key vector K1(1) and the first query vector Q1(1), the first attention score vector A1(1) of the first sequence n_seq=1 is generated. In the second head HEAD_2 of the first sequence n_seq=1, as a result of the first matrix-vector multiplication of the second key vector K2(1) and the second query vector Q2(1), a second attention score vector A2(1) of the first sequence n_seq=1 is generated. In addition, in the third head HEAD_3 of the first sequence n_seq=1, as a result of the first matrix-vector multiplication of the third key vector K3(1) and the third query vector Q3(1), a third attention score vector A3(1) of the first sequence n_seq=1 is generated.

[0098] Each of the first key matrix K1(1), the second key matrix K2(1) and the third key matrix K3(1) for the first matrix-vector multiplication of the first sequence n_seq=1 has a size of n_seq×d_head. Here, "n_seq" represents the number of the current sequence, and when the number of tokens input in the first sequence is 1, as shown in this example, the number of the current sequence corresponds to the number of tokens input so far. Therefore, each of the first key vector, the second key vector and the third key vector (or the first key matrix, the second key matrix and the third key matrix) K1(1), K2(1) and K3(1) for the first matrix-vector multiplication of the first sequence n_seq=1 has a size of 1×d_head. In addition, each of the first query vector, the second query vector and the third query vector Q1(1), Q2(1) and Q3(1) for the first sequence n_seq=1 has a size of d_head×1. Therefore, each of the first attention score vector, second attention score vector and third attention score vector A1(1), A2(1) and A3(1) obtained from the first matrix-vector multiplication of the first head, second head and third head HEAD_1, HEAD_2 and HEAD_3 of the first sequence n_seq=1 has a dimension of 1×n_seq, i.e. 1×1.

[0099] Next refer to Figure 8In the second sequence n_seq=2 in which the result of the first sequence n_seq=1 is used as an input token, the first query vector, the second query vector and the third query vector Q1(2), Q2(2) and Q3(2) and the first key vector, the second key vector and the third key vector K1(2), K2(2) and K3(2) are generated. The first matrix-vector multiplication in the first head HEAD_1 of the second sequence n_seq=2 is performed on the first key matrix CONCAT(K1(1), K1(2)) of the second sequence n_seq=2 and the first query vector Q1(2) generated in the second sequence n_seq=2. Here, the first key matrix CONCAT(K1(1), K1(2)) of the second sequence n_seq=2 is formed by concatenating the first key vector K1(1) generated in the first sequence n_seq=1 and the first key vector K1(2) generated in the second sequence n_seq=2 in the direction of increasing rows (i.e., the vertical direction in the figure) (hereinafter referred to as the "row direction"). The first matrix-vector multiplication in the second header HEAD_2 of the second sequence n_seq=2 is performed on the second key matrix CONCAT(K2(1), K2(2)) of the second sequence n_seq=2 and the second query vector Q2(2) generated in the second sequence n_seq=2. Here, the second key matrix CONCAT(K2(1), K2(2)) is a matrix generated by concatenating the second key vector K2(1) generated in the first sequence n_seq=1 and the second key vector K2(2) generated in the second sequence n_seq=2 in the row direction. The first matrix-vector multiplication in the third header HEAD_3 of the second sequence n_seq=2 is performed on the third key matrix CONCAT(K3(1), K3(2)) of the second sequence n_seq=2 and the third query vector Q3(2) generated in the second sequence n_seq=2. Here, the third key matrix CONCAT(K3(1), K3(2)) is a matrix generated by concatenating the third key vector K3(1) generated in the first sequence n_seq=1 and the third key vector K3(2) generated in the second sequence n_seq=2 in the row direction.

[0100] Each of the first key vector K1(2), the second key vector K2(2), and the third key vector K3(2) generated in the second sequence n_seq=2 has a size of 1×d_head, but each of the first key matrix CONCAT(K1(1), K1(2)), the second key matrix CONCAT(K2(1), K2(2)), and the third key matrix CONCAT(K3(1), K3(2)) has a size of n_seq×d_head, that is, 2×d_head. On the other hand, each of the first query vector Q1(2), the second query vector Q2(2), and the third query vector Q3(2) generated in the second sequence n_seq=2 has a size of d_head×1 and is used in the first matrix-vector multiplication with the same size. Therefore, in the first head HEAD_1, as a result of the first matrix-vector multiplication of the first key matrix CONCAT(K1(1), K1(2)) of the second sequence n_seq=2 and the first query vector Q1(2), a first attention score vector CONCAT(A1(2)_1, A1(2)_2) of size n_seq×1 (i.e., size 2×1) is generated. The first attention score vector CONCAT(A1(2)_1, A1(2)_2) of the second sequence n_seq=2 is formed by concatenating the first part A1(2)_1 and the second part A1(2)_2 of the first attention score vector CONCAT(A1(2)_1, A1(2)_2) in the row direction. Here, the first part A1(2)_1 of the first attention score vector CONCAT(A1(2)_1, A1(2)_2) is a vector generated by the first matrix-vector multiplication of the first key vector K1(1) of the first sequence (n_seq=1) and the first query vector Q1(2) of the second sequence (n_seq=2). In addition, the second part A1(2)_2 of the first attention score vector CONCAT(A1(2)_1, A1(2)_2) is a vector generated by the first matrix-vector multiplication of the first key vector K1(2) of the second sequence (n_seq=2) and the first query vector Q1(2) of the second sequence (n_seq=2).

[0101] In the second head HEAD_2, as a result of the first matrix-vector multiplication of the second key matrix CONCAT(K2(1), K2(2)) of the second sequence n_seq=2 and the second query vector Q2(2), a second attention score vector CONCAT(A2(2)_1, A2(2)_2) of size 2×1 is generated. The second attention score vector CONCAT(A2(2)_1, A2(2)_2) of the second sequence n_seq=2 is formed by concatenating the first part A2(2)_1 and the second part A2(2)_2 of the second attention score vector CONCAT(A2(2)_1, A2(2)_2) in the row direction. Here, the first part A2(2)_1 of the second attention score vector CONCAT(A2(2)_1, A2(2)_2) is a vector generated by performing the first matrix-vector multiplication of the second key vector K2(1) of the first sequence n_seq=1 and the second query vector Q2(2) of the second sequence n_seq=2. In addition, the second part A2(2)_2 of the second attention score vector CONCAT(A2(2)_1, A2(2)_2) is a vector generated by performing a first matrix-vector multiplication on the second key vector K2(2) of the second sequence n_seq=2 and the second query vector Q2(2) of the second sequence n_seq=2.

[0102] In the same manner, in the third head HEAD_3, as a result of the first matrix-vector multiplication of the third key matrix CONCAT(K3(1), K3(2)) of the second sequence n_seq=2 and the third query vector Q3(2), a third attention score vector CONCAT(A3(2)_1, A3(2)_2) of size 2×1 is generated. The third attention score vector CONCAT(A3(2)_1, A3(2)_2) of the second sequence n_seq=2 is constructed by concatenating the first part A3(2)_1 and the second part A3(2)_2 of the third attention score vector CONCAT(A3(2)_1, A3(2)_2) in the row direction. Here, the first part A3(2)_1 of the third attention score vector CONCAT(A3(2)_1, A3(2)_2) is a vector generated by performing a first matrix-vector multiplication on the third key vector K3(1) of the first sequence n_seq=1 and the third query vector Q3(2) of the second sequence n_seq=2. In addition, the second part A3(2)_2 of the third attention score vector CONCAT(A3(2)_1, A3(2)_2) is a vector generated by performing a first matrix-vector multiplication on the third key vector K3(2) of the second sequence n_seq=2 and the third query vector Q3(2) of the third sequence n_seq=3.

[0103] Next refer to Figure 9In the third sequence n_seq=3 in which the result of the second sequence n_seq=2 is used as an input token, a first query vector Q1(3), a second query vector Q2(3), and a third query vector Q3(3) and a first key vector K1(3), a second key vector K2(3), and a third key vector K3(3) are generated. The first matrix-vector multiplication in the first header HEAD_1 of the third sequence n_seq=3 is performed on the first key matrix CONCAT(K1(1), K1(2), K1(3)) of the third sequence n_seq=3 and the first query vector Q1(3) generated in the third sequence n_seq=3. Here, the first key matrix CONCAT(K1(1), K1(2), K1(3)) of the third sequence n_seq=3 is formed by concatenating the first key vector K1(1) generated in the first sequence n_seq=1, the first key vector K1(2) generated in the second sequence n_seq=2, and the first key vector K1(3) generated in the third sequence n_seq=3 in the row direction. The first matrix-vector multiplication in the second head HEAD_2 of the third sequence n_seq=3 is performed on the second key matrix CONCAT(K2(1), K2(2), K2(3)) of the third sequence n_seq=3 and the second query vector Q2(3) generated in the third sequence n_seq=3. Here, the second key matrix CONCAT(K2(1), K2(2), K2(3)) of the third sequence n_seq=3 is a matrix generated by concatenating in the row direction the second key vector K2(1) generated in the first sequence n_seq=1, the second key vector K2(2) generated in the second sequence n_seq=2, and the second key vector K2(3) generated in the third sequence n_seq=3. The first matrix-vector multiplication in the third header HEAD_3 of the third sequence n_seq=3 is performed on the third key matrix CONCAT(K3(1), K3(2), K3(3)) of the third sequence n_seq=3 and the third query vector Q3(3) generated in the third sequence n_seq=3. Here, the third key matrix CONCAT(K3(1), K3(2), K3(3)) of the third sequence n_seq=3 is a matrix generated by concatenating the third key vector K3(1) generated in the first sequence n_seq=1, the third key vector K3(2) generated in the second sequence n_seq=2, and the third key vector K3(3) generated in the third sequence n_seq=3 in the row direction.

[0104] Each of the first, second and third key vectors K1(3), K2(3) and K3(3) generated in the third sequence n_seq=3 has a size of 1×d_head, but each of the first, second and third key matrices CONCAT(K1(1), K1(2), K1(3)), CONCAT(K2(1), K2(2), K2(3)) and CONCAT(K3(1), K3(2), K3(3)) used for the first matrix-vector multiplication in the third sequence n_seq has a size of n_seq×d_head, that is, a size of 3×d_head. On the other hand, each of the first, second and third query vectors Q1(3), Q2(3) and Q3(3) generated in the third sequence n_seq=3 has a size of d_head×1 and is used for the first matrix-vector multiplication with the same size. Therefore, in the first head HEAD_1, as a result of the first matrix-vector multiplication of the first key matrix CONCAT(K1(1), K1(2), K1(3)) of the third sequence n_seq=3 and the first query vector Q1(3), a first attention score vector CONCAT(A1(3)_1, A1(3)_2, A1(3)_3) of size n_seq×1 (i.e., a size of 3×1) is generated. In the second head HEAD_2, as a result of the first matrix-vector multiplication of the second key matrix CONCAT(K2(1), K2(2), K2(3)) of the third sequence n_seq=3 and the second query vector Q2(3), a second attention score vector CONCAT(A2(3)_1, A2(3)_2, A2(3)_3) of size 3×1 is generated. Similarly, in the third head HEAD_3, as a result of the first matrix-vector multiplication of the third key matrix CONCAT(K3(1), K3(2), K3(3)) of the third sequence n_seq=3 and the third query vector Q3(3), a third attention score vector CONCAT(A3(3)_1, A3(3)_2, A3(3)_3) of size 3×1 is generated.

[0105] The first attention score vector CONCAT(A1(3)_1, A1(3)_2, A1(3)_3) generated in the first head HEAD_1 of the third sequence n_seq=3 is formed by concatenating the first part A1(3)_1, the second part A1(3)_2 and the third part A1(3)_3 of the first attention score vector CONCAT(A1(3)_1, A1(3)_2, A1(3)_3) in the row direction. Here, the first part A1(3)_1 of the first attention score vector CONCAT(A1(3)_1, A1(3)_2, A1(3)_3) is a vector generated by performing the first matrix-vector multiplication on the first key vector K1(1) of the first sequence n_seq=1 and the first query vector Q1(3) of the third sequence n_seq=3. The second part A1(3)_2 of the first attention score vector CONCAT(A1(3)_1, A1(3)_2, A1(3)_3) is a vector generated by performing a first matrix-vector multiplication on the first key vector K1(2) of the second sequence n_seq=2 and the first query vector Q1(3) of the third sequence n_seq=3. In addition, the third part A1(3)_3 of the first attention score vector CONCAT(A1(3)_1, A1(3)_2, A1(3)_3) is a vector generated by performing a first matrix-vector multiplication on the first key vector K1(3) of the third sequence n_seq=3 and the first query vector Q1(3) of the third sequence n_seq=3.

[0106] The second attention score vector CONCAT(A2(3)_1, A2(3)_2, A2(3)_3) generated in the second head HEAD_2 of the third sequence n_seq=3 is formed by concatenating the first part A2(3)_1, the second part A2(3)_2 and the third part A2(3)_3 of the second attention score vector CONCAT(A2(3)_1, A2(3)_2, A2(3)_3) in the row direction. Here, the first part A2(3)_1 of the second attention score vector CONCAT(A2(3)_1, A2(3)_2, A2(3)_3) is a vector generated by performing the first matrix-vector multiplication on the second key vector K2(1) of the first sequence n_seq=1 and the second query vector Q2(3) of the third sequence n_seq=3. The second part A2(3)_2 of the second attention score vector CONCAT(A2(3)_1, A2(3)_2, A2(3)_3) is a vector generated by performing a first matrix-vector multiplication on the second key vector K2(2) of the second sequence n_seq=2 and the second query vector Q2(3) of the third sequence n_seq=3. In addition, the third part A2(3)_3 of the second attention score vector CONCAT(A2(3)_1, A2(3)_2, A2(3)_3) is a vector generated by performing a first matrix-vector multiplication on the second key vector K2(3) of the third sequence n_seq=3 and the second query vector Q2(3) of the third sequence n_seq=3.

[0107] The third attention score vector CONCAT(A3(3)_1, A3(3)_2, A3(3)_3) generated in the third head HEAD_3 of the third sequence n_seq=3 is formed by concatenating the first part A3(3)_1, the second part A3(3)_2 and the third part A3(3)_3 of the third attention score vector CONCAT(A3(3)_1, A3(3)_2, A3(3)_3) in the row direction. Here, the first part A3(3)_1 of the third attention score vector CONCAT(A3(3)_1, A3(3)_2, A3(3)_3) is a vector generated by performing the first matrix-vector multiplication on the third key vector K3(1) of the first sequence n_seq=1 and the third query vector Q3(3) of the third sequence n_seq=3. The second part A3(3)_2 of the third attention score vector CONCAT(A3(3)_1, A3(3)_2, A3(3)_3) is a vector generated by performing a first matrix-vector multiplication on the third key vector K3(2) of the second sequence n_seq=2 and the third query vector Q3(3) of the third sequence n_seq=3. In addition, the third part A3(3)_3 of the third attention score vector CONCAT(A3(3)_1, A3(3)_2, A3(3)_3) is a vector generated by performing a first matrix-vector multiplication on the third key vector K3(3) of the third sequence n_seq=3 and the third query vector Q3(3) of the third sequence n_seq=3.

[0108] Next refer to Figure 10 In the fourth sequence n_seq=4 in which the result of the third sequence n_seq=3 is used as an input token, the first, second and third query vectors Q1(4), Q2(4) and Q3(4) and the first, second and third key vectors K1(4), K2(4) and K3(4) are generated. The first matrix-vector multiplication in the first header HEAD_1 of the fourth sequence n_seq=4 is performed on the first key matrix CONCAT(K1(1), K1(2), K1(3), K1(4)) of the fourth sequence n_seq=4 and the first query vector Q1(4) generated in the fourth sequence n_seq=4. Here, the first key matrix CONCAT(K1(1), K1(2), K1(3), K1(4)) of the fourth sequence n_seq=4 is formed by concatenating in the row direction the first key vector K1(1) generated in the first sequence n_seq=1, the first key vector K1(2) generated in the second sequence n_seq=2, the first key vector K1(3) generated in the third sequence n_seq=3, and the first key vector K1(4) generated in the fourth sequence n_seq=4.

[0109] The first matrix-vector multiplication in the second head HEAD_2 of the fourth sequence n_seq=4 is performed on the second key matrix CONCAT(K2(1), K2(2), K2(3), K2(4)) of the fourth sequence n_seq=4 and the second query vector Q2(4) generated in the fourth sequence n_seq=4. Here, the second key matrix CONCAT(K2(1), K2(2), K2(3), K2(4)) of the fourth sequence n_seq=4 is formed by concatenating the second key vector K2(1) generated in the first sequence n_seq=1, the second key vector K2(2) generated in the second sequence n_seq=2, the second key vector K2(3) generated in the third sequence n_seq=3, and the second key vector K2(4) generated in the fourth sequence n_seq=4 in the row direction.

[0110] The first matrix-vector multiplication in the third header HEAD_3 of the fourth sequence n_seq=4 is performed on the third key matrix CONCAT(K3(1), K3(2), K3(3), K3(4)) of the fourth sequence n_seq=4 and the third query vector Q3(4) generated in the fourth sequence n_seq=4. Here, the third key matrix CONCAT(K3(1), K3(2), K3(3), K3(4)) of the fourth sequence n_seq=4 is formed by concatenating in the row direction the third key vector K3(1) generated in the first sequence n_seq=1, the third key vector K3(2) generated in the second sequence n_seq=2, the third key vector K3(3) generated in the third sequence n_seq=3, and the third key vector K3(4) generated in the fourth sequence n_seq=4.

[0111] Each of the first, second and third key vectors K1(4), K2(4) and K3(4) generated in the fourth sequence n_seq=4 has a size of 1×d_head, but each of the first, second and third key matrices CONCAT(K1(1), K1(2), K1(3), K1(4)), CONCAT(K2(1), K2(2), K2(3), K2(4)) and CONCAT(K3(1), K3(2), K3(3), K3(4)) used in the first matrix operation in the fourth sequence has a size of n_seq×d_head, that is, a size of 4×d_head. On the other hand, each of the first, second and third query vectors Q1(4), Q2(4) and Q3(4) generated in the fourth sequence n_seq=4 has a size of d_head×1 and is used in the first matrix-vector multiplication with the same size. Therefore, in the first head HEAD_1, as a result of the first matrix-vector multiplication of the first key matrix CONCAT(K1(1), K1(2), K1(3), K1(4)) of the fourth sequence n_seq=4 and the first query vector Q1(4), a first attention score vector CONCAT(A1(4)_1, A1(4)_2, A1(4)_3, A1(4)_4) of size n_seq×1 (i.e., size 4×1) is generated. In the second head HEAD_2, as a result of the first matrix-vector multiplication of the second key matrix CONCAT(K2(1), K2(2), K2(3), K2(4)) of the fourth sequence n_seq=4 and the second query vector Q2(4), a second attention score vector CONCAT(A2(4)_1, A2(4)_2, A2(4)_3, A2(4)_4) of size n_seq×1 (i.e., size 4×1) is generated. In addition, in the third head HEAD_3, as a result of the first matrix-vector multiplication of the third key matrix CONCAT(K3(1), K3(2), K3(3), K3(4)) of the fourth sequence n_seq=4 and the third query vector Q3(4), a third attention score vector CONCAT(A3(4)_1, A3(4)_2, A3(4)_3, A3(4)_4) of size n_seq×1 (i.e., size 4×1) is generated.

[0112] The first attention score vector CONCAT(A1(4)_1, A1(4)_2, A1(4)_3, A1(4)_4) generated in the first head HEAD_1 of the fourth sequence n_seq=4 is formed by concatenating the first part A1(4)_1, the second part A1(4)_2, the third part A1(4)_3 and the fourth part A1(4)_4 of the first attention score vector CONCAT(A1(4)_1, A1(4)_2, A1(4)_3, A1(4)_4) in the row direction. Here, the first part A1(4)_1 of the first attention score vector CONCAT(A1(4)_1, A1(4)_2, A1(4)_3, A1(4)_4) is a vector generated by performing the first matrix-vector multiplication on the first key vector K1(1) of the first sequence n_seq=1 and the first query vector Q1(4) of the fourth sequence n_seq=4. The second part A1(4)_2 of the first attention score vector CONCAT(A1(4)_1, A1(4)_2, A1(4)_3, A1(4)_4) is a vector generated by performing a first matrix-vector multiplication on the first key vector K1(2) of the second sequence n_seq=2 and the first query vector Q1(4) of the fourth sequence n_seq=4. The third part A1(4)_3 of the first attention score vector CONCAT(A1(4)_1, A1(4)_2, A1(4)_3, A1(4)_4) is a vector generated by performing a first matrix-vector multiplication on the first key vector K1(3) of the third sequence n_seq=3 and the first query vector Q1(4) of the fourth sequence n_seq=4. In addition, the fourth part A1(4)_4 of the first attention score vector CONCAT(A1(4)_1, A1(4)_2, A1(4)_3, A1(4)_4) is a vector generated by performing the first matrix-vector multiplication on the first key vector K1(4) of the fourth sequence n_seq=4 and the first query vector Q1(4) of the fourth sequence n_seq=4.

[0113] The second attention score vector CONCAT(A2(4)_1, A2(4)_2, A2(4)_3, A2(4)_4) generated in the second head HEAD_2 of the fourth sequence n_seq=4 is formed by concatenating the first part A2(4)_1, the second part A2(4)_2, the third part A2(4)_3 and the fourth part A2(4)_4 of the second attention score vector CONCAT(A2(4)_1, A2(4)_2, A2(4)_3, A2(4)_4) in the row direction. Here, the first part A2(4)_1 of the second attention score vector CONCAT(A2(4)_1, A2(4)_2, A2(4)_3, A2(4)_4) is a vector generated by performing the first matrix-vector multiplication on the second key vector K2(1) of the first sequence n_seq=1 and the second query vector Q2(4) of the fourth sequence n_seq=4. The second part A2(4)_2 of the second attention score vector CONCAT(A2(4)_1, A2(4)_2, A2(4)_3, A2(4)_4) is a vector generated by performing a first matrix-vector multiplication on the second key vector K2(2) of the second sequence n_seq=2 and the second query vector Q2(4) of the fourth sequence n_seq=4. The third part A2(4)_3 of the second attention score vector CONCAT(A2(4)_1, A2(4)_2, A2(4)_3, A2(4)_4) is a vector generated by performing a first matrix-vector multiplication on the second key vector K2(3) of the third sequence n_seq=3 and the second query vector Q2(4) of the fourth sequence n_seq=4. In addition, the fourth part A2(4)_4 of the second attention score vector CONCAT(A2(4)_1, A2(4)_2, A2(4)_3, A2(4)_4) is a vector generated by performing a first matrix-vector multiplication on the second key vector K2(4) of the fourth sequence n_seq=4 and the second query vector Q2(4) of the fourth sequence n_seq=4.

[0114] The third attention score vector CONCAT(A3(4)_1, A3(4)_2, A3(4)_3, A3(4)_4) generated in the third head HEAD_3 of the fourth sequence n_seq=4 is formed by concatenating the first part A3(4)_1, the second part A3(4)_2, the third part A3(4)_3 and the fourth part A3(4)_4 of the third attention score vector CONCAT(A3(4)_1, A3(4)_2, A3(4)_3, A3(4)_4) in the row direction. Here, the first part A3(4)_1 of the third attention score vector CONCAT(A3(4)_1, A3(4)_2, A3(4)_3, A3(4)_4) is a vector generated by performing the first matrix-vector multiplication on the third key vector K3(1) of the first sequence n_seq=1 and the third query vector Q3(4) of the fourth sequence n_seq=4. The second part A3(4)_2 of the third attention score vector CONCAT(A3(4)_1, A3(4)_2, A3(4)_3, A3(4)_4) is a vector generated by performing a first matrix-vector multiplication on the third key vector K3(2) of the second sequence n_seq=2 and the third query vector Q3(4) of the fourth sequence n_seq=4. The third part A3(4)_3 of the third attention score vector CONCAT(A3(4)_1, A3(4)_2, A3(4)_3, A3(4)_4) is a vector generated by performing a first matrix-vector multiplication on the third key vector K3(3) of the third sequence n_seq=3 and the third query vector Q3(4) of the fourth sequence n_seq=4. In addition, the fourth part A3(4)_4 of the third attention score vector CONCAT(A3(4)_1, A3(4)_2, A3(4)_3, A3(4)_4) is a vector generated by performing the first matrix-vector multiplication on the third key vector K3(4) of the fourth sequence n_seq=4 and the third query vector Q3(4) of the fourth sequence n_seq=4.

[0115] As reference Figures 7 to 10As described above, for the first sequence (i.e., when the number of input tokens is 1), the key vector generated in the first sequence can be used in the same matrix form as the operand of the first matrix-vector multiplication to generate the attention score vector for multi-head attention. On the other hand, for sequences other than the first sequence, a key matrix is ​​used instead of the key vector generated in each sequence. The key matrix is ​​obtained by concatenating all key vectors generated in the previous sequence and the key vector generated in the current sequence. Therefore, the dimensions of the attention score vector generated by the first matrix-vector multiplication, particularly the dimensions in the row direction, vary with the numbered sequence. The key vectors and query vectors generated in all sequences have the same dimensions, so the size of the key vector written to the memory bank of the PIM device for each sequence is fixed. The key vectors numbered up to the current sequence are concatenated into a key matrix, which is used as the operand of the first matrix-vector multiplication performed in the processing unit of the PIM device. Therefore, when writing the key vectors to the memory bank of the PIM device, the key vector generated for each sequence is written to one memory bank of one PIM device, so that the first matrix-vector multiplication can be performed in units of key matrices without requiring separate data alignment. Furthermore, as the number of sequences increases, the degree of parallel execution of the first matrix-vector multiplication can be increased. The process of writing the key vector to the memory bank and the first matrix operation process will be described in more detail below.

[0116] Figures 11 to 14 To show Figure 5 Graph of the second matrix-vector multiplication process of multi-head attention. Specifically, Figure 11 It shows Figure 5 Diagram of the second matrix-vector multiplication process in the first sequence of multi-head attention. Figure 12 It shows Figure 5 Diagram of the second matrix-vector multiplication process in the second sequence of multi-head attention. Figure 13 It shows Figure 5 Graph of the second matrix-vector multiplication process in the third sequence of multi-head attention. In addition, Figure 14 It shows Figure 5 Graph of the second matrix-vector multiplication process in the fourth sequence of multi-head attention. Figures 11 to 14 As an example, the head of the multi-head attention ( Figure 5 The number of n_head) is three, but the embodiment is not limited to only three heads.

[0117] First refer to Figure 11In the first sequence n_seq=1 in which the first input token is input, the second matrix-vector multiplication is performed on the first head HEAD_1, the second head HEAD_2, and the third head HEAD_3, that is, the second matrix-vector multiplication is performed on the value matrix and the attention distribution vector. The first value matrix, the second value matrix, and the third value matrix used as operands in the second matrix-vector multiplication in the first sequence n_seq=1 are respectively the same as the first value vector V1(1), the second value vector V2(1), and the third value vector V3(1) generated in the first sequence n_seq=1.

[0118] By referring to Figure 5 The softmax function block ( Figure 5 The scaling and softmax function processing in 422-2) generates the first, second and third attention distribution vectors S1(1), S2(1) and S3(1) used as operands in the second matrix-vector multiplication in the first sequence n_seq=1 together with the value matrix. Specifically, as shown in FIG. Figure 7 As described, the first attention score vector A1(1) of the first sequence n_seq=1 is generated by the first matrix-vector multiplication of the first head HEAD_1 of the first sequence n_seq=1. The first attention score vector A1(1) of the first sequence n_seq=1 undergoes scaling processing and softmax function processing in the softmax function block 422-2 to obtain the first attention distribution vector S1(1) of the first sequence n_seq=1. The second attention score vector A2(1) of the first sequence n_seq=1 is generated by the first matrix-vector multiplication of the second head HEAD_2 in the first sequence n_seq=1. The second attention score vector A2(1) of the first sequence n_seq=1 is also subjected to scaling processing and softmax function processing in the softmax function block 422-2 to obtain the second attention distribution vector S2(1) of the first sequence n_seq=1. The third attention score vector A3(1) of the first sequence n_seq=1 is generated by the first matrix-vector multiplication of the third head HEAD_3 in the first sequence n_seq=1. The third attention score vector A3(1) of the first sequence n_seq=1 is also subjected to scaling processing and softmax function processing in the softmax function block 422-2 to obtain the third attention distribution vector S3(1) of the first sequence n_seq=1. Therefore, the first, second and third attention distribution vectors S1(1), S2(1) and S3(1) of the first sequence n_seq=1 have the same dimensions as the first, second and third attention score vectors A1(1), A2(1) and A3(1) of the first sequence n_seq=1, i.e., dimensions of 1×1.

[0119] In the first head HEAD_1 of the first sequence n_seq=1, the second matrix-vector multiplication is performed on the first value matrix V1(1) of the first sequence n_seq=1 and the first attention distribution vector S1(1) of the first sequence n_seq=1, obtaining the first multi-head attention value vector Z1(1) of the first sequence n_seq=1. Because the first value matrix V1(1) of the first sequence n_seq=1 has a size of d_head×n_seq, and the first attention distribution vector S1(1) of the first sequence n_seq=1 has a size of n_seq×1, the first multi-head attention value vector Z1(1) of the first sequence n_seq=1 has a size of d_head×1. Because the size d_head of the head is a preset value for multi-head attention, the first multi-head attention value vector Z1(1) has a fixed size.

[0120] In the second head HEAD_2 of the first sequence n_seq=1, the second matrix-vector multiplication is performed on the second value matrix V2(1) of the first sequence n_seq=1 and the second attention distribution vector S2(1) of the first sequence n_seq=1, obtaining the second multi-head attention value vector Z2(1) of the first sequence n_seq=1. Because the second value matrix V2(1) of the first sequence n_seq=1 has a size of d_head×n_seq, and the second attention distribution vector S2(1) of the first sequence n_seq=1 has a size of n_seq×1, the second multi-head attention value vector Z2(1) of the first sequence n_seq=1 also has a fixed size of d_head×1.

[0121] In the third head HEAD_3 of the first sequence n_seq=1, the second matrix-vector multiplication is performed on the third value matrix V3(1) of the first sequence n_seq=1 and the third attention distribution vector S3(1) of the first sequence n_seq=1, obtaining the third multi-head attention value vector Z3(1) of the first sequence n_seq=1. Because the third value matrix V3(1) of the first sequence n_seq=1 has a size of d_head×n_seq, and the third attention distribution vector S3(1) of the first sequence n_seq=1 has a size of n_seq×1, the third multi-head attention value vector Z3(1) of the first sequence n_seq=1 also has a fixed size of d_head×1. Therefore, the first, second and third multi-head attention value vectors Z1(1), Z2(1) and Z3(1) generated by the second matrix-vector multiplication of the first head HEAD_1, the second head HEAD_2 and the third head HEAD_3 of the first sequence n_seq=1 respectively have the same size d_head×1.

[0122] Next refer to Figure 12, in the second sequence n_seq=2 in which the second input token is input, for each of the first head HEAD_1, the second head HEAD_2 and the third head HEAD_3, the second matrix-vector multiplication is performed on the value matrix and the attention distribution vector. Specifically, the second matrix-vector multiplication for the first head HEAD_1 in the second sequence n_seq=2 is performed on the first value matrix CONCAT(V1(1), V1(2)) of the second sequence n_seq=2 and the first attention distribution vector CONCAT(S1(2)_1, S1(2)_2)) of the second sequence n_seq=2. Here, the first value matrix CONCAT(V1(1), V1(2)) of the second sequence n_seq=2 is formed by concatenating the first value vector V1(1) generated in the first sequence n_seq=1 and the first value vector V1(2) generated in the second sequence n_seq=2 in the direction of increasing columns (hereinafter referred to as "column direction"). The first attention distribution vector CONCAT(S1(2)_1, S1(2)_2) of the second sequence n_seq=2 is formed by concatenating the first part S1(2)_1 and the second part S1(2)_2 of the first attention distribution vector CONCAT(S1(2)_1, S1(2)_2) generated in the first head HEAD_1 of the second sequence n_seq=2 in the row direction.

[0123] The second matrix-vector multiplication for the second head HEAD_2 of the second sequence n_seq=2 is performed on the second value matrix CONCAT(V2(1), V2(2)) of the second sequence n_seq=2 and the second attention distribution vector CONCAT(S2(2)_1, S2(2)_2) of the second sequence n_seq=2. Here, the second value matrix CONCAT(V2(1), V2(2)) of the second sequence n_seq=2 is formed by concatenating the second value vector V2(1) generated in the first sequence n_seq=1 and the second value vector V2(2) generated in the second sequence n_seq=2 in the column direction. The second attention distribution vector CONCAT(S2(2)_1, S2(2)_2) of the second sequence n_seq=2 is formed by concatenating the first part S2(2)_1 and the second part S2(2)_2 of the second attention distribution vector CONCAT(S2(2)_1, S2(2)_2) generated in the second head HEAD_2 of the second sequence n_seq=2 in the row direction.

[0124] The second matrix-vector multiplication for the third head HEAD_3 of the second sequence n_seq=2 is performed on the third value matrix CONCAT(V3(1), V3(2)) of the second sequence n_seq=2 and the third attention distribution vector CONCAT(S3(2)_1, S3(2)_2) of the second sequence n_seq=2. Here, the third value matrix CONCAT(V3(1), V3(2)) of the second sequence n_seq=2 is formed by concatenating the third value vector V3(1) generated in the first sequence n_seq=1 and the third value vector V3(2) generated in the second sequence n_seq=2 in the column direction. The third attention distribution vector CONCAT(S3(2)_1, S3(2)_2) of the second sequence n_seq=2 is formed by concatenating the first part S3(2)_1 and the second part S3(2)_2 of the third attention distribution vector CONCAT(S3(2)_1, S3(2)_2) generated in the third head HEAD_3 of the second sequence n_seq=2 in the row direction.

[0125] By referring to the above Figure 8 The scaling processing and softmax function processing of the first, second and third attention score vectors CONCAT(A1(2)_1, A1(2)_2), CONCAT(A2(2)_1, A2(2)_2) and CONCAT(A3(2)_1, A3(2)_2) generated by the first matrix-vector multiplication of the second sequence (n_seq=2) described are performed, and the first, second and third attention distribution vectors CONCAT(S1(2)_1, S1(2)_2), CONCAT(S2(2)_1, S2(2)_2) and CONCAT(S3(2)_1, S3(2)_2) used as operands in the second matrix-vector multiplication in the second sequence n_seq=2 are generated together with the value matrix.

[0126] Specifically, as reference Figure 8 As described above, in the second sequence n_seq=2, when the first matrix-vector multiplication of the first head HEAD_1 is performed in the first matrix-vector multiplication block 422-1 of the multi-head attention block 420, the first part A1(2)_1 and the second part A1(2)_2 of the first attention score vector constituting the first attention score vector CONCAT(A1(2)_1, A1(2)_2) are generated. As shown in FIG. Figure 5As described above, the first part A1(2)_1 and the second part A1(2)_2 of the first attention score vector of the second sequence n_seq=2 undergo scaling processing and softmax function processing in the softmax function block 422-2, thereby obtaining the first part S1(2)_1 and the second part S1(2)_2 of the first attention distribution vector of the second sequence n_seq=2. The first attention distribution vector CONCAT(S1(2)_1, S1(2)_2) of the second sequence n_seq=2 is formed by concatenating the first part S1(2)_1 and the second part S1(2)_2 of the first attention distribution vector of the second sequence n_seq=2 in the row direction.

[0127] In the second sequence n_seq=2, when the first matrix-vector multiplication of the second head HEAD_2 is performed in the first matrix-vector multiplication block 422-1 of the multi-head attention block 420, the first part A2(2)_1 and the second part A2(2)_2 constituting the second attention score vector CONCAT(A2(2)_1, A2(2)_2) are generated. Scaling processing and softmax function processing are also performed on the first part A2(2)_1 and the second part A2(2)_2 of the second attention score vector of the second sequence n_seq=2 in the softmax function block 422-2, thereby obtaining the first part S2(2)_1 and the second part S2(2)_2 of the second attention distribution vector of the second sequence n_seq=2. The second attention distribution vector CONCAT(S2(2)_1, S2(2)_2) of the second sequence n_seq=2 is formed by concatenating the first part S2(2)_1 and the second part S2(2)_2 of the second attention distribution vector of the second sequence n_seq=2 in the row direction.

[0128] In the second sequence n_seq=2, when the first matrix-vector multiplication for the third head HEAD_3 is performed in the first matrix-vector multiplication block 422-1 of the multi-head attention block 420, the first part A3(2)_1 and the second part A3(2)_2 of the third attention score vector constituting the third attention score vector CONCAT(A3(2)_1, A3(2)_2) are generated. Scaling and softmax function processing are also performed on the first part A3(2)_1 and the second part A3(2)_2 of the third attention score vector of the second sequence n_seq=2, thereby obtaining the first part S3(2)_1 and the second part S3(2)_2 of the third attention distribution vector of the second sequence n_seq=2. The third attention distribution vector (CONCAT(S3(2)_1, S3(2)_2) of the second sequence n_seq=2 is formed by concatenating the first part S3(2)_1 and the second part S3(2)_2 of the third attention distribution vector of the second sequence n_seq=2 in the row direction.

[0129] As a result of the second matrix-vector multiplication of the first head HEAD_1 of the second sequence n_seq=2, the first multi-head attention value vector Z1(2) is generated. The first value matrix CONCAT(V1(1), V1(2)) of the second sequence n_seq=2 has a size of d_head×n_seq, that is, a size of d_head×2, and the first attention distribution vector CONCAT(S1(2)_1, S1(2)_2) of the second sequence n_seq=2 has a size of n_seq×1, that is, a size of 2×1, so the first multi-head attention value vector Z1(2) of the second sequence n_seq=2 has a fixed size of d_head×1.

[0130] As a result of the second matrix-vector multiplication of the second head HEAD_2 of the second sequence n_seq=2, the second multi-head attention value vector Z2(2) is generated. The second value matrix (V2(1), V2(2)) of the second sequence n_seq=2 has a size of d_head×n_seq, that is, a size of d_head×2, and the second attention distribution vector CONCAT(S2(2)_1, S2(2)_2) of the second sequence n_seq=2 has a size of n_seq×1, that is, a size of 2×1, so the second multi-head attention value vector Z2(2) of the second sequence n_seq=2 also has a fixed size of d_head×1.

[0131] As a result of the second matrix-vector multiplication of the third head HEAD_3 of the second sequence n_seq=2, the third multi-head attention value vector Z3(2) is generated. The third value matrix (V3(1), V3(2)) of the second sequence n_seq=2 has a size of d_head×n_seq, that is, a size of d_head×2, and the third attention distribution vector CONCAT(S3(2)_1, S3(2)_2) of the second sequence n_seq=2 has a size of n_seq×1, that is, a size of 2×1, so the third multi-head attention value vector Z3(2) of the second sequence n_seq=2 also has a fixed size of d_head×1. Therefore, the first, second and third multi-head attention value vectors Z1(2), Z2(2) and Z3(2) of the second sequence n_seq=2, generated by the second matrix-vector multiplication of the first head HEAD_1, the second head HEAD_2 and the third head HEAD_3 of the second sequence n_seq=2, respectively, all have the same size d_head×1.

[0132] Next, refer to Figure 13 , in the third sequence n_seq=3, for the first head HEAD_1, the second head HEAD_2, and the third head HEAD_3, the second matrix-vector multiplication is performed on the value matrix and the attention distribution vector. The second matrix-vector multiplication for the first head HEAD_1 in the third sequence n_seq=3 is performed on the first value matrix CONCAT(V1(1), V1(2), V1(3)) of the third sequence n_seq=3 and the first attention distribution vector CONCAT(S1(3)_1, S1(3)_2, S1(3)_3) of the third sequence n_seq=3. Here, the first value matrix CONCAT(V1(1), V1(2), V1(3)) of the third sequence n_seq=3 is formed by concatenating the first value vector V1(1) generated in the first sequence n_seq=1, the first value vector V1(2) generated in the second sequence n_seq=2, and the first value vector V1(3) generated in the third sequence n_seq=3 in the column direction. The first attention distribution vector CONCAT(S1(3)_1, S1(3)_2, S1(3)_3)) of the third sequence n_seq=3 has a vector format and is formed by concatenating the first part S1(3)_1, the second part S1(3)_2, and the third part S1(3)_3 of the first attention distribution vector generated in the first head HEAD_1 of the third sequence n_seq=3 in the row direction.

[0133] The second matrix-vector multiplication of the second head HEAD_2 of the third sequence n_seq=3 is performed on the second value matrix CONCAT(V2(1), V2(2), V2(3)) of the third sequence n_seq=3 and the second attention distribution vector CONCAT(S2(3)_1, S2(3)_2, S2(3)_3) of the third sequence n_seq=3. Here, the second value matrix CONCAT(V2(1), V2(2), V2(3)) of the third sequence n_seq=3 is formed by concatenating the second value vector V2(1) generated in the first sequence n_seq=1, the second value vector V2(2) generated in the second sequence n_seq=2, and the second value vector V2(3) generated in the third sequence n_seq=3 in the column direction. The second attention distribution vector CONCAT(S2(3)_1, S2(3)_2, S2(3)_3) of the third sequence n_seq=3 has a vector format and is formed by concatenating the first part S2(3)_1, the second part S2(3)_2 and the third part S2(3)_3 of the second attention distribution vector generated in the second head HEAD_2 of the third sequence n_seq=3 in the row direction.

[0134] The second matrix-vector multiplication for the third head HEAD_3 of the third sequence n_seq=3 is performed on the third value matrix CONCAT(V3(1), V3(2), V3(3)) of the third sequence n_seq=3 and the third attention distribution vector CONCAT(S3(3)_1, S3(3)_2, S3(3)_3) of the third sequence n_seq=3. Here, the third value matrix CONCAT(V3(1), V3(2), V3(3)) of the third sequence n_seq=3 is formed by concatenating the third value vector V3(1) generated in the first sequence n_seq=1, the third value vector V3(2) generated in the second sequence n_seq=2, and the third value vector V3(3) generated in the third sequence n_seq=3 in the column direction. The third attention distribution vector CONCAT(S3(3)_1, S3(3)_2, S3(3)_3)) of the third sequence n_seq=3 has a vector format and is formed by concatenating the first part S3(3)_1, the second part S3(3)_2 and the third part S3(3)_3 of the third attention distribution vector generated in the third head HEAD_3 of the third sequence n_seq=3 in the row direction.

[0135] Together with the first value matrix, the second value matrix and the third value matrix, the first to third parts S1(3)_1 to S1(3)_3 of the first attention distribution vector CONCAT(S1(3)_1, S1(3)_2, S1(3)_3), the first to third parts S2(3)_1 to S2(3)_3 of the second attention distribution vector CONCAT(S2(3)_1, S2(3)_2, S2(3)_3) and the first to third parts S3(3)_1 to S3(3)_3 of the third attention distribution vector CONCAT(S3(3)_1, S3(3)_2, S3(3)_3) are as described above with reference to Figure 5 The ground is generated by scaling and softmax function processing in the softmax function block 422-2.

[0136] Specifically, as reference Figure 9 As described above, in the third sequence n_seq=3, when the first matrix-vector multiplication is performed on the first head HEAD_1 in the first matrix-vector multiplication block 422-1 of the multi-head attention block 420, the first part A1(3)_1, the second part A1(3)_2 and the third part A1(3)_3 of the first attention score vector CONCAT(A1(1), A1(2), A1(3)) are generated. The first to third parts A1(3)_1 to A1(3)_3 of the first attention score vector in the third sequence n_seq=3 undergo scaling processing and softmax function processing in the softmax function block 422-2, as described above with reference to Figure 5 As described above, the first to third parts S1(3)_1 to S1(3)_3 of the first attention distribution vector in the third sequence n_seq=3 are obtained. The first to third parts S1(3)_1 to S1(3)_3 of the first attention distribution vector in the third sequence n_seq=3 are concatenated in the row direction to form the first attention distribution vector CONCAT(S3(3)_1, S3(3)_2, S3(3)_3) in the third sequence n_seq=3.

[0137] In the third sequence n_seq=3, when the first matrix-vector multiplication is performed on the second head HEAD_2 in the first matrix-vector multiplication block 422-1 of the multi-head attention block 420, the first part A2(3)_1, the second part A2(3)_2 and the third part A2(3)_3 of the second attention score vector CONCAT(A2(3)_1, A2(3)_2, A2(3)_3) are generated. Scaling and softmax function processing are also performed on the first to third parts A2(3)_1 to A2(3)_3 of the second attention score vector of the third sequence n_seq=3 in the softmax function block 422-2 to obtain the first part S2(3)_1, the second part S2(3)_2 and the third part S2(3)_3 of the second attention distribution vector of the third sequence n_seq=3. The first to third parts S2(3)_1 to S2(3)_3 of the second attention distribution vector of the third sequence n_seq=3 are concatenated in the row direction to form the second attention distribution vector CONCAT(S2(3)_1, S2(3)_2, S2(3)_3) in the third sequence n_seq=3.

[0138] In the third sequence n_seq=3, when the first matrix-vector multiplication is performed on the third head HEAD_3 in the first matrix-vector multiplication block 422-1 of the multi-head attention block 420, the first part A3(3)_1, the second part A3(3)_2 and the third part A3(3)_3 of the third attention score vector CONCAT(A3(3)_1, A3(3)_2, A3(3)_3) are generated. Scaling and softmax function processing are also performed on the first to third parts A3(3)_1 to A3(3)_3 of the third attention score vector in the third sequence n_seq=3 in the softmax function block 422-2, thereby obtaining the first to third parts S3(3)_1 to S3(3)_3 of the third attention distribution vector in the third sequence n_seq=3. The first to third parts S3(3)_1 to S3(3)_3 of the third attention distribution vector of the third sequence n_seq=3 are concatenated in the row direction to form the third attention distribution vector CONCAT(S3(3)_1, S3(3)_2, S3(3)_3) in the third sequence n_seq=3.

[0139] As a result of the second matrix-vector multiplication of the first head HEAD_1 of the third sequence n_seq=3, a first multi-head attention value vector Z1(3) is generated. The first value matrix CONCAT(V1(1), V1(2), V1(3)) of the third sequence n_seq=3 has a size of d_head×n_seq, i.e., a size of d_head×3, and the first attention distribution vector CONCAT(S1(3)_1, S1(3)_2, S1(3)_3) has a size of n_seq×1, i.e., a size of 3×1, so the first multi-head attention value vector Z1(3) of the third sequence n_seq=3 has a fixed size of d_head×1. As a result of the second matrix-vector multiplication of the second head HEAD_2 of the third sequence n_seq=3, a second multi-head attention value vector Z2(3) is generated. The second value matrix CONCAT(V2(1), V2(2), V2(3)) of the third sequence n_seq=3 has a size of d_head×n_seq, i.e., a size of d_head×3, and the second attention distribution vector CONCAT(S2(3)_1, S2(3)_2, S2(3)_3)) has a size of n_seq×1, i.e., a size of 3×1, so the second multi-head attention value vector Z2(3) also has a size of d_head×1. As a result of the second matrix-vector multiplication of the third head HEAD_3 of the third sequence n_seq=3, the third multi-head attention value vector Z3(3) is generated. The third value matrix CONCAT(V3(1), V3(2), V3(3)) of the third sequence n_seq=3 has a size of d_head×n_seq, that is, a size of d_head×3, and the third attention distribution vector CONCAT(S3(3)_1, S3(3)_2, S3(3)_3) of the third sequence n_seq=3 has a size of n_seq×1, that is, a size of 3×1, so the third multi-head attention value vector Z3(3) of the third sequence n_seq=3 also has a fixed size of d_head×1. Therefore, the first, second and third multi-head attention value vectors Z1(3), Z2(3) and Z3(3) generated by the second matrix-vector multiplication of the first head HEAD_1, the second head HEAD_2 and the third head HEAD_3 of the third sequence n_seq=3 respectively all have the same size of d_head×1.

[0140] Next refer to Figure 14, in the fourth sequence n_seq=4, for the first head HEAD_1, the second head HEAD_2, and the third head HEAD_3, the second matrix-vector multiplication is performed on the value matrix and the attention distribution matrix. The second matrix-vector multiplication for the first head HEAD_1 in the fourth sequence n_seq=4 is performed on the first value matrix CONCAT(V1(1), V1(2), V1(3), V1(4)) and the first attention distribution vector CONCAT(S1(4)_1, S1(4)_2, S1(4)_3, S1(4)_4) of the fourth sequence n_seq=4. Here, the first value matrix CONCAT(V1(1), V1(2), V1(3), V1(4)) of the fourth sequence n_seq=4 is formed by concatenating in the column direction the first value vector V1(1) generated in the first sequence n_seq=1, the first value vector V1(2) generated in the second sequence n_seq=2, the first value vector V1(3) generated in the third sequence n_seq=3, and the first value vector V1(4) generated in the fourth sequence n_seq=4. The first attention distribution vector CONCAT(S1(4)_1, S1(4)_2, S1(4)_3, S1(4)_4) of the fourth sequence n_seq=4 is formed by concatenating in the row direction the first to fourth parts S1(4)_1 to S1(4)_4 of the first attention distribution vector generated in the first head HEAD_1 of the fourth sequence n_seq=4.

[0141] The second matrix-vector multiplication of the second head HEAD_2 for the fourth sequence n_seq=4 is performed on the second value matrix CONCAT(V2(1), V2(2), V2(3), V2(4)) and the second attention distribution vector CONCAT(S2(4)_1, S2(4)_2, S2(4)_3, S2(4)_4). Here, the second value matrix CONCAT(V2(1), V2(2), V2(3), V2(4)) of the fourth sequence n_seq=4 is formed by concatenating the second value vector V2(1) generated in the first sequence n_seq=1, the second value vector V2(2) generated in the second sequence n_seq=2, the second value vector V2(3) generated in the third sequence n_seq=3, and the second value vector V2(4) generated in the fourth sequence n_seq=4 in the column direction. The second attention distribution vector CONCAT(S2(4)_1, S2(4)_2, S2(4)_3, S2(4)_4) of the fourth sequence n_seq=4 is formed by concatenating the first to fourth parts S2(4)_1 to S2(4)_4 of the second attention distribution vector generated in the second head HEAD_2 of the fourth sequence n_seq=4 in the row direction.

[0142] The second matrix-vector multiplication of the third head HEAD_3 for the fourth sequence n_seq=4 is performed on the third value matrix CONCAT(V3(1), V3(2), V3(3), V3(4)) and the third attention distribution vector CONCAT(S3(4)_1, S3(4)_2, S3(4)_3, S3(4)_4). Here, the third value matrix CONCAT(V3(1), V3(2), V3(3), V3(4)) for the fourth sequence n_seq=4 is formed by concatenating the third value vector V3(1) generated in the first sequence n_seq=1, the third value vector V3(2) generated in the second sequence n_seq=2, the third value vector V3(3) generated in the third sequence n_seq=3, and the fourth value vector V3(4) generated in the fourth sequence n_seq=4 in the column direction. The third attention distribution vector CONCAT(S3(4)_1, S3(4)_2, S3(4)_3, S3(4)_4) of the fourth sequence n_seq=4 is formed by concatenating the first to fourth parts S3(4)_1 to S3(4)_4 of the third attention distribution vector generated in the third head HEAD_3 of the fourth sequence n_seq=4 in the row direction.

[0143] The first to fourth parts S1(4)_1 to S1(4)_4 of the first attention distribution vector, the first to fourth parts S2(4)_1 to S2(4)_4 of the second attention distribution vector, and the first to fourth parts S3(4)_1 to S3(4)_4 of the third attention distribution vector, which are used as operands in the second matrix-vector multiplication in the fourth sequence n_seq=4 together with the first value matrix, the second value matrix, and the third value matrix, are generated by scaling and softmax function processing in the softmax function block 422-2, as described above with reference to Figure 5 As stated.

[0144] Specifically, as mentioned above Figure 10 As described above, in the fourth sequence n_seq=4, when the first matrix-vector multiplication is performed on the first head HEAD_1 in the first matrix-vector multiplication block 422-1 of the multi-head attention block 420, the first to fourth parts A1(4)_1 to A1(4)_4 of the first attention score vector CONCAT(A1(4)_1, A1(4)_2, A1(4)_3, A1(4)_4) are generated. As described above with reference to Figure 5As described above, scaling and softmax function processing are performed on the first to fourth parts A1(4)_1 to A1(4)_4 of the first attention score vector of the fourth sequence n_seq=4 in the softmax function block 422-2. As a result, the first to fourth parts S1(4)_1 to S1(4)_4 of the first attention distribution vector of the fourth sequence n_seq=4 are generated. The first to fourth parts S1(4)_1 to S1(4)_4 of the first attention distribution vector of the fourth sequence n_seq=4 are concatenated in the row direction to form the first attention distribution vector CONCAT(S1(4)_1, S1(4)_2, S1(4)_3, S1(4)_4) of the fourth sequence n_seq=4.

[0145] In the fourth sequence n_seq=4, when the first matrix-vector multiplication is performed on the second head HEAD_2 in the first matrix-vector multiplication block 422-1 of the multi-head attention block 420, the first to fourth parts A2(4)_1, A2(4)_2, A2(4)_3, A2(4)_4 of the second attention score vector CONCAT(A2(4)_1, A2(4)_2, A2(4)_3, A2(4)_4) are generated. In the softmax function block 422-2, scaling and softmax function processing are also performed on the first to fourth parts A2(4)_1 to A2(4)_4 of the second attention score vector of the fourth sequence n_seq=4. As a result, the first to fourth parts S2(4)_1 to S2(4)_4 of the second attention distribution vector of the fourth sequence n_seq=4 are generated. The first to fourth parts S2(4)_1 to S2(4)_4 of the second attention distribution vector in the fourth sequence n_seq=4 are concatenated in the row direction to form the second attention distribution vector CONCAT(S2(4)_1, S2(4)_2, S2(4)_3, S2(4)_4) in the fourth sequence n_seq=4.

[0146] In the fourth sequence n_seq=4, when the first matrix-vector multiplication is performed on the third head HEAD_3 in the first matrix-vector multiplication block 422-1 of the multi-head attention block 420, the first to fourth parts A3(4)_1 to A3(4)_4 of the third attention score vector CONCAT(A3(4)_1, A3(4)_2, A3(4)_3, A3(4)_4) are generated. In the softmax function block 422-2, scaling and softmax function processing are also performed on the first to fourth parts A3(4)_1 to A3(4)_4 of the third attention score vector of the fourth sequence n_seq=4. As a result, the first to fourth parts S3(4)_1 to S3(4)_4 of the third attention distribution vector of the fourth sequence n_seq=4 are generated. The first to fourth parts S3(4)_1 to S3(4)_4 of the third attention distribution vector in the fourth sequence n_seq=4 are concatenated in the row direction to form the third attention distribution vector CONCAT(S3(4)_1, S3(4)_2, S3(4)_3, S3(4)_4) in the fourth sequence n_seq=4.

[0147] As a result of the second matrix-vector multiplication of the first head HEAD_1 of the fourth sequence n_seq=4, the first multi-head attention value vector Z1(4) of the fourth sequence n_seq=4 is generated. The first value matrix CONCAT(V1(1), V1(2), V1(3), V1(4)) of the fourth sequence n_seq=4 has a size of d_head×n_seq, that is, a size of d_head×4, and the first attention distribution vector CONCAT(S1(4)_1, S1(4)_2, S1(4)_3, S1(4)_4) of the fourth sequence n_seq=4 has a size of n_seq×1, that is, a size of 4×1, so the first multi-head attention value vector Z1(4) of the fourth sequence n_seq=4 has a fixed size of d_head×1. As a result of the second matrix-vector multiplication of the second head HEAD_2 of the fourth sequence n_seq=4, the second multi-head attention value vector Z2(4) of the fourth sequence n_seq=4 is generated. The second value matrix CONCAT(V2(1), V2(2), V2(3), V2(4)) of the fourth sequence n_seq=4 has a size of d_head×n_seq, i.e., a size of d_head×4, and the second attention distribution vector CONCAT(S2(4)_1, S2(4)_2, S2(4)_3, S2(4)_4) has a size of n_seq×1, i.e., a size of 4×1, so the second multi-head attention value vector Z2(4) of the fourth sequence n_seq=4 also has a fixed size of d_head×1. As a result of the second matrix-vector multiplication of the third head HEAD_3 of the fourth sequence n_seq=4, the third multi-head attention value vector Z3(4) of the fourth sequence (n_seq=4) is generated. The third value matrix CONCAT(S2(4)_1, S2(4)_2, S2(4)_3, S2(4)_4) has a size of n_seq×1, i.e., a size of 4×1. CAT(V3(1), V3(2), V3(3), V3(4)) has a size of d_head×n_seq, i.e., a size of d_head×4, and the third attention distribution vector CONCAT(S3(4)_1, S3(4)_2, S3(4)_3, S3(4)_4) has a size of n_seq×1, i.e., a size of 4×1, so the third multi-head attention value vector Z3(4) of the fourth sequence n_seq=4 also has a fixed size of d_head×1. Therefore, the first, second, and third multi-head attention value vectors Z1(4), Z2(4), and Z3(4) generated by the second matrix-vector multiplication of the first head HEAD_1, the second head HEAD_2, and the third head HEAD_3 of the fourth sequence n_seq=4, respectively, all have the same size of d_head×1.

[0148] As reference Figures 11 to 14As described above, in the first sequence (i.e., when the number of input tokens is 1), the value vector generated in the first sequence can be used in the same matrix format as the operand of the second matrix-vector multiplication to generate the multi-head attention value vector of the multi-head attention. On the other hand, in sequences other than the first sequence, a value matrix is ​​used as the operand of the second matrix-vector multiplication instead of the value vector generated in each sequence, in which the value vector generated in the previous sequence and the value vector generated in the current sequence are concatenated. However, the attention distribution vector in the vector format used as the operand of the second matrix-vector multiplication together with the value matrix is ​​formed by concatenating all parts of the attention distribution vector generated by applying scaling and softmax functions. Therefore, as the operand of the second matrix-vector multiplication, the value matrix and the attention distribution vector increase together in the column direction and the row direction in the sequence number, respectively, so that the dimension of the multi-head attention value vector generated as a result of the second matrix-vector multiplication is constant, regardless of the sequence number. Because the dimension of the value vector generated in all sequences is the same, the size of each key vector written to the memory bank of the PIM device for each sequence is a fixed size. On the other hand, the attention distribution matrix stored in the global buffer of the PIM device for each sequence has different sizes depending on the number of sequences. Therefore, when writing the value vectors to the memory banks of the PIM device, the value vectors generated for each sequence can be distributed and stored across multiple PIM devices and multiple memory banks, thereby increasing the degree of parallel execution of the second matrix-vector multiplication as the number of sequences increases. The process of distributing and storing the value vectors across the PIM devices and memory banks will be described in more detail below.

[0149] Figure 15 Flowchart showing the process of performing multi-head attention in a neural network architecture according to an embodiment of the present disclosure. The following description relates to multi-head attention in one sequence, but the process can be applied to each numbered sequence accordingly.

[0150] Reference Figure 15 as well as Figure 1 and Figure 2 , first, a query vector, a key vector, and a value vector are generated in the PIM devices 110(0), ..., and 110(15) (operation 510). The query vector, the key vector, and the value vector are generated in the PIM devices 110(0), ..., and 110(15), but in other embodiments, the query vector, the key vector, and the value vector may be generated in a separate computing device other than the PIM devices 110(0), ..., and 110(15). The process of generating the query vector, the key vector, and the value vector is the same as described above with reference to Figure 6The query vector, key vector, and value vector are generated as many times as the number of heads, respectively. Then, the query vector, key vector, and value vector generated by the first to sixteenth PIM devices 110 (0) to 110 (15) are written to the local memory 130 (operation 520). To this end, the PIM controller 120 reads the query vector, key vector, and value vector from the first to sixteenth PIM devices 110 (0) to 110 (15). The PIM controller 120 then writes the query vector, key vector, and value vector to the local memory 130.

[0151] Next, the key vector is written to the memory banks of the PIM devices 110(0), ..., and 110(15) in a first manner (operation 530). To this end, the PIM controller 120 reads the key vector from the local memory 130. The PIM controller 120 then writes the key vector read from the local memory 130 to the memory banks BK of the first to sixteenth PIM devices 110(0) to 110(15). In one example, the key vector is written to one of the memory banks BK of one of the first to sixteenth PIM devices 110(0) to 110(15) in a first manner. Next, the value vector is written to the memory banks BK of the PIM devices 110(0) to 110(15) in a second manner (operation 540). To this end, the PIM controller 120 reads the value vector from the local memory 130. The PIM controller 120 then writes the value vector read from the local memory 130 to the memory banks BK of the first to sixteenth PIM devices 110(0) to 110(15). In one example, the value vector is written in a second manner such that the value vector is distributed to the memory banks BK of the first to sixteenth PIM devices 110(0) to 110(15). Then, the query data is written to the global buffer GB of the first to sixteenth PIM devices 110(0) to 110(15) (operation 550). Specifically, the PIM controller 120 reads the query data from the local memory 130. The PIM controller 120 then writes the query data read from the local memory 130 to the global buffer GB of the first to sixteenth PIM devices 110(0) to 110(15). The processes of operations 530, 540, and 550 may be performed in different ways. Figure 15 Execute in the order shown.

[0152] Next, the processing units PU of the first to sixteenth PIM devices 110 (0) to 110 (15) perform a first matrix-vector multiplication to generate an attention score vector (operation 560). Specifically, the PIM controller 120 transmits a command for the first matrix-vector multiplication to the first to sixteenth PIM devices 110 (0) to 110 (15). Among the storage banks of the first to sixteenth PIM devices 110 (0) to 110 (15), the storage bank storing the key vector provides the key vector to the processing unit PU. In addition, the first to sixteenth PIM devices 110 (0) to 110 (15) also provide the query vector from the global buffer GB to the first to sixteenth processing units PU0 to PU15. In the first to sixteenth processing units PU0 to PU15, the processing units receiving the key vector and the query vector perform the first matrix-vector multiplication on the query vector and the key vector, and generate the resulting data in the form of an attention score vector. The attention score vectors generated by the first to sixteenth PIM devices 110 ( 0 ) to 110 ( 15 ) are written to the local memory 130 .

[0153] Next, the local processing unit 140 applies the softmax function to the attention score vector to generate an attention distribution vector (operation 570). Specifically, the PIM controller 120 transmits a control signal to the local memory 130 so that the attention score vector is transmitted from the local memory 130 to the local processing unit 140. Next, the PIM controller 120 transmits a control signal to the local processing unit 140 to apply the softmax function. As a result, the local processing unit 140 performs a scaling operation and a softmax function application operation on the attention score vector. Specifically, the local processing unit 140 performs a scaling operation on the attention score vector transmitted from the local memory 130 to generate a scaled attention score vector. Next, the local processing unit 140 applies the softmax function to the scaled attention score vector to generate an attention distribution vector, which is a weighted average of the attention score vectors.

[0154] Next, the attention distribution vector is written into the global buffer GB of the first to sixteenth PIM devices 110 (0) to 110 (15) (operation 580). The processing units PU of the first to sixteenth PIM devices 110 (0) to 110 (15) perform a second matrix-vector multiplication to generate a multi-head attention value vector (operation 590). Specifically, the PIM controller 120 transmits a command for matrix-vector multiplication to the first to sixteenth PIM devices 110 (0) to 110 (15). The storage bodies of the first to sixteenth PIM devices 110 (0) to 110 (15) provide the value vector to the processing unit. The global buffer GB of the first to sixteenth PIM devices 110 (0) to 110 (15) provides the attention distribution vector to the processing unit. The first to sixteenth processing units PU0 to PU15 perform a second matrix-vector multiplication on the value vector and the attention distribution vector, and generate the resulting data in the form of a multi-head attention value vector. As shown in FIG. Figure 5 As described above, n_head multi-head attention value vectors are generated by n_head multi-head attention layers 422, where n_head is the number of heads. All n_head multi-head attention value vectors are concatenated and output from the multi-head attention layer 422 in the format of a concatenated multi-head attention matrix, whose dimension is equal to the dimension of the embedding vector.

[0155] Figure 16 : is a diagram showing an example of a key matrix used as an operand in the first matrix-vector multiplication according to an embodiment of the present disclosure. As an example, Figure 16 Assume that the number of sequences n_seq of the multi-head attention is 257, and the number of heads n_head is 8. In addition, it is also assumed that the size of the head d_head is 128, and each head size is floating point data with a 16-bit BFloat16 (BF16) format. Figure 16 Only the first head HEAD_1 and the eighth head HEAD_8 are shown, but key vectors are generated in the same manner for the remaining second to seventh heads HEAD_2 to HEAD_7.

[0156] Reference Figure 16 , in the first sequence n_seq=1, the first to eighth key vectors K1(1) to K8(1) are generated for each of the first to eighth heads HEAD_1 to HEAD_8. The process of generating the first to eighth key vectors K1(1) to K8(1) is the same as that described above with reference to Figure 6 The same as described. Since the head size d_head of the multi-head attention is 128, each of the first to eighth key vectors K1(1) to K8(1) has the first to 128th elements. Figure 16As shown, the first key vector K1(1) has first to 128th elements K1(1)_1 to K1(1)_128. Similarly, the eighth key vector K8(1) has first to 128th elements K8(1)_1 to K8(1)_128. Each of the first to 128th elements of each of the first to eighth key vectors K1(1) to K8(1) includes 16-bit floating-point data in BF16 format. The first to eighth key vectors K1(1) to K8(1) generated in the first sequence n_seq=1 are used as the first to eighth key matrices, which are the operands in the first matrix-vector multiplication in the first to eighth headers HEAD_1 to HEAD_8 of the first sequence n_seq=1. For example, the first key vector K1(1) generated in the first sequence n_seq=1 is used as the first key matrix, which is the operand in the first matrix-vector multiplication in the first header HEAD_1 of the first sequence n_seq. In the same way, the eighth key vector K8(1) generated in the first sequence n_seq=1 is used as the eighth key matrix, which is the operand in the first matrix-vector multiplication in the eighth head HEAD_8 of the first sequence n_seq=1.

[0157] In the second sequence n_seq=2, the first to eighth key vectors K1(2) to K8(2) are generated for each of the first to eighth heads HEAD_1 to HEAD_8. The first to eighth key matrices used as operands in the first matrix-vector multiplication in the second sequence n_seq=2 are formed by concatenating the first to eighth key vectors K1(1) to K8(1) generated in the first sequence n_seq=1 and the first to eighth key vectors K1(2) to K8(2) generated in the second sequence n_seq=2, respectively. For example, the first key matrix CONCAT(K1(1), K1(2)) used as an operand in the first matrix-vector multiplication of the first head HEAD_1 of the second sequence n_seq=2 is formed by concatenating the first key vector K1(1) of the first sequence n_seq=1 and the first key vector K1(2) of the second sequence n_seq=2 in the row direction. In the same manner, the eighth key matrix CONCAT(K8(1), K8(2)) used as an operand in the first matrix-vector multiplication in the eighth head HEAD_8 of the second sequence n_seq=2 is formed by concatenating the eighth key vector K8(1) of the first sequence n_seq=1 and the eighth key vector K8(2) of the second sequence n_seq=2 in the row direction.

[0158] In the third sequence n_seq=3, first to eighth key vectors K1(3) to K8(3) are generated for each of the first to eighth heads HEAD_1 to HEAD_8. The first to eighth key matrices used as operands in the first matrix-vector multiplication in the third sequence n_seq=3 are formed by concatenating the first to eighth key vectors K1(1) to K8(1) generated in the first sequence n_seq=1, the first to eighth key vectors K1(2) to K8(2) generated in the second sequence n_seq=2, and the first to eighth key vectors K1(3) to K8(3) generated in the third sequence n_seq=3, respectively. For example, by concatenating the first key vector K1(1) of the first sequence n_seq=1, the first key vector K1(2) of the second sequence n_seq=2, and the first key vector K1(3) of the third sequence n_seq=3 in the row direction, a first key matrix CONCAT(K1(1), K1(2), K1(3)) used as an operand in the first matrix-vector multiplication of the first head HEAD_1 of the third sequence n_seq=3 is formed. In the same manner, by concatenating the eighth key vector K8(1) of the first sequence n_seq=1, the eighth key vector K8(2) of the second sequence n_seq=2, and the eighth key vector K8(3) of the third sequence n_seq=3 in the row direction, an eighth key matrix CONCAT(K8(1), K8(2), K8(3)) used as an operand in the first matrix-vector multiplication in the third sequence n_seq=3 is formed.

[0159] Similarly, in the 257th sequence n_seq=257, the first to eighth key vectors K1(257) to K8(257) are generated for each of the first to eighth heads HEAD_1 to HEAD_8. The first to eighth key matrices used as operands in the first matrix-vector multiplication in the 257th sequence n_seq=257 are formed by concatenating the first to eighth key vectors of all sequences in the row direction, from the first to eighth key vectors K1(1) to K8(1) in the first sequence n_seq=1 to the first to eighth key vectors K1(257) to K8(257) in the 257th sequence n_seq=257, respectively. For example, the first key matrix CONCAT(K1(1), ..., K1(257)) used as an operand in the first matrix-vector multiplication in the first header HEAD_1 of the 257th sequence n_seq=257 is formed by concatenating the first key vectors K1(1) to K1(257) of the first to 257th sequences n_seq=1 to n_seq=257 in the row direction. In the same manner, the eighth key matrix CONCAT(K8(1), K8(2), K8(3, ..., K8(257)) is formed by concatenating the eighth key vectors K8(1) to K8(257) of the first to 257th sequences n_seq=1 to n_seq=257 in the row direction.

[0160] To perform transformer-based multi-head attention in a neural network architecture according to the present disclosure, multiple memory banks of multiple PIM devices store key vectors in a first manner. Here, the first manner can be defined as a set of operations in which a key vector generated in one of multiple sequences of multi-head attention is stored in one of the multiple memory banks of each of the multiple PIM devices. In one example, when "P" is the number of multiple PIM devices included in the neural network architecture, "M" is the number of multiple memory banks included in each PIM device, and "L" is a natural number from "1" to "P," the "L"th PIM device is configured to store key vectors generated in the "(L-1)·M+1"-"L·M"th sequences in the multiple sequences. Specifically, the key vectors generated in the "(L-1)·M+1"-"L·M"th sequences are sequentially stored in the first to "M"th memory banks of the "L"th PIM device, using the key vectors generated in one of the "(L-1)·M+1"-"L·M"th sequences as units.

[0161] In one example, when the total size d_head×n_head of the key vectors generated in one sequence is less than or equal to the page size of each of a plurality of memory banks, the key vectors generated in the “(L-1)·M+1”-th to “L·M”-th sequences are stored in the same first row in the first to “M”-th memory banks of the “L”-th PIM device. Here, “first row” does not mean the physical first row in the memory bank, and the same applies to the following. When the number of heads is “N”, “N” query vectors generated in each of the “(L-1)·M+1”-th to “L·M”-th sequences are stored in a global buffer. The “N” query vectors stored in the global buffer can be used as operands in the first matrix operation.

[0162] Specifically, first to Nth key vectors generated in a first sequence among the plurality of sequences are sequentially stored in a first row of a first memory bank of a first PIM device among the plurality of PIM devices in a column-increasing direction, and first to Nth query vectors generated in the first sequence are sequentially stored in a global buffer of the first PIM device. Then, when "K" is a natural number from "1" to "N", a first processing unit of the first PIM device performs a first matrix-vector multiplication on the "K"th key vector and the "K"th query vector to generate a "K"th attention score vector in the first sequence.

[0163] The first through Nth key vectors generated in the second sequence are sequentially stored in a first row of a second memory bank of a first PIM device among the plurality of PIM devices in a column-increasing direction, and the first through Nth query vectors generated in the second sequence are sequentially stored in a global buffer of the first PIM device. When "K" is a natural number from "1" to "N," a first processing unit of the first PIM device performs a first matrix-vector multiplication on the "K"th key vector generated in the first sequence and the "K"th query vector generated in the second sequence to generate a first portion of a "K"th attention score vector in the second sequence. Furthermore, a second processing unit of the first PIM device performs a first matrix-vector multiplication on the "K"th key vector generated in the second sequence and the "K"th query vector generated in the second sequence to generate a second portion of the "K"th attention score vector in the second sequence.

[0164] That is, the first to Nth key vectors generated in the Mth sequence are sequentially stored in a first row of an Mth memory bank of a first PIM device among the plurality of PIM devices in a column-increasing direction, and the first to Nth query vectors generated in the Mth sequence are sequentially stored in a global buffer of the first PIM device. When "K" is a natural number from "1" to "N", the first to Mth processing units of the first PIM device perform first matrix-vector multiplications on the Kth key vectors generated in the first to Mth sequences and on the Kth query vector generated in the Mth sequence in parallel to generate the first to Nth parts of the Kth attention score vector in the Mth sequence.

[0165] In one example, key vectors generated in the (P+(L-1))·M+1"-(P+1)·M"-th sequences are sequentially stored in the first to M-th memory banks of the L-th PIM device, using the key vectors generated in one of the (P+(L-1))·M+1"-(P+1)·M"-th sequences as units. Key vectors generated in the (P+(L-1))·M+1"-(P+1)·M"-th sequences are stored in the second row, which is the next row after the first row, of the first to M-th memory banks of the L-th PIM device. In particular, the first to N-th key vectors generated in the (P+(L-1))·M+1"-(P+1)·M"-th sequence are sequentially stored in the second row of the first memory bank of the L-th PIM device in an increasing column direction. In addition, the first to Nth query vectors generated in the (P+(L-1))·M+1)-th sequence are sequentially stored in the global buffer of the Lth PIM device. When "K" is a natural number from "1" to "N", the first to Mth processing units of each of the first to Pth PIM devices perform a first operation of matrix-vector multiplication on the Kth key vector generated in the first to (P+(L-1))·M"-th sequence and the Kth query vector generated in the (P+(L-1))·M+1)-th sequence in parallel to generate the first to (P+(L-1))·M"-th parts of the Kth attention score vector in the (P+(L-1))·M"-th sequence. The first processing unit of the first PIM device sequentially performs a second operation of the first matrix-vector multiplication on the “K”-th key vector generated in the “(P+(L-1))·M+1”-th sequence and the “K”-th query vector generated in the “(P+(L-1))·M+1”-th sequence to generate the “(P+(L-1))·M+1”-th part of the “K”-th attention score vector in the “(P+(L-1))·M+1”-th sequence.

[0166] In one example, when the total size of key vectors generated in a sequence is greater than the page size of each of a plurality of memory banks, key vectors generated in the (L-1)·M+1-L·M sequences are stored in the first and second consecutive rows of each of the first to M-th memory banks of the L-th PIM device. Key vectors generated in the (P+(L-1))·M+1-(P+1)·M sequences are sequentially stored in the first to M-th memory banks of the L-th PIM device, using key vectors generated in one of the (P+(L-1))·M+1-(P+1)·M sequences as units. Key vectors generated in the (P+(L-1))·M+1-(P+1)·M sequences are stored in the third and fourth consecutive rows following the second row of the first to M-th memory banks of the L-th PIM device.

[0167] Figures 17 to 27 A method of writing key vectors generated in multiple sequences of multi-head attention in a neural network architecture according to the present disclosure in a first manner, and a first matrix-vector multiplication process performed using the written key vectors are shown. In the following description, the process of generating key vectors is the same as that described above with reference to Figure 16 The same as described above. Therefore, in this example, the number of sequences n_seq of the multi-head attention is 257, the number of heads n_head is 8, the size of the head d_head is 128, and each element of the head size is a 16-bit floating point data in BF16 format. In addition, it is assumed that the memory bank and the global buffer each have a page size of 2KB.

[0168] first, Figure 17 A diagram illustrating a method of writing first to eighth key vectors generated in a first sequence of multi-head attention and a first matrix-vector multiplication process performed on the first head according to an embodiment of the present disclosure. Figure 18 is a diagram illustrating a first matrix-vector multiplication performed on a second head when first to eighth key vectors generated in a first sequence of multi-head attention are written according to an embodiment of the present disclosure. Figure 19 is a diagram illustrating a first matrix-vector multiplication performed on the eighth head when first to eighth key vectors generated in a first sequence of multi-head attention are written according to an embodiment of the present disclosure.

[0169] First refer to Figure 17 , the first to eighth key vectors K1(1) to K8(1) generated in the first sequence n_seq=1 are written into the first memory bank BK0 of the first PIM device PIM0. In one example, the first to eighth key vectors K1(1) to K8(1) of the first sequence n_seq=1 are stored in the same row (hereinafter referred to as "first row") of the first memory bank BK0 of the first PIM device PIM0. Figure 17In the embodiment, since the total size of the first to eighth key vectors K1(1) to K8(1) (16 bits (i.e., 2 bytes) (BF16)×128 bits (i.e., 16 bytes) (d_head)×8 bits (i.e., 1 byte) (n_head)) is the same as 2KB, which is the page size of the first memory bank BK0, the first row of the first memory bank BK0 of the first PIM device PIM0 is filled with all of the first to eighth key vectors K1(1) to K8(1) of the first sequence n_seq=1. The first to eighth query vectors Q1(1) to Q8(1) generated in the first to eighth heads of the first sequence n_seq=1 are written to the global buffer GB of the first PIM device PIM0.

[0170] When the first to eighth key vectors K1(1) to K8(1) are stored in the first memory bank BK0 of the first PIM device PIM0, and the first to eighth query vectors Q1(1) to Q8(1) are stored in the global buffer GB of the first PIM device PIM0, the first matrix-vector multiplication in the first sequence n_seq=1 is performed in sequence. That is, first, the first matrix-vector multiplication is performed on the first head HEAD_1 of the first sequence n_seq=1. Next, the first matrix-vector multiplication is performed on the second head HEAD_2 of the first sequence n_seq=1. Then, the first matrix-vector multiplication is performed sequentially on the remaining third to eighth heads HEAD_3 to HEAD_8 of the first sequence n_seq=1.

[0171] like Figure 17 As shown, the first matrix-vector multiplication of the first head HEAD_1 of the first sequence n_seq=1 is performed by the first processing unit PU0 of the first PIM device PIM0. Specifically, the first processing unit PU0 of the first PIM device PIM0 receives the first key vector K1(1) and the first query vector Q1(1) from the first storage body BK0 and the global buffer GB of the first PIM device PIM0, respectively. The first processing unit PU0 of the first PIM device PIM0 performs the first matrix-vector multiplication on the first key vector K1(1) and the first query vector Q1(1) to generate the first attention score vector A1(1) of the first sequence n_seq=1. The first attention score vector A1(1) of the first sequence n_seq=1 generated by the first processing unit PU0 of the first PIM device PIM0 can be transmitted via the PIM controller ( Figure 1 The control operation of 120) is transferred to the local memory ( Figure 1 130).

[0172] Next, if Figure 18As shown, the first matrix-vector multiplication of the second head HEAD_2 of the first sequence n_seq=1 is performed by the first processing unit PU0 of the first PIM device PIM0. Specifically, the first processing unit PU0 of the first PIM device PIM0 receives the second key vector K2(1) and the second query vector Q2(1) from the first storage body BK0 and the global buffer GB of the first PIM device PIM0, respectively. The first processing unit PU0 of the first PIM device PIM0 performs the first matrix-vector multiplication on the second key vector K2(1) and the second query vector Q2(1) to generate the second attention score vector A2(1) of the first sequence n_seq=1. The second attention score vector A2(1) of the first sequence n_seq=1 generated by the first processing unit PU0 of the first PIM device PIM0 can be transmitted via the PIM controller ( Figure 1 The control operation of 120) is transferred to the local memory ( Figure 1 130).

[0173] Although not shown in the figure, in the same manner, the first matrix-vector multiplications of the third to seventh heads of the first sequence n_seq=1 are sequentially performed by the first processing unit PU0 of the first PIM device PIM0. That is, the first processing unit PU0 of the first PIM device PIM0 performs the first matrix-vector multiplication on the third key vector K3(1) of the first sequence n_seq=1 and the third query vector Q3(1) of the first sequence n_seq=1 to generate the third attention score vector of the first sequence n_seq=1. The first processing unit PU0 of the first PIM device PIM0 performs the first matrix-vector multiplication on the fourth key vector of the first sequence n_seq=1 and the fourth query vector of the first sequence n_seq=1 to generate the fourth attention score vector of the first sequence n_seq=1. The first processing unit PU0 of the first PIM device PIM0 performs the first matrix-vector multiplication on the fifth key vector of the first sequence n_seq=1 and the fifth query vector of the first sequence n_seq=1 to generate the fifth attention score vector of the first sequence n_seq=1. The first processing unit PU0 of the first PIM device PIM0 performs a first matrix-vector multiplication on the sixth key vector of the first sequence n_seq=1 and the sixth query vector of the first sequence n_seq=1 to generate a sixth attention score vector of the first sequence n_seq=1. Then, the first processing unit PU0 of the first PIM device PIM0 performs a first matrix-vector multiplication on the seventh key vector of the first sequence n_seq=1 and the seventh query vector of the first sequence n_seq=1 to generate a seventh attention score vector of the first sequence n_seq=1.

[0174] Next, if Figure 19As shown, the first matrix-vector multiplication of the eighth head HEAD_8 of the first sequence n_seq=1 is performed by the first processing unit PU0 of the first PIM device PIM0. Specifically, the first processing unit PU0 of the first PIM device PIM0 receives the eighth key vector K8(1) of the first sequence n_seq=1 and the eighth query vector Q8(1) of the first sequence n_seq=1 from the first storage body BK0 and the global buffer GB of the first PIM device PIM0, respectively. The first processing unit PU0 of the first PIM device PIM0 performs the first matrix-vector multiplication on the eighth key vector K8(1) and the eighth query vector Q8(1) to generate the eighth attention score vector A8(1) of the first sequence n_seq=1. The eighth attention score vector A8(1) generated by the first processing unit PU0 of the first PIM device PIM0 can be transmitted to the PIM controller ( Figure 1 The control operation of 120) is transferred to the local memory ( Figure 1 130). Therefore, as referenced Figures 17 to 19 As described above, the first to eighth attention score vectors A1(1) to A8(1) in the first sequence n_seq=1 are generated by eight first matrix-vector multiplications in the first sequence n_seq=1 and are transferred to the local memory ( Figure 1 130).

[0175] Figure 20 is a diagram illustrating a method of writing first to eighth key vectors generated in a second sequence of multi-head attention and a first matrix-vector multiplication process performed on the first head according to an embodiment of the present disclosure.

[0176] Reference Figure 20, the first to eighth key vectors K1(2) to K8(2) generated in the second sequence n_seq=2 are written to the first row of the second memory bank BK1 of the first PIM device PIM0. In addition, the first to eighth query vectors Q1(2) to Q8(2) generated in the second sequence n_seq=2 are stored in the global buffer GB of the first PIM device PIM0. The first processing unit PU0 of the first PIM device PIM0 performs a first matrix-vector multiplication using the first to eighth key vectors K1(1) to K8(1) of the first sequence n_seq=1 and the first to eighth query vectors Q1(2) to Q8(2) of the second sequence n_seq=2 as operands. In addition, the second processing unit PU1 of the first PIM device PIM0 performs a first matrix-vector multiplication using the first to eighth key vectors K1(2) to K8(2) of the second sequence n_seq=2 and the first to eighth query vectors Q1(2) to Q8(2) of the second sequence n_seq=2 as operands. The first matrix-vector multiplication in the first processing unit PU0 of the first PIM device PIM0 and the first matrix-vector multiplication in the second processing unit PU1 of the first PIM device PIM0 are performed in parallel.

[0177] First, the first processing unit PU0 and the second processing unit PU1 of the first PIM device PIM0 perform a first matrix-vector multiplication on the first head HEAD_1 of the second sequence n_seq=2 in parallel. Specifically, the first processing unit PU0 of the first PIM device PIM0 receives a first key vector K1(1) of the first sequence n_seq=1 from the first storage bank BK0 of the first PIM device PIM0. The second processing unit PU1 of the first PIM device PIM0 receives a first key vector K1(2) of the second sequence n_seq=2 from the second storage bank BK1 of the first PIM device PIM0. The first and second processing units PU0 and PU1 of the first PIM device PIM0 jointly receive a first query vector Q1(2) of the second sequence n_seq=2 from the global buffer GB of the first PIM device PIM0. The first processing unit PU0 of the first PIM device PIM0 performs a first matrix-vector multiplication on the first key vector K1(1) of the first sequence n_seq=1 and the first query vector Q1(2) of the second sequence n_seq=2 in the first header HEAD_1 to generate a first part A1(2)_1 of the first attention score vector A1(2) of the second sequence n_seq=2. The second processing unit PU1 of the first PIM device PIM0 performs a first matrix-vector multiplication on the first key vector K1(2) of the second sequence n_seq=2 and the first query vector Q1(2) of the second sequence n_seq=2 in the first header HEAD_1 to generate a second part A1(2)_2 of the first attention score vector A1(2) of the second sequence n_seq=2. The attention score vector A1(2) of the second sequence n_seq=2 is obtained by concatenating the first part A1(2)_1 and the second part A1(2)_2 in the row direction. The first part A1(2)_1 and the second part A1(2)_2 of the first attention score vector A1(2) in the first head HEAD_1 generated by the first processing unit PU0 and the second processing unit PU1 of the first PIM device PIM0 are transferred to the local memory ( Figure 1 130).

[0178] Although not shown in the figure, the first matrix-vector multiplication is performed on the second to eighth heads HEAD2 to HEAD8 in the same manner as the first matrix-vector multiplication is performed on the first head HEAD_1. For example, the first processing unit PU0 of the first PIM device PIM0 performs the first matrix-vector multiplication on the second key vector K2(1) of the first sequence n_seq=1 and the second query vector Q2(2) of the second sequence n_seq=2 in the second head HEAD_2 to generate a first part of the second attention score vector of the second sequence n_seq=2. The second processing unit PU1 of the first PIM device PIM0 performs the first matrix-vector multiplication on the second key vector K2(2) of the second sequence n_seq=2 and the second query vector Q2(2) of the second sequence n_seq=2 in the second head HEAD_2 to generate a second part of the second attention score vector of the second sequence n_seq=2. In the same manner, the first processing unit PU0 of the first PIM device PIM0 performs a first matrix-vector multiplication on the eighth key vector K8(1) of the first sequence n_seq=1 and the eighth query vector Q8(2) of the second sequence n_seq=2 in the second head HEAD_2 to generate a first part of the eighth attention score vector of the second sequence n_seq=2. The second processing unit PU1 of the first PIM device PIM0 performs a first matrix-vector multiplication on the eighth key vector K8(2) of the second sequence n_seq=2 and the eighth query vector Q8(2) of the second sequence n_seq=2 in the eighth head HEAD_8 to generate a second part of the eighth attention score vector of the second sequence n_seq=2.

[0179] As mentioned above Figures 17 to 20As described above, the first matrix-vector multiplication in the first sequence n_seq=1 and the second sequence n_seq=2 is performed in the same manner for the third to fifteenth sequences n_seq=3 to n_seq=15. Therefore, the first to eighth attention score vectors for the third sequence n_seq=3 are generated in the first to third processing units of the first PIM device PIM0. The first to eighth attention score vectors for the fourth sequence n_seq=4 are generated by the first to fourth processing units of the first PIM device PIM0. The first to eighth attention score vectors for the fifth sequence n_seq=5 are generated by the first to fifth processing units of the first PIM device PIM0. The first to eighth attention score vectors for the sixth sequence n_seq=6 are generated by the first to sixth processing units of the first PIM device PIM0. The first to eighth attention score vectors for the seventh sequence n_seq=7 are generated by the first to seventh processing units of the first PIM device PIM0. The first to eighth attention score vectors for the eighth sequence n_seq=8 are generated by the first to eighth processing units of the first PIM device PIM0. The first to eighth attention score vectors of the ninth sequence n_seq=9 are generated by the first to ninth processing units of the first PIM device PIM0. The first to eighth attention score vectors of the tenth sequence n_seq=10 are generated by the first to tenth processing units of the first PIM device PIM0. The first to eighth attention score vectors of the eleventh sequence n_seq=11 are generated by the first to eleventh processing units of the first PIM device PIM0. The first to eighth attention score vectors of the twelfth sequence n_seq=12 are generated by the first to twelfth processing units of the first PIM device PIM0. The first to eighth attention score vectors of the thirteenth sequence n_seq=13 are generated by the first to thirteenth processing units of the first PIM device PIM0. The first to eighth attention score vectors of the fourteenth sequence n_seq=14 are generated by the first to fourteenth processing units of the first PIM device PIM0. Furthermore, the first to eighth attention score vectors of the fifteenth sequence n_seq=15 are generated by the first to fifteenth processing units of the first PIM device PIM0.

[0180] Figure 21 is a diagram illustrating a method of writing first to eighth key vectors generated in a sixteenth sequence of multi-head attention and a first matrix-vector multiplication process performed on a first head according to an embodiment of the present disclosure.

[0181] Reference Figure 21, the first to eighth key vectors K1(16) to K8(16) generated in the sixteenth sequence n_seq=16 are stored in the first row of the sixteenth memory bank BK15 of the first PIM device PIM0. The first to eighth query vectors Q1(16) to Q8(16) generated in the sixteenth sequence n_seq=16 are stored in the global buffer GB of the first PIM device PIM0. For each of the first to eighth heads, the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 perform the first matrix-vector multiplication in parallel to generate the first to sixteenth parts of each of the first to eighth attention score vectors of the sixteenth sequence n_seq=16.

[0182] First, the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 perform the first matrix-vector multiplication on the first head HEAD_1 of the sixteenth sequence n_seq=16 in parallel. Specifically, the first processing unit PU0 of the first PIM device PIM0 receives the first key vector K1(1) of the first sequence n_seq=1 from the first memory bank BK0 of the first PIM device PIM0. The second processing unit PU1 of the first PIM device PIM0 receives the first key vector K1(2) of the second sequence n_seq=2 from the second memory bank BK1 of the first PIM device PIM0. Although not shown, the third to fifteenth processing units of the first PIM device PIM0 receive the first key vectors of the third to fifteenth sequences from the third to fifteenth memory banks of the first PIM device PIM0, respectively. The sixteenth processing unit PU15 of the first PIM device PIM0 receives the first key vector K1(16) of the sixteenth sequence n_seq=16 from the sixteenth memory bank BK15 of the first PIM device PIM0. The first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 jointly receive a first query vector Q1 (16) of a sixteenth sequence n_seq=16 from the global buffer GB of the first PIM device PIM0.

[0183] The first processing unit PU0 of the first PIM device PIM0 performs a first matrix-vector multiplication on the first key vector K1(1) of the first sequence n_seq=1 and the first query vector Q1(16) of the sixteenth sequence n_seq=16 to generate a first part A1(16)_1 of the first attention score vector A1(16) of the sixteenth sequence n_seq=16. The second processing unit PU1 of the first PIM device PIM0 performs a first matrix-vector multiplication on the first key vector K1(2) of the second sequence n_seq=2 and the first query vector Q1(16) of the sixteenth sequence n_seq=16 to generate a second part A1(16)_2 of the first attention score vector A1(16) of the sixteenth sequence n_seq=16. Although not shown, the third to fifteenth processing units of the first PIM device PIM0 perform first matrix-vector multiplication on the first key vectors generated in the third to fifteenth sequences n_seq=3 to n_seq=15 and the first query vector Q1(16) of the sixteenth sequence n_seq=16 to generate third to fifteenth parts A1(16)_3 to A1(16)_15 of the first attention score vector A1(16) of the sixteenth sequence n_seq=16. In addition, the sixteenth processing unit PU15 of the first PIM device PIM0 performs first matrix-vector multiplication on the first key vector K1(16) of the sixteenth sequence n_seq=16 and the first query vector Q1(16) of the sixteenth sequence n_seq=16 to generate the sixteenth part A1(16)_16 of the first attention score vector A1(16) of the sixteenth sequence n_seq=16. The attention score vector A1(16) of the sixteenth sequence n_seq=16 can be obtained by concatenating the first to sixteenth parts A1(16)_1 to A1(16)_16 in the row direction. The first to sixteenth parts A1(16)_1 to A1(16)_16 of the first head HEAD_1 of the sixteenth sequence n_seq=16 generated by the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 are transferred to the local memory ( Figure 1 130). Although not shown, the first matrix-vector multiplication is performed for the second to eighth heads HEAD2 to HEAD8 in the same manner as the first matrix-vector multiplication performed for the first head HEAD_1. Thus, for each of the second to eighth heads, the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 perform the first matrix-vector multiplication in parallel to generate the first to sixteenth parts of each of the second to eighth attention score vectors of the sixteenth sequence n_seq=16.

[0184] Figure 22is a diagram illustrating a method of writing first to eighth key vectors generated in a seventeenth sequence of multi-head attention and a first matrix-vector multiplication process performed on a first head according to an embodiment of the present disclosure.

[0185] Reference Figure 22 , during the execution of the first to sixteenth sequences n_seq=1 to n_seq=16, the key vectors are stored in all the first rows of the first to sixteenth memory banks BK0 to BK15 of the first PIM device PIM0. The first to eighth key vectors K1(17) to K8(17) generated in the seventeenth sequence n_seq=17 are stored in the first row of the first memory bank BK0 of the second PIM device PIM1. The first to eighth query vectors Q1(17) to Q8(17) generated in the first to eighth heads of the seventeenth sequence n_seq=17 are stored in the global buffer GB of the first PIM device PIM0 and the second PIM device PIM1. For each of the first to eighth heads, the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 and the first processing unit PU0 of the second PIM device PIM1 perform the first matrix-vector multiplication in parallel to generate the first to seventeenth parts of each of the first to eighth attention score vectors of the seventeenth sequence n_seq=17.

[0186] First, for the first matrix-vector multiplication in the first header HEAD_1 of the seventeenth sequence n_seq=17, the first processing unit PU0 of the first PIM device PIM0 receives the first key vector K1(1) generated in the first sequence n_seq=1 from the first memory bank BK0 of the first PIM device PIM0. The second processing unit PU1 of the first PIM device PIM0 receives the first key vector K1(2) generated in the second sequence n_seq=2 from the second memory bank BK1 of the first PIM device PIM0. Although not shown, the third to fifteenth processing units of the first PIM device PIM0 receive the first key vectors generated in the third to fifteenth sequences from the third to fifteenth memory banks of the first PIM device PIM0, respectively. The sixteenth processing unit PU15 of the first PIM device PIM0 receives the first key vector K1(16) generated in the sixteenth sequence n_seq=16 from the sixteenth memory bank BK15 of the first PIM device PIM0. The first processing unit PU0 of the second PIM device PIM1 receives the first key vector K1(17) generated in the seventeenth sequence n_seq=17 from the first memory bank BK0 of the second PIM device PIM1. The first to sixteenth processing units PU0 to PU15 of each of the first PIM device PIM0 and the second PIM device PIM1 jointly receive the first query vector Q1(17) generated in the seventeenth sequence n_seq=17 from the global buffer GB.

[0187] The first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 and the first processing unit PU0 of the second PIM device PIM1 use the received first key vector and the first query vector as operands to perform the first matrix-vector multiplication in the first head HEAD_1 in parallel. Specifically, the first processing unit PU0 of the first PIM device PIM0 performs the first matrix-vector multiplication in the first HEAD_1 on the first key vector K1(1) of the first sequence n_seq=1 and the first query vector Q1(17) of the seventeenth sequence n_seq=17 to generate a first part A1(17)_1 of the first attention score vector A1(17) of the seventeenth sequence n_seq=17. The second processing unit PU1 of the first PIM device PIM0 performs the first matrix-vector multiplication in the first head HEAD_1 on the first key vector K1(2) of the second sequence n_seq=2 and the first query vector Q1(17) of the seventeenth sequence n_seq=17 to generate the second part A1(17)_2 of the first attention score vector A1(17) of the seventeenth sequence n_seq=17. Although not shown, the third to fifteenth processing units of the first PIM device PIM0 perform the first matrix-vector multiplication on the first key vectors generated in the third to fifteenth sequences and the first query vector Q1(17) generated in the seventeenth sequence n_seq=17 to generate the third to fifteenth parts of the first attention score vector A1(17) of the seventeenth sequence n_seq=17. The sixteenth processing unit PU15 of the first PIM device PIM0 performs the first matrix-vector multiplication in the first header HEAD_1 on the first key vector K1(16) of the sixteenth sequence n_seq=16 and the first query vector Q1(17) of the seventeenth sequence n_seq=17 to generate the sixteenth part A1(17)_16 of the first attention score vector A1(17) of the seventeenth sequence n_seq=17. In addition, the first processing unit PU0 of the second PIM device PIM1 performs the first matrix-vector multiplication in the first header HEAD_1 on the first key vector K1(17) of the seventeenth sequence n_seq=17 and the first query vector Q1(17) of the seventeenth sequence n_seq=17 to generate the seventeenth part A1(17)_17 of the first attention score vector A1(17) of the seventeenth sequence n_seq=17. The attention score vector A1(17) of the 17th sequence n_seq=17 can be obtained by concatenating the first to seventeenth parts A1(17)_1 to A1(17)_17 in the row direction. The first to seventeenth parts A1(17)_1 to A1(17)_17 of the first attention score vector A1(17) in the seventeenth sequence n_seq=17 generated in the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 and the first processing unit PU0 of the second PIM device PIM1 are transferred to the local memory ( ). Figure 1 130 in ). Although not shown, the first matrix-vector multiplication is performed on the second to eighth heads HEAD2 to HEAD8 in the same manner as the first matrix-vector multiplication performed on the first head HEAD_1. Therefore, for each of the second to eighth heads, the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 and the first processing unit PU0 of the second PIM device PIM1 perform the first matrix-vector multiplication in parallel to generate the first to seventeenth parts of each of the second to eighth attention score vectors of the seventeenth sequence n_seq=17.

[0188] Figure 23 is a diagram illustrating a method of writing first to eighth key vectors generated in an eighteenth sequence of multi-head attention and a first matrix-vector multiplication process performed on the first head according to an embodiment of the present disclosure.

[0189] Reference Figure 23 , the first to eighth key vectors K1(18) to K8(18) generated in the eighteenth sequence n_seq=18 are stored in the first row of the second memory bank BK1 of the second PIM device PIM1. The first to eighth query vectors Q1(18) to Q8(18) generated in the eighteenth sequence n_seq=18 are stored in the global buffer GB of the first PIM device PIM0 and the second PIM device PIM1. For each of the first to eighth heads, the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 and the first and second processing units PU0 and PU1 of the second PIM device PIM1 perform the first matrix-vector multiplication in parallel to generate the first to eighteenth parts of each of the first to eighth attention score vectors of the eighteenth sequence n_seq=18.

[0190] First, the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 and the first and second processing units PU0 and PU1 of the second PIM device PIM1 perform the first matrix-vector multiplication on the first header HEAD_1 of the eighteenth sequence n_seq=18 in parallel. Specifically, for the first matrix-vector multiplication in the first header HEAD_1 of the eighteenth sequence n_seq=18, the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 receive the first key vectors K1(1) to K1(16) generated in the first to sixteenth sequences n_seq=1 to n_seq=16 from the first to sixteenth memory banks BK0 to BK15 of the first PIM device PIM0, respectively. The first processing unit PU0 of the second PIM device PM1 receives the first key vector K1(17) generated in the seventeenth sequence n_seq=17 from the first memory bank BK0 of the second PIM device PIM1. In addition, the second processing unit PU1 of the second PIM device PIM1 receives the first key vector K1 (18) generated in the eighteenth sequence n_seq=18 from the second memory bank BK1 of the second PIM device PIM1. The first to sixteenth processing units PU0 to PU15 of each of the first PIM device PIM0 and the second PIM device PIM1 jointly receive the first query vector Q1 (18) generated in the eighteenth sequence n_seq=18 from the global buffer GB.

[0191] The first processing unit PU0 of the first PIM device PIM0 performs the first matrix-vector multiplication in the first header HEAD_1 on the first key vector K1(1) generated in the first sequence n_seq=1 and the first query vector Q1(18) generated in the eighteenth sequence n_seq=18 to generate the first part A1(18)_1 of the first attention score vector A1(18) of the eighteenth sequence n_seq=18. The second processing unit PU1 of the first PIM device PIM0 performs the first matrix-vector multiplication in the first header HEAD_1 on the first key vector K1(2) generated in the second sequence n_seq=2 and the first query vector Q1(18) generated in the eighteenth sequence n_seq=18 to generate the second part A1(18)_2 of the first attention score vector A1(18) of the eighteenth sequence n_seq=18. Although not shown, the third to fifteenth processing units of the first PIM device PIM0 perform first matrix-vector multiplication on the first key vectors generated in the third to fifteenth sequences and the first query vector Q1(18) generated in the eighteenth sequence n_seq=18 to generate third to fifteenth parts of the first attention score vector A1(18) of the eighteenth sequence n_seq=18. The sixteenth processing unit PU15 of the first PIM device PIM0 performs the first matrix-vector multiplication in the first head HEAD_1 on the first key vector K1(16) generated in the sixteenth sequence n_seq=16 and the first query vector Q1(18) generated in the sixteenth sequence n_seq=16 to generate the sixteenth part A1(18)_16 of the first attention score vector A1(18) of the eighteenth sequence n_seq=18. The first processing unit PU0 of the second PIM device PIM1 performs the first matrix-vector multiplication in the first header HEAD_1 on the first key vector K1(17) generated in the seventeenth sequence n_seq=17 and the first query vector Q1(18) generated in the eighteenth sequence n_seq=18 to generate the seventeenth part A1(18)_17 of the first attention score vector A1(18) of the eighteenth sequence n_seq=18. In addition, the second processing unit PU1 of the second PIM device PIM1 performs the first matrix-vector multiplication in the first header HEAD_1 on the first key vector K1(18) generated in the eighteenth sequence n_seq=18 and the first query vector Q1(18) generated in the eighteenth sequence n_seq=18 to generate the eighteenth part A1(18)_18 of the first attention score vector A1(18) of the eighteenth sequence n_seq=18. The first attention score vector A1(18) of the eighteenth sequence n_seq=18 can be obtained by concatenating the first to eighteenth parts A1(18)_1 to A1(18)_18 in the row direction.The first to eighteenth parts A1(18)_1 to A1(18)_18 of the first attention score vector A1(18) in the first head HEAD_1 of the eighteenth sequence n_seq=18 generated in the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 and the first and second processing units PU0 and PU1 of the second PIM device PIM1 are transmitted to the local memory (. Figure 1 130). Although not shown, the first matrix-vector multiplication is performed for the second to eighth heads HEAD2 to HEAD8 in the same manner as the first matrix-vector multiplication performed for the first head HEAD_1. Thus, for each of the second to eighth heads, the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 and the first and second processing units PU0 and PU1 of the second PIM device PIM1 perform the first matrix-vector multiplication in parallel to generate the first to eighteenth parts of each of the second to eighth attention score vectors of the eighteenth sequence n_seq=18.

[0192] Reference Figures 21 to 23The described process is performed identically for sequences 19 through 31. Thus, in sequence 19, the first through nineteenth portions of the first through eighth attention score vectors of the nineteenth sequence are generated in the first through sixteenth processing units of the first PIM device PIM0 and the first through third processing units of the second PIM device PIM1. In sequence 20, the first through twentieth portions of the first through eighth attention score vectors of the 20th sequence are generated in the first through sixteenth processing units of the first PIM device PIM0 and the first through fourth processing units of the second PIM device PIM1. In sequence 21, the first through twenty-first portions of the first through eighth attention score vectors of the 21st sequence are generated in the first through sixteenth processing units of the first PIM device PIM0 and the first through fifth processing units of the second PIM device PIM1. In sequence 22, the first through twenty-second portions of the first through eighth attention score vectors of the 22nd sequence are generated in the first through sixteenth processing units of the first PIM device PIM0 and the first through sixth processing units of the second PIM device PIM1. In the 23rd sequence, the first to twenty-third portions of the first to eighth attention score vectors of the 23rd sequence are generated in the first to sixteenth processing units of the first PIM device PIM0 and the first to seventh processing units of the second PIM device PIM1. In the 24th sequence, the first to twenty-fourth portions of the first to eighth attention score vectors of the 24th sequence are generated in the first to sixteenth processing units of the first PIM device PIM0 and the first to eighth processing units of the second PIM device PIM1. In the 25th sequence, the first to twenty-fifth portions of the first to eighth attention score vectors of the 25th sequence are generated in the first to sixteenth processing units of the first PIM device PIM0 and the first to ninth processing units of the second PIM device PIM1. In the 26th sequence, the first to twenty-sixth portions of the first to eighth attention score vectors of the 26th sequence are generated in the first to sixteenth processing units of the first PIM device PIM0 and the first to tenth processing units of the second PIM device PIM1. In the 27th sequence, the first to twenty-seventh portions of the first to eighth attention score vectors of the 27th sequence are generated in the first to sixteenth processing units of the first PIM device PIM0 and the first to eleventh processing units of the second PIM device PIM1. In the 28th sequence, the first to twenty-eighth portions of the first to eighth attention score vectors of the 28th sequence are generated in the first to sixteenth processing units of the first PIM device PIM0 and the first to twelfth processing units of the second PIM device PIM1. In the 29th sequence, the first to twenty-ninth portions of the first to eighth attention score vectors of the 29th sequence are generated in the first to sixteenth processing units of the first PIM device PIM0 and the first to thirteenth processing units of the second PIM device PIM1.In the 30th sequence, the first to 30th parts of the first to eighth attention score vectors of the 30th sequence are generated in the first to sixteenth processing units of the first PIM device PIM0 and the first to fourteenth processing units of the second PIM device PIM1. In addition, in the 31st sequence, the first to thirty-first parts of the first to eighth attention score vectors of the 31st sequence are generated in the first to sixteenth processing units of the first PIM device PIM0 and the first to fifteenth processing units of the second PIM device PIM1.

[0193] Figure 24 is a diagram illustrating a method of writing first to eighth key vectors generated in the 32nd sequence of multi-head attention and a first matrix-vector multiplication process performed on the first head according to an embodiment of the present disclosure.

[0194] Reference Figure 24 , the first to eighth key vectors K1(32) to K8(32) generated in the 32nd sequence n_seq=32 are stored in the first row of the 16th memory bank BK15 of the second PIM device PIM1. The first to eighth query vectors Q1(32) to Q8(32) generated in the 32nd sequence n_seq=32 are stored in the global buffer GB of the first PIM device PIM0 and the second PIM device PIM1. For each of the first to eighth heads, the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 and the first to sixteenth processing units PU0 to PU15 of the second PIM device PIM1 perform the first matrix-vector multiplication in parallel to generate the first to eighth parts of each of the first to eighth attention score vectors of the 32nd sequence n_seq=32.

[0195] First, the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 and the first to sixteenth processing units PU0 to PU15 of the second PIM device PIM1 concurrently perform a first matrix-vector multiplication on the first head HEAD_1 of the 32nd sequence n_seq=32. Specifically, for the first matrix-vector multiplication performed on the first head HEAD_1 of the 32nd sequence n_seq=32, the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 receive first key vectors K1(1) to K1(16) of the first to sixteenth sequences n_seq=1 to n_seq=16 from the memory banks BK0 to BK15 of the first PIM device PIM0, respectively. The first to sixteenth processing units PU0 to PU15 of the second PIM device PIM1 receive the first key vectors K1(17) to K1(32) of the seventeenth to thirty-second sequences n_seq=17 to n_seq=32 from the first to sixteenth memory banks BK0 to BK15 of the second PIM device PIM1, respectively. The first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 and the second PIM device PIM1 jointly receive the first query vector Q1(32) of the thirty-second sequence n_seq=32 from the global buffer GB.

[0196] The first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 perform the first matrix-vector multiplication on the first head HEAD_1 for the first key vectors K1(1) to K1(16) of the first to sixteenth sequences n_seq=1 to n_seq=16 and the first query vector Q1(32) of the 32nd sequence n_seq=32 to generate the first to sixteenth parts A1(32)_1 to A1(32)_16 of the first attention score vector A1(32) of the 32nd sequence n_seq=32. The first to sixteenth processing units PU0 to PU15 of the second PIM device PIM1 perform the first matrix-vector multiplication on the first head HEAD_1 with respect to the first key vectors K1(17) to K1(32) of the seventeenth to thirty-second sequences n_seq=17 to n_seq=32 and the first query vector Q1(32) of the thirty-second sequence n_seq=32 to generate the seventeenth to thirty-second parts A1(32)_17 to A1(32)_32 of the first attention score vector A1(32) of the thirty-second sequence n_seq=32. The first attention score vector A1(32) of the thirty-second sequence n_seq=32 can be obtained by concatenating the first to thirty-second parts A1(32)_1 to A1(32)_32 in the row direction. The first to thirty-second parts A1(32)_1 to A1(32)_32 of the first attention score vector A1(32) in the first head HEAD_1 of the thirty-second sequence n_seq=32 generated in the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 and the first to sixteenth processing units PU0 to PU15 of the second PIM device PIM1 are transferred to the local memory ( Figure 1 130). Although not shown in the figure, the first matrix-vector multiplication process is performed for the second to eighth heads HEAD2 to HEAD8 in the same manner as the first matrix-vector multiplication for the first head HEAD_1. Therefore, for each of the second to eighth heads, the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 and the first to sixteenth processing units PU0 to PU15 of the second PIM device PIM1 perform the first matrix-vector multiplication in parallel to generate the first to sixteenth parts of each of the second to eighth attention score vectors of the thirty-second sequence n_seq=32.

[0197] With reference to the above Figures 17 to 24In a similar manner to the described process, the first to eighth key vectors generated in the 33rd to 240th sequences, n_seq=33 to n_seq=240, are respectively stored in the first row of the memory banks of the third to fifteenth PIM devices. The first to eighth query vectors generated in the 33rd to 48th sequences, n_seq=33 to n_seq=48, are stored in the global buffers of the first to third PIM devices. The first to eighth query vectors generated in the 49th to 64th sequences, n_seq=49 to n_seq=64, are stored in the global buffers of the first to fourth PIM devices. Similarly, the first to eighth query vectors generated in the 225th to 240th sequences, n_seq=225 to n_seq=240, are stored in the global buffers of the first to fifteenth PIM devices. For each of the 33rd to 240th sequences n_seq=33 to n_seq=240, a first matrix-vector multiplication is performed using the first to eighth key vectors and the first to eighth query vectors as operands, thereby generating portions of the first to eighth attention score vectors for each of the 33rd to 240th sequences n_seq=33 to n_seq=240.

[0198] Figure 25 2 is a diagram illustrating a method of writing the first to eighth key vectors generated in the 241st to 256th sequences of multi-head attention and a first matrix-vector multiplication process performed on the first head of the 256th sequence according to an embodiment of the present disclosure.

[0199] Reference Figure 25 , the first to eighth key vectors K1(241) to K8(241) generated in the 241st sequence n_seq=241 are stored in the first row of the first memory bank BK0 of the sixteenth PIM device PIM15. Although not shown in the figure, the first to eighth query vectors generated in the 241st sequence n_seq=241 are stored in the global buffer GB of the first to sixteenth PIM devices PIM0 to PIM15. The first to sixteenth processing units PU0 to PU15 of each of the first to fifteenth PIM devices PIM0 to PIM14 and the first processing unit PU0 of the sixteenth PIM device PIM15 perform the first matrix-vector multiplication in the 241st sequence n_seq=241 to generate the first to 241st parts of each of the first to eighth attention score vectors of the 241st sequence n_seq=241.

[0200] Next, the first to eighth key vectors K1 (242) to K8 (242) generated in the 242nd sequence n_seq = 242 are stored in the first row of the second memory bank BK1 of the sixteenth PIM device PIM15. Although not shown in the figure, the first to eighth query vectors generated in the 242nd sequence n_seq = 242 are stored in the global buffer GB of the first to sixteenth PIM devices PIM0 to PIM15. The first to sixteenth processing units PU0 to PU15 of each of the first to fifteenth PIM devices and the first and second processing units PU0 and PU1 of the sixteenth PIM device PIM15 perform the first matrix-vector multiplication in the 242nd sequence n_seq = 242 to generate the first to 242nd parts of each of the first to eighth attention score vectors of the 242nd sequence n_seq = 242.

[0201] Although not shown in the figure, the first to eighth key vectors generated in the 243rd to 255th sequences n_seq=243 to n_seq=255 are respectively stored in the first rows of the third to fifteenth memory banks of the sixteenth PIM device PIM15. In addition, the first to eighth query vectors generated in the 243rd to 255th sequences n_seq=243 to n_seq=255 are stored in the global buffer GB of the first to sixteenth PIM devices PIM0 to PIM15. For the 243rd to 255th sequences n_seq=243 to n_seq=255, a first matrix-vector multiplication of the key vector and the query vector is also performed to obtain a portion of each of the first to eighth attention score vectors of the 243rd to 255th sequences n_seq=243 to n_seq=255.

[0202] Next, the first to eighth key vectors K1 (256) to K8 (256) generated in the 256th sequence n_seq = 256 are stored in the first row of the 16th memory bank BK15 of the 16th PIM device PIM15. The first to eighth query vectors Q1 (256) to Q8 (256) generated in the 256th sequence n_seq = 256 are stored in the global buffer GB of the first to sixteenth PIM devices PIM0 to PIM15. The first to sixteenth processing units PU0 to PU15 of each of the first to fifteenth PIM devices and the first to sixteenth processing units PU0 to PU15 of the sixteenth PIM device PIM15 perform the first matrix-vector multiplication in the 256th sequence n_seq = 256 to generate the first to 256th parts of each of the first to eighth attention score vectors of the 256th sequence n_seq = 256.

[0203] Specifically, the first matrix-vector multiplication of the first head HEAD_1 of the 256th sequence n_seq=256 is performed in parallel in the first to sixteenth processing units PU0 to PU15 of each of the first to sixteenth PIM devices PIM0 to PIM15. The first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 perform the first matrix-vector multiplication on the first key vectors K1(1) to K1(16) generated in the first to sixteenth sequences n_seq=1 to n_seq=16 and the first query vector Q1(256) generated in the 256th sequence n_seq=256 to generate the first to sixteenth parts A1(256)_1 to A1(256)_16 of the first attention score vector A1(256) for the first head HEAD_1 of the 256th sequence n_seq=256. The first to sixteenth processing units PU0 to PU15 of the second PIM device PIM1 perform a first matrix-vector multiplication on the first key vectors K1(17) to K1(32) generated in the 17th to 32nd sequences n_seq=17 to n_seq=32 and the first query vector Q1(256) generated in the 256th sequence n_seq=256 to generate the 17th to 32nd parts A1(256)_17 to A1(256)_32 of the first attention score vector A1(256) for the first head HEAD_1 of the 256th sequence n_seq=256. Although not shown in the figure, the first to sixteenth processing units of each of the third to fifteenth PIM devices perform the first matrix-vector multiplication on the first key vector generated in the 33rd to 240th sequences n_seq=33 to n_seq=240 and the first query vector Q1(256) generated in the 256th sequence n_seq=256 to generate the 33rd to 240th parts of the first attention score vector A1(256) for the first head HEAD_1 of the 256th sequence n_seq=256. The first to sixteenth processing units PU0 to PU15 of the sixteenth PIM device PIM15 perform a first matrix-vector multiplication on the first key vectors K1(241) to K1(256) generated in the 241st to 256th sequences n_seq=241 to n_seq=256 and the first query vector Q1(256) generated in the 256th sequence n_seq=256 to generate the 241st to 256th parts of the first attention score vector A1(256) for the first head HEAD_1 of the 256th sequence n_seq=256. The first attention score vector A1(256) of the 256th sequence n_seq=256 can be obtained by concatenating the first to 256th parts A1(256)_1 to A1(256)_256 in the row direction.

[0204] Figure 26is a diagram illustrating a method of writing first to eighth key vectors generated in the 257th sequence of multi-head attention and a first process of a first matrix-vector multiplication performed on a first head according to an embodiment of the present disclosure. Figure 27 is a diagram illustrating a second process of the first matrix-vector multiplication performed on the first head when the first to eighth key vectors generated in the 257th sequence of the multi-head attention are written according to an embodiment of the present disclosure.

[0205] Reference Figure 26 and Figure 27 , when multi-head attention is performed for the first to 256th sequences n_seq=1 to n_seq=256, the first to eighth key vectors generated in the first to 256th sequences n_seq=1 to n_seq=256 are written to all first rows of the first to sixteenth memory banks BK0 to BK15 of each of the first to sixteenth PIM devices PIM0 to PIM15. In this state, the first to eighth key vectors K1(257) to K8(257) generated in the 257th sequence n_seq=257 are written to the first memory bank BK0 of the first PIM device PIM0. The first to eighth key vectors K1(257) to K8(257) of the 257th sequence n_seq=257 are written to the second row of the rows of the first memory bank BK0 of the first PIM device PIM0, the second row being different from the first row of the first to eighth key vectors of the first sequence being written. In one example, the second row may be a row after the first row. The first to eighth query vectors Q1 (257) to Q(257) generated in the 257th sequence n_seq=257 are stored in the global buffers GB of the first to sixteenth PIM devices PIM0 to PIM15.

[0206] In the 257th sequence n_seq=257, since the key vectors K1(1) to K8(1) generated in the first sequence n_seq=1 and the key vectors K1(257) to K8(257) generated in the 257th sequence n_seq=257 are respectively stored in the first row and the second row of the first memory bank BK0 of the first PIM device PIM0, the first matrix-vector multiplication in the first to eighth heads HEAD1 to HEAD8 of the 257th sequence n_seq=257 is divided into two operations and performed. The first operation of the first matrix-vector multiplication in the first head HEAD_1 of the 257th sequence n_seq=257 is performed using the first key vectors K1(1) to K1(256) generated in the first to 256th sequences n_seq=1 to n_seq=256 and stored in the first row of the first to sixteenth memory banks BK0 to BK15 of each of the first to sixteenth PIM devices PM0 to PM15 as operands. Furthermore, the second operation of the first matrix-vector multiplication in the first head HEAD_1 of the 257th sequence n_seq=257 is performed using the first key vector K1 (257) generated in the 257th sequence and stored in the second row of the first memory bank BK0 of the first PIM device PIM0 as an operand.

[0207] First, if Figure 26As shown, to perform the first operation of the first matrix-vector multiplication in the first header HEAD_1 of the 257th sequence n_seq=257, the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 receive the first key vectors K1(1) to K1(16) generated in the first to sixteenth sequences n_seq=1 to n_seq=16 from the first to sixteenth memory banks BK0 to BK15 of the first PIM device PIM0, respectively. The first to sixteenth processing units PU0 to PU15 of the second PIM device PIM1 receive the first key vectors K1(17) to K1(32) generated in the seventeenth to thirty-second sequences n_seq=17 to n_seq=32 from the first to sixteenth memory banks BK0 to BK15 of the second PIM device PIM1, respectively. In the same manner, the first to sixteenth processing units PU0 to PU15 of the sixteenth PIM device PIM15 receive the first key vectors K1 (241) to K1 (256) generated in the 241st to 256th sequences n_seq = 241 to n_seq = 256 from the first to sixteenth memory banks BK0 to BK15 of the sixteenth PIM device PIM15, respectively. Although not shown, the first to sixteenth processing units of the third to fifteenth PIM devices receive the first key vectors of the 33rd to 240th sequences in the same manner. The first to sixteenth processing units PU0 to PU15 of each of the first to sixteenth PIM devices PIM0 to PIM15 collectively receive the first query vector Q1 (257) generated in the 257th sequence n_seq = 257 from the global buffer GB. The first to sixteenth processing units PU0 to PU15 of the first to sixteenth PIM devices PIM0 to PIM15 perform the first matrix-vector multiplication in the first head HEAD_1 on the first key vectors K1(1) to K1(256) of the first to 256th sequences n_seq=1 to n_seq=256 and the first query vector Q1(257) of the 257th sequence n_seq=257 to generate the first to 256th parts A1(257)_1 to A1(257)_256 of the first attention score vector A1(257) of the 257th sequence n_seq=257.

[0208] Next, if Figure 27As shown, in order to perform the second operation of the first matrix-vector multiplication in the first header HEAD_1 of the 257th sequence n_seq=257, the first processing unit PU0 of the first PIM device PIM0 receives the first key vector K1(257) generated in the 257th sequence n_seq=257 from the first memory bank BK0 of the first PIM device PIM0, and receives the first query vector Q1(257) generated in the 257th sequence n_seq=257 from the global buffer GB of the first PIM device PIM0. The first processing unit PU0 of the first PIM device PIM0 performs the first matrix-vector multiplication in the first header HEAD_1 on the first key vector K1(257) of the 257th sequence n_seq=257 and the first query vector Q1(257) of the 257th sequence n_seq=257 to generate the 257th part A1(257)_257 of the first attention score vector A1(257) of the 257th sequence n_seq=257. The attention score vector A1(257) of the 257th sequence n_seq=257 can be obtained by concatenating the first to 256th parts A1(257)_1 to A1(257)_256 generated by the first operation of the first matrix-vector multiplication and the 257th part A1(257)_257 generated by the second operation of the first matrix-vector multiplication in the row direction. The first operation and the second operation of the first matrix-vector multiplication for the first head of the 257th sequence n_seq=257 are also applied to the second to eighth heads of the 257th sequence n_seq=257. Therefore, for each of the second to eighth heads, the first to 257th parts of each of the second to eighth attention score vectors of the 257th sequence n_seq=257 are generated.

[0209] As reference Figure 26 and Figure 27 As described, when the key vector is stored in two different rows of the memory bank of at least one of the first to sixteenth PIM devices PIM0 to PIM15, the first matrix-vector multiplication is performed in two operations. In this case, the two operations of the first matrix-vector multiplication can be performed for each of the first to eighth heads HEAD1 to HEAD8. However, in another example, the first operation of the first matrix-vector multiplication can be performed continuously for the first to eighth heads HEAD1 to HEAD8, and the second operation of the first matrix-vector multiplication can be performed continuously for the first to eighth heads HEAD1 to HEAD8. In this case, the first to 256th parts of each of the first to eighth attention score vectors of the 257th sequence n_seq=257 are first generated sequentially, and then the 257th parts of the first to eighth attention score vectors of the 257th sequence n_seq=257 are subsequently generated sequentially.

[0210] Figure 281 is a diagram illustrating another example of a method of writing the first to eighth key vectors generated in the first to 257th sequences of multi-head attention into a memory bank of a PIM device according to an embodiment of the present disclosure. Figure 28 In the figure, the global buffer and processing unit are omitted for clarity. In this example, it is assumed that in the multi-head attention process of the neural network architecture, the total size of the key vector generated in one sequence is smaller than the page size of the memory bank. That is, in one example, the page size of the memory bank is 2KB, the number of sequences n_seq of the multi-head attention is 257, the number of heads n_head is 6, and the size of the head d_head is 128. The key vector K generated in one sequence has a size of n_seq×d_head, that is, a size of n_seq×128. Assume that each of the 128 elements included in a single row of the key vector K is 16-bit floating-point data in BF16 format. In the following, for ease of explanation, the access unit for the column of the memory bank will not be considered, so the column of the memory bank will be simply expressed in bits.

[0211] Reference Figure 28 , the first to sixth key vectors K1(1) to K6(1) generated in the first sequence n_seq=1 are stored in the first row of the first memory bank BK0 of the first PIM device PIM0. Because the total size of the first to sixth key vectors K1(1) to K6(1) is 16×128×6, the first to sixth key vectors K1(1) to K6(1) are stored in the first to 12,288th columns of the first row of the first memory bank BK0. Because the page size of the first memory bank BK0 is 2KB, the remaining columns (e.g., 12289th to 16384th columns) of the first row in the first memory bank BK0 are empty. Although not shown, the first to sixth key vectors generated in the second to fifteenth sequences are stored in the first to 12,288th columns of the first row of the second to fifteenth memory banks of the first PIM device PIM0. The first to sixth key vectors K1(16) to K6(16) generated in the first sequence n_seq=1 are stored in the first row, first to 12288th columns of the sixteenth memory bank BK15 of the first PIM device PIM0.

[0212] The first to sixth key vectors K1(17) to K6(17) generated in the seventeenth sequence n_seq=17 are stored in the first row, first to 12288th columns of the first memory bank BK0 of the second PIM device PIM1. Although not shown, the first to sixth key vectors generated in the eighteenth to thirty-first sequences are stored in the first row, first to 12288th columns of the second to fifteenth memory banks of the second PIM device PIM1. The first to sixth key vectors K1(32) to K6(32) generated in the thirty-second sequence n_seq=32 are stored in the first row, first to 12288th columns of the sixteenth memory bank BK15 of the second PIM device PIM1. As with the first PIM device PIM0 and the second PIM device PIM1, the first to sixth key vectors generated in the 33rd to 240th sequences are stored in the first row, first to 12288th columns of the first to sixteenth memory banks BK0 to BK15 of each of the third to fifteenth PIM devices PIM2 to PIM14.

[0213] The first to sixth key vectors K1 (241) to K6 (241) generated in the 241st sequence n_seq=241 are stored in the first row, first column to the 12288th column of the first memory bank BK0 of the sixteenth PIM device PIM15. Although not shown, the first to sixth key vectors generated in each of the 242nd to 255th sequences are stored in the first row, first column to the 12288th column of the second to fifteenth memory banks of the sixteenth PIM device PIM15. The first to sixth key vectors K1 (256) to K6 (256) generated in the 256th sequence n_seq=256 are stored in the first row, first column to the 12288th column of the sixteenth memory bank BK15 of the sixteenth PIM device PIM15. Because the first to sixth key vectors generated in each of the first to 256th sequences are stored in all first rows of the first to 16th memory banks BK0 to BK15 of each of the first to 16th PIM devices PIM0 to PIM15, the first to sixth key vectors K1 (257) to K6 (257) generated in the 257th sequence n_seq=257 are stored in the first to 12288th columns of the second row of the first memory bank BK0 in the first PIM device PIM0. Here, the second row is the row whose row address follows the row address of the first row. However, in other embodiments, the row address of the first row and the row address of the second row may not be consecutive row addresses.

[0214] Figure 29 1 is a diagram showing another example of a method of writing the first to eighth key vectors generated in the first to 257th sequences of multi-head attention into the memory bank of the PIM device. Figure 29In the figure, the global buffer and processing unit are omitted for clarity. In this example, it is assumed that in the multi-head attention of the neural network architecture, the total size of the key vector generated in one sequence is larger than the page size of the memory bank. That is, in one example, the page size of the memory bank is 2KB, the number of sequences n_seq of the multi-head attention is 257, the number of heads n_head is 9, and the size of the head d_head is 128. The key vector K generated in one sequence has a size of n_seq×d_head, that is, a size of n_seq×128. It is assumed that each of the 128 elements included in a row of the key vector K is 16-bit floating point data in the BF16 format.

[0215] Reference Figure 29 , the size of the first row of the first memory bank BK0 of the first PIM device PIM0 is 2KB, and the total size of the first to ninth key vectors K1(1) to K9(1) generated in the first sequence n_seq=1 is 16(BF16)×128(d_head)×9(n_head) bits, which is larger than 2KB. Therefore, among the first to ninth key vectors K1(1) to K9(1) generated in the first sequence n_seq=1, the first to eighth key vectors K1(1) to K8(1) having a size of 2KB are written to the first row of the first memory bank BK0 of the first PIM device PIM0. In addition, the remaining ninth key vector K9(1) among the first to ninth key vectors K1(1) to K9(1) generated in the first sequence n_seq=1 is written to the second row of the first memory bank BK0 of the first PIM device PIM0. Among the first to ninth key vectors K1(257) to K9(257) generated in the 257th sequence n_seq=257, the first to eighth key vectors K1(257) to K8(257) having a size of 2KB are written to the third row of the first memory bank BK0 of the first PIM device PIM0. The remaining ninth key vector K9(257) among the first to ninth key vectors K1(257) to K9(257) generated in the 257th sequence n_seq=257 is written to the fourth row of the first memory bank BK0 of the first PIM device PIM0.

[0216] The first to ninth key vectors generated in the second to 256th sequences are also written in the same manner as the first to ninth key vectors generated in the first sequence. Figure 29As shown, among the first to ninth key vectors K1(16) to K9(16) generated in the sixteenth sequence n_seq=16, the first to eighth key vectors K1(16) to K8(16) having a size of 2KB are written to the first row of the sixteenth memory bank BK15 of the first PIM device PIM0. The remaining ninth key vector K9(16) among the first to ninth key vectors K1(16) to K9(16) generated in the sixteenth sequence n_seq=16 is written to the second row of the sixteenth memory bank BK15 of the first PIM device PIM0. The first to eighth key vectors K1(17) to K8(17) generated in the seventeenth sequence n_seq=17 are written to the first row of the first memory bank BK0 of the second PIM device PIM1, and the ninth key vector K9(17) is written to the second row of the first memory bank BK0 of the second PIM device PIM1. The first to eighth key vectors K1(32) to K8(32) generated in the 32nd sequence n_seq=32 are written to the first row of the sixteenth memory bank BK15 of the second PIM device PIM1, and the ninth key vector K9(32) is written to the second row of the sixteenth memory bank BK15 of the second PIM device PIM1. The first to eighth key vectors K1(241) to K8(241) generated in the 241st sequence n_seq=241 are written to the first row of the first memory bank BK0 of the sixteenth PIM device PIM15, and the ninth key vector K9(241) is written to the second row of the first memory bank BK0 of the sixteenth PIM device PIM15. The first to eighth key vectors K1 (256) to K8 (256) generated in the 256th sequence n_seq=256 are written to the first row of the sixteenth memory bank BK15 of the sixteenth PIM device PIM15, and the ninth key vector K9 (256) is written to the second row of the sixteenth memory bank BK15 of the sixteenth PIM device PIM15.

[0217] Figure 30 This is a diagram showing an embodiment of the present disclosure. Figure 5 Figure 2 shows an example of a value vector generated in a sequence of multi-head attention and a value matrix used as an operand in the second matrix-vector multiplication. Figure 30 In the multi-head attention, the number of sequences n_seq in the multi-head attention is 257, and the number of heads n_head in the multi-head attention is 8. In addition, it is assumed that the size of the head d_head of the multi-head attention is 128, so that the value vector V has 128 elements. In the following, each of the 128 elements of the value vector V is 16-bit floating point data in the BF16 format. Figure 30Only the first head HEAD_1, the second head HEAD_2 and the eighth head HEAD_8 are shown, but the following description can be applied in the same manner to the remaining third to seventh heads HEAD_3 to HEAD_7 not shown, and other embodiments may vary in the number and configuration of the components shown.

[0218] Reference Figure 30 , for each of the first to 257th sequences n_seq=1 to n_seq=257, generate as many value vectors as the number of heads of the multi-head attention, that is, eight value vectors. Figure 30 As shown, in the first sequence n_seq=1, the first to eighth value vectors V1(1) to V8(1) are generated. The first to eighth value vectors V1(1) to V8(1) generated in the first sequence n_seq=1 are used as operands for the second matrix-vector multiplication of the first sequence n_seq=1 performed on the first to eighth heads HEAD_1 to HEAD_8. In the second sequence n_seq=2, the first to eighth value vectors V1(2) to V8(2) are generated. The first to eighth value vectors V1(2) to V8(2) generated in the second sequence n_seq=2 are used together with the first to eighth value vectors V1(1) to V8(1) generated in the first sequence n_seq=1 as operands for the second matrix-vector multiplication of the second sequence n_seq=2 performed on the first to eighth heads HEAD_1 to HEAD_8. In the 257th sequence n_seq=257, the first to eighth value vectors V1(257) to V8(257) are generated. The first to eighth value vectors V1(257) to V8(257) generated in the 257th sequence n_seq=257 are used together with the first to eighth value vectors generated in the first to 256th sequences n_seq=1 to n_seq=256 as operands for the second matrix-vector multiplication of the 257th sequence n_seq=257 performed on the first to eighth heads HEAD_1 to HEAD_8. The process of generating the first to eighth value vectors in each sequence is the same as that described above with reference to Figure 6 Same as described.

[0219] More specifically, each of the first to eighth value vectors V1(1) to V8(1) generated in the first sequence n_seq=1 has first to 128th elements V1(1)_1 to V1(1)_128 arranged in the row direction. Each of the first to 128th elements V1(1)_1 to V1(1)_128 consists of 16-bit floating-point data in the BF16 format. The first to eighth value vectors V1(1) to V8(1) generated in the first sequence n_seq=1 are used as first to eighth value matrices, which are operands in the second matrix-vector multiplication in the first sequence n_seq=1. For example, the first value vector V1(1) generated in the first sequence n_seq=1 is used as the first value matrix, which is the operand in the second matrix-vector multiplication performed on the first head HEAD_1 of the first sequence n_seq=1. The second value vector V2(1) generated in the first sequence n_seq=1 is used as a second value matrix, which is an operand in the second matrix-vector multiplication performed on the second head HEAD_2 of the first sequence n_seq=1. Similarly, the eighth value vector V8(1) generated in the first sequence n_seq=1 is used as an eighth value matrix, which is an operand in the second matrix-vector multiplication performed on the eighth head HEAD_8 of the first sequence n_seq=1.

[0220] The first to eighth value vectors V1(2) to V8(2) generated in the second sequence n_seq=2 are used together with the first to eighth value vectors V1(1) to V8(1) generated in the first sequence n_seq=1 as operands in the second matrix-vector multiplication in the second sequence n_seq=2. That is, the first to eighth value matrices used as operands in the second matrix-vector multiplication in the second sequence n_seq=2 are formed by concatenating the first to eighth value vectors V1(1) to V8(1) generated in the first sequence n_seq=1 and the first to eighth value vectors V1(2) to V8(2) generated in the second sequence n_seq=2 in the column direction, respectively.

[0221] For example, a first value matrix CONCAT(V1(1), V1(2)) used as an operand in a second matrix-vector multiplication performed on a first header HEAD_1 of a second sequence n_seq=2 is formed by concatenating in the column direction the first value vector V1(1) generated in the first sequence n_seq=1 and the first value vector V1(2) generated in the second sequence n_seq=2. A second value matrix CONCAT(V2(1), V2(2)) used as an operand in a second matrix-vector multiplication in a second header HEAD_2 of a second sequence n_seq=2 is formed by concatenating in the column direction the second value vector V2(1) generated in the first sequence n_seq=1 and the second value vector V2(2) generated in the second sequence n_seq=2. In the same manner, the eighth value matrix CONCAT(V8(1), V8(2)) used as an operand in the second matrix-vector multiplication in the eighth head HEAD_8 of the second sequence n_seq=2 is formed by concatenating the eighth value vector V8(1) generated in the first sequence n_seq=1 and the eighth value vector V8(2) generated in the second sequence n_seq=2 in the column direction.

[0222] Similarly, the first to eighth value vectors V1(257) to V8(257) generated in the 257th sequence n_seq=257 are used as operands in the second matrix-vector multiplication in the 257th sequence n_seq=257, together with the first to eighth value vectors generated in the first to 256th sequences. That is, the first to eighth value matrices used as operands in the second matrix-vector multiplication in the 257th sequence n_seq=257 are formed by concatenating the first to eighth value vectors of all 257 sequences in the column direction, i.e., from the first to eighth value vectors V1(1) to V8(1) generated in the first sequence n_seq=1 to the first to eighth value vectors V1(257) to V8(257) generated in the 257th sequence n_seq=257.

[0223] For example, a first value matrix CONCAT(V1(1), V1(2), ... V1(257)) serving as an operand in the second matrix-vector multiplication performed on the first head HEAD_1 of the 257th sequence n_seq=257 is formed by concatenating in the column direction the first value vectors V1(1) to V1(257) generated in the first to 257th sequences n_seq=1 to n_seq=257. A second value matrix CONCAT(V2(1), V2(2), ... V2(257)) serving as an operand in the second matrix-vector multiplication performed on the second head HEAD_2 of the 257th sequence n_seq=257 is formed by concatenating in the column direction all the second value vectors V2(1) to V2(257) generated in the first to 257th sequences n_seq=1 to n_seq=257. In the same manner, the eighth value matrix CONCAT(V8(1), V8(2), ...V8(257)) used as an operand in the second matrix-vector multiplication performed on the eighth head HEAD_8 of the 257th sequence n_seq=257 is formed by concatenating the eighth value vectors V8(1) to V8(257) generated in the first to 257th sequences n_seq=1 to n_seq=257 in the column direction.

[0224] In order to perform transformer-based multi-head attention in the neural network architecture according to the present disclosure, the plurality of memory banks of each of the plurality of PIM devices write value vectors in a second manner. Here, the second manner can be defined as a set of operations in which the value vectors generated in one of the plurality of sequences of the multi-head attention are distributed and stored in the plurality of memory banks of each of the plurality of PIM devices in units of elements.

[0225] In one example, assume that the number of PIM devices is "P," the number of memory banks included in each of the PIM devices is "M," the number of heads of multi-head attention is "N," one head of the multi-head attention includes "D" elements, "K" is a natural number from 1 to "N," and "D / M" is less than "P." In this case, in the N value vectors generated in a first sequence, one of the multiple sequences, the first through "D" elements of the "K"-th value vector are distributed and stored in "M"-element increments across the first through "D / M"-th PIM devices in the plurality of PIM devices. Within the first through "D" elements of the "K"-th value vector, the "M" elements written into each of the first through "D / M" PIM devices are distributed and written in "one" increments across the "M" memory banks included in each of the first through "D / M" PIM devices. Specifically, the "M" elements written in "one" increments across the "M" memory banks included in each of the first through "D / M" PIM devices are written to the same first row and first column.

[0226] Similarly, in the N value vectors generated in the second sequence (which is a sequence subsequent to the first sequence), the 1st to 'D'th elements of the 'K'th value vector are distributed as 'M' elements and written to the 1st to 'D / M'th PIM devices among the plurality of PIM devices. The 'M' elements of the 1st to 'D'th elements of the 'K'th value vector that are written to each of the 1st to 'D / M'th PIM devices are distributed as single elements and written to the 'M'th memory banks included in each of the 1st to 'D / M'th PIM devices. In the second sequence, the 'M'th elements distributed and written to the 'M'th memory banks included in each of the 1st to 'D / M'th PIM devices are written to the first row and the second column, the second column being the next column after the first column.

[0227] In one example, it is assumed that each of the N heads of the multi-head attention includes (P / 2)·M elements, i.e., D=(P / 2)·M. In this case, the first to Dth elements of the odd-numbered value vectors in the N value vectors generated in a first sequence as one of the plurality of sequences may be stored in the first to P / 2th PIM devices among the plurality of PIM devices, and the first to Dth elements of the even-numbered value vectors in the N value vectors generated in the first sequence may be stored in the (P / 2)+1th to Pth PIM devices among the plurality of PIM devices. That is, the first to Dth elements of the odd-numbered value vectors generated in the first sequence may be distributed in units of M elements and stored in the first to P / 2th PIM devices, and the first to Dth elements of the even-numbered value vectors generated in the first sequence may be distributed in units of M elements and stored in the (P / 2)+1th to Pth PIM devices. In particular, the “M” elements of the odd-numbered value vectors generated, distributed, and written to the first through “P / 2”th PIM devices in the first sequence may be distributed in units of single elements and stored in the first through “M”th memory banks of each of the first through “P / 2”th PIM devices, and the “M” elements of the even-numbered value vectors generated, distributed, and stored in the “(P / 2)+1”th through “P”th PIM devices in the first sequence may be distributed in units of single elements and stored in the first through “M”th memory banks of each of the “(P / 2)+1”th through “P”th PIM devices.

[0228] In one example, odd-numbered value vectors generated in the first sequence, from the first odd-numbered value vector to the last odd-numbered value vector, may be stored in different rows of the first through "M"th memory banks of each of the first through "P / 2"th PIM devices. Furthermore, even-numbered value vectors generated in the first sequence, from the first even-numbered value vector to the last even-numbered value vector, may be stored in different rows of the first through "M"th memory banks of each of the "(P / 2)+1"th through "P"th PIM devices. In this case, the odd-numbered value vectors generated in the first sequence are stored and designated by the same first column address in the first through "M"th memory banks of each of the first through "P / 2"th PIM devices, while the even-numbered value vectors generated in the first sequence are stored and designated by the same first column address in the first through "M"th memory banks of each of the "(P / 2)+1"th through "P"th PIM devices.

[0229] The odd-numbered value vectors generated in a second sequence (a sequence subsequent to the first sequence) of the plurality of sequences are stored in the same rows as the rows in the first through Mth memory banks of each of the first through P / 2th PIM devices in which the odd-numbered value vectors generated in the first sequence were stored, but are stored and designated by a second column address (an address subsequent to the first column address). Furthermore, the even-numbered value vectors generated in the second sequence are stored in the same rows as the rows in the first through Mth memory banks of each of the (P / 2)+1 through Pth PIM devices in which the even-numbered value vectors generated in the first sequence were stored, but are stored and designated by a second column address (an address subsequent to the first column address). To perform the second matrix-vector multiplication, the first through Dth elements of the odd-numbered attention distribution vectors are stored in the global buffers of the first through P / 2th PIM devices, while the first through Dth elements of the even-numbered attention distribution vectors are stored in the global buffers of the (P / 2)+1 through Pth PIM devices.

[0230] In one example, the D elements of a first value vector in the N value vectors generated in a first sequence of the plurality of sequences are distributed, stored, and designated by a first column address in a first row of first through M-th memory banks of each of the first through P / 2-th PIM devices. The D elements of a second value vector in the N value vectors generated in the first sequence are distributed, stored, and designated by a first column address in a first row of first through M-th memory banks of each of the (P / 2)+1-th through P-th PIM devices. The D elements of a third value vector in the N value vectors generated in the first sequence are distributed, stored, and designated by a first column address in a second row of first through M-th memory banks of each of the first through P / 2-th PIM devices. In addition, the "D" element of the fourth value vector among the "N" value vectors generated in the first sequence is distributed, stored, and designated by the first column address in the second row of the first to "M"th memory banks of each of the "(P / 2)+1"th to "P"th PIM devices. In this case, the odd-numbered attention distribution vectors among the attention distribution vectors are stored in the global buffer of the first to "P / 2"th PIM devices, and the even-numbered attention distribution vectors among the attention distribution vectors are stored in the global buffer of the "(P / 2)+1"th to "P"th PIM devices.

[0231] The first to Mth processing units of each of the first to P / 2th PIM devices perform a second matrix-vector multiplication on the M elements of the first value vector stored in the first to Mth memory banks of each of the first to P / 2th PIM devices and the first attention distribution vector stored in the global buffer of each of the first to P / 2th PIM devices to generate first to Dth segments of the first multi-head attention value vector. The first to Mth processing units of each of the (P / 2)+1th to Pth PIM devices perform a second matrix-vector multiplication on the M elements of the second value vector stored in the first to Mth memory banks of each of the (P / 2)+1th to Pth PIM devices and the second attention distribution vector stored in the global buffer of each of the (P / 2)+1th to Pth PIM devices to generate first to Dth segments of the second multi-head attention value vector.

[0232] The first to Mth processing units of each of the first to P / 2th PIM devices perform a second matrix-vector multiplication on the Mth elements of the third value vector stored in the first to Mth memory banks of each of the first to P / 2th PIM devices and the third attention distribution vector stored in the global buffer of each of the first to P / 2th PIM devices to generate first to Dth segments of the third multi-head attention value vector. The first to Mth processing units of each of the (P / 2)+1th to Pth PIM devices perform a second matrix-vector multiplication on the Mth elements of the fourth value vector stored in the first to Mth memory banks of each of the (P / 2)+1th to Pth PIM devices and the second attention distribution vector stored in the global buffer of each of the (P / 2)+1th to Pth PIM devices to generate first to Dth segments of the fourth multi-head attention value vector.

[0233] Figures 31 to 46 is a diagram showing how the value vectors generated in multiple sequences of multi-head attention are distributed and stored in the PIM device in the neural network architecture according to an embodiment of the present disclosure. The following description is assumed to apply to the above reference Figure 1 The neural network architecture described ( Figure 1 10), but the embodiment is not limited to this configuration. The neural network architecture 10 includes first to sixteenth PIM devices PIM0 to PIM15, and each PIM device includes 16 memory banks and 16 processing units. In this example, the process of generating the value vector and the matrix format of the value matrix are the same as those in the reference Figure 30 Therefore, in the following, the example shown assumes that the number of sequences n_seq of the multi-head attention is 257, the number of heads n_head is 8, the size of the head d_head is 128, and each of the 128 elements of the value vector is 16-bit floating-point data in BF16 format. In addition, it is assumed that each of the memory bank and the global buffer has a page size of 2KB.

[0234] Figure 31 is a diagram illustrating how first value vectors generated in a first sequence of multi-head attention are distributed and stored in a first group of PIM devices according to an embodiment of the present disclosure. Figure 32 is a diagram illustrating a process according to an embodiment of the present disclosure, in which a first value vector generated in a first sequence of multi-head attention is distributed in units of single elements and stored in a memory bank of a first group of PIM devices, and in which a second matrix-vector multiplication is performed on the first head.

[0235] First refer to Figure 31, the first value vector V1(1) generated in the first sequence n_seq=1 is distributed in element units and written to the memory banks BK of the first group of PIM devices in the first to sixteenth PIM devices PIM0 to PIM15. The first to 128th elements V1(1)_1 to V1(1)_128 of the first value vector V1(1) are distributed and written to each memory bank BK of the first group of PIM devices. That is, one of the first to 128th elements V1(1)_1 to V1(1)_128 of the first value vector V1(1) is written to each memory bank of the first to sixteenth memory banks BK0 to BK15 in each of the PIM devices in the first group of PIM devices. Therefore, since one PIM device includes 16 memory banks, the first group of PIM devices includes the first to eighth PIM devices PIM0 to PIM7 into which the first to 128th elements V1(1)_1 to V1(1)_128 of the first value vector V1(1) are written.

[0236] More specifically, since each of the first to sixteenth PIM devices PIM0 to PIM15 includes the first to sixteenth memory banks BK0 to BK15, the first to 128th elements V1(1)_1 to V1(1)_128 of the first value vector V1(1) are divided into 16-element units and written to different PIM devices. In other words, Figure 31 As shown, the first to sixteenth elements V1(1)_1 to V1(1)_16 of the first value vector V1(1) are respectively distributed and stored in the first to sixteenth memory banks BK0 to BK15 of the first PIM device PIM0. The 17th to 32nd elements V1(1)_17 to V1(1)_32 of the first value vector V1(1) are respectively distributed and stored in the first to sixteenth memory banks BK0 to BK15 of the second PIM device PIM1. In the same manner, the 113th to 128th elements V1(1)_113 to V1(1)_128 of the first value vector V1(1) are respectively distributed and stored in the first to sixteenth memory banks BK0 to BK15 of the eighth PIM device PIM7. Although Figure 31 Although not shown in FIG. 1 , the 33rd to 112th elements of the first value vector V1 ( 1 ) are distributed in units of 16 elements and are stored in the third to seventh PIM devices, respectively.

[0237] More specifically, refer to Figure 32, the first element V1(1)_1 of the first value vector V1(1) is written to the first memory bank BK0 of the first PIM device PIM0. The first element V1(1)_1 of the first value vector V1(1) is stored in one of the rows of the first memory bank BK0 of the first PIM device PIM0, for example, the first row. The first element V1(1)_1 of the first value vector V1(1) can be specified by the first column address in the first row of the first memory bank BK0 of the first PIM device PIM0. Because one element of the first value vector V1(1) is in BF16 format, that is, a 16-bit floating point format, the first column address specifies 16 columns, for example, the first to sixteenth columns in the first row of the first memory bank BK0 of the first PIM device PIM0. The second element V1(1)_2 of the first value vector V1(1) is stored in the first to sixteenth columns of the first row of the second memory bank BK1 of the first PIM device PIM0. Although not shown, in the same manner, the third to fifteenth elements of the first value vector V1(1) are stored in the first row, first to sixteenth columns of the third to fifteenth memory banks of the first PIM device PIM0, respectively. The sixteenth element V1(1)_16 of the first value vector V1(1) is stored in the first row, first to sixteenth columns of the sixteenth memory bank BK15 of the first PIM device PIM0.

[0238] The 17th element V1(1)_17 of the first value vector V1(1) is stored in the first row, first to sixteenth columns of the first memory bank BK0 of the second PIM device PIM1. The 18th element V1(1)_18 of the first value vector V1(1) is stored in the first row, first to sixteenth columns of the second memory bank BK1 of the second PIM device PIM1. Although not shown, in the same manner, the 19th to 31st elements of the first value vector V1(1) are stored in the first row, first to sixteenth columns of the third to fifteenth memory banks of the second PIM device PIM1, respectively. The 32nd element V1(1)_32 of the first value vector V1(1) is stored in the first row, first to sixteenth columns of the sixteenth memory bank BK15 of the second PIM device PIM1.

[0239] Although not shown, the 33rd to 48th elements of the first value vector V1(1) are stored in the first row, first to sixteenth columns of the first to sixteenth memory banks of the third PIM device PIM2, respectively. The 49th to 64th elements of the first value vector V1(1) are stored in the first row, first to sixteenth columns of the first to sixteenth memory banks of the fourth PIM device PIM3, respectively. The 65th to 80th elements of the first value vector V1(1) are stored in the first row, first to sixteenth columns of the first to sixteenth memory banks of the fifth PIM device PIM4, respectively. The 81st to 96th elements of the first value vector V1(1) are stored in the first row, first to sixteenth columns of the first to sixteenth memory banks of the sixth PIM device PIM5, respectively. In addition, the 97th to 112th elements of the first value vector V1(1) are stored in the first row, first to sixteenth columns of the first to sixteenth memory banks of the seventh PIM device PIM6, respectively.

[0240] The 113th element V1(1)_113 of the first value vector V1(1) is stored in the first row, first to sixteenth columns of the first memory bank BK0 of the eighth PIM device PIM7. The 114th element V1(1)_114 of the first value vector V1(1) is stored in the first row, first to sixteenth columns of the second memory bank BK1 of the eighth PIM device PIM7. Although not shown, in the same manner, the 115th to 127th elements of the first value vector V1(1) are stored in the first row, first to sixteenth columns of the third to fifteenth memory banks of the eighth PIM device PIM7, respectively. The 128th element V1(1)_128 of the first value vector V1(1) is stored in the first row, first to sixteenth columns of the sixteenth memory bank BK15 of the eighth PIM device PIM7, respectively.

[0241] In the same manner as described above, when the first value vector V1(1) generated in the first sequence n_seq=1 is written to the first group of PIM devices, i.e., the first to eighth PIM devices PIM0 to PIM7, the first attention distribution vector S1(1) is stored in each of the global buffers GB of the first to eighth PIM devices PIM0 to PIM7. Figure 17 As described, the first attention distribution vector S1(1) has the same size as the first attention score vector A1(1), which is the result of the first matrix-vector multiplication of the first key vector K1(1) and the first query vector Q1(1) generated in the first sequence n_seq=1. Therefore, the first attention distribution vector S1(1) generated in the first sequence n_seq=1 has a size of n_seq×1, that is, a size of 1×1. Because the first attention distribution vector S1(1) is in BF16 format, the first attention distribution vector S1(1) is stored in the first to sixteenth columns of each global buffer GB.

[0242] The first to sixteenth processing units PU0 to PU15 of each of the first to eighth PIM devices PIM0 to PIM7 perform a second matrix-vector multiplication on the first to 128th elements V1(1)_1 to V1(1)_128 of the first value vector V1(1) and the first attention distribution vector S1(1) to generate a first multi-head attention value vector Z1(1). Since the first value vector V1(1) generated in the first sequence n_seq=1 has a size of d_head×1, that is, a size of 128×1, and the first attention distribution vector S1(1) generated in the first sequence n_seq=1 has a size of 1×1, the first multi-head attention value vector Z1(1) generated by the second matrix-vector multiplication on the first head HEAD_1 of the first sequence n_seq=1 has a size of d_head×1, that is, a size of 128×1. That is, when the second matrix-vector multiplication of the first head HEAD_1 of the first sequence n_seq=1 is performed in the first to sixteenth processing units PU0 to PU15 of each of the first to eighth PIM devices PIM0 to PIM7, the first to 128th elements Z1(1)_1 to Z1(1)_128 of the first multi-head attention value vector Z1(1) are generated.

[0243] More specifically, the first processing unit PU0 of the first PIM device PIM0 receives the first element V1(1)_1 of the first value vector V1(1) and the first attention distribution vector S1(1) from the first memory bank BK0 and the global buffer GB, respectively. The first processing unit PU0 of the first PIM device PIM0 performs a second matrix-vector multiplication on the first element V1(1)_1 of the first value vector V1(1) and the first attention distribution vector S1(1) to generate the first element Z1(1)_1 of the first multi-head attention value vector Z1(1) of the first sequence n_seq=1. The second processing unit PU1 of the first PIM device PIM0 receives the second element V1(1)_2 of the first value vector V1(1) and the first attention distribution vector S1(1) from the second memory bank BK1 and the global buffer GB, respectively. The second processing unit PU1 of the first PIM device PIM0 performs a second matrix-vector multiplication on the second element V1(1)_2 of the first value vector V1(1) and the first attention distribution vector S1(1) to generate the second element Z1(1)_2 of the first multi-head attention value vector Z1(1) of the first sequence n_seq=1. In the same manner, the sixteenth processing unit PU15 of the first PIM device PIM0 receives the sixteenth element V1(1)_16 of the first value vector V1(1) and the first attention distribution vector S1(1) from the sixteenth storage body BK15 and the global buffer GB, respectively. The sixteenth processing unit PU15 of the first PIM device PIM0 performs a second matrix-vector multiplication on the sixteenth element V1(1)_16 of the first value vector V1(1) and the first attention distribution vector S1(1) to generate the sixteenth element Z1(1)_16 of the first multi-head attention value vector Z1(1) of the first sequence n_seq=1. Therefore, the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 generate the first to sixteenth elements Z1(1)_1 to Z1(1)_16 of the first multi-head attention value vector Z1(1) of the first sequence n_seq=1.

[0244] The first processing unit PU0 of the second PIM device PIM1 performs a second matrix-vector multiplication on the 17th element V1(1)_17 of the first value vector V1(1) and the first attention distribution vector S1(1) to generate the 17th element Z1(1)_17 of the first multi-head attention value vector Z1(1) of the first sequence n_seq=1. The second processing unit PU1 of the second PIM device PIM1 performs a second matrix-vector multiplication on the 18th element V1(1)_18 of the first value vector V1(1) and the first attention distribution vector S1(1) to generate the 18th element 1(1)_18 of the first multi-head attention value vector Z1(1) of the first sequence n_seq=1. In the same manner, the sixteenth processing unit PU15 of the second PIM device PIM1 performs a second matrix-vector multiplication on the 32nd element V1(1)_32 of the first value vector V1(1) and the first attention distribution vector S1(1) to generate the 32nd element Z1(1)_32 of the first multi-head attention value vector Z1(1) of the first sequence n_seq=1. In this manner, the first to sixteenth processing units PU0 to PU15 of the second PIM device PIM1 generate the 17th to 32nd elements Z1(1)_17 to Z1(1)_32 of the first multi-head attention value vector Z1(1) of the first sequence n_seq=1. Although not shown, in the same manner, the first to sixteenth processing units of the third to seventh PIM devices generate the 33rd to 112th elements Z1(1)_33 to Z1(1)_112 of the first multi-head attention value vector Z1(1) of the first sequence n_seq=1.

[0245] The first processing unit PU0 of the eighth PIM device PIM7 performs a second matrix-vector multiplication on the 113th element V1(1)_113 of the first value vector V1(1) and the first attention distribution vector S1(1) to generate the 113th element Z1(1)_113 of the first multi-head attention value vector Z1(1) of the first sequence n_seq=1. The second processing unit PU1 of the eighth PIM device PIM7 performs a second matrix-vector multiplication on the 114th element V1(1)_114 of the first value vector V1(1) and the first attention distribution vector S1(1) to generate the 114th element Z1(1)_114 of the first multi-head attention value vector Z1(1) of the first sequence n_seq=1. In the same manner, the sixteenth processing unit PU15 of the eighth PIM device PIM7 performs a second matrix-vector multiplication on the 128th element V1(1)_128 of the first value vector V1(1) and the first attention distribution vector S1(1) to generate the 128th element Z1(1)_128 of the first multi-head attention value vector Z1(1) of the first sequence n_seq=1. In this manner, the first to sixteenth processing units PU0 to PU15 of the eighth PIM device PIM7 generate the 113th to 128th elements Z1(1)_113 to Z1(1)_128 of the first multi-head attention value vector Z1(1) of the first sequence n_seq=1. The second matrix-vector multiplication for generating the first to 128th elements Z1(1)_1 to Z1(1)_128 of the first multi-head attention value vector Z1(1) of the first sequence n_seq=1 is performed in parallel on the first to eighth PIM devices PIM0 to PIM7.

[0246] Figure 33 is a diagram illustrating how the second value vectors generated in the first sequence of multi-head attention are distributed and stored in the second group of PIM devices according to an embodiment of the present disclosure. Figure 34 is a diagram illustrating a process according to an embodiment of the present disclosure, in which a second value vector generated in a first sequence of multi-head attention is distributed element-wise and stored in a memory bank of a second group of PIM devices, and in which a second matrix-vector multiplication is performed on the second head.

[0247] First refer to Figure 33 , the second value vector V2(1) generated in the first sequence n_seq=1 is the same as Figure 31 and Figure 3215. The second value vector V2(1) is written in a similar manner to the first value vector V1(1) shown in FIG. 15, except that the second value vector V2(1) is written to the second group of PIM devices, for example, the ninth to sixteenth PIM devices PIM8 to PIM15. Therefore, the second value vector V2(1) generated in the first sequence n_seq=1 is distributed element by element and written to the memory bank BK of each PIM device in the second group of PIM devices, that is, the ninth to sixteenth PIM devices PIM8 to PIM15. The first to 128th elements V2(1)_1 to V2(1)_128 of the second value vector V2(1) are divided, and each element is written to one memory bank BK. That is, one element of the first to 128th elements V2(1)_1 to V2(1)_128 of the second value vector V2(1) generated in the first sequence n_seq=1 is written to each of the first to sixteenth memory banks BK0 to BK15 of each of the ninth to sixteenth PIM devices PIM8 to PIM15.

[0248] like Figure 33 As shown, the first to sixteenth elements V2(1)_1 to V2(1)_16 of the second value vector V2(1) generated in the first sequence n_seq=1 are respectively distributed and stored in the first to sixteenth memory banks BK0 to BK15 of the ninth PIM device PIM8. The 17th to 32nd elements V2(1)_17 to V2(1)_32 of the second value vector V2(1) generated in the first sequence n_seq=1 are respectively distributed and stored in the first to sixteenth memory banks BK0 to BK15 of the tenth PIM device PIM9. In the same manner, the 33rd to 112th elements of the second value vector V2(1) generated in the first sequence n_seq=1 are respectively distributed and stored in units of 16 elements in the eleventh to fifteenth PIM devices. The 113th to 128th elements V2(1)_113 to V2(1)_128 of the second value vector V2(1) generated in the first sequence n_seq=1 are respectively distributed and stored in the first to sixteenth memory banks BK0 to BK15 of the sixteenth PIM device PIM15.

[0249] See Figure 34, the first element V2(1)_1 of the second value vector V2(1) is written to the first memory bank BK0 of the ninth PIM device PIM8. The first element V2(1)_1 of the second value vector V2(1) is stored in the first row, first to sixteenth columns of the first memory bank BK0 of the ninth PIM device PIM8. The second element V2(1)_2 of the second value vector V2(1) is stored in the first row, first to sixteenth columns of the second memory bank BK1 of the ninth PIM device PIM8. Although not shown, in the same manner, the third to fifteenth elements of the second value vector V2(1) are stored in the first row, first to sixteenth columns of the third to fifteenth memory banks of the ninth PIM device PIM8, respectively. The sixteenth element V2(1)_16 of the second value vector V2(1) is stored in the first row, first to sixteenth columns of the sixteenth memory bank BK15 of the ninth PIM device PIM8.

[0250] The 17th element V2(1)_17 of the second value vector V2(1) is stored in the first row, first to sixteenth columns of the first memory bank BK0 of the tenth PIM device PIM9. The 18th element V2(1)_18 of the second value vector V2(1) is stored in the first row, first to sixteenth columns of the second memory bank BK1 of the tenth PIM device PIM9. Although not shown, in the same manner, the 19th to 31st elements of the second value vector V2(1) are stored in the first row, first to sixteenth columns of the third to fifteenth memory banks of the tenth PIM device PIM9, respectively. The 32nd element V2(1)_32 of the second value vector V2(1) is stored in the first row, first to sixteenth columns of the sixteenth memory bank BK15 of the tenth PIM device PIM9.

[0251] Although not shown, the 33rd to 48th elements of the second value vector V2(1) are stored in the first row, first to sixteenth columns of the first to sixteenth memory banks of the eleventh PIM device PIM10, respectively. The 49th to 64th elements of the second value vector V2(1) are stored in the first row, first to sixteenth columns of the first to sixteenth memory banks of the twelfth PIM device PIM11, respectively. The 65th to 80th elements of the second value vector V2(1) are stored in the first row, first to sixteenth columns of the first to sixteenth memory banks of the thirteenth PIM device PIM12, respectively. The 81st to 96th elements of the second value vector V2(1) are stored in the first row, first to sixteenth columns of the first to sixteenth memory banks of the fourteenth PIM device PIM13, respectively. In addition, the 97th to 112th elements of the second value vector V2(1) are stored in the first row, first to sixteenth columns of the first to sixteenth memory banks of the fifteenth PIM device PIM14, respectively.

[0252] The 113th element V2(1)_113 of the second value vector V2(1) is stored in the first row, first to sixteenth columns of the first memory bank BK0 of the sixteenth PIM device PIM15. The 114th element V2(1)_114 of the second value vector V2(1) is stored in the first row, first to sixteenth columns of the second memory bank BK1 of the sixteenth PIM device PIM15. Although not shown, in the same manner, the 115th to 127th elements of the second value vector V2(1) are stored in the first row, first to sixteenth columns of the third to fifteenth memory banks of the sixteenth PIM device PIM15, respectively. The 128th element V2(1)_128 of the second value vector V2(1) is stored in the first row, first to sixteenth columns of the sixteenth memory bank BK15 of the sixteenth PIM device PIM15, respectively.

[0253] In the same manner as described above, when the second value vector V2(1) generated in the first sequence _seq=1 is written to the second group of PIM devices, namely the ninth to sixteenth PIM devices PIM8 to PIM15, the second attention distribution vector S2(1) generated in the first sequence n_seq=1 is stored in each of the global buffers GB of the ninth to sixteenth PIM devices PIM8 to PIM15. The second attention distribution vector S2(1) generated in the first sequence n_seq=1 has the same size as the second attention score vector A2(1), which is the result of the first matrix-vector multiplication of the second key vector K2(1) and the second query vector Q2(1) generated in the first sequence n_seq=1, as described above with reference to Figure 18 Therefore, the second attention distribution vector S2(1) generated in the first sequence n_seq=1 has a size of n_seq×1, that is, a size of 1×1.

[0254] The first to sixteenth processing units PU0 to PU15 of each of the ninth to sixteenth PIM devices PIM8 to PIM15 perform a second matrix-vector multiplication on the first to 128th elements V2(1)_1 to V2(1)_128 of the second value vector V2(1) and the second attention distribution vector S2(1) to generate a second multi-head attention value vector Z2(1). Since the second value vector V2(1) generated in the first sequence n_seq=1 has a size of d_head×1, that is, a size of 128×1, and the second attention distribution vector S2(1) generated in the first sequence n_seq=1 has a size of 1×1, the second multi-head attention value vector Z2(1) generated by the second matrix-vector multiplication of the second head HEAD_2 of the first sequence n_seq=1 has a size of d_head×1, that is, a size of 128×1. That is, when the second matrix-vector multiplication of the second head HEAD_2 for the first sequence n_seq=1 is performed in the first to sixteenth processing units PU0 to PU15 of each of the ninth to sixteenth PIM devices PIM8 to PIM15, the first to 128th elements Z2(1)_1 to Z2(1)_128 of the second multi-head attention value vector Z2(1) are generated.

[0255] More specifically, the first processing unit PU0 of the ninth PIM device PIM8 receives the first element V2(1)_1 of the second value vector V2(1) and the second attention distribution vector S2(1) from the first storage body BK0 and the global buffer GB, respectively. The first processing unit PU0 of the ninth PIM device PIM8 performs a second matrix-vector multiplication on the first element V2(1)_1 of the second value vector V2(1) and the second attention distribution vector S2(1) to generate the first element Z2(1)_1 of the second multi-head attention value vector Z2(1) of the first sequence n_seq=1. Similarly, the second processing unit PU1 of the ninth PIM device PIM8 performs a second matrix-vector multiplication on the second element V2(1)_2 of the second value vector V2(1) and the second attention distribution vector S2(1) to generate the second element Z2(1)_2 of the second multi-head attention value vector Z2(1) of the first sequence n_seq=1. In the same manner, the sixteenth processing unit PU15 of the ninth PIM device PIM8 performs a second matrix-vector multiplication on the sixteenth element V2(1)_16 of the second value vector V2(1) and the second attention distribution vector S2(1) to generate the sixteenth element Z2(1)_16 of the second multi-head attention value vector Z2(1) of the first sequence n_seq=1. In this manner, the first to sixteenth processing units PU0 to PU15 of the ninth PIM device PIM8 generate the first to sixteenth elements Z2(1)_1 to Z2(1)_16 of the second multi-head attention value vector Z2(1) of the first sequence n_seq=1.

[0256] The first processing unit PU0 of the tenth PIM device PIM9 performs a second matrix-vector multiplication on the 17th element V2(1)_17 of the second value vector V2(1) and the second attention distribution vector S2(1) to generate the 17th element Z2(1)_17 of the second multi-head attention value vector Z2(1) of the first sequence n_seq=1. The second processing unit PU1 of the tenth PIM device PIM9 performs a second matrix-vector multiplication on the 18th element V2(1)_18 of the second value vector V2(1) and the second attention distribution vector S2(1) to generate the 18th element Z2(1)_18 of the second multi-head attention value vector Z2(1) of the first sequence n_seq=1. In the same manner, the sixteenth processing unit PU15 of the tenth PIM device PIM9 performs a second matrix-vector multiplication on the 32nd element V2(1)_32 of the second value vector V2(1) and the second attention distribution vector S2(1) to generate the 32nd element Z2(1)_32 of the second multi-head attention value vector Z2(1) of the first sequence n_seq=1. In this manner, the first to sixteenth processing units PU0 to PU15 of the tenth PIM device PIM9 generate the 17th to 32nd elements Z2(1)_17 to Z2(1)_32 of the second multi-head attention value vector Z2(1) of the first sequence n_seq=1. Although not shown, the first to sixteenth processing units of the eleventh to fifteenth PIM devices generate the 33rd to 112th elements Z2(1)_33 to Z2(1)_112 of the second multi-head attention value vector Z2(1) of the first sequence n_seq=1.

[0257] The first processing unit PU0 of the sixteenth PIM device PIM15 performs a second matrix-vector multiplication on the 113th element V2(1)_113 of the second value vector V2(1) and the second attention distribution vector S2(1) to generate the 113th element Z2(1)_113 of the second multi-head attention value vector Z2(1) of the first sequence n_seq=1. The second processing unit PU1 of the sixteenth PIM device PIM15 performs a second matrix-vector multiplication on the 114th element V2(1)_114 of the second value vector V2(1) and the second attention distribution vector S2(1) to generate the 114th element Z2(1)_114 of the second multi-head attention value vector Z2(1) of the first sequence n_seq=1. In the same manner, the sixteenth processing unit PU15 of the sixteenth PIM device PIM15 performs a second matrix-vector multiplication on the 128th element V2(1)_128 of the second value vector V2(1) and the second attention distribution vector S2(1) to generate the 128th element Z2(1)_128 of the second multi-head attention value vector Z2(1) of the first sequence n_seq=1. In this manner, the first to sixteenth processing units PU0 to PU15 of the sixteenth PIM device PIM15 generate the 113th to 128th elements Z2(1)_113 to Z2(1)_128 of the second multi-head attention value vector Z2(1) of the first sequence n_seq=1.

[0258] The first to 128th elements V2(1)_1 to V2(1)_128 of the second value vector V2(1) of the first sequence n_seq=1 can be specified by the same row address and the same column address in the first to sixteenth memory banks BK0 to BK15 of each of the ninth to sixteenth PIM devices PIM8 to PIM15. Therefore, the second matrix-vector multiplication for generating the first to 128th elements Z2(1)_1 to Z2(1)_128 of the second multi-head attention value vector Z2(1) of the first sequence n_seq=1 is performed in parallel in the ninth to sixteenth PIM devices PIM8 to PIM15. In addition, the first to 128th elements V1(1)_1 to V1(1)_128 of the first value vector V1(1) of the first sequence n_seq=1 and the first to 128th elements V2(1)_1 to V2(1)_128 of the second value vector V2(1) can be respectively specified by the same row address and the same column address in the first to sixteenth memory banks BK0 to BK15 of each of the first to eighth PIM devices PIM0 to PIM7 and the first to sixteenth memory banks BK0 to BK15 of each of the ninth to sixteenth PIM devices PIM8 to PIM15. Therefore, the second matrix-vector multiplication of the first to 128th elements Z1(1)_1 to Z1(1)_128 of the first multi-head attention value vector Z1(1) of the first sequence n_seq=1 generated in the first to eighth PIM devices PIM0 to PIM7 and the second matrix-vector multiplication of the first to 128th elements Z2(1)_1 to Z2(1)_128 of the second multi-head attention value vector Z2(1) of the first sequence n_seq=1 generated in the ninth to sixteenth PIM devices PIM8 to PIM15 can also be executed in parallel.

[0259] Figure 35 is a diagram illustrating how the third value vector generated by the first sequence of multi-head attention is distributed and stored in the first group of PIM devices according to an embodiment of the present disclosure. Figure 36 is a diagram illustrating a process according to an embodiment of the present disclosure, in which a third value vector generated in a first sequence of multi-head attention is distributed element-wise and stored in a memory bank of a first group of PIM devices, and in which a second matrix-vector multiplication is performed on a third head.

[0260] First refer to Figure 35 , the third value vector V3(1) generated in the first sequence n_seq=1 is the same as Figure 31 and Figure 32The first value vector V1(1) of the first sequence n_seq=1 shown in FIG is stored in a similar manner, except that the third value vector V3(1) is stored in a row of the memory bank that is different from the first row used to store the first value vector V1(1). That is, Figure 35 As shown, the first to sixteenth elements V3(1)_1 to V3(1)_16 of the third value vector V3(1) are distributed and stored in the first to sixteenth memory banks BK0 to BK15 of the first PIM device PIM0, respectively. The seventeenth to thirty-second elements V3(1)_17 to V3(1)_32 of the third value vector V3(1) are distributed and stored in the first to sixteenth memory banks BK0 to BK15 of the second PIM device PIM1, respectively. The 113th to 128th elements V3(1)_113 to V3(1)_128 of the third value vector V3(1) are distributed and stored in the first to sixteenth memory banks BK0 to BK15 of the eighth PIM device PIM7, respectively. Although not shown, in the same manner, the 33rd to 112th elements of the third value vector V3(1) are distributed in units of 16 elements and stored in the first to sixteenth memory banks BK0 to BK15 of the third to seventh PIM devices, respectively.

[0261] Specific reference Figure 36 , the first element V3(1)_1 of the third value vector V3(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the second row of the first memory bank BK0 of the first PIM device PIM0. Here, the second row may be a row having a row address subsequent to the row address of the first row. The second element V3(1)_2 of the third value vector V3(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the second row of the second memory bank BK1 of the first PIM device PIM0. Although not shown, in the same manner, the third to fifteenth elements of the third value vector V3(1) of the first sequence n_seq=1 are respectively stored in the first to sixteenth columns of the second row of the third to fifteenth memory banks of the first PIM device PIM0. The sixteenth element V3(1)_16 of the third value vector V3(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the second row of the sixteenth memory bank BK15 of the first PIM device PIM0.

[0262] The 17th element V3(1)_17 of the third value vector V3(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the second row of the first memory bank BK0 of the second PIM device PIM1. The 18th element V3(1)_18 of the third value vector V3(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the second row of the second memory bank BK1 of the second PIM device PIM1. Although not shown, in the same manner, the 19th to 31st elements of the third value vector V3(1) of the first sequence n_seq=1 are stored in the first to sixteenth columns of the second row of the third to fifteenth memory banks of the second PIM device PIM1, respectively. The 32nd element V3(1)_32 of the third value vector V3(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the second row of the sixteenth memory bank BK15 of the second PIM device PIM1.

[0263] Although not shown, the 33rd to 48th elements of the third value vector V3(1) of the first sequence n_seq=1 are stored in the second row, first to sixteenth columns of the first to sixteenth memory banks of the third PIM device PIM2, respectively. The 49th to 64th elements of the third value vector V3(1) of the first sequence n_seq=1 are stored in the second row, first to sixteenth columns of the first to sixteenth memory banks of the fourth PIM device PIM3, respectively. The 65th to 80th elements of the third value vector V3(1) of the first sequence n_seq=1 are stored in the second row, first to sixteenth columns of the first to sixteenth memory banks of the fifth PIM device PIM4, respectively. The 81st to 96th elements of the third value vector V3(1) of the first sequence n_seq=1 are stored in the second row, first to sixteenth columns of the first to sixteenth memory banks of the sixth PIM device PIM5, respectively. Furthermore, the 97th to 112th elements of the third value vector V3(1) of the first sequence n_seq=1 are stored in the second row and first to sixteenth columns of the first to sixteenth memory banks of the seventh PIM device PIM6, respectively.

[0264] The 113th element V3(1)_113 of the third value vector V3(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the second row of the first memory bank BK0 of the eighth PIM device PIM7. The 114th element V3(1)_114 of the third value vector V3(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the second row of the second memory bank BK1 of the eighth PIM device PIM7. Although not shown, in the same manner, the 115th to 127th elements of the third value vector V3(1) of the first sequence n_seq=1 are stored in the first to sixteenth columns of the second row of the third to fifteenth memory banks of the eighth PIM device PIM7, respectively. The 128th element V3(1)_128 of the third value vector V3(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the second row of the sixteenth memory bank BK15 of the eighth PIM device (PIM7).

[0265] In the same manner as described above, when the third value vector V3(1) generated in the first sequence n_seq=1 is written to the first group of PIM devices, i.e., the first to eighth PIM devices PIM0 to PIM7, the third attention distribution vector S3(1) generated in the first sequence n_seq=1 is stored in each of the global buffers GB of the first to eighth PIM devices PIM0 to PIM7. The third attention distribution vector S3(1) has the same size as the third attention score vector, which is the result of the first matrix-vector multiplication of the third key vector K3(1) and the third query vector Q3(1) generated in the first sequence n_seq=1. Therefore, the third attention distribution vector S3(1) generated in the first sequence n_seq=1 has a size of n_seq×1, i.e., a size of 1×1.

[0266] The first to sixteenth processing units PU0 to PU15 of the first to eighth PIM devices PIM0 to PIM7 perform a second matrix-vector multiplication on the first to 128th elements V3(1)_1 to V3(1)_128 of the third value vector V3(1) and the third attention distribution vector S3(1) to generate a third multi-head attention value vector Z3(1) of the first sequence n_seq=1. Similar to the first multi-head attention value vector Z1(1) and the second multi-head attention value vector Z2(1) of the first sequence n_seq=1, the third multi-head attention value vector Z3(1) of the first sequence n_seq=1 also has a size of d_head×1, that is, a size of 128×1. That is, when the second matrix-vector multiplication of the third head HEAD_3 of the first sequence n_seq=1 is performed in the first to sixteenth processing units PU0 to PU15 of each of the first to eighth PIM devices PIM0 to PIM7, the first to 128th elements Z3(1)_1 to Z3(1)_128 of the third multi-head attention value vector Z3(1) of the first sequence n_seq=1 are generated.

[0267] More specifically, the first processing unit PU0 of the first PIM device PIM0 performs a second matrix-vector multiplication on the first element V3(1)_1 of the third value vector V3(1) and the third attention distribution vector S3(1) to generate the first element Z3(1)_1 of the third multi-head attention value vector Z3(1) of the first sequence n_seq=1. The second processing unit PU1 of the first PIM device PIM0 performs a second matrix-vector multiplication on the second element V3(1)_2 of the third value vector V3(1) and the third attention distribution vector S3(1) to generate the second element Z3(1)_2 of the third multi-head attention value vector Z3(1) of the first sequence n_seq=1. Similarly, the sixteenth processing unit PU15 of the first PIM device PIM0 performs a second matrix-vector multiplication on the sixteenth element V3(1)_16 of the third value vector V3(1) and the third attention distribution vector S3(1) to generate the sixteenth element Z3(1)_16 of the third multi-head attention value vector Z3(1) of the first sequence n_seq=1. In this way, the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 generate the first to sixteenth elements Z3(1)_1 to Z3(1)_16 of the third multi-head attention value vector Z3(1) of the first sequence n_seq=1.

[0268] The first processing unit PU0 of the second PIM device PIM1 performs a second matrix-vector multiplication on the 17th element V3(1)_17 of the third value vector V3(1) and the third attention distribution vector S3(1) to generate the 17th element Z3(1)_17 of the third multi-head attention value vector Z3(1) of the first sequence n_seq=1. The second processing unit PU1 of the second PIM device PIM1 performs a second matrix-vector multiplication on the 18th element V3(1)_18 of the third value vector V3(1) and the third attention distribution vector S3(1) to generate the 18th element Z3(1)_18 of the third multi-head attention value vector Z3(1) of the first sequence n_seq=1. In the same manner, the sixteenth processing unit PU15 of the second PIM device PIM1 performs a second matrix-vector multiplication on the 32nd element V3(1)_32 of the third value vector V3(1) and the third attention distribution vector S3(1) to generate the 32nd element Z3(1)_32 of the third multi-head attention value vector Z3(1) of the first sequence n_seq=1. In this manner, the first to sixteenth processing units PU0 to PU15 of the second PIM device PIM1 generate the 17th to 32nd elements Z3(1)_17 to Z3(1)_32 of the third multi-head attention value vector Z3(1) of the first sequence n_seq=1. Although not shown, in the same manner, the first to sixteenth processing units of the third to seventh PIM devices generate the 33rd to 112th elements Z3(1)_33 to Z3(1)_112 of the third multi-head attention value vector Z3(1) of the first sequence n_seq=1.

[0269] The first processing unit PU0 of the eighth PIM device PIM7 performs a second matrix-vector multiplication on the 113th element V3(1)_113 of the third value vector V3(1) and the third attention distribution vector S3(1) to generate the 113th element Z3(1)_113 of the third multi-head attention value vector Z3(1) of the first sequence n_seq=1. The second processing unit PU1 of the eighth PIM device PIM7 performs a second matrix-vector multiplication on the 114th element V3(1)_114 of the third value vector V3(1) and the third attention distribution vector S3(1) to generate the 114th element Z3(1)_114 of the third multi-head attention value vector Z3(1) of the first sequence n_seq=1. In the same manner, the sixteenth processing unit PU15 of the eighth PIM device PIM7 performs a second matrix-vector multiplication on the 128th element V3(1)_128 of the third value vector V3(1) and the third attention distribution vector S3(1) to generate the 128th element Z3(1)_128 of the third multi-head attention value vector Z3(1) of the first sequence n_seq=1. In this manner, the first to sixteenth processing units PU0 to PU15 of the eighth PIM device PIM7 generate the 113th to 128th elements Z3(1)_113 to Z3(1)_128 of the third multi-head attention value vector Z3(1) of the first sequence n_seq=1. The second matrix-vector multiplication for generating the first to 128th elements Z3(1)_1 to Z3(1)_128 of the third multi-head attention value vector Z3(1) of the first sequence n_seq=1 is performed in parallel in the first to eighth PIM devices PIM0 to PIM7.

[0270] Figure 37 is a diagram illustrating how the fourth value vector generated in the first sequence of multi-head attention is distributed and stored in the second group of PIM devices according to an embodiment of the present disclosure. Figure 38 is a diagram illustrating a process according to an embodiment of the present disclosure, in which a fourth value vector generated in a first sequence of multi-head attention is distributed element-wise and stored in a memory bank of a second group of PIM devices, and in which a second matrix-vector multiplication is performed on the fourth head.

[0271] First refer to Figure 37 , the fourth value vector V4(1) generated in the first sequence n_seq=1 is the same as Figure 35 and Figure 36 The third value vector V3(1) generated in the first sequence n_seq=1 is stored in a similar manner, except that the fourth value vector V4(1) is stored in the ninth to sixteenth PIM devices PIM8 to PIM15. That is, Figure 37As shown, the first to sixteenth elements V4(1)_1 to V4(1)_16 of the fourth value vector V4(1) are distributed and stored in the first to sixteenth memory banks BK0 to BK15 of the ninth PIM device PIM8, respectively. The seventeenth to thirty-second elements V4(1)_17 to V4(1)_32 of the fourth value vector V4(1) are distributed and stored in the first to sixteenth memory banks BK0 to BK15 of the tenth PIM device PIM9, respectively. The 113th to 128th elements V4(1)_113 to V4(1)_128 of the fourth value vector V4(1) are distributed and stored in the first to sixteenth memory banks BK0 to BK15 of the sixteenth PIM device PIM15, respectively. Although not shown, in the same manner, the 33rd to 112th elements of the fourth value vector V4(1) are distributed and stored in units of 16 elements in the eleventh to fifteenth PIM devices, respectively.

[0272] Specific reference Figure 38 , the first element V4(1)_1 of the fourth value vector V4(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the second row of the first memory bank BK0 of the ninth PIM device PIM8. The second element V4(1)_2 of the fourth value vector V4(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the second row of the second memory bank BK1 of the ninth PIM device PIM8. Although not shown, in the same manner, the third to fifteenth elements of the fourth value vector V4(1) of the first sequence n_seq=1 are stored in the first to sixteenth columns of the second row of the third to fifteenth memory banks of the ninth PIM device PIM8, respectively. The sixteenth element V4(1)_16 of the fourth value vector V4(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the second row of the sixteenth memory bank BK15 of the ninth PIM device PIM8.

[0273] The 17th element V4(1)_17 of the fourth value vector V4(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the second row of the first memory bank BK0 of the tenth PIM device PIM9. The 18th element V4(1)_18 of the fourth value vector V4(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the second row of the second memory bank BK1 of the tenth PIM device PIM9. Although not shown, in the same manner, the 19th to 31st elements of the fourth value vector V4(1) of the first sequence n_seq=1 are stored in the first to sixteenth columns of the second row of the third to fifteenth memory banks of the tenth PIM device PIM9, respectively. The 32nd element V4(1)_32 of the fourth value vector V4(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the second row of the sixteenth memory bank BK15 of the tenth PIM device PIM9.

[0274] Although not shown, the 33rd to 48th elements of the fourth value vector V4(1) of the first sequence n_seq=1 are stored in the second row, first to sixteenth columns of the first to sixteenth memory banks of the eleventh PIM device PIM10, respectively. The 49th to 64th elements of the fourth value vector V4(1) of the first sequence n_seq=1 are stored in the second row, first to sixteenth columns of the first to sixteenth memory banks of the twelfth PIM device PIM11, respectively. The 65th to 80th elements of the fourth value vector V4(1) of the first sequence n_seq=1 are stored in the second row, first to sixteenth columns of the first to sixteenth memory banks of the thirteenth PIM device PIM12, respectively. The 81st to 96th elements of the fourth value vector V4(1) of the first sequence n_seq=1 are stored in the second row, first to sixteenth columns of the first to sixteenth memory banks of the fourteenth PIM device PIM13, respectively. Furthermore, the 97th to 112th elements of the fourth value vector V4(1) of the first sequence n_seq=1 are stored in the second row, first to sixteenth columns of the first to sixteenth memory banks of the fifteenth PIM device PIM14, respectively.

[0275] The 113th element V4(1)_113 of the fourth value vector V4(1) of the first sequence n_seq=1 is stored in the second row, first to sixteenth columns of the first memory bank BK0 of the sixteenth PIM device PIM15. The 114th element V4(1)_114 of the fourth value vector V4(1) of the first sequence n_seq=1 is stored in the second row, first to sixteenth columns of the second memory bank BK1 of the sixteenth PIM device PIM15. Although not shown, in the same manner, the 115th to 127th elements of the fourth value vector V4(1) of the first sequence n_seq=1 are stored in the second row, first to sixteenth columns of the third to fifteenth memory banks of the sixteenth PIM device PIM15, respectively. The 128th element V4(1)_128 of the fourth value vector V4(1) of the first sequence n_seq=1 is stored in the second row, first to sixteenth columns of the sixteenth memory bank BK15 of the sixteenth PIM device PIM15.

[0276] In the same manner as described above, when the fourth value vector V4(1) generated in the first sequence n_seq=1 is written to the second group of PIM devices, i.e., the ninth to sixteenth PIM devices PIM8 to PIM15, the fourth attention distribution vector S4(1) generated in the first sequence n_seq=1 is stored in each of the global buffers GB of the ninth to sixteenth PIM devices PIM8 to PIM15. The fourth attention distribution vector S4(1) has the same size as the fourth attention score vector, which is the result of the first matrix-vector multiplication of the fourth key vector K4(1) and the fourth query vector Q4(1) generated in the first sequence n_seq=1. Therefore, the fourth attention distribution vector S4(1) generated in the first sequence n_seq=1 has a size of n_seq×1, i.e., a size of 1×1.

[0277] The first to sixteenth processing units PU0 to PU15 of the ninth to sixteenth PIM devices PIM8 to PIM15 perform a second matrix-vector multiplication on the first to 128th elements V4(1)_1 to V4(1)_128 of the fourth value vector V4(1) and the fourth attention distribution vector S4(1) to generate a fourth multi-head attention value vector Z4(1) of the first sequence n_seq = 1. Similar to the first to third multi-head attention value vectors Z1(1) to Z3(1) of the first sequence n_seq = 1, the fourth multi-head attention value vector Z4(1) of the first sequence n_seq = 1 also has a size of d_head×1, that is, a size of 128×1. That is, when the second matrix-vector multiplication of the fourth head HEAD_4 of the first sequence n_seq=1 is performed in the first to sixteenth processing units PU0 to PU15 of each of the ninth to sixteenth PIM devices PIM8 to PIM15, the first to 128th elements Z4(1)_1 to Z4(1)_128 of the fourth multi-head attention value vector Z4(1) of the first sequence n_seq=1 are generated.

[0278] More specifically, the first to sixteenth processing units PU0 to PU15 of the ninth PIM device PIM8 perform a second matrix-vector multiplication on the first to sixteenth elements V4(1)_1 to V4(1)_16 of the fourth value vector V4(1) and the fourth attention distribution vector S4(1) to generate the first to sixteenth elements Z4(1)_1 to Z4(1)_16 of the fourth multi-head attention value vector Z4(1) of the first sequence n_seq=1. The first to sixteenth processing units PU0 to PU15 of the tenth PIM device PIM9 perform a second matrix-vector multiplication on the seventeenth to thirty-second elements V4(1)_17 to V4(1)_32 of the fourth value vector V4(1) and the fourth attention distribution vector S4(1) to generate the seventeenth to thirty-second elements Z4(1)_17 to Z4(1)_32 of the fourth multi-head attention value vector Z4(1) of the first sequence n_seq=1. The first to sixteenth processing units PU0 to PU15 of the sixteenth PIM device PIM15 perform a second matrix-vector multiplication on the 113th to 128th elements V4(1)_113 to V4(1)_128 of the fourth value vector V4(1) and the fourth attention distribution vector S4(1) to generate the 113th to 128th elements Z4(1)_113 to Z4(1)_128 of the fourth multi-head attention value vector Z4(1) of the first sequence n_seq=1. Although not shown, the first to sixteenth processing units of the eleventh to fifteenth PIM devices perform a second matrix-vector multiplication on the 33rd to 112th elements V4(1) of the fourth value vector V4(1) and the fourth attention distribution vector S4(1) to generate the 33rd to 112th elements Z4(1)_33 to Z4(1)_112 of the fourth multi-head attention value vector Z4(1) of the first sequence n_seq=1. The second matrix-vector multiplications for generating the first to 128th elements Z4(1)_1 to Z4(1)_128 of the fourth multi-head attention value vector Z4(1) of the first sequence n_seq=1 are performed in parallel in the ninth to sixteenth PIM devices PIM8 to PIM15.

[0279] The second matrix-vector multiplication of the first to 128th elements Z3(1)_1 to Z3(1)_128 of the third multi-head attention value vector Z3(1) for generating the first sequence n_seq=1 in the first to eighth PIM devices PIM0 to PIM7, and the second matrix-vector multiplication of the first to 128th elements Z4(1)_1 to Z4(1)_128 of the fourth multi-head attention value vector Z4(1) for generating the first sequence n_seq=1 in the ninth to sixteenth PIM devices PIM8 to PIM15 can be performed in parallel.

[0280] Figure 39is a diagram illustrating a process according to an embodiment of the present disclosure, in which a fifth value vector generated in a first sequence of multi-head attention is distributed element-wise and stored in a memory bank of a first group of PIM devices, and a second matrix-vector multiplication is performed on the fifth head in the process.

[0281] Reference Figure 39 , the first to 128th elements V5(1)_1 to V5(1)_128 of the fifth value vector V5(1) generated in the first sequence n_seq=1 are stored in units of 16 elements in the first to eighth PIM devices PIM0 to PIM7. The first element V5(1)_1 of the fifth value vector V5(1) is stored in the first to sixteenth columns of the third row of the first memory bank BK0 of the first PIM device PIM0. Here, the third row may be a row having a row address subsequent to the row address of the second row. The second element V5(1)_2 of the fifth value vector V5(1) is stored in the first to sixteenth columns of the third row of the second memory bank BK1 of the first PIM device PIM0. Although not shown, in the same manner, the third to fifteenth elements of the fifth value vector V5(1) are respectively stored in the first to sixteenth columns of the third row of each of the third to fifteenth memory banks of the first PIM device PIM0. The sixteenth element V5(1)_16 of the fifth value vector V5(1) is stored in the third row, first to sixteenth columns of the sixteenth memory bank BK15 of the first PIM device PIM0.

[0282] The 17th element V5(1)_17 of the fifth value vector V5(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the third row of the first memory bank BK0 of the second PIM device PIM1. The 18th element V5(1)_18 of the fifth value vector V5(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the third row of the second memory bank BK1 of the second PIM device PIM1. The 32nd element V5(1)_32 of the fifth value vector V5(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the third row of the sixteenth memory bank BK15 of the second PIM device PIM1. Although not shown, in the same manner, the 19th to 31st elements of the fifth value vector V5(1) of the first sequence n_seq=1 are respectively stored in the first to sixteenth columns of the third row of each of the third to fifteenth memory banks of the second PIM device PIM1.

[0283] Although not shown, the 33rd to 48th elements of the fifth value vector V5(1) of the first sequence n_seq=1 are stored in the third row, first to sixteenth columns of each of the first to sixteenth memory banks of the third PIM device PIM2, respectively. The 49th to 64th elements of the fifth value vector V5(1) of the first sequence n_seq=1 are stored in the third row, first to sixteenth columns of each of the first to sixteenth memory banks of the fourth PIM device PIM3, respectively. The 65th to 80th elements of the fifth value vector V5(1) of the first sequence n_seq=1 are stored in the third row, first to sixteenth columns of each of the first to sixteenth memory banks of the fifth PIM device PIM4, respectively. The 81st to 96th elements of the fifth value vector V5(1) of the first sequence n_seq=1 are stored in the third row, first to sixteenth columns of each of the first to sixteenth memory banks of the sixth PIM device PIM5, respectively. Furthermore, the 97th to 112th elements of the fifth value vector V5(1) of the first sequence n_seq=1 are respectively stored in the third row, first to sixteenth columns of each of the first to sixteenth memory banks of the seventh PIM device PIM6.

[0284] The 113th element V5(1)_113 of the fifth value vector V5(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the third row of the first memory bank BK0 of the eighth PIM device PIM7. The 114th element V5(1)_114 of the fifth value vector V5(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the third row of the second memory bank BK1 of the eighth PIM device PIM7. The 128th element V5(1)_128 of the fifth value vector V5(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the third row of the sixteenth memory bank BK15 of the eighth PIM device PIM7. Although not shown, in the same manner, the 115th to 127th elements of the fifth value vector V5(1) of the first sequence n_seq=1 are stored in the first to sixteenth columns of the third row of the third to fifteenth memory banks of the eighth PIM device PIM7, respectively.

[0285] In the same manner as described above, when the fifth value vector V5(1) generated in the first sequence n_seq=1 is written to the first group of PIM devices, i.e., the first to eighth PIM devices PIM0 to PIM7, the fifth attention distribution vector S5(1) generated in the first sequence n_seq=1 is stored in each of the global buffers GB of the first to eighth PIM devices PIM0 to PIM7. The fifth attention distribution vector S5(1) has the same size as the fifth attention score vector, which is the result of the first matrix-vector multiplication of the fifth key vector and the fifth query vector generated in the first sequence n_seq=1. Therefore, the fifth attention distribution vector S5(1) generated in the first sequence n_seq=1 has a size of n_seq×1, i.e., a size of 1×1.

[0286] The first to sixteenth processing units PU0 to PU15 of each of the first to eighth PIM devices PIM0 to PIM7 perform a second matrix-vector multiplication on the first to 128th elements V5(1)_1 to V5(1)_128 of the fifth value vector V5(1) and the fifth attention distribution vector S5(1) to generate a fifth multi-head attention value vector Z5(1) of the first sequence n_seq=1. The fifth multi-head attention value vector Z5(1) of the first sequence n_seq=1 has a size of d_head×1, i.e., a size of 128×1. That is, when the second matrix-vector multiplication for the fifth head HEAD_5 of the first sequence n_seq=1 is performed in the first to sixteenth processing units PU0 to PU15 of each of the first to eighth PIM devices PIM0 to PIM7, the first to 128th elements Z5(1)_1 to Z5(1)_128 of the fifth multi-head attention value vector Z5(1) of the first sequence n_seq=1 are generated.

[0287] More specifically, the first to sixteenth processing units PU0 to PU15 of the first PIM device PIM0 perform the second matrix-vector multiplication on the first to sixteenth elements V5(1)_1 to V5(1)_16 of the fifth value vector V5(1) and the fifth attention distribution vector S5(1) to generate the first to sixteenth elements Z5(1)_1 to Z5(1)_16 of the fifth multi-head attention value vector Z5(1) of the first sequence n_seq=1. The first to sixteenth processing units PU0 to PU15 of the second PIM device PIM1 perform the second matrix-vector multiplication on the seventeenth to thirty-second elements V5(1)_17 to V5(1)_32 of the fifth value vector V5(1) and the fifth attention distribution vector S5(1) to generate the seventeenth to thirty-second elements Z5(1)_17 to Z5(1)_32 of the fifth multi-head attention value vector Z5(1) of the first sequence n_seq=1. The first to sixteenth processing units PU0 to PU15 of the eighth PIM device PIM7 perform second matrix-vector multiplication on the 113th to 128th elements V5(1)_113 to V5(1)_128 of the fifth value vector V5(1) and the fifth attention distribution vector S5(1) to generate the 113th to 128th elements Z5(1)_113 to Z5(1)_128 of the fifth multi-head attention value vector Z5(1) of the first sequence n_seq=1. Although not shown, the first to sixteenth processing units of each of the third to seventh PIM devices perform second matrix-vector multiplication on the 33rd to 112th elements V5(1) of the fifth value vector V5(1) and the fifth attention distribution vector S5(1) to generate the 33rd to 112th elements Z5(1)_33 to Z5(1)_112 of the fifth multi-head attention value vector Z5(1) of the first sequence n_seq=1. The second matrix-vector multiplications for generating the first to 128th elements Z5(1)_1 to Z5(1)_128 of the fifth multi-head attention value vector Z5(1) of the first sequence n_seq=1 are performed in parallel in the first to eighth PIM devices PIM0 to PIM7.

[0288] Figure 40 is a diagram illustrating a process according to an embodiment of the present disclosure, in which a sixth value vector generated in a first sequence of multi-head attention is distributed element-wise and stored in a memory bank of a second group of PIM devices, and in which a second matrix-vector multiplication is performed on the sixth head.

[0289] Reference Figure 40, the first to 128th elements V6(1)_1 to V6(1)_128 of the sixth value vector V6(1) generated in the first sequence n_seq=1 are divided into units of 16 elements and stored in the ninth to sixteenth PIM devices PIM8 to PIM15. The first element V6(1)_1 of the sixth value vector V6(1) of the first sequence n_seq=1 is stored in the third row, first to sixteenth columns of the first memory bank BK0 of the ninth PIM device PIM8. The second element V6(1)_2 of the sixth value vector V6(1) of the first sequence n_seq=1 is stored in the third row, first to sixteenth columns of the second memory bank BK1 of the ninth PIM device PIM8. The sixteenth element V6(1)_16 of the sixth value vector V6(1) of the first sequence n_seq=1 is stored in the third row, first to sixteenth columns of the sixteenth memory bank BK15 of the ninth PIM device PIM8. Although not shown, in the same manner, the third to fifteenth elements of the sixth value vector V6(1) of the first sequence n_seq=1 are respectively stored in the third to fifteenth memory banks of the ninth PIM device PIM8. Each of the third to fifteenth elements of the sixth value vector V6(1) of the first sequence n_seq=1 is stored in the third row, first to sixteenth columns of the corresponding memory bank of the ninth PIM device PIM8.

[0290] The 17th element V6(1)_17 of the sixth value vector V6(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the third row of the first memory bank BK0 of the tenth PIM device PIM9. The 18th element V6(1)_18 of the sixth value vector V6(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the third row of the second memory bank BK1 of the tenth PIM device PIM9. The 32nd element V6(1)_32 of the sixth value vector V6(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the third row of the sixteenth memory bank BK15 of the tenth PIM device PIM9. Although not shown, in the same manner, the 19th to 31st elements of the sixth value vector V6(1) of the first sequence n_seq=1 are stored in the third to fifteenth memory banks of the tenth PIM device PIM9, respectively. Each of the 19th to 31st elements of the sixth value vector V6(1) of the first sequence n_seq=1 is stored in the third row, first to sixteenth columns of the corresponding memory bank of the tenth PIM device PIM9.

[0291] Although not shown, the 33rd to 48th elements of the sixth value vector V6(1) of the first sequence n_seq=1 are stored in the first to sixteenth memory banks of the eleventh PIM device PIM10, respectively. Each of the 33rd to 48th elements of the sixth value vector V6(1) of the first sequence n_seq=1 is stored in the third row, first to sixteenth columns of the corresponding memory bank of the eleventh PIM device PIM10. The 49th to 64th elements of the sixth value vector V6(1) of the first sequence n_seq=1 are stored in the first to sixteenth memory banks of the twelfth PIM device PIM11, respectively. Each of the 49th to 64th elements of the sixth value vector V6(1) of the first sequence n_seq=1 is stored in the third row, first to sixteenth columns of the corresponding memory bank of the twelfth PIM device PIM11. The 65th to 80th elements of the sixth value vector V6(1) of the first sequence n_seq=1 are stored in the first to sixteenth memory banks of the thirteenth PIM device PIM12, respectively. Each of the 65th to 80th elements of the sixth value vector V6(1) of the first sequence n_seq=1 is stored in the third row, first to sixteenth columns of the corresponding memory bank of the thirteenth PIM device PIM12. The 81st to 96th elements of the sixth value vector V6(1) of the first sequence n_seq=1 are stored in the first to sixteenth columns of the corresponding memory bank of the fourteenth PIM device PIM13, respectively. Each of the 81st to 96th elements of the sixth value vector V6(1) of the first sequence n_seq=1 is stored in the third row, first to sixteenth columns of the corresponding memory bank of the fourteenth PIM device PIM13. In addition, the 97th to 112th elements of the sixth value vector V6(1) of the first sequence n_seq=1 are stored in the first to sixteenth columns of the corresponding memory bank of the fifteenth PIM device PIM14, respectively. Each of the 97th to 112th elements of the sixth value vector V6(1) of the first sequence n_seq=1 is stored in the third row, first to sixteenth columns of the corresponding memory bank of the fifteenth PIM device PIM14.

[0292] The 113th element V6(1)_113 of the sixth value vector V6(1) of the first sequence n_seq=1 is stored in the third row, first to sixteenth columns of the first memory bank BK0 of the sixteenth PIM device PIM15. The 114th element V6(1)_114 of the sixth value vector V6(1) of the first sequence n_seq=1 is stored in the third row, first to sixteenth columns of the second memory bank BK1 of the sixteenth PIM device PIM15. The 128th element V6(1)_128 of the sixth value vector V6(1) of the first sequence n_seq=1 is stored in the third row, first to sixteenth columns of the sixteenth memory bank BK15 of the sixteenth PIM device PIM15. Although not shown, in the same manner, the 115th to 127th elements of the sixth value vector V6(1) of the first sequence n_seq=1 are stored in the third to fifteenth memory banks of the sixteenth PIM device PIM15, respectively. Each of the 115th to 127th elements of the sixth value vector V6(1) of the first sequence n_seq=1 is stored in the third row, first to sixteenth columns of the corresponding memory bank of the sixteenth PIM device PIM15.

[0293] In the same manner as described above, when the sixth value vector V6(1) generated in the first sequence n_seq=1 is written to the second group of PIM devices, namely the ninth to sixteenth PIM devices PIM8 to PIM15, the sixth attention distribution vector S6(1) generated in the first sequence n_seq=1 is stored in each of the global buffers GB of the ninth to sixteenth PIM devices PIM8 to PIM15. The sixth attention distribution vector S6(1) has the same size as the sixth attention score vector, which is the result of the first matrix-vector multiplication of the sixth key vector K6(1) and the sixth query vector Q6(1) generated in the first sequence n_seq=1. Therefore, the sixth attention distribution vector S6(1) generated in the first sequence n_seq=1 has a size of n_seq×1, that is, a size of 1×1.

[0294] The first to sixteenth processing units PU0 to PU15 of each of the ninth to sixteenth PIM devices PIM8 to PIM15 perform a second matrix-vector multiplication on the first to 128th elements V6(1)_1 to V6(1)_128 of the sixth value vector V6(1) and the sixth attention distribution vector S6(1) to generate a sixth multi-head attention value vector Z6(1) of the first sequence n_seq=1. The sixth multi-head attention value vector Z6(1) of the first sequence n_seq=1 has a size of d_head×1, that is, a size of 128×1. That is, when the second matrix-vector multiplication of the sixth head HEAD_6 of the first sequence n_seq=1 is performed in the first to sixteenth processing units PU0 to PU15 of each of the ninth to sixteenth PIM devices PIM8 to PIM15, the first to 128th elements Z6(1)_1 to Z6(1)_128 of the sixth multi-head attention value vector Z6(1) of the first sequence n_seq=1 are generated.

[0295] More specifically, the first to sixteenth processing units PU0 to PU15 of the ninth PIM device PIM8 perform a second matrix-vector multiplication on the first to sixteenth elements V6(1)_1 to V6(1)_16 and the sixth attention distribution vector S6(1) to generate the first to sixteenth elements Z6(1)_1 to Z6(1)_16 of the sixth multi-head attention value vector Z6(1) of the first sequence n_seq=1. The first to sixteenth processing units PU0 to PU15 of the tenth PIM device PIM9 perform a second matrix-vector multiplication on the seventeenth to thirty-second elements V6(1)_17 to V6(1)_32 of the sixth value vector V6(1) and the sixth attention distribution vector S6(1) to generate the seventeenth to thirty-second elements Z6(1)_17 to Z6(1)_32 of the sixth multi-head attention value vector Z6(1) of the first sequence n_seq=1. The first to sixteenth processing units PU0 to PU15 of the sixteenth PIM device PIM15 perform a second matrix-vector multiplication on the 113th to 128th elements V6(1)_113 to V6(1)_128 of the sixth value vector V6(1) and the sixth attention distribution vector S6(1) to generate the 113th to 128th elements Z6(1)_113 to Z6(1)_128 of the sixth multi-head attention value vector Z6(1) of the first sequence n_seq=1. Although not shown, the first to sixteenth processing units of each of the eleventh to fifteenth PIM devices perform a second matrix-vector multiplication on the 33rd to 112th elements V6(1) of the sixth value vector V6(1) and the sixth attention distribution vector S6(1) to generate the 33rd to 112th elements Z6(1)_33 to Z6(1)_112 of the sixth multi-head attention value vector Z6(1) of the first sequence n_seq=1. The second matrix-vector multiplications for generating the first to 128th elements Z6(1)_1 to Z6(1)_128 of the sixth multi-head attention value vector Z6(1) of the first sequence n_seq=1 are performed in parallel in the ninth to sixteenth PIM devices PIM8 to PIM15.

[0296] The second matrix-vector multiplications of the first to 128th elements Z5(1)_1 to Z5(1)_128 of the fifth multi-head attention value vector Z5(1) for generating the first sequence n_seq=1 in the first to eighth PIM devices PIM0 to PIM7, and the second matrix-vector multiplications of the first to 128th elements Z6(1)_1 to Z6(1)_128 of the sixth multi-head attention value vector Z6(1) for generating the first sequence n_seq=1 in the ninth to sixteenth PIM devices PIM8 to PIM15 can be performed in parallel.

[0297] Figure 41is a diagram illustrating a process according to an embodiment of the present disclosure, in which a seventh value vector generated in a first sequence of multi-head attention is distributed element-wise and stored in a memory bank of a first group of PIM devices, and in which a second matrix-vector multiplication is performed on the seventh head.

[0298] Reference Figure 41 , the first to 128th elements V7(1)_1 to V7(1)_128 of the seventh value vector V7(1) generated in the first sequence n_seq=1 are divided into units of 16 elements and stored in the first to eighth PIM devices PIM0 to PIM7. The first element V7(1)_1 of the seventh value vector V7(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the fourth row of the first memory bank BK0 of the first PIM device PIM0. Here, the fourth row may be a row having a row address subsequent to the row address of the third row. The second element V7(1)_2 of the seventh value vector V7(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the fourth row of the second memory bank BK1 of the first PIM device PIM0. The sixteenth element V7(1)_16 of the seventh value vector V7(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the fourth row of the sixteenth memory bank BK15 of the first PIM device PIM0. Although not shown, in the same manner, the third to fifteenth elements of the seventh value vector V7(1) of the first sequence n_seq=1 are stored in the third to fifteenth memory banks of the first PIM device PIM0, respectively. Each of the third to fifteenth elements of the seventh value vector V7(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the fourth row of the corresponding memory bank of the first PIM device PIM0.

[0299] The 17th element V7(1)_17 of the seventh value vector V7(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the fourth row of the first memory bank BK0 of the second PIM device PIM1. The 18th element V7(1)_18 of the seventh value vector V7(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the fourth row of the second memory bank BK1 of the second PIM device PIM1. The 32nd element V7(1)_32 of the seventh value vector V7(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the fourth row of the sixteenth memory bank BK15 of the second PIM device PIM1. Although not shown, in the same manner, the 19th to 31st elements of the seventh value vector V7(1) of the first sequence n_seq=1 are stored in the third to fifteenth memory banks of the second PIM device PIM1, respectively. Each of the 19th to 31st elements of the seventh value vector V7(1) of the first sequence n_seq=1 is stored in the fourth row, first to sixteenth columns of the corresponding memory bank of the second PIM device PIM1.

[0300] Although not shown, the 33rd to 48th elements of the seventh value vector V7(1) of the first sequence n_seq=1 are stored in the first to sixteenth memory banks of the third PIM device PIM2, respectively. Each of the 33rd to 48th elements of the seventh value vector V7(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the fourth row of the corresponding memory bank of the third PIM device PIM2. The 49th to 64th elements of the seventh value vector V7(1) of the first sequence n_seq=1 are stored in the first to sixteenth columns of the fourth row of the corresponding memory bank of the fourth PIM device PIM3, respectively. Each of the 49th to 64th elements of the seventh value vector V7(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the fourth row of the corresponding memory bank of the fourth PIM device PIM3. The 65th to 80th elements of the seventh value vector V7(1) of the first sequence n_seq=1 are stored in the first to sixteenth banks of the fifth PIM device PIM4, respectively. Each of the 65th to 80th elements of the seventh value vector V7(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the fourth row of the corresponding memory bank of the fifth PIM device PIM4. The 81st to 96th elements of the seventh value vector V7(1) of the first sequence n_seq=1 are stored in the first to sixteenth columns of the sixth PIM device PIM5, respectively. Each of the 81st to 96th elements of the seventh value vector V7(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the fourth row of the corresponding memory bank of the sixth PIM device PIM5. In addition, the 97th to 112th elements of the seventh value vector V7(1) of the first sequence n_seq=1 are stored in the first to sixteenth columns of the fourth row of the corresponding memory bank of the seventh PIM device PIM6, respectively. Each of the 97th to 112th elements of the seventh value vector V7(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the fourth row of the corresponding memory bank of the seventh PIM device PIM6.

[0301] The 113th element V7(1)_113 of the seventh value vector V7(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the fourth row of the first memory bank BK0 of the eighth PIM device PIM7. The 114th element V7(1)_114 of the seventh value vector V7(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the fourth row of the second memory bank BK1 of the eighth PIM device PIM7. The 128th element V7(1)_128 of the seventh value vector V7(1) of the first sequence n_seq=1 is stored in the first to sixteenth columns of the fourth row of the sixteenth memory bank BK15 of the eighth PIM device PIM7. Although not shown, in the same manner, the 115th to 127th elements of the seventh value vector V7(1) of the first sequence n_seq=1 are stored in the third to fifteenth memory banks of the eighth PIM device PIM7, respectively. Each of the 115th to 127th elements of the seventh value vector V7(1) of the first sequence n_seq=1 is stored in the fourth row, first to sixteenth columns of the corresponding memory bank of the eighth PIM device PIM7.

[0302] In the same manner as described above, when the seventh value vector V7(1) generated in the first sequence n_seq=1 is written to the first group of PIM devices, i.e., the first to eighth PIM devices PIM0 to PIM7, the seventh attention distribution vector S7(1) generated in the first sequence n_seq=1 is stored in each of the global buffers GB of the first to eighth PIM devices PIM0 to PIM7. The seventh attention distribution vector S7(1) has the same size as the seventh attention score vector, which is the result of the first matrix-vector multiplication of the seventh key vector K7(1) and the seventh query vector Q7(1) generated in the first sequence n_seq=1. Therefore, the seventh attention distribution vector S7(1) generated in the first sequence n_seq=1 has a size of n_seq×1, i.e., a size of 1×1.

[0303] The first to sixteenth processing units PU0 to PU15 of each of the first to eighth PIM devices PIM0 to PIM7 perform a second matrix-vector multiplication on the first to 128th elements V7(1)_1 to V7(1)_128 of the seventh value vector V7(1) and the seventh attention distribution vector S7(1) to generate the seventh multi-head attention value vector Z7(1) of the first sequence n_seq=1. The seventh multi-head attention value vector Z7(1) of the first sequence n_seq=1 has a size of d_head×1, that is, a size of 128×1. That is, when the second matrix-vector multiplication is performed on the seventh head HEAD_7 of the first sequence n_seq=1 in the first to sixteenth processing units PU0 to PU15 of each of the first to eighth PIM devices PIM0 to PIM7, the first to 128th elements Z7(1)_1 to Z7(1)_128 of the seventh multi-head attention value vector Z7(1) of t...

Claims

1. A neural network architecture comprising a plurality of PIM devices that perform multi-head attention on a transformer model, wherein the PIM is in-memory processing, in, Each of the plurality of PIM devices comprises: a plurality of memory banks storing key vectors and value vectors used as input data for the multi-head attention; and a plurality of processing units corresponding to the plurality of memory banks, the plurality of processing units performing the multi-head attention via the transformer model using the key vector and the value vector, and The plurality of storage bodies store the key vectors in a first manner, and store the value vectors in a second manner different from the first manner.

2. The neural network architecture according to claim 1, in, The first mode is a set of operations in which the key vector generated in one of the plurality of sequences of the multi-head attention is stored in one of the plurality of memory banks of each of the plurality of PIM devices, and The second manner is a set of operations in which the value vector generated in one of the multiple sequences of the multi-head attention is distributed and stored in the multiple storage banks of each of the multiple PIM devices.

3. The neural network architecture of claim 1, wherein: When the number of the plurality of PIM devices is "P", the number of the plurality of memory banks is "M", and "L" is a natural number from "1" to "P", The "L"th PIM device stores the key vectors generated in the "(L-1)·M+1"th to "L·M"th sequences among the plurality of sequences.

4. The neural network structure according to claim 3, wherein: The key vectors generated in the (L-1)·M+1”-L·M”th sequences are sequentially stored in the first to Mth memory banks of the Lth PIM device in units of the key vectors generated in one of the (L-1)·M+1”-L·M”th sequences.

5. The neural network architecture of claim 4, wherein: The key vectors generated in the "(L-1)·M+1"th to "L·M"th sequence are stored in the same first row in the first to "M"th memory banks of the "L"th PIM device.

6. The neural network architecture according to claim 1, in, The plurality of PIM devices further include a global buffer storing query vectors used as input data for the multi-head attention, When the number of the plurality of PIM devices is "P", the number of the plurality of memory banks is "M", the number of heads in the multi-head attention is "N", and "L" is a natural number from "1" to "P", Wherein, whenever each of the plurality of sequences is executed, 'N' query vectors, 'N' key vectors, and 'N' value vectors are generated, wherein the "N" key vectors generated in the "(L-1)·M+1"th to "L·M"th sequences among the plurality of sequences are sequentially stored in the first to "M"th memory banks of the "L"th PIM device in units of the "N" key vectors generated in a single sequence, and The "N" query vectors generated in each of the "(L-1)·M+1"th to "L·M"th sequences are stored in a global buffer of the "L"th PIM device.

7. The neural network architecture according to claim 6, in, The first to "N"th key vectors generated in the first sequence are sequentially stored in a first row of a first memory bank of a first PIM device in a column increasing direction, wherein the first to "N"th query vectors generated in the first sequence are sequentially stored in the global buffer of the first PIM device, and When "K" is a natural number from "1" to "N", The first processing unit of the first PIM device performs a first matrix-vector multiplication on the "K"th key vector and the "K"th query vector to generate a "K"th attention score vector in the first sequence.

8. The neural network architecture according to claim 7, in, The first to “N”th query vectors generated in the second sequence are sequentially stored in a first row of a second memory bank of the first PIM device among the plurality of PIM devices in a column increasing direction, wherein the first to "N"th query vectors generated in the second sequence are sequentially stored in the global buffer of the first PIM device, When "K" is a natural number from "1" to "N", wherein the first processing unit of the first PIM device performs the first matrix-vector multiplication on the "K"th key vector and the "K"th query vector generated in the first sequence to generate a first part of the "K"th attention score vector in the second sequence, and The second processing unit of the first PIM device performs the first matrix-vector multiplication on the "K"th key vector generated in the second sequence and the "K"th query vector generated in the second sequence to generate a second part of the "K"th attention score vector in the second sequence.

9. The neural network architecture according to claim 8, in, the first to "N"th key vectors generated in the "M"th sequence are sequentially stored in the first row of the "M"th memory bank of the first PIM device among the plurality of PIM devices in the direction of increasing the columns, wherein the first to "N"th query vectors generated in the "M"th sequence are sequentially stored in the global buffer of the first PIM device, and When "K" is a natural number from "1" to "N", Wherein, the first to "M"th processing units of the first PIM device sequentially perform a first matrix-vector multiplication on the Kth key vector generated in the first to "M"th sequences and the "K"th query vector generated in the "M"th sequence to generate the first to "N"th parts of the "K"th attention score vector in the "M"th sequence.

10. The neural network architecture according to claim 1, When the number of the plurality of PIM devices is "P", the number of the plurality of memory banks is "M", and "L" is a natural number from "1" to "P", in, The key vectors generated in the "(L-1)·M+1"th to "L·M"th sequences among the plurality of sequences and the key vectors generated in the "(P+(L-1))·M+1"th to "(P+1)·M"th sequences are sequentially stored in the first to "M"th memory banks of the "L"th PIM device, with the key vectors generated in one sequence as a unit.

11. The neural network architecture according to claim 10, in, The key vectors generated in the "(L-1)·M+1"th to "L·M"th sequences among the plurality of sequences are stored in the first rows of the first to "M"th memory banks of the "L"th PIM device, and The key vectors generated in the (P+(L-1))·M+1th to (P+1)·M"th sequence are stored in the second row of each of the first to Mth memory banks of the Lth PIM device, where the second row is the next row after the first row.

12. The neural network architecture according to claim 11, in, the first to "N"th key vectors generated in the "(P+(L-1))·M+1"th sequence are sequentially stored in the second row of the first memory bank of the "L"th PIM device in a column increasing direction, wherein the first to "N"th query vectors generated in the "(P+(L-1))·M+1"th sequence are sequentially stored in the global buffer of the "N"th PIM device, And when "K" is a natural number from "1" to "N", wherein the first to the "M"th processing units of each of the first to the "P"th PIM devices sequentially perform a first operation of a first matrix-vector multiplication on the "K"th key vector generated in the first to the "(P+(L-1))·M"th sequences and the "K"th query vector generated in the "M"th sequence to generate the first to the "(P+(L-1))·M+1"th parts of the "K"th attention score vector in the "(P+(L-1))·M+1"th sequence, and The first processing unit of the first PIM device sequentially performs the second operation of the first matrix-vector multiplication on the "K"th key vector generated in the "(P+(L-1))·M"th sequence and the "K"th query vector generated in the "(P+(L-1))·M"th sequence to generate the "(P+(L-1))·M+1"th part of the "K"th attention score vector in the "(P+(L-1))·M+1"th sequence.

13. The neural network architecture of claim 1, in, When the number of the plurality of PIM devices is "P", the number of the plurality of memory banks is "M", and "L" is a natural number from "1" to "P", The key vectors generated in the "(L-1)·M+1"th to "L·M"th sequences among the plurality of sequences are stored in consecutive first and second rows in each of the first to "M"th memory banks of the "L"th PIM device.

14. The neural network architecture according to claim 13, in, The key vectors generated in the "(P+(L-1))·M+1"th to "(P+1)·M"th sequences are sequentially stored in the first to "M"th memory banks of the "L"th PIM device in units of key vectors generated in a single sequence.

15. The neural network architecture of claim 14, in, The key vectors generated in the "(P+(L-1))·M+1"-"(P+1)·M"-th sequence are stored in third and fourth rows consecutive after the second row in each of the first to "M"-th memory banks of the "L"-th PIM device.

16. The neural network architecture of claim 1, in, When the number of the plurality of PIM devices is "P", the number of the plurality of memory banks included in each of the plurality of PIM devices is "M", the number of heads of the multi-head attention is "N", one head of the multi-head attention includes "D" elements, "K" is a natural number from 1 to "N", and "D / M" is less than "P", First to "D"th elements of a "K"th value vector among "N" value vectors generated in a first sequence as one of the plurality of sequences are distributed as "M" elements and written to first to "D / M"th PIM devices among the plurality of PIM devices.

17. The neural network architecture of claim 16, in, The "M" elements of the first to "D"-th elements of the "K"-th value vector written into each of the first to "D / M"-th PIM devices are distributed in units of single elements and written into the "M" memory banks included in each of the first to "D / M"-th PIM devices.

18. The neural network architecture of claim 17, in, The "M" elements of the "M" memory banks included in each of the first to "D / M"th PIM devices distributed in units of single elements and written to the same first row and first column.

19. The neural network architecture of claim 18, in, the first to "D"th elements of the "K"th value vector in the "N"th value vector generated in a second sequence which is a next sequence of the first sequence are distributed in units of "M" elements and written into the first to "D / M"th PIM devices among the plurality of PIM devices, wherein the "M" elements of the first to "D"th elements of the first value vector written into each of the first to "D / M"th PIM devices are distributed in units of single elements and written into the "M" memory banks included in each of the first to "D / M"th PIM devices, and The "M" elements of the "M" memory banks included in each of the first to "D / M" th PIM devices distributed in units of single elements are written into the first row and second column, which is the next column to the first column.

20. The neural network architecture of claim 1, in, When the number of the plurality of PIM devices is "P", the number of the plurality of memory banks included in each of the plurality of PIM devices is "M", the number of heads in the multi-head attention is "N", one head in the multi-head attention includes "D" elements, and "D = (P / 2)·M", First to "D"th elements of odd-numbered value vectors among "N" value vectors generated in a first sequence as one of the plurality of sequences are stored in first to "P / 2"th PIM devices among the plurality of PIM devices, and The first to "D"th elements of the even-numbered value vectors among the "N" value vectors generated in the first sequence are stored in "(P / 2)+1"th to "P"th PIM devices among the plurality of PIM devices.

21. The neural network architecture of claim 20, in, The first to "D"th elements of the odd-numbered value vectors generated in the first sequence are distributed in units of "M" elements and stored in the first to "P / 2"th PIM devices, and The first to "D"th elements of the even-numbered value vectors generated in the first sequence are distributed in units of "M" elements and stored in the "(P / 2)+1"th to "P"th PIM devices.

22. The neural network architecture of claim 21, in, the 'M' elements of the odd-numbered value vectors generated in the first sequence and distributed and stored in the first to 'P / 2'th PIM devices are distributed and stored in units of single elements in the first to 'M'th memory banks of each of the first to 'P / 2'th PIM devices, and wherein the 'M' elements of the even-numbered value vectors generated, distributed, and stored in the '(P / 2)+1'th to 'P'th PIM devices in the first sequence are distributed and stored in units of single elements in the first to 'M'th memory banks of each of the '(P / 2)+1'th to 'P'th PIM devices.

23. The neural network architecture of claim 22, in, The odd-numbered value vectors generated in the first sequence are stored in different rows of the first through "M"th memory banks of each of the first through "P / 2"th PIM devices, from a first odd-numbered value vector to a last odd-numbered value vector, and Wherein, from the first even-numbered value vector to the last even-numbered value vector, the even-numbered value vectors generated in the first sequence are stored in different rows of the first to "M"th memory banks of each of the "(P / 2)+1"th to "P"th PIM devices.

24. The neural network architecture of claim 23, in, the odd-numbered value vectors generated in the first sequence are stored in the first through "M"th memory banks of each of the first through "P / 2"th PIM devices as specified by the same first column address, and The even-numbered value vectors generated in the first sequence are stored in the first to "M"th memory banks of each of the "(P / 2)+1"th to "P"th PIM devices as specified by the same first column address.

25. The neural network architecture of claim 24, in, the odd-numbered value vectors generated in a second sequence following the first sequence among the plurality of sequences are stored in the same row as the row in which the odd-numbered value vectors generated in the first sequence were stored in the first through "M"th memory banks of each of the first through "P / 2"th PIM devices, the odd-numbered value vectors generated in the second sequence are stored as specified by a second column address following the first column address, and The even-numbered value vectors generated in the second sequence are stored in the same row as the row in which the even-numbered value vectors generated in the first sequence are stored in the first to "M"th memory banks of each of the "(P / 2)+1"th to "P"th PIM devices, and the even-numbered value vectors generated in the second sequence are stored as specified by the second column address.

26. The neural network architecture of claim 24, in, Each of the plurality of PIM devices further includes a global buffer storing an attention distribution vector used as input data in a second matrix-vector multiplication of the multi-head attention. wherein the first to "D"th elements of the odd-numbered attention distribution vectors are stored in the global buffer of the first to "P / 2"th PIM devices, and Among them, the first to "D"th elements of the even-numbered attention distribution vectors are stored in the global buffer of the "(P / 2)+1"th to "P"th PIM devices.

27. The neural network architecture of claim 22, in, the 'M' elements of the first value vector of the 'N' value vectors generated in the first sequence of the plurality of sequences are distributed and stored in a first row of each of the first to 'M'th memory banks of each of the first to 'P / 2'th PIM devices as specified by a first column address, wherein the 'M' elements of the second value vector of the 'N' value vectors generated in the first sequence are distributed and stored in the first row of each of the first to 'M'th memory banks of each of the '(P / 2)+1'th to 'P'th PIM devices as specified by the first column address, wherein the "M" elements of a third value vector among the "N" value vectors generated in the first sequence are distributed and stored in the second row of each of the first to "M"th memory banks of each of the first to "P / 2"th PIM devices as specified by the first column address, and wherein the 'M' elements of a fourth value vector among the 'N' value vectors generated in the first sequence are distributed and stored in the second row of each of the first to 'M'th memory banks of each of the '(P / 2)+1'th to 'P'th PIM devices as specified by the first column address.

28. The neural network architecture of claim 27, in, Each of the plurality of PIM devices further includes a global buffer storing an attention distribution vector used as input data in a second matrix-vector multiplication of the multi-head attention. wherein odd-numbered attention distribution vectors are stored in the global buffer of each of the first to "P / 2"th PIM devices, and Among them, the even-numbered attention distribution vectors are stored in the global buffer of each PIM device from the "(P / 2)+1"th to the "P"th PIM devices.

29. The neural network architecture of claim 28, in, performing the second matrix-vector multiplication on the M elements of the first value vector stored in the first to M memory banks of each of the first to P / 2 PIM devices and the first attention distribution vector stored in the global buffer of each of the first to P / 2 PIM devices to generate first to D segments of the first multi-head attention value vector, and Wherein, the first to "M"th processing units of each of the "(P / 2)+1"th to "P"th PIM devices perform the second matrix-vector multiplication on the "M" elements of the second value vector stored in the first to "M"th storage bodies of each of the "(P / 2)+1"th to "P"th PIM devices and the second attention distribution vector stored in the global buffer of each of the "(P / 2)+1"th to "P"th PIM devices to generate the first to "D"th segments of the second multi-head attention value vector.

30. The neural network architecture of claim 29, in, the first to "M"th processing units of each of the first to "P / 2"th PIM devices performing the second matrix-vector multiplication on the "M" elements of the third value vector stored in the first to "M"th memory banks of each of the first to "P / 2"th PIM devices and the third attention distribution vector stored in the global buffer of each of the first to "P / 2"th PIM devices to generate the first to "D"th segments of the third multi-head attention value vector, and Wherein, the first to "M"th processing units of each of the "(P / 2)+1"th to "P"th PIM devices perform the second matrix-vector multiplication on the "M" elements of the fourth value vector stored in the first to "M"th storage bodies of each of the "(P / 2)+1"th to "P"th PIM devices and the second attention distribution vector stored in the global buffer of each of the "(P / 2)+1"th to "P"th PIM devices to generate the first to "D"th segments of the fourth multi-head attention value vector.

Images