Matrix multiplier and operation method of matrix multiplication device including matrix multiplier

Abstract

A matrix multiplier includes: an input vector scaler configured to generate a first scaled input vector based on a first input vector and a plurality of quantized scaling coefficients; a first data type converter configured to generate a first fixed point scaled input vector based on the first scaled input vector; an array of processing elements including a first processing element configured to generate a first fixed-point output element based on the first fixed-point scaled input vector and the first plurality of quantized symbol values and a second processing element configured to generate a second fixed-point output element based on the first fixed-point scaled input vector and the second plurality of quantized symbol values; and a second data type converter configured to generate first and second output elements by converting data types of the first and second fixed-point output elements, and output a first output vector including the first and second output elements.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of Korean Patent Application No. 10-2023-0143978 filed in the Korean Intellectual Property Office on October 25, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] Various example embodiments generally relate to semiconductor devices and more particularly to a matrix multiplier that performs matrix multiplication and / or a matrix multiplication apparatus including the matrix multiplier. Background Art

[0004] With the recent development of artificial intelligence technology, the computational complexity of artificial intelligence models has increased rapidly. Therefore, various technologies are being studied to shorten the running time of artificial intelligence models.

[0005] Typically, most of the running time of an AI model is spent on matrix multiplication. For example, an AI model may spend most of its running time computing the output matrix by performing multiplication of the input matrix and the weight matrix. Therefore, various algorithms such as binary coded quantization (BCQ) are being studied to perform multiplication of the input matrix and the weight matrix with less computation. Summary of the invention

[0006] Some example embodiments solve or improve the above technical problems. More specifically, the purpose of various example embodiments is to provide a matrix multiplier and / or a matrix multiplication device including the matrix multiplier, which is configured to perform matrix multiplication at a faster speed and / or with a smaller amount of calculation.

[0007] According to some example embodiments, a matrix multiplier includes: an input vector scaler configured to generate a first scaled input vector based on a first input vector and a plurality of quantization scaling coefficients; a first data type converter configured to generate a first fixed-point scaled input vector based on the first scaled input vector; a processing element array including a first processing element and a second processing element, the first processing element configured to generate a first fixed-point output element based on the first fixed-point scaled input vector and a first plurality of quantization symbol values, the second processing element configured to generate a second fixed-point output element based on the first fixed-point scaled input vector and the second plurality of quantization symbol values; and a second data type converter configured to generate a first output element and a second output element by converting a data type of the first fixed-point output element and a data type of the second fixed-point output element, respectively, and configured to output a first output vector including the first output element and the second output element.

[0008] Optionally or additionally, a matrix multiplier according to some example embodiments includes: an input vector scaler configured to generate a first plurality of scaled input elements based on a first input element and a first plurality of quantization scaling coefficients, and to generate a second plurality of scaled input elements based on a second input element and a second plurality of quantization scaling coefficients; a first data type converter configured to generate a first plurality of fixed-point scaled input elements based on the first plurality of scaled input elements, and to generate a second plurality of fixed-point scaled input elements based on the second plurality of scaled input elements; a first processing element configured to generate a first fixed-point output element by accumulating the first plurality of fixed-point scaled input elements and the second plurality of fixed-point scaled input elements based on a plurality of quantization symbol values; and a second data type converter configured to generate a first output element by converting the data type of the first fixed-point output element.

[0009] Optionally or additionally, the operating method of the matrix multiplication device according to some example embodiments includes: receiving first to Nth weights from an external device, (where N is an integer greater than or equal to 2); generating first to (N×R)th quantization symbol values ​​and first to (N×R)th quantization scaling coefficients, (where R is an integer greater than or equal to 2) by binary-coding quantization of the first to Nth weights; receiving first to Nth input elements from an external device; generating first to (N×R)th scaled input elements by scaling the first to Nth input elements based on the first to (N×R)th quantization scaling coefficients; and outputting a first output element generated by accumulating the first to (N×R)th scaled input elements based on the first to (N×R)th quantization symbol values.

[0010] Optionally or additionally, according to some example embodiments, a matrix multiplication device configured to receive a weight matrix and a first input vector from the outside includes: a binary coded quantization (BCQ) circuit configured to generate a plurality of quantization symbol values ​​and a plurality of quantization scaling coefficients by binary coded quantization of the weight matrix; and a matrix multiplier configured to calculate a first output vector corresponding to the product of the first input vector and the weight matrix based on the plurality of quantization symbol values ​​and the plurality of quantization scaling coefficients. The matrix multiplier includes: an input vector scaler configured to generate a first scaled input vector by scaling the first input vector based on the plurality of quantization scaling coefficients; a first data type converter configured to generate a first fixed-point scaled input vector based on the first scaled input vector; a processing element array configured to calculate a first fixed-point output vector based on the plurality of quantization symbol values ​​and the first fixed-point scaled input vector; and a second data type converter configured to generate the first output vector by converting the data type of the first fixed-point output vector.

[0011] Optionally or additionally, according to some example embodiments, a matrix multiplication device, wherein the matrix multiplication device is configured to receive an n-dimensional input vector and a weight matrix having n times m dimensions, and output an m-dimensional output vector, wherein n and m are integers greater than or equal to 2, the matrix multiplication device includes: a binary coded quantization (BCQ) circuit, configured to generate first to R-th quantization symbol matrices having n times m dimensions and first to (N×R)-th quantization scaling coefficients corresponding to different rows of the first to R-th quantization symbol matrices, respectively (where R is an integer greater than or equal to 2); an input vector sealer, configured to scale elements of the n-dimensional input vector based on the first to (N×R)-th quantization scaling coefficients; and a processing element array comprising a plurality of processing elements, wherein each of the plurality of processing elements is configured to output output elements different from each other included in the m-dimensional output vector by accumulating elements of the input vector scaled based on the first to R-th quantization symbol matrices. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Various example embodiments will be described with reference to one or more drawings, in which:

[0013] Figure 1 is a block diagram of a matrix multiplication apparatus according to some example embodiments;

[0014] Figure 2 is a diagram illustrating the operation of a matrix multiplication device, which is implemented to convert Figure 1 The input matrix and weight matrix are directly multiplied;

[0015] Figure 3 The diagram shows Figure 1 A diagram of the operation of the BCQ circuit;

[0016] Figure 4 The diagram shows Figure 1 A diagram of the operation of a BCQ circuit that performs a binary coded quantization operation on each column of a weight matrix;

[0017] Figure 5 The diagram shows Figure 4 A diagram of the operation of a matrix multiplier of some example embodiments;

[0018] Figure 6 The diagram shows Figure 4 Embodiment Figure 1 A block diagram of a configuration of a matrix multiplier;

[0019] Figure 7 The diagram shows Figure 6 a diagram of a configuration of a first data type converter;

[0020] Figure 8 The diagram shows Figure 7A diagram of the operation of an index extraction circuit;

[0021] Fig. 9 The diagram shows Figure 7 A diagram of the operation of a data type conversion circuit;

[0022] Fig.10 A more detailed map shows Figure 6 A block diagram of a configuration of an array of processing elements;

[0023] Fig.11 A more detailed map shows Figure 6 Block diagram of some operations of the matrix multiplier;

[0024] Fig.12 yes Fig.11 A diagram of the operation of a processing element;

[0025] Fig.13 is implemented according to some example embodiments Fig.12 a diagram of a configuration of one of the processing elements in;

[0026] Fig.14 The diagram shows Figure 6 a diagram of the operation of a second data type converter;

[0027] Fig.15 is a diagram illustrating a method according to some example embodiments Figure 1 A diagram of the operation of the BCQ circuit;

[0028] Fig.16 The diagram shows Fig.15 A graph of an embodiment of an approximate weight matrix;

[0029] Fig.17 The diagram shows Fig.16 A diagram of the quantized symbol matrix of ;

[0030] Fig.18 is implemented according to some example embodiments Figure 1 A block diagram of a configuration of a matrix multiplier;

[0031] Fig.19 The diagram shows Fig.18 A block diagram of a configuration of an input vector scaler;

[0032] Fig. 20 A more detailed map shows Fig.19 A diagram of the operation of an input vector scaling circuit;

[0033] Fig.21 The diagram shows Fig.18 a block diagram of a configuration of a first data type converter;

[0034] Fig. 22A more detailed map shows Fig.18 A block diagram of a configuration of an array of processing elements;

[0035] Fig.23 A more detailed map shows Fig. 22 A block diagram of an output of a processing element;

[0036] Fig.24 A more detailed map shows Fig. 22 a block diagram of the operation of a first processing element row;

[0037] Fig.25 is a diagram illustrating a method according to some example embodiments Fig.24 a diagram of a configuration of a processing element;

[0038] Fig.26 The diagram shows Fig.18 a diagram of the operation of a second data type converter;

[0039] Fig. 27 The diagram shows Figure 1 A flowchart of the operation of a matrix multiplication device;

[0040] Fig.28 A more detailed map shows Fig. 27 Flow chart of step S150;

[0041] Fig.29 The diagram shows Figure 1 A flowchart of the operation of a matrix multiplication device;

[0042] Fig.30 A more detailed map shows Fig.29 Flow chart of step S250;

[0043] Fig.31 is a diagram showing the implementation of a systolic array method Fig. 22 A block diagram of an array of processing elements;

[0044] Fig.32 A more detailed map shows Fig.31 a diagram of a configuration of a processing element;

[0045] Fig.33 is a diagram illustrating a method according to some example embodiments Fig.19 A diagram of the operation of an input vector scaling circuit;

[0046] Fig.34 is a diagram illustrating a method according to some example embodiments Figure 1 A diagram of the operation of a matrix multiplication device;

[0047] Fig.35 The diagram shows Fig.34 A graph of the full input matrix of ;

[0048] Fig.36 The diagram shows Fig.34 Graph of the full weight matrix of ;

[0049] Fig.37 The diagram shows Fig.34 A graph of the full output matrix of ;

[0050] Fig.38 is a block diagram illustrating a matrix multiplication apparatus according to some example embodiments;

[0051] Fig.39 is a block diagram illustrating a neural processing system implemented in accordance with some example embodiments; and

[0052] Fig.40 The diagram shows Fig.39 Block diagram of an artificial intelligence model driven by a neural processing system. DETAILED DESCRIPTION

[0053] Below, various example embodiments will be described clearly and in detail to the extent that those of ordinary skill in the art can easily perform the example embodiments. Details such as detailed configuration and structure are provided only to facilitate the overall understanding of the example embodiments. Therefore, without departing from the technical spirit and scope of the present invention, those of ordinary skill in the art may make modifications to the described example embodiments. In addition, for the sake of clarity and / or brevity, the description of some functions and / or structures (such as some well-known functions and / or structures) may be omitted. The configurations in the drawings and / or detailed description may be connected to elements different from those shown in the drawings or described in the detailed description. The terms used below are defined with respect to the functions of the various described example embodiments, without being limited to specific functions. The definition of the terms may be determined based on the details described in the detailed description.

[0054] Elements described with reference to terms such as drivers, blocks, or the like used in the detailed description may be implemented in the form of software, hardware, or a combination thereof. For example, software may be machine code, firmware, embedded code, and application software. For example, hardware may include circuits, electronic circuits, processors, computers, integrated circuit cores, pressure sensors, inertial sensors, micro-electromechanical systems (MEMS), passive components, or a combination thereof.

[0055] Hereinafter, for a more concise description, matrices are referenced by square brackets “[” and “]”, and sets are referenced by curly brackets “{” and “}”. However, the scope of example embodiments is not limited to the notation method.

[0056] Figure 1 is a block diagram illustrating a matrix multiplication apparatus according to some example embodiments. Figure 1, the matrix multiplication device MMD may include a matrix multiplier 100 and a binary coded quantization (BCQ) circuit 200. The matrix multiplication device MMD may also be referred to as a matrix multiplication device MMD.

[0057] The matrix multiplication device MMD can receive and / or obtain an input matrix XM. The input matrix XM can include multiple input vectors. Each of the multiple input vectors can include multiple input elements. For example, the input matrix XM can be expressed as the following equation 1.

[0058] (Equation 1)

[0059]

[0060] Referring to Equation 1, XM can represent an input matrix XM with dimension n×h, arrive can represent the first to hth input vectors, x 11 to x hn can represent different input elements. For example, x 11 to x 1n It can be represented by the first input vector (for example, ), x h1 to x hn can represent the input vector (e.g., ). In Equation 1, h can be the same as n, less than n, or greater than n.

[0061] Below, for a more concise description, an example embodiment in which the dimension of each input vector included in the input matrix XM is "n" is representatively described. For example, various example embodiments in which each input vector includes n input elements are representatively described. An example embodiment in which the input matrix XM includes n columns will be representatively described.

[0062] In various example embodiments, each input element included in the input matrix XM may have a 16-bit floating point (FP16) data type and / or a 32-bit floating point (FP32) data type. However, example embodiments are not limited thereto.

[0063] The matrix multiplication device MMD may receive a weight matrix WM. The weight matrix WM may include a plurality of weights. For example, the weight matrix WM may be expressed as the following equation 2.

[0064] (Equation 2)

[0065]

[0066] Referring to Equation 2, WM can be expressed as the weight matrix WM, w 11 townm can represent weights, wherein at least two weights can be different from each other. ij It may be the weight set in the i-th row and j-th column of the weight matrix WM. Here, m may be the same as n, less than n, or greater than n.

[0067] In various example embodiments, each weight included in the weight matrix WM may have a 16-bit floating point (FP16) data type and / or a 32-bit floating point (FP32) data type. However, example embodiments are not limited thereto.

[0068] The BCQ circuit 200 may perform binary coded quantization (BCQ) on the weight matrix WM. For example, the BCQ circuit 200 may determine a plurality of quantization symbol values ​​QSV and a plurality of quantization scaling coefficients QSC based on the weight matrix WM.

[0069] For example, the BCQ circuit 200 may convert each of the plurality of weights into a plurality of "quantization scaling coefficient (QSC)-quantization symbol value (QSV)" pairs. In some example embodiments, the BCQ circuit 200 may approximate each weight of the weight matrix WM with a plurality of quantization scaling coefficient (QSC) and quantization symbol value (QSV) pairs.

[0070] In various example embodiments, each of the plurality of quantization symbol values ​​QSV may be or correspond to or represent -1 or +1 (or one of -1 or +1).

[0071] In some example embodiments, each of the plurality of quantized scaling coefficients QSC may have the same data type as the data type of the corresponding weight of the weight matrix WM. For example, each of the plurality of quantized scaling coefficients QSC may have an FP16 and / or FP32 data type. However, example embodiments are not limited thereto. Figure 3 The specific operation of the BCQ circuit 200 is described in more detail.

[0072] The matrix multiplier 100 may receive a plurality of quantization symbol values ​​QSV and a plurality of quantization scaling coefficients QSC. The matrix multiplier 100 may perform matrix multiplication on an input matrix XM and a weight matrix WM based on the plurality of quantization symbol values ​​QSV and the plurality of quantization scaling coefficients QSC. For example, the matrix multiplier 100 may generate an output matrix YM by multiplying the input matrix XM by a weight matrix WM approximated based on the plurality of quantization symbol values ​​QSV and the plurality of quantization scaling coefficients QSC. A quantization symbol bit having a code length of 1 bit may be provided to the matrix multiplier 100. However, example embodiments are not limited thereto.

[0073] The output matrix YM may include multiple output vectors. Each of the multiple output vectors may include multiple output elements. For example, the output matrix YM may be expressed as the following equation 3.

[0074] (Equation 3)

[0075]

[0076] In this case, YM can represent the output matrix YM, y 11 To hm can represent different output elements. For example, y 11 To 1m can be represented by including in the first output vector (ie, ) in the output element, y h1 To hm can be represented by the h-th output vector (i.e., ) Here, h can be the same as m, greater than m, or less than m.

[0077] For example, in various example embodiments, "n" and "m" may be the same integer. For example, the weight matrix WM may be implemented as a block matrix. In this case, the dimension of the output vector included in the output matrix YM may be the same as the dimension of the input vector. However, example embodiments are not limited thereto.

[0078] In various exemplary embodiments, if the matrix multiplication device MMD directly multiplies the input matrix XM and the weight matrix WM to calculate the output matrix YM, the matrix multiplication device MMD may have to process a very large number of calculations. In this case, the operating speed of the matrix multiplication device MMD may be degraded. Figure 2 The operation of the matrix multiplication device MMD for calculating the output matrix YM by directly multiplying the input matrix XM and the weight matrix WM is described in more detail.

[0079] On the other hand, if the matrix multiplication device MMD calculates the output matrix YM by multiplying the input matrix XM by a weight matrix WM approximated based on a plurality of quantization symbol values ​​QSV and a plurality of quantization scaling coefficients QSC, the amount of calculation of the matrix multiplication device MMD can be reduced, for example, can be greatly reduced. The operation of the matrix multiplication device MMD for calculating the output matrix YM based on a plurality of quantization symbol values ​​QSV and a plurality of quantization scaling coefficients QSC will be described in more detail with reference to the following drawings.

[0080] Figure 2 is a diagram showing a method implemented as Figure 1 FIG. 1 is a diagram of the operation of a matrix multiplication device in which the input matrix and the weight matrix are directly multiplied. Figure 1 and Figure 2 The matrix multiplication device MMD can be implemented by, for example, using a matrix multiplication algorithm (such as but not limited to n^3 algorithm, Strassen algorithm, divide and conquer algorithm, Winogra algorithm, etc.) directly multiplies the input matrix XM and the weight matrix WM to calculate the output matrix YM.

[0081] In order to calculate one output element included in the output matrix YM, the matrix multiplication device MMD may have to perform n floating point multiplications and then may have to perform n-1 floating point sums. For example, the matrix multiplication device MMD may calculate one of the output elements (e.g., y) included in the output matrix YM in a manner similar to the following equation 4: 11 ).

[0082] (Equation 4)

[0083] y 11 =x 11 × 11 +x 12 × 21 +…+x 1n × n1

[0084] In this way, the matrix multiplication device MMD may have to perform m×n floating-point multiplications and m×(n-1) floating-point sums to calculate the matrix corresponding to one input vector (e.g. ) output vector (for example, ). In this case, the operation speed of the artificial intelligence model driven by the matrix multiplication device MMD may be degraded due to excessive calculations performed by the matrix multiplication device MMD.

[0085] Figure 3 The diagram shows Figure 1 The operation of the BCQ circuit is shown in Figure 1. Figure 1 and Figure 3 , the horizontal axis represents the size of the weights included in the weight matrix WM.

[0086] The BCQ circuit 200 may approximate one or more weights included in the weight matrix WM as a plurality of quantum levels QL. The BCQ circuit 200 may determine the number of the plurality of quantum levels QL based on a resolution (such as a predetermined BCQ resolution). For example, the BCQ circuit 200 may approximate each of the plurality of weights as 2 R The quantum level QL, where R is the BCQ resolution. In this case, it can be determined based on the combination of the R quantization scaling factor QSC and the R quantization symbol value QSV. R Each in the quantum level QL.

[0087] However, hereinafter, for a more concise description, an example embodiment in which R is 3 will be representatively described. For example, the BCQ circuit 200 may approximate each weight included in the weight matrix WM to the first to eighth quantum levels QL1-QL8. In this case, each of the first to eighth quantum levels QL1-QL8 may be determined based on the following equation 5.

[0088] (Equation 5)

[0089] QL=α 1 ×b 1 +α 2 ×b 2 +α 3 ×b 3

[0090] Referring to Equation 5, QL can represent any quantum level, α 1 To α 3 The quantization scaling factor QSC can be expressed as 1 To b 3 The quantized symbol value QSV may be represented. In this case, the first to eighth quantum levels QL1-QL8 may correspond to different combinations of b1 to b3.

[0091] For example, the first quantum level QL1 may correspond to b 1 To b 3 are all "-1". In this case, the first quantum level QL1 can correspond to "-α 1 -α 2 -α 3 Similarly, the eighth quantum level QL8 can correspond to b 1 To b 3 In this case, the eighth quantum level QL8 can correspond to "+α 1 +α 2 +α 3 In this way, the seventh quantum level QL7 can correspond to b 1 To b 3 In this case, the seventh quantum level QL7 can correspond to "+α 1 +α 2 -α 3 In some example embodiments, the signs of QL and QL may be based on binary encoding of the quantum level QL. However, example embodiments are not limited thereto.

[0092] The BCQ circuit 200 may approximate each weight included in the weight matrix WM to a quantum level having the closest value among the first to eighth quantum levels QL1-QL8. For example, if the weight w 11The size of is closest to the seventh quantum level QL7 among the first to eighth quantum levels QL1-QL8, then the BCQ circuit 200 can change the weight w 11 Approximately the seventh quantum level QL7. In this case, the BCQ circuit 200 can be weighted by 11 It is represented by a plurality of quantized scaling coefficients QSC (e.g., α 1 To α 3 ) and a combination of multiple quantization symbol values ​​QSV (e.g., "+1", "+1" and "-1"). Similarly, the BCQ circuit 200 can approximate each weight included in the weight matrix WM based on multiple quantization symbol values ​​QSV and multiple quantization scaling coefficients QSC.

[0093] In various example embodiments, the plurality of quantization scaling coefficients QSC may have different sizes, for example, may have decreasing sizes (such as arithmetically decreasing or geometrically decreasing). 1 The size can be larger than α 2 The size of α 2 The size can be larger than α 3 size.

[0094] For a more concise description, Figure 3 Various example embodiments in which the BCQ resolution is 3 are representatively described in , but example embodiments are not limited thereto. For example, depending on the type of artificial intelligence model driven by the matrix multiplication device MMD, the BCQ resolution may be determined to be an integer of 2 or greater.

[0095] In some example embodiments, if the artificial intelligence model driven by the matrix multiplication device MMD is or includes or is included in a large language model (LLM), the BCQ resolution may be 3. However, example embodiments are not limited thereto.

[0096] Alternatively or additionally, in some example embodiments, if an artificial intelligence model driven by a matrix multiplication device MMD is or includes or is included in the image object recognition model, the BCQ resolution may be 2. However, example embodiments are not limited thereto.

[0097] In some example embodiments, the BCQ circuit 200 may perform a binary coded quantization operation on each column of the weight matrix WM. For example, the BCQ circuit 200 may determine a different plurality of quantization scaling coefficients QSC for each column of the weight matrix WM. In this case, the quantization scaling coefficients of the weights included in the first column of the weight matrix WM may be different from the quantization scaling coefficients of the weights included in the second column of the weight matrix WM. Figures 4 to 14An example embodiment in which the BCQ circuit 200 performs a binary coded quantization operation on each column of the weight matrix WM is described in more detail. However, example embodiments are not limited thereto.

[0098] In some example embodiments, the BCQ circuit 200 may perform a binary coded quantization operation on each row of the weight matrix WM. For example, the BCQ circuit 200 may determine a different plurality of quantization scaling coefficients QSC for each row of the weight matrix WM. In this case, the quantization scaling coefficients of the weights included in the first row of the weight matrix WM may be different from the quantization scaling coefficients of the weights included in the second row of the weight matrix WM. Figures 15 to 40 An embodiment in which the BCQ circuit 200 performs a binary coded quantization operation on each row of the weight matrix WM is described in more detail. However, example embodiments are not limited thereto.

[0099] In some example embodiments, the BCQ circuit 200 may approximate each of the plurality of weights as the first to eighth quantum levels QL1-QL8 based on a uniform BCQ algorithm. For example, the BCQ circuit 200 may approximate the plurality of weights based on the first to eighth quantum levels QL1-QL8 having uniform intervals. In this case, the sizes of the plurality of quantization scaling coefficients QSC may be implemented as a geometric sequence having a common ratio of 2. For example, α 1 The size can be α 2 Twice the size of α 2 The size can be α 3 However, example embodiments are not limited thereto.

[0100] Figure 4 The diagram shows Figure 1 The operation of the BCQ circuit performs a binary coded quantization operation on each column of the weight matrix. Figure 1 , Figure 3 and Figure 4 , the BCQ circuit 200 may perform a binary coded quantization operation on each column of the weight matrix WM.

[0101] First, the weight matrix WM can be expressed as the following Equation 6.

[0102] (Equation 6)

[0103]

[0104] in this case, can represent the j-th column vector of the weight matrix WM. For example, Can include w 1j tow nj .

[0105] The BCQ circuit 200 may perform a binary coded quantization operation on each column of the weight matrix WM based on Equation 7 below.

[0106] (Equation 7)

[0107]

[0108] Referring to Equation 7, R can represent the BCQ resolution. k_cj The k-th quantized scaling coefficient of the j-th column vector of the weight matrix WM may be represented. It can be expressed as corresponding to α k_cj A quantized symbol vector of . For example, The quantization symbol vector corresponding to the k-th quantization scaling coefficient of the j-th column vector of the weight matrix WM may be represented. may include a plurality of quantized symbol values ​​corresponding to the kth quantized scaling coefficient of the jth column vector. More specifically, It can be expressed as the following equation 8.

[0109] (Equation 8)

[0110]

[0111] b 1_k_cj To b n_k_cj Each of b can represent a different quantization symbol value QSV. More specifically, b 1_k_cj To b n_k_cj can be the quantized symbol values ​​of the weights arranged in different rows of the weight matrix WM. 1_k_cj To b n_k_cj Each of can be "+1" or "-1".

[0112] For example, the BCQ circuit 200 may approximate each weight of the weight matrix WM based on a plurality of quantization symbol values ​​QSV (eg, “b” values) and a plurality of quantization scaling coefficients QSC (eg, “α” values).

[0113] The BCQ circuit 200 may provide a plurality of quantization symbol values ​​QSV (e.g., "b" values) and a plurality of quantization scaling coefficients QSC (e.g., "α" values) to the matrix multiplier 100. Hereinafter, the operation of the matrix multiplier 100 performing a matrix multiplication calculation based on the plurality of quantization symbol values ​​QSV and the plurality of quantization scaling coefficients QSC will be described.

[0114] Figure 5 The figure shows that according to the reference Figure 4 FIG. 1 is a diagram of the operation of a matrix multiplier of an exemplary embodiment described herein. Figure 1 and Figures 3 to 5, the matrix multiplier 100 can calculate the output matrix YM based on multiple quantization symbol values ​​QSV and multiple quantization scaling coefficients QSC.

[0115] In the following, for a more concise description, the operation of the matrix multiplier 100 is representatively described, where the matrix multiplier 100 converts a first input vector (eg, ) is multiplied by a first column vector (e.g., ) to calculate the output elements of the first row and first column of the output matrix YM (e.g., y 11 ). However, example embodiments are not limited thereto.

[0116] The matrix multiplier 100 can calculate y according to the following equation 9: 11 .

[0117] (Equation 9)

[0118]

[0119] In this case, the first input vector (e.g., ) and quantized symbol vectors (e.g., ) can be called a partial sum (hereinafter referred to as "PSM"). For example, and The product of the partial sum can be called the "PSM 1_c1_X1 ”, and The product of can be called "PSM 2_c1_X1 In this way, the first input vector (e.g., ) can be expressed as the following equation 10.

[0120] (Equation 10)

[0121]

[0122] In this case, the first input vector (e.g., ) can be obtained based on the quantized symbol value QSV by x 11 to x 1n For example, the matrix multiplier 100 may determine whether to invert the sign bit of each input element based on the quantization sign value QSV. Thereafter, the matrix multiplier 100 may sequentially accumulate the input elements whose sign bits are determined to calculate each partial sum of the above equation 10. Therefore, the matrix multiplier 100 may have to perform (n-1) floating-point summations to calculate one partial sum. As a result, the matrix multiplier 100 may have to perform R×(n-1) floating-point summations to calculate all partial sums of equation 10.

[0123] In some example embodiments, if the data type of each of the plurality of input elements is converted to fixed point, the matrix multiplier 100 may calculate the partial sums of Equation 10 with a small amount of calculation. For example, if the data type of each of the plurality of input elements is fixed point corresponding to the same exponent value, the matrix multiplier 100 may have to perform R×(n−1) fixed point summations to calculate all the partial sums of Equation 10. Figures 6 to 15 A more detailed operation of the matrix multiplier 100 for converting the data type of each of the plurality of input elements into a fixed point is described in more detail.

[0124] The matrix multiplier 100 may calculate an output element by multiplying each of the multiple partial sums of the above equation 10 by the quantization scaling factor QSC and then accumulating them. For example, the matrix multiplier 100 may calculate an output element (e.g., y) by performing R floating-point multiplications and then performing R-1 floating-point sums. 11 ). In this case, with reference Figure 2 Unlike the above description, the number of floating-point multiplications performed by the matrix multiplier 100 can be reduced or minimized. For example, the matrix multiplier 100 can calculate the output element by performing R floating-point multiplications and R-1 floating-point summations, instead of performing m×n floating-point multiplication calculations and m×(n-1) floating-point summations. Figure 4 and Figure 5 The described example embodiments can improve the operating speed of the matrix multiplication device MMD.

[0125] Figure 6 The diagram shows Figure 4 Embodiment Figure 1 A block diagram of the configuration of a matrix multiplier. Figure 1 and Figures 3 to 6 , Figure 1 The matrix multiplier 100 may be implemented as the matrix multiplier 10 .

[0126] The matrix multiplier 10 may include a first data type converter 11 , a quantization symbol value buffer 12 , a processing element array 13 , a second data type converter 14 , a quantization scaling factor buffer 15 , a partial sum scaler 16 , and an accumulator 17 .

[0127] The first data type converter 11 may receive an input matrix XM. For example, the first data type converter 11 may receive a plurality of input vectors (eg, to ).

[0128] The first data type converter 11 can extract the exponent EXP from each of the plurality of input vectors. For example, the first data type converter 11 can extract the exponent EXP from each of the plurality of input vectors. The first index of the input element in the input vector can be extracted from the second input vector The first data type converter 11 may provide the extracted exponent EXP to the second data type converter 14.

[0129] The first data type converter 11 may convert the data type of each of the plurality of input vectors into a fixed-point data type based on the extracted exponent. For example, the first data type converter 11 may convert the data type of each of the plurality of input elements into a fixed-point. For example, the first data type converter 11 may receive an input matrix XM and may output a fixed-point input matrix XM_fxp.

[0130] The fixed-point input matrix XM_fxp may include multiple fixed-point input vectors. Each of the multiple fixed-point input vectors may include multiple fixed-point input elements. For example, the fixed-point input matrix XM_fxp may be expressed as the following equation 11.

[0131] (Equation 11)

[0132]

[0133] Here, h can be equal to, greater than, or less than n. In this case, XM_fxp can represent the fixed-point input matrix XM_fxp, to can represent the first to hth fixed-point input vectors, x′ 11 to x′ hn Can represent different fixed-point input elements. x′ 11 to x′ hn Can represent x converted to fixed-point data types respectively 11 to x hn . Will refer to Figures 7 to 9 The configuration and operation of the first data type converter 11 are described in more detail.

[0134] The quantization symbol value buffer 12 may store a plurality of quantization symbol values ​​QSV provided from the BCQ circuit 200. The quantization symbol value buffer 12 may provide a plurality of quantization symbol values ​​QSV to the processing element array 13.

[0135] The processing element array 13 may receive a plurality of quantized symbol values ​​QSV and a fixed-point input matrix XM_fxp. The processing element array 13 may calculate a plurality of fixed-point parts and PSM_fxp based on a plurality of fixed-point input elements and a plurality of quantized symbol values ​​QSV included in the fixed-point input matrix XM_fxp. For example, the processing element array 13 may calculate a vector corresponding to the first input vector (e.g., )'s fixed-point portion and PSM_fxp.

[0136] (Equation 12)

[0137]

[0138] In this case, PSM′ 1_c1_X1 To PSM′ R__c1_X1 The above PSMs can be 1_c1_X1 To PSM 2_c1_X1 Fixed-point form.

[0139] The processing element array 13 may include a plurality of processing elements arranged or arranged in the row direction and the column direction. Each processing element may calculate the different fixed point parts and PSM_fxp described above by equation 12. Figures 10 to 13 A more detailed configuration and operation of each processing element is described in more detail.

[0140] The second data type converter 14 may receive a plurality of exponents EXP from the first data type converter 11. The second data type converter 14 may receive a plurality of fixed-point parts and PSM_fxp from the processing element array 13. The second data type converter 14 may convert the data types of the plurality of fixed-point parts and PSM_fxp into floating points based on the plurality of exponents EXP. For example, the second data type converter 14 may output the plurality of parts and PSM in floating point format. For example, the second data type converter 14 may convert PSM′ into floating point data. 1_c1_X1 ~PSM' R__c1_X1 Convert to PSM 1_c1_X1 ~PSM 2_c1_x1 . Will refer to Fig.14 A more detailed configuration and operation of the second data type converter 14 are described in more detail.

[0141] The quantization scaling coefficient buffer 15 may store a plurality of quantization scaling coefficients QSC provided from the BCQ circuit 200. The quantization scaling coefficient buffer 15 may provide the plurality of quantization scaling coefficients QSC to the partial sum scaler 16.

[0142] The partial sum sealer 16 may receive a plurality of quantization scaling coefficients QSC and a plurality of partial sum PSMs.

[0143] The part sum scaler 16 may scale the plurality of parts and PSMs based on the plurality of quantization scaling coefficients QSC. For example, the part sum scaler 16 may generate a plurality of scaled parts and SCPSMs by multiplying each of the plurality of parts and PSMs by a quantization scaling coefficient QSC corresponding to each of the plurality of parts and PSMs. Fig.11 A more detailed configuration and operation of the parts and sealer 16 are described in more detail.

[0144] In some example embodiments, the partial sum scaler 16 may temporarily store the plurality of scaled partial sum SCPSMs in a volatile memory device (e.g., a static random access memory (SRAM) device and / or a dynamic random access memory (DRAM) device) external to the matrix multiplication device MMD.

[0145] The accumulator 17 may receive a plurality of scaled parts and SCPSMs. The accumulator 17 may calculate a plurality of output elements based on the plurality of scaled parts and SCPSMs. For example, the accumulator 17 may calculate one output element by adding the scaled parts and SCPSMs. For example, the accumulator 17 may calculate one output element by accumulating R scaled parts and SCPSMs. In this manner, the accumulator 17 may calculate a plurality of output elements to output the output matrix YM. Fig.11 A more detailed operation of the accumulator 17 is described.

[0146] In some example embodiments, the accumulator 17 may read the plurality of scaled portions and the SCPSM from a volatile memory device external to the matrix multiplication device MMD.

[0147] In some example embodiments, the operation speed of the processing element array 13 may be faster than the speed at which the matrix multiplication device MMD accesses the external volatile memory device. In this case, the operation speed of the matrix multiplication device MMD may be bottlenecked due to the speed at which the multiple scaled parts and the SCPSM are read from the external volatile memory device. Figures 15 to 40 The configuration and operation of a matrix multiplication device MMD having a reduced or minimized access time to an external volatile memory device is described.

[0148] Figure 7 The diagram shows Figure 6 FIG. 1 is a diagram of a configuration of a first data type converter. Figure 1 and Figures 3 to 7 , the first data type converter 11 may include first to h-th exponent extraction circuits 11a_1 to 11a_h and first to h-th data type conversion circuits 11b_1 to 11b_h.

[0149] The first to h-th index extraction circuits 11a_1 to 11a_h may receive different input vectors, respectively. For example, the first to h-th index extraction circuits 11a_1 to 11a_h may receive first to h-th input vectors, respectively (eg, to ).

[0150] Each of the first to h-th exponent extraction circuits 11a_1 to 11a_h may extract an exponent from a plurality of input elements included in a received input vector. For example, the first exponent extraction circuit 11a_1 may extract an exponent from an input element (eg, x) included in the first input vector. 11 to x 1n ) extracts the first index EXP1; the second index extraction circuit 11a_2 can extract the first index EXP1 from the input element (eg, x) included in the second input vector. 21 to x 2n ) extracts the second exponent EXP2. In this manner, the first to h-th exponent extraction circuits 11a_1 to 11a_h can extract the first to h-th exponents EXP1-EXPh, respectively.

[0151] The first to h-th exponent extraction circuits 11a_1 to 11a_h may provide the extracted exponents to the first to h-th data type conversion circuits 11b_1 to 11b_h, respectively. In addition, each of the first to h-th exponent extraction circuits 11a_1 to 11a_h may provide the extracted exponents to the second data type converter 14. Figure 8 A more detailed operation of each of the first to h-th index extraction circuits 11 a_1 to 11 a_h is described in more detail.

[0152] The first to h-th data type conversion circuits 11b_1 to 11b_h may receive first to h-th exponents EXP1-EXPh, respectively. The first to h-th data type conversion circuits 11b_1 to 11b_h may receive first to h-th input vectors (eg, to ).

[0153] Each of the first to h-th data type conversion circuits 11b_1 to 11b_h may convert the data type of the received input vector into a fixed point based on the received exponent. The first to h-th data type conversion circuits 11b_1 to 11b_h may output first to h-th fixed-point input vectors (e.g., to ). For example, the first data type conversion circuit (or first data type conversion circuit) 11b_1 may convert each input element included in the first input vector into a fixed-point format based on the first exponent EXP1. Fig. 9A more detailed operation of each of the first to h-th data type conversion circuits 11 b_1 to 11 b_h is described in more detail.

[0154] Figure 8 The diagram shows Figure 7 FIG. 1 is a diagram of the operation of the index extraction circuit of FIG. 1. In the following, for a more concise description, the operation of the first index extraction circuit 11a_1 will be representatively described. However, example embodiments are not limited thereto.

[0155] Reference Figure 1 and Figures 3 to 8 , the first index extraction circuit 11a_1 may receive a plurality of input elements. For example, the first index extraction circuit 11a_1 may receive x 11 to x 1n .

[0156] The data type of each of the multiple input elements can be a floating point. For example, x 11 to x 1n Each of may include a sign part SP, an exponent part EXPP and a mantissa part MTSP.

[0157] The first exponent extraction circuit 11a_1 may identify the maximum value among the values ​​of the exponent part EXPP of the received plurality of input elements. In this case, the first exponent extraction circuit 11a_1 may determine the exponent of the identified input element as the first exponent EXP1. For example, the first exponent extraction circuit 11a_1 may extract the exponent x 11 to x 1n The largest index among them.

[0158] For a more concise description, Figure 8 , the first exponent extraction circuit 11a_1 extracts the maximum value among the values ​​of the exponent part of the plurality of input elements, but the example embodiment is not limited thereto. For example, the first exponent extraction circuit 11a_1 may be implemented to extract the minimum value among the values ​​of the exponent part of the plurality of input elements.

[0159] Fig. 9 The diagram shows Figure 7 FIG. 1 is a diagram of the operation of the data type conversion circuit of FIG. 1. Below, for a more concise description, the operation of the first data type conversion circuit 11b_1 will be representatively described. However, example embodiments are not limited thereto.

[0160] Reference Figure 1 and Figures 3 to 9 , the first data type conversion circuit 11b_1 can receive multiple input elements. For example, the first data type conversion circuit 11b_1 can receive x 11 to x 1n .

[0161] The first data type conversion circuit 11b_1 may receive a first exponent EXP1. The first data type conversion circuit 11b_1 may convert each of the received plurality of input elements into a fixed-point data type based on the first exponent EXP1. For example, the first data type conversion circuit 11b_1 may convert x 11 ~x 1n Convert to x′ 11 ~x′ 1n However, in the following, for a more concise description, the input element (x 11 ) is converted to a fixed-point input element (x′ 11 )'s operation of the first data type conversion circuit 11b_1.

[0162] The first data type conversion circuit 11b_1 can convert the input element (x 11 ) is shifted toward the least significant bit (LSB) by the first exponent EXP1 and x 11 For example, if the first exponent exp1 and x 11 is “4”, the first data type conversion circuit 11 b_1 may insert four “0” bits into the most significant bit (MSB) of the first mantissa part MTSPa.

[0163] The first data type conversion circuit 11b_1 may determine the fixed-point input element (x′) based on the shifted first mantissa part MTSPa. 11 ) of the mantissa part (hereinafter referred to as the second mantissa part MTSPb). For example, the first data type conversion circuit 11b_1 may truncate the low-order bits of the shifted first mantissa part MTSPa according to the code length of the second mantissa part MTSPb. Alternatively or additionally, the first data type conversion circuit 11b_1 may determine the low-order bits of the shifted first mantissa part MTSPa according to the code length of the second mantissa part MTSPb based on various types of rounding algorithms (such as "nearest even rounding" and / or the like). However, example embodiments are not limited thereto.

[0164] In some example embodiments, the code length of the first mantissa part MTSPa may be 10 bits or 23 bits. However, example embodiments are not limited thereto.

[0165] In some example embodiments, the code length of the second mantissa part MTSPb may be 7 bits. However, example embodiments are not limited thereto.

[0166] In some example embodiments, a data type of each of the plurality of fixed-point input elements may be an 8-bit integer (INT8). However, example embodiments are not limited thereto.

[0167] Fig.10 is shown in more detail Figure 6 A block diagram of a configuration of a processing element array. Figure 1 and Figures 3 to 10 , the processing element array 13 may include a plurality of processing elements PE arranged in the row direction and the column direction. In the following, for a more concise description, it is assumed that the plurality of processing elements PE are arranged in p rows and q columns. In addition, the processing element arranged in the i-th row and j-th column of the processing element array 13 will be referred to as "PEij". For example, the processing element arranged in the first row and second column of the processing element array 13 will be referred to as "PE12".

[0168] The processing element array 13 may include first to p-th processing element rows PER1-PERp. Each of the first to p-th processing element rows PER1-PERp may include a different plurality of processing elements PE. For example, the first processing element row PER1 may include processing elements PE11-PE1q.

[0169] In some example embodiments, “p” may be an integer having a size equal to or smaller than “h.” However, example embodiments are not limited thereto.

[0170] The first to p-th processing element rows PER1-PERp may receive different fixed-point input vectors, respectively. For example, the first to p-th processing element rows PER1-PERp may receive first to p-th fixed-point input vectors, respectively (eg, to ). For example, each of the first to p-th processing element rows PER1-PERp may perform a multiplication operation on a weight matrix WM of a different input vector.

[0171] Each of the plurality of processing elements PE may receive a plurality of quantized symbol values ​​QSV from the quantized symbol value buffer 12 .

[0172] Processing elements arranged in the same column of processing element array 13 may receive the same quantization symbol value. For a more detailed example, the quantization symbol value received by processing element PE11 may be the same as the quantization symbol value received by processing element PE21.

[0173] Processing elements arranged in different columns of processing element array 13 may receive different quantization symbol values. For example, the quantization symbol value received by processing element PE11 may be different from the quantization symbol value received by processing element PE12.

[0174] Reference Fig.12The quantized symbol values ​​provided to the different columns of the processing element array 13 are described in more detail.

[0175] Each of the plurality of processing elements PE may calculate a different fixed-point partial sum PSM_fxp based on the received fixed-point input vector and the quantized symbol value QSV. The plurality of processing elements PE may provide the calculated fixed-point partial sum (PSM_fxp) to the second data type converter 14. Fig.11 and 13 The specific method by which multiple processing elements PE calculate different fixed-point parts and PSM_fxp is described in more detail.

[0176] Fig.11 A more detailed map shows Figure 6 The following is a block diagram of some operations of the matrix multiplier. The operations of the matrix multiplier 10 are related, but the example embodiments are not limited thereto. For example, the matrix multiplier 10 may operate on other fixed-point input vectors in the same or similar manner.

[0177] Reference Figure 1 and Figures 3 to 11 , the first processing element row PER1 may receive a first fixed-point input vector (eg, ). For example, each of the processing elements PE11-PE1q may receive x′ 11 to x′ 1n .

[0178] Processing elements PE11-PE1q may receive different quantized symbol vectors, respectively. For example, processing elements PE11-PE1R may receive quantized symbol vectors of the first to Rth quantized scaling coefficients, respectively corresponding to the first column vector of the weight matrix WM (eg, to Similarly, processing elements PE1(R+1)-PE1(2R) may receive quantized symbol vectors (eg, to ). In this way, the processing elements included in the first processing element row PER1 can receive different quantization symbol vectors from the quantization symbol value buffer 12.

[0179] Each processing element included in the first processing element row PER1 can calculate a different fixed point part and PSM_fxp. For example, the processing elements PE11-PE1R can calculate PSM′ 1_c1_X1 To PSM′ R__c1_X1 Similarly, processing elements PE1(R+1)-PE1(2R) can calculate PSM′ respectively.1_c2_X1 To PSM′ R__c2_X1 .

[0180] The second data type converter 14 may receive the first exponent EXP1. The second data type converter 14 may receive a plurality of fixed-point parts and PSM_fxp from the first processing element row PER1. The second data type converter 14 may convert the received fixed-point parts and PSM_fxp into floating-point data types based on the first exponent EXP1. For example, the second data type converter 14 may convert PSM′ 1_c1_X1 ~PSM' R__c1_X1 Convert to PSM 1_c1_X1 ~PSM R__c1_X1 Similarly, the second data type converter 14 can convert PSM′ 1_c2_X1 ~PSM' R__c2_X1 Convert to PSM 1_c2_X1 ~PSM R__c2_X1 .

[0181] The partial sum scaler 16 may include a plurality of multiplication circuits MUL. The plurality of multiplication circuits MUL may receive different partial sums. Hereinafter, for a more concise description, the multiplication circuit MUL that receives the partial sum PSM generated based on the processing element set in the j-th column of the processing element array 13 will be referred to as “MUL_j”. For example, the multiplication circuits MUL_1-MUL_R may receive PSMs, respectively. 1_c1_X1 To PSM R__c1_X1 , the multiplication circuit MUL_R+1-MUL_2R can receive PSM 1_c2_X1 To PSM R__c2_X1 .

[0182] Each of the plurality of multiplication circuits MUL may receive a quantization scaling factor QSC corresponding to the received partial sum PSM. For example, the multiplication circuits MUL_1-MUL_R may receive α 1_c1 To α R_c1 , the multiplication circuit MUL_R+1-MUL_2R can receive α 1_c2 To α R_c2 .

[0183] Each of the plurality of multiplication circuits MUL may multiply the received partial sum PSM by the quantization scaling factor QSC to output a scaled partial sum SCPSM. For example, the multiplication circuits MUL_1-MUL_R may output SCPSM respectively. 1__c1_X1 To SCPSM R__c1_X1 , and the multiplication circuit MUL_R+1-MUL_2R can output SCPSM respectively 1__c2_X1 To SCPSM R__c2_X1 .

[0184] The accumulator 17 may receive a plurality of scaled partial sum SCPSMs from the partial sum scaler 16. The accumulator 17 may calculate an output element by accumulating the scaled partial sums corresponding to the same input vector and the same column vector among the plurality of scaled partial sum SCPSMs. For example, the accumulator 17 may calculate an output element by accumulating the scaled partial sums corresponding to the first input vector (e.g., ) corresponds to the first column vector of the weight matrix WM (e.g., ) Corresponding SCPSM 1__c1_X1 To SCPSM R__c1_X1 Sum to calculate y 11 Similarly, the accumulator 17 may be configured to generate the first input vector (eg, ) and the second column vector of the weight matrix WM (e.g., ) 1__c2_X1 To SCPSM R__c2_X1 Sum to calculate y 12 .

[0185] For example, the matrix multiplier 10 may use the processing elements included in the first processing element row PER1 to calculate the matrix corresponding to the first input vector (eg, ) output element (e.g., y 11 To 1m ). Similarly, the matrix multiplier 10 may use the processing elements included in the second processing element row PER2 to calculate the matrix corresponding to the second input vector (eg, ) output element (e.g., y 21 To 2m ).

[0186] In some example embodiments, if the product of the number of columns (e.g., m) of the weight matrix WM and the BCQ resolution (e.g., R) is greater than the number of columns (e.g., q) of the processing element array 13, the matrix multiplier 10 may calculate the output elements based on various tiling techniques. Figure 34 to Figure 37 Describes tiling technology in more detail.

[0187] In some example embodiments, the product of the number of columns (e.g., m) of the weight matrix WM and the BCQ resolution (e.g., R) may not be an integer multiple of the number of columns (e.g., q) of the processing element array 13. In this case, some columns of the processing element array 13 may not perform the above-mentioned partial and calculation operations. For example, according to the reference Figures 4 to 11 According to the described example embodiment, the operation efficiency of the processing element array 13 may be deteriorated, and the operation speed of the matrix multiplication device MMD may be deteriorated.

[0188] Fig.12 The diagram shows Fig.11A diagram of the operation of a processing element. Figure 1 and Figures 3 to 12 , the first processing element row PER1 may include processing elements PE11, PE12, PE1R, and PE1(R+1). Below, for a more concise description, operations of the processing elements PE11, PE12, PE1R, and PE1(R+1) are representatively described, but example embodiments are not limited thereto.

[0189] Each processing element included in the first processing element row PER1 may receive a first fixed-point input vector (eg, ). For example, each of the processing elements PE11, PE12, PE1R, and PE1(R+1) may sequentially receive x′ 11 to x′ 1n .

[0190] Each of the processing elements PE11, PE12, PE1R, and PE1(R+1) may receive a different quantized symbol vector. For example, the processing element PE11 may sequentially receive the vectors included in The quantized symbol value in (for example, b 1_1_c1 To b n_1_c1 ). Similarly, processing element PE12 may sequentially receive The quantized symbol value included in 1_2_c1 To b n_2_c1 ), the processing element PE1R can sequentially receive The quantized symbol value included in 1_R_c1 To b n_R_c1 ), processing element PE1(R+1) can sequentially receive The quantized symbol value included in 1_1_c1 To b n_1_c1 ).

[0191] Each of the processing elements PE11, PE12, PE1R, and PE1(R+1) may calculate a different fixed-point part and PSM_fxp based on the order in which the fixed-point input elements and quantized symbol values ​​are received. For example, the processing element PE11 may calculate PSM′ according to the following equation 13: 1__c1_X1 .

[0192] (Equation 13)

[0193] PSM′ 1__c1_X1 = x′ 11 ×b 1_1_c1 +x′ 12 ×b 2_1_c1 +…+x′ 1n ×b n_1_c1

[0194] The method in which another processing element PE calculates the fixed-point part and PSM_fxp is similar to the method in which the processing element PE11 calculates the fixed-point part and PSM_fxp, and thus a detailed description thereof is omitted.

[0195] To describe it more concisely, Fig.12 Various example embodiments are shown in which the same fixed-point input element is individually provided to each processing element included in the first processing element row PER1, but example embodiments are not limited thereto. For example, each processing element included in the first processing element row PER1 may be implemented to sequentially transmit the received fixed-point input element to the processing element adjacent to the row direction. However, example embodiments are not limited thereto.

[0196] In some example embodiments, processing elements arranged in the same column of processing element array 13 may receive the same quantized symbol vector. In this case, each processing element arranged in the same column may be implemented to sequentially transmit the received quantized symbol vector to a processing element adjacent to the column direction. However, example embodiments are not limited thereto.

[0197] Fig.13 is a diagram illustrating implementation according to various example embodiments Fig.12 FIG. 1 is a diagram of a configuration of one of the processing elements of FIG. Figure 1 and Figures 3 to 13 , the processing element PE may include an arithmetic logic unit ALU.

[0198] The arithmetic logic unit ALU may include first to third input terminals TI1-TI3 and an output terminal TO. The first input terminal TI1 may receive a fixed-point input element IE_fxp (eg, x′ 11 ). The second input terminal TI2 may receive the quantization symbol value QSV. The third input terminal TI3 may be connected to the output terminal TO.

[0199] The arithmetic logic unit ALU may output a value obtained by adding a product of values ​​received by the first input terminal TI1 and the second input terminal TI2 and a value received by the third input terminal TI3 to the output terminal TO. However, example embodiments are not limited thereto.

[0200] Fig.14 The diagram shows Figure 6 . Below, for a more concise description, the operation corresponding to the first input vector (e.g., ) of the second data type converter 14. However, example embodiments are not limited thereto.

[0201] Reference Figure 1 and Figures 3 to 14, the second data type converter 14 may receive the first exponent EXP1 from the first data type converter 11. The second data type converter 14 may receive the fixed point part and PSM_fxp from one processing element PE.

[0202] The second data type converter 14 may generate a partial sum PSM by converting the data type of the fixed-point partial sum PSM_fxp into a floating point.

[0203] The second data type converter 14 may add the exponent part EXPP to the fixed point part and PSM_fxp. The second data type converter 14 may determine the exponent part EXPP of the part and PSM as a first exponent EXP1.

[0204] The second data type converter 14 may add a plurality of 0 bits to the LSB bits of the mantissa part of the partial sum PSM according to the difference in code length between the mantissa part of the partial sum PSM and the mantissa part of the fixed point part and PSM_fxp. However, example embodiments are not limited thereto.

[0205] Fig.15 is a diagram illustrating a method according to some example embodiments Figure 1 The operation of the BCQ circuit is shown in Figure 1. Figure 1 , Figure 3 and Fig.15 , the BCQ circuit 200 may perform a binary coded quantization operation on each row of the weight matrix WM.

[0206] First, the weight matrix WM can be expressed as the following Equation 14.

[0207] (Equation 14)

[0208]

[0209] Referring to Equation 14, It can represent the i-th row vector of the weight matrix WM. For example, Can include w i1 tow im .

[0210] The BCQ circuit 200 may perform a binary coded quantization operation on each row of the weight matrix WM based on Equation 15 below.

[0211] (Equation 15)

[0212]

[0213] Referring to Equation 15, R can represent the BCQ resolution. k_ri The k-th quantized scaling coefficient of the i-th row vector of the weight matrix WM may be represented. It can be expressed as corresponding to αk_ri A quantized symbol vector of . For example, The quantization symbol vector corresponding to the k-th quantization scaling coefficient of the ith row vector of the weight matrix WM may be represented.

[0214] Fig.16 The diagram shows Fig.15 An example of an approximate weight matrix is ​​shown in FIG. Figure 1 , Figure 3 , Fig.15 and Fig.16 , quantized symbol vector can be implemented as a row vector with the same dimension as the number of columns of the weight matrix WM. For example, the quantized symbol vector It can be expressed as the following equation 16.

[0215] (Equation 16)

[0216]

[0217] b 1_k_ri to b m_k_ri can represent different quantization symbol values ​​QSV. 1_k_ri to b m_k_ri can be the quantized symbol values ​​of the weights set in different columns of the weight matrix WM. 1_k_ri to b m_k_ri Each of can be "+1" or "-1".

[0218] In this manner, the BCQ circuit 200 may approximate each row vector included in the weight matrix WM with a combination of a plurality of quantization symbol values ​​QSV and a plurality of quantization scaling coefficients QSC.

[0219] For a more concise description, the following representatively describes the calculation of the output element of the first row and first column of the output matrix YM (e.g., y 11 ) of the matrix multiplier 100. However, example embodiments are not limited thereto.

[0220] The BCQ circuit 200 may provide a plurality of quantized symbol values ​​QSV (eg, “b” values) and a plurality of quantized scaling coefficients QSC (eg, “α” values) to the matrix multiplier 100 .

[0221] The matrix multiplier 100 may calculate the output matrix YM based on the plurality of quantized symbol values ​​QSV and the plurality of quantized scaling coefficients QSC. For example, the matrix multiplier 100 may calculate the output matrix YM by multiplying the input matrix XM by the weight matrix WM as described above with reference to equations 14 to 16 and Fig.16 The method described is approximated.

[0222] For a more detailed example, the matrix multiplier 100 may calculate y according to the following equation 17: 11 .

[0223] (Equation 17)

[0224] y 11 ≈x 11 ×(α 1_r1 ×b 1_2_r1 +α 2_r1 ×b 1_2_r1 +…+α R_r1 ×b 1_R_r1 )

[0225] +x 12 ×(α 1_r2 ×b 1_2_r2 +α 2_r2 ×b 1_2_r2 +…+α r_r2 ×b 1_R_r2 )

[0226] +…

[0227] +x 1n ×(α 1_rn ×b 1_2_rn +α 2_rn ×b 1_2_rn +…+α R_rn ×b 1_R_rn )

[0228] In some example embodiments, the matrix multiplier 100 may be implemented to calculate output elements by first calculating the product of the quantized scaling coefficient QSC and the input element (hereinafter referred to as the scaled input element (SCIE)) and then accumulating the calculated product based on the quantized symbol value. In this case, the matrix multiplier 100 may use the same scaled input elements to calculate an output vector (e.g., ) include different output elements. For example, the matrix multiplier 100 can use (α 1_r1 × 11 ) to calculate y 11 To 1n Therefore, according to the embodiment of the present disclosure, the amount of calculation of the matrix multiplier 100 can be reduced or minimized. Figures 16 to 33 A more detailed configuration and operation of the matrix multiplier 100 is described in more detail.

[0229] On the other hand, continue to refer to Fig.16 and Equations 14 to 16, with the first quantized scaling coefficients (e.g., α 1_r1 To α 1_rn ) multiplied by a quantized symbol vector (e.g., to ) can be expressed as a first quantization symbol matrix QSM_1. For example, the first quantization symbol matrix QSM_1 can be expressed as the following equation 18.

[0230] (Equation 18)

[0231]

[0232] In a similar manner, the Rth quantized scaling coefficients (e.g., α R_r1 To α R_rn ) multiplied by a quantized symbol vector (e.g., to ) can be expressed as a quantization symbol matrix QSM_R.

[0233] In some example embodiments, the first to Rth quantization symbol matrices QSM_1-QSM_R may be implemented with the same number of rows and columns as the weight matrix WM. For example, the number of rows of the first quantization symbol matrix QSM_1 may be "n", and the number of columns of the first quantization symbol matrix QSM_1 may be "m". Fig.17 The first to Rth quantization symbol matrices QSM_1-QSM_R are described in more detail.

[0234] Fig.17 The diagram shows Fig.16 Figure 1 shows the quantization symbol matrix of . Figure 1 , Figure 3 and Figures 15 to 17 , the BCQ circuit 200 may perform binary coded quantization (BCQ) on the weight matrix WM into a plurality of quantization symbol matrices QSM and a plurality of quantization scaling coefficients QSC.

[0235] The number of quantization symbol matrices QSM may be determined according to a BCQ resolution, such as R. For example, the BCQ circuit 200 may perform binary coded quantization on the weight matrix WM to generate first to Rth quantization symbol matrices QSM_1 -QSM_R.

[0236] Each of the first to Rth quantization symbol matrices QSM_1-QSM_R may be implemented with the same number of rows and columns as the weight matrix WM. For example, the number of rows of each of the first to Rth quantization symbol matrices QSM_1-QSM_R may be "n", and the number of columns of each of the first to Rth quantization symbol matrices QSM_1-QSM_R may be "m". In this case, the quantization symbol value QSV set in the i-th row and j-th column of the k-th quantization symbol matrix QSM_k may be b j_k_ri .

[0237] Each weight of the weight matrix WM can be approximated based on the quantization symbol value QSV set at the corresponding position of the first to R quantization symbol matrices QSM_1-QSM_R. For example, the weight (e.g., w ij ) can be based on the first to R quantization symbol matrices QSM_1-QSM_R (for example, b j_1_ri To b j_R_ri ) is approximated by the quantized symbol value in the i-th row and j-th column.

[0238] Each of the first to Rth quantization symbol matrices QSM_1-QSM_R may correspond to a plurality of quantization scaling coefficients. Figure 15 to Figure 16 As described above, if the BCQ circuit 200 performs a binary coded quantization operation on each row of the weight matrix WM, different rows of the weight matrix WM can be approximated based on different quantization scaling factors, and weights included in the same row of the weight matrix WM can be approximated based on the same quantization scaling factor.

[0239] For example, all quantization symbol values ​​(eg, b 1_k_ri To b m_k_ri ) can correspond to the quantization scaling factor (α k_ri ). For example, the first to nth rows of the k-th quantization symbol matrix QSM_k may correspond to the quantization scaling coefficients α k_r1 To α k_rn .

[0240] For a more detailed example, the first to nth rows of the first quantization symbol matrix QSM_1 may correspond to α 1_r1 To α 1_rn In this case, all quantization symbol values ​​set in the first row of the first quantization symbol matrix QSM_1 may correspond to α 1_r1 .

[0241] In this way, the weights (e.g., w ij ) may be based on the quantization scaling coefficient QSC (eg, α 1_ri To α R_ri According to various example embodiments, the quantization scaling factor may be defined independently of the number of columns of the weight matrix.

[0242] Therefore, according to the following Equation 19, each of the plurality of weights may be approximated by a plurality of quantization scaling coefficients QSC and a plurality of quantization symbol values ​​QSV.

[0243] (Equation 19)

[0244] w ij ≈α 1_ri ×b j_1_ri +α 2_ri ×b j_2_ri +…+α R_ri ×b j_R_ri

[0245] According to various example embodiments, a weight may be approximated by R 'quantization scaling coefficient (QSC) - quantization symbol value (QSV)' pairs. Figures 18 to 33 The operation of the matrix multiplier 100 based on weights approximated by a plurality of quantization scaling coefficients QSC and a plurality of quantization symbol values ​​QSV is described in more detail.

[0246] Fig.18 is a diagram illustrating an implementation according to some example embodiments Figure 1 A block diagram of the configuration of a matrix multiplier. Figure 1 , Figure 3 and Figures 15 to 18 , the matrix multiplier 100 may include a quantization scaling coefficient buffer 110 , an input vector scaler 120 , a first data type converter 130 , a quantization symbol value buffer 140 , a processing element array 150 , and a second data type converter 160 .

[0247] The quantization scaling coefficient buffer 110 may store a plurality of quantization scaling coefficients QSC provided from the BCQ circuit 200. The quantization scaling coefficient buffer 110 may provide the plurality of quantization scaling coefficients QSC to the input vector scaler 120.

[0248] The input vector scaler 120 may receive an input matrix XM. For example, the input vector scaler 120 may receive a plurality of input vectors (eg, to ).

[0249] The input vector scaler 120 may scale the input matrix XM based on a plurality of quantized scaling coefficients QSC. For example, the input vector scaler 120 may generate a plurality of scaled input vectors SCX based on a plurality of input vectors. In this case, the plurality of scaled input vectors SCX may correspond to a plurality of input vectors (e.g., to ). In the following, corresponding to to The scaled input vectors will be referred to as to

[0250] In some example embodiments, a plurality of scaled input vectors SCX may be included in a scaled input matrix. In this case, the row size of the scaled input matrix may be an integer multiple of the row size of the input matrix XM, and the column size of the scaled input matrix may be the same as the column size of the input matrix XM. However, example embodiments are not limited thereto.

[0251] Each of the plurality of scaled input vectors SCX may be implemented as a row vector having a dimension that is R times the dimension of the corresponding input vector. )'s first scaled input vector (e.g., ) may include R×n scaled input elements, as shown in Equation 20 below.

[0252] (Equation 20)

[0253]

[0254] Referring to Equation 20, a plurality of scaled input elements included in the first scaled input vector may be generated by multiplying each of the plurality of input elements included in the first input vector by an 'R' quantization scaling coefficient QSC.

[0255] In this way, the second to h-th scaled input vectors (e.g., to ) may include R×n scaled input elements. For the sake of simplicity, the second to h-th scaled input vectors (eg, to ) in each of the input elements included in the detailed description. In addition, reference will be made to Fig.19 and Fig. 20 The configuration and operation of the input vector sealer 120 is described in more detail.

[0256] In some example embodiments, the data type of each of the plurality of input elements and the data type of each of the plurality of quantization scaling coefficients QSC may be a floating point. In this case, the data type of each scaled input element may be a floating point. However, example embodiments are not limited thereto.

[0257] In some example embodiments, the code length of each of the plurality of input elements may be 16 bits or 32 bits. However, example embodiments are not limited thereto.

[0258] In some example embodiments, the code length of each of the plurality of quantization scaling coefficients QSC may be 16 bits or 32 bits. However, example embodiments are not limited thereto.

[0259] The first data type converter 130 may receive a plurality of scaled input vectors SCX. For example, the first data type converter 130 may receive first to h-th scaled input vectors (eg, to ). Each of the first to h-th scaled input vectors may include multiple scaled input elements.

[0260] The first data type converter 130 may extract an exponent EXP from each of the plurality of scaled input vectors SCX. For example, the first data type converter 130 may extract an exponent EXP from the first scaled input vector SCX. Extract the first index and the second scaled input vector from Extracting the Second Exponent The first data type converter 130 may provide the extracted exponent EXP to the second data type converter 160 .

[0261] The first data type converter 130 may convert the data types of the plurality of scaled input vectors SCX into fixed point. For example, the first data type converter 130 may receive the plurality of scaled input vectors SCX and may output a plurality of fixed point scaled input vectors SCX_fxp. For example, the first data type converter 130 may generate the first to hth fixed point scaled input vectors (e.g., to ).

[0262] More specifically, the first data type converter 130 may convert the data type of each scaled input element included in the plurality of scaled input vectors SCX into a fixed point based on the extracted exponent. In this case, the first fixed point scaled input vector (e.g., ) may include elements of Equation 20 in fixed-point format. Figures 19 to 21 The configuration and operation of the first data type converter 130 are described in more detail.

[0263] The quantization symbol value buffer 140 may store a plurality of quantization symbol values ​​QSV provided from the BCQ circuit 200. The quantization symbol value buffer 140 may provide a plurality of quantization symbol values ​​QSV to the processing element array 150.

[0264] The processing element array 150 may receive a plurality of quantized symbol values ​​QSV and a plurality of fixed-point scaled input vectors SCX_fxp. The processing element array 150 may generate a fixed-point output matrix YM_fxp based on a plurality of fixed-point scaled input elements (e.g., a fixed-point scaled input vector SCX_fxp) and a plurality of quantized symbol values ​​QSV. The fixed-point output matrix YM_fxp may be expressed as the following equation 21.

[0265] (Equation 21)

[0266]

[0267] Referring to Equation 21, YM_fxp can represent the fixed-point output matrix YM_fxp, and to Can represent different output vectors with fixed-point data types. y′ 11 to y′ hm Each of can represent a different output element with a fixed-point data type.

[0268] The processing element array 150 may include a plurality of processing elements arranged in a row direction and a column direction. Each of the plurality of processing elements may calculate and output a different fixed-point output element of the above equation 21. Figure 22 to Figure 25 A more detailed configuration and operation of each processing element is described in more detail.

[0269] Second data type converter 160 may receive a plurality of exponents EXP from first data type converter 130. Second data type converter 160 may receive a fixed-point output matrix YM_fxp from processing element array 150. For example, second data type converter 160 may receive a plurality of fixed-point output elements from processing element array 150.

[0270] The second data type converter 160 may convert the data type of the fixed-point output matrix YM_fxp into a floating point based on the plurality of exponents EXP. For example, the second data type converter 160 may output the output matrix YM having a floating point data type. For example, the second data type converter 160 may generate a plurality of output elements by converting the data type of each of the plurality of fixed-point output elements received into a floating point. Fig.26 A more detailed configuration and operation of the second data type converter 160 is described in more detail.

[0271] Fig.19 The diagram shows Fig.18 A block diagram of the configuration of the input vector scaler. Figure 1 , Figure 3 and Figures 15 to 19 , the input vector scaler 120 may include first to h-th input vector scaling circuits 121 to 12h.

[0272] The first to h-th input vector scaling circuits 121 to 12h may receive different input vectors, respectively. For example, the first to h-th input vector scaling circuits 121 to 12h may receive the first to h-th input vectors, respectively (eg, to ).

[0273] Each of the first to h-th input vector scaling circuits 121 to 12h may sequentially receive a plurality of input elements. For example, the first input vector scaling circuit 121 may sequentially receive x 11 to x 1n , the hth input vector scaling circuit 12h can sequentially receive x h1 to x hn .

[0274] Each of the first to h-th input vector scaling circuits 121 - 12 h may sequentially receive a plurality of quantization scaling coefficients QSC from the quantization scaling coefficient buffer 110 .

[0275] The quantization scaling coefficients QSC received by the first to h-th input vector scaling circuits 121-12h and provided from the quantization scaling coefficient buffer 110 may be the same. For example, the quantization scaling coefficients QSC sequentially received by the first input vector scaling circuit 121 may be the same as the quantization scaling coefficients QSC sequentially received by the second input vector scaling circuit 122.

[0276] The order in which each of the first to h-th input vector scaling circuits 121 to 12h receives the plurality of quantization scaling coefficients QSC may be the same. For example, the quantization scaling coefficient QSC first received by the first input vector scaling circuit 121 may be the same as the quantization scaling coefficient QSC first received by the second input vector scaling circuit 122. Additionally or alternatively, the quantization scaling coefficient QSC second received by the first input vector scaling circuit 121 may be the same as the quantization scaling coefficient QSC second received by the second input vector scaling circuit 122.

[0277] The first to h-th input vector scaling circuits 121-12h may respectively generate first to h-th scaled input vectors (eg, to ). For example, the first input vector scaling circuit 121 may generate

[0278] Fig. 20 A more detailed map shows Fig.19 The operation of the input vector scaling circuit is shown in FIG. Below, for a more concise description, reference will be made to Figure 1 , Figure 3 and Figures 15 to 20 The operation of the first input vector scaling circuit 121 is representatively described. However, the scope of example embodiments is not limited thereto, and the second to h-th input vector scaling circuits 122 - 12h may operate in a similar manner to the operation of the first input vector scaling circuit 121 .

[0279] The first input vector scaling circuit 121 may receive a first input vector (eg, ). For example, the first input vector scaling circuit 121 may sequentially receive x 11 to x 1n .

[0280] The first input vector scaling circuit 121 may sequentially receive a plurality of quantized scaling coefficients QSC. For example, the first input vector scaling circuit 121 may receive a first row vector corresponding to a weight matrix WM (eg, ) of the quantized scaling factor (e.g., α 1_r1 To α R_r1 ), then may receive a second row vector corresponding to the weight matrix WM (e.g., ) of the quantized scaling factor (e.g., α 1_r2 To α R_r2 In this way, the first input vector scaling circuit 121 can sequentially receive all quantized scaling coefficients QSC (eg, α 1_r1 To α R_rn ).

[0281] In some example embodiments, the second to hth input vector scaling circuits 122-12h may sequentially receive the above-mentioned α 1_r1 To α R_rn .

[0282] In some example embodiments, each of the first to hth input vector scaling circuits 122-12h receives the above-mentioned α 1_r1 To α R_rn However, example embodiments are not limited thereto.

[0283] The first input vector scaling circuit 121 may generate a first scaled input vector based on the order in which the input elements and the plurality of quantized scaling coefficients QSC are received. For example, the first input vector scaling circuit 121 may sequentially multiply the received input elements by the received plurality of quantization scaling coefficients QSC to calculate a plurality of scaled input elements SCIE.

[0284] More specifically, the first input vector scaling circuit 121 can convert x 11 Multiply by the vectors corresponding to the first row (e.g., ) of the quantized scaling factor (e.g., α 1_r1 To α R_r1 ), to sequentially calculate the corresponding 11 A number of scaled input elements SCIE. (Where the calculation is shown as a slash.)

[0285] Thereafter, the first input vector scaling circuit 121 may convert x 12Multiply by the vector corresponding to the second row (for example, ) of the quantized scaling factor (e.g., α 1_r2 To α R_r2 ), to sequentially calculate the corresponding 12 of multiple scaled input elements SCIE (where the calculation is shown as a dot pattern).

[0286] In this way, the first input vector scaling circuit 121 can sequentially calculate the vectors corresponding to x 13 to x 1n Multiple scaled input elements SCIE.

[0287] The first input vector scaling circuit 121 may sequentially output the calculated plurality of scaled input elements SCIE.

[0288] For example, according to various example embodiments, the first input vector scaling circuit 121 may be based on a single input element (eg, x 11 ) generates multiple scaled input elements (e.g., x shown with slashes). 11 SCIE). For example, the first input vector scaling circuit 121 may generate a plurality of scaled input elements by reusing a single input element. Therefore, according to various example embodiments, input reuse of the matrix multiplier 100 may be improved or maximized, so that the number of times the matrix multiplier 100 receives input elements from the outside is reduced or minimized. In this case, the number of times the matrix multiplier 100 accesses an external memory device storing input elements may be reduced or minimized, thereby improving the operating efficiency and operating speed of the matrix multiplication device MMD.

[0289] In some example embodiments, the BCQ circuit 200 may generate a plurality of quantization scaling coefficients QSC and a plurality of quantization symbol values ​​QSV from a plurality of weights based on a uniform BCQ algorithm. In this case, the calculation may be performed with a smaller amount of calculation. Fig. 20 The product of the quantization scaling coefficient QSC and the input element shown. For example, if the size ratio between the multiple quantization scaling coefficients is a power of 2, the mantissas of the multiple quantization scaling coefficients QSC may be the same, and the exponents of the multiple quantization scaling coefficients QSC may differ by "1". In this case, it is easier to perform a single input element (e.g., x 11 ) and different multiple quantization scaling factors (e.g., α 1_r1 To α R_r1 However, example embodiments are not limited thereto.

[0290] Fig.21 The diagram shows Fig.18 A block diagram of a configuration of a first data type converter. Figure 1 , Figure 3 , Figures 15 to 18 and Fig.21 , the first data type converter 130 may include first to h-th exponent extraction circuits 131_1 to 131_h and first to h-th data type conversion circuits 132_1 to 132_h.

[0291] The first to h-th exponent extraction circuits 131_1 to 131_h may receive different scaled input vectors SCX, respectively. For example, the first to h-th exponent extraction circuits 131_1 to 131_h may receive first to h-th scaled input vectors (eg, to ).

[0292] Each of the first to h-th exponent extraction circuits 131_1 to 131_h may extract an exponent from a plurality of scaled input elements SCIE included in the received scaled input vector SCX. For example, the first exponent extraction circuit 131_1 may sequentially receive the scaled input elements SCIE included in the first scaled input vector SCX. The first index extraction circuit 131_1 can extract the first scaled input vector SCIE from the first scaled input vector received. The first exponent EXP1 is extracted from the multiple scaled input elements SCIE in the first exponent extraction circuit 131_2. The second exponent extraction circuit 131_2 can sequentially receive the received multiple scaled input elements SCIE. The second exponent extraction circuit 131_2 can extract the second exponent EXP2 from the received multiple scaled input elements SCIE. In this way, the first to h-th exponent extraction circuits 131_1-131_h can respectively extract the first to h-th exponents EXP1-EXPh.

[0293] The first to hth exponent extraction circuits 131_1 to 131_h may provide the extracted exponents to the first to hth data type conversion circuits 132_1 to 132_h, respectively. In addition, each of the first to hth exponent extraction circuits 131_1 to 131_h may provide each of the extracted exponents to the second data type converter 160.

[0294] The detailed method of extracting the index from the received element by each of the first to h-th index extraction circuits 131_1-131_h is similar to the above reference numeral 131_1. Figures 7 and 8 The operation of the exponent extraction circuit is described, and thus a detailed description thereof is omitted.

[0295] The first to h-th data type conversion circuits 132_1 to 132_h may receive first to h-th exponents EXP1-EXPh, respectively. The first to h-th data type conversion circuits 132_1 to 132_h may receive first to h-th scaled input vectors (eg, to ).

[0296] Each of the first to h-th data type conversion circuits 132_1 to 132_h may convert the data type of the received scaled input vector into a fixed point based on the received exponent. For example, the first to h-th data type conversion circuits 132_1 to 132_h may output first to h-th fixed-point scaled input vectors (e.g., to For example, the first data type conversion circuit 132_1 may convert the input vector (eg, ) converts each scaled input element SCIE in fixed-point format.

[0297] The specific method of converting the data type of the received element into a fixed point based on the received index by each of the first to h-th data type conversion circuits 132_1 to 132_h is similar to that described above with reference to Figure 7 and Fig. 9 The operation of the data type conversion circuit is described, and thus a detailed description thereof is omitted.

[0298] Fig. 22 A more detailed map shows Fig.18 A block diagram of a configuration of a processing element array. Figure 1 , Figure 3 and Figures 15 to 22 , the processing element array 150 may include a plurality of processing elements PE arranged in the row direction and the column direction. Hereinafter, for a more concise description, it is assumed that the plurality of processing elements PE are arranged in h rows and m columns. In addition, the processing element arranged in the i-th row and j-th column of the processing element array 150 will be referred to as "PEij". For example, the processing element arranged in the first row and the second column of the processing element array 150 will be referred to as "PE12".

[0299] The processing element array 150 may include first to hth processing element rows PER1-PERh. Each of the first to hth processing element rows PER1-PERh may include a plurality of processing elements PE. For example, the first processing element row PER1 may include processing elements PE11-PE1m.

[0300] The processing element array 150 may include first to mth processing element columns PEC1-PECm. Each of the first to mth processing element columns PEC1-PECm may include a plurality of processing elements PE. For example, the first processing element column PEC1 may include processing elements PE11-PEh1.

[0301] Different processing element rows may receive different fixed-point input vectors SCX_fxp. For example, the first to h-th processing element rows PER1-PERh may receive first to h-th fixed-point scaled input vectors (eg, to ).

[0302] The processing elements included in the same processing element row may receive the same fixed-point scaled input vector SCX_fxp. For example, each of the processing elements PE11-PE1m may receive a first fixed-point scaled input vector (eg, ).

[0303] Different processing element columns may receive different quantization symbol values ​​QSV. For example, the first to mth processing element columns PEC1-PECm may receive the first to mth quantization symbol values ​​QSVs_1-QSVs_m, respectively.

[0304] Processing elements arranged in the same processing element column may receive the same plurality of quantization symbol values ​​QSV. For example, each of processing elements PE11-PEh1 may receive a first plurality of quantization symbol values ​​QSVs_1, and each of processing elements PE12-PEh2 may receive a second plurality of quantization symbol values ​​QSVs_2.

[0305] Each of the plurality of processing elements PE may calculate a different fixed-point output element based on the received fixed-point scaled input element SCIE and the received plurality of quantized symbol values ​​QSV. For example, according to various example embodiments, one processing element PE may calculate one fixed-point output element. For example, a processing element (PEij) may calculate y′ ij In the following, the fixed-point output elements calculated in each processing element PE will be described in more detail.

[0306] Each of the first to hth processing element rows PER1-PERh may calculate a different fixed-point output vector. For example, the first to hth processing element rows PER1-PERh may respectively output first to hth fixed-point output vectors (eg, to ).

[0307] Processing elements arranged in the same processing element row and different processing element columns can calculate different fixed-point output elements. For example, processing elements PE11-PE1m can calculate y′ respectively. 11 to y′ 1m Similarly, processing elements PE21-PE2m can calculate y′ respectively. 21 to y′ 2m , the processing elements PEh1-PEhm can calculate y′ respectively h1 to y′hm .

[0308] Fig.23 A more detailed map shows Fig. 22 A block diagram of the output of the processing element. Figure 1 , Figure 3 and Figures 15 to 23 , each of the plurality of processing elements PE may provide the calculated fixed-point output element to the second data type converter 160 .

[0309] For a more concise description, some exemplary embodiments will be representatively described below, in which each of the plurality of processing elements PE directly provides the calculated fixed-point output element to the second data type converter 160, such as Fig.23 However, the scope of the example embodiment is not limited thereto, and each of the plurality of processing elements PE may transmit the calculated fixed-point output element to the second data type converter 160 in a systolic array method. Fig.31 and Fig.32 An embodiment of implementing a plurality of processing elements PEs in a systolic array approach is described in more detail.

[0310] Fig.24 A more detailed diagram is shown Fig. 22 For example, in the following, reference will be made to Figure 1 , Figure 3 and Figures 15 to 24 The operation of the first processing element row PER1 is representatively described. However, the scope of example embodiments is not limited thereto, and the second to h-th processing element rows PER2 -PERh may operate in a similar manner to the operation of the first processing element row PER1.

[0311] The first processing element row PER1 may receive a first fixed-point scaled input vector (eg, For example, the first processing element row PER1 may sequentially receive a plurality of fixed-point scaled input elements (SCIE_fxp). For example, the first processing element row PER1 may sequentially receive the above-referenced fixed-point scaled input elements (SCIE_fxp) in fixed-point form. Fig. 20 Describes the scaled input element SCIE.

[0312] Processing elements PE11 - PE1m may receive a first plurality of quantized symbol values ​​QSVs_1 to an mth plurality of quantized symbol values ​​QSVs_m, respectively. For example, processing element PE11 may receive a first plurality of quantized symbol values ​​QSVs_1 and processing element PE12 may receive a second plurality of quantized symbol values ​​QSVs_2.

[0313] The first plurality of quantized symbol values ​​QSVs_1 may include the values ​​set above with reference to Fig.17The quantization symbol values ​​QSV in the first column of the first to R th quantization symbol matrices QSM_1-QSM_R are described. For example, the first plurality of quantization symbol values ​​QSVs_1 may include quantization symbol values ​​(eg, b 1_1_r1 To b 1_R_rn ).

[0314] More specifically, the processing element PE11 may sequentially receive the quantization symbol values ​​(eg, b ) set in the first row and first column of each of the first to Rth quantization symbol matrices QSM_1-QSM_R. 1_1_r1 To b 1_R_r1 ), and then the quantization symbol values ​​(eg, b 1_1_r2 To b 1_R_r2 In this manner, the processing element PE11 may sequentially receive the quantization symbol values ​​(eg, b ) set in the nth row and first column of each of the first to Rth quantization symbol matrices QSM_1-QSM_R. 1_1_rn To b 1_R_rn ).

[0315] The processing element PE11 may calculate a fixed-point output element (eg, y′) based on the order in which the quantized symbol value QSV and a plurality of fixed-point scaled input elements SCIE_fxp are received. 11 ). For example, the processing element PE11 may sequentially accumulate the sum of the sums based on the input element “x 11 "Generate multiple fixed-point scaled input elements SCIE_fxp (for example, in Fig.24 Indicated as "x 11 SCIE_fxp”) is multiplied by the quantization symbol value (for example, b 1_1_r1 To b 1_R_r1 ) and the value obtained. In this way, the processing element PE11 can sequentially accumulate the fixed-point format α 1_r1 × 11 Multiply by b 1_1_r1 The value obtained by passing the fixed-point format α R_rn × 1n Multiply by b 1_R_rn The obtained value is used to calculate and output the fixed-point output element (y′ 11 ).

[0316] Similarly, the processing element PE1j may receive a j-th plurality of quantized symbol values ​​QSVs_j, the j-th plurality of quantized symbol values ​​QSVs_j including quantized symbol values ​​(eg, b ) disposed in the j-th column of each of the first to R-th quantized symbol matrices QSM_1-QSM_R. j_1_r1 To b j_R_rn ). In this case, the processing element PE1j may include the input vector (eg, ) are multiplied by the j-th quantization symbol values ​​QSVs_j, and the multiplied values ​​can be accumulated to calculate and output the fixed-point output element (y′ 1j ).

[0317] Fig.25 is a diagram illustrating a method according to some example embodiments Fig.24 Figure 1 shows the configuration of the processing elements. Figure 1 , Figure 3 and Figures 15 to 25 The processing element PE may include an arithmetic logic unit ALU and an accumulation register REG_ACC.

[0318] The arithmetic logic unit ALU may include first to third input terminals TI1 - TI3 and an output terminal TO.

[0319] The arithmetic logic unit ALU may receive a fixed-point scaled input element SCIE_fxp through a first input terminal TI1. For example, the arithmetic logic unit ALU may sequentially receive a plurality of fixed-point scaled input elements SCIE_fxp through a first input terminal TI1.

[0320] In some example embodiments, each fixed-point scaled input element SCIE_fxp may have a code length of 8 bits, and the arithmetic logic unit ALU may be configured to receive 8-bit data through the first input terminal TI1 .

[0321] The arithmetic logic unit ALU may receive the quantization symbol value QSV through the second input terminal TI2. For example, the arithmetic logic unit ALU may sequentially receive a plurality of quantization symbol values ​​QSV through the second input terminal TI2.

[0322] In some example embodiments, the arithmetic logic unit ALU may receive each quantization symbol value QSV in the form of a control signal indicating a logic low or a logic high through the second input terminal TI2. For example, if the control signal provided to the second input terminal TI2 is a logic high, the arithmetic logic unit ALU may determine that a quantization symbol value QSV indicating "+1" is received. Conversely, if the control signal received through the second input terminal TI2 is a logic low, the arithmetic logic unit ALU may determine that a quantization symbol value QSV indicating "-1" is received. However, example embodiments are not limited thereto.

[0323] The third input terminal TI3 may be connected to the accumulation register REG_ACC. The arithmetic logic unit ALU may receive data stored in the accumulation register REG_ACC through the third input terminal TI3.

[0324] The arithmetic logic unit ALU may calculate a value obtained by adding the product of the quantization symbol value QSV received via the second input terminal TI2 and the fixed-point scaled input element SCIE_fxp received via the first input terminal TI1 to the data received via the third input terminal TI3. ​​The arithmetic logic unit ALU may update the value stored in the accumulation register REG_ACC by providing the calculated value to the accumulation register REG_ACC via the output terminal TO. For example, the arithmetic logic unit ALU may update the value calculated by the following equation 22 in the accumulation register REG_ACC.

[0325] (Equation 22)

[0326] ACC_post=QSVin×SCIE_fxp_in+ACC_pre

[0327] QSVin may refer to a quantized symbol value QSV received by the arithmetic logic unit ALU through the second input terminal TI2, SCIE_fxp_in may refer to a fixed-point scaled input element SCIE_fxp received by the arithmetic logic unit ALU through the first input terminal TI1, ACC_pre may refer to an accumulated value received by the arithmetic logic unit ALU from the accumulation register REG_ACC through the third input terminal TI3, and ACC_post may refer to a value calculated by the arithmetic logic unit ALU and provided to the accumulation register REG_ACC.

[0328] For example, the arithmetic logic unit ALU may accumulate a plurality of fixed-point scaled input elements SCIE_fxp sequentially received through the first input terminal TI1 based on a plurality of quantized symbol values ​​QSV sequentially received through the second input terminal TI2. In this manner, the arithmetic logic unit ALU may calculate a fixed-point output element OE_fxp (e.g., y′) by accumulating a plurality of fixed-point scaled input elements SCIE_fxp in the accumulation register REG_ACC based on the plurality of quantized symbol values ​​QSV. 11 ).

[0329] For example, the arithmetic logic unit ALU may store the fixed-point output element OE_fxp in the accumulation register REG_ACC. The accumulation register REG_ACC may provide the fixed-point output element OE_fxp to the second data type converter 160.

[0330] In some example embodiments, the fixed-point output element OE_fxp may have a code length of 8 bits or more. For example, the fixed-point output element OE_fxp may have a code length long enough to represent the cumulative size of multiple fixed-point scaled input elements SCIE_fxp.

[0331] In some example embodiments, the accumulation register REG_ACC may have a size of 8 bits or more. For example, the accumulation register REG_ACC may have a size large enough to store the fixed-point output element OE_fxp.

[0332] In some example embodiments, the fixed-point output element OE_fxp may have a code length of 10 to 12 bits. However, example embodiments are not limited thereto.

[0333] In some example embodiments, the accumulation register REG_ACC may have a size of 10 bits to 12 bits. However, example embodiments are not limited thereto.

[0334] In some example embodiments, a plurality of fixed-point scaled input elements SCIE_fxp received by the arithmetic logic unit ALU may correspond to the same exponent value. In this case, the arithmetic logic unit ALU may perform the calculation of equation 22 above, even without considering the bit value of each of the plurality of fixed-point scaled input elements SCIE_fxp. Therefore, the arithmetic logic unit ALU may calculate the fixed-point output element OE_fxp with a reduced or minimal amount of calculation.

[0335] Fig.26 The diagram shows Fig.18. Below, for a more concise description, the operation of the second data type converter 160 of one fixed-point output element OE_fxp will be representatively described. However, example embodiments are not limited thereto. For example, the operation of the second data type converter 160 for any fixed-point output element OE_fxp may be similar to the operation of the second data type converter 160 for one fixed-point output element OE_fxp.

[0336] Reference Figure 1 , Figure 3 and Figures 15 to 26 , the second data type converter 160 may receive the fixed-point output element OE_fxp. For example, the second data type converter 160 may receive y′ 11 to y′ hn one of the.

[0337] The second data type converter 160 may receive the first to h-th exponents EXP1-EXPh from the first data type converter 130. In this case, the first to h-th exponents EXP1-EXPh may correspond to the first fixed-point output vector (eg, ) to the hth fixed-point output vector (e.g., ).

[0338] The fixed-point output element OE_fxp may include a sign part SP and a mantissa part MTSP. The second data type converter 160 may generate an output element OE by converting the data type of the fixed-point output element OE_fxp into a floating point. Hereinafter, a specific method of converting the data type of the fixed-point output element OE_fxp by the second data type converter 160 is described.

[0339] The second data type converter 160 may add the exponent part EXPP to the fixed-point output element OE_fxp. For example, the second data type converter 160 may determine the exponent corresponding to the fixed-point output element OE_fxp among the exponents received from the first data type converter 130 as the exponent part EXPP of the output element OE. For a more detailed example, if the fixed-point output element OE_fxp is included in the first fixed-point output vector (e.g., ) (for example, if the fixed-point output element OE_fxp is y′ 11 to y′ 1n ), the exponent part EXPP of the output element OE can be determined as the first exponent EXP1. Similarly, if the fixed-point output element OE_fxp is included in the second fixed-point output vector , the exponent part EXPP of the output element OE can be determined as the second exponent EXP2.

[0340] The second data type converter 160 may add a plurality of '0' bits to the LSB bits of the mantissa part of the output element OE according to the difference in code length between the mantissa part of the fixed-point output element OE_fxp and the mantissa part of the output element OE. However, example embodiments are not limited thereto.

[0341] Fig. 27 The diagram shows Figure 1 Flow chart of the operation of the matrix multiplication device. Figure 1 , Figure 3 and Figures 15 to 27 Describes a method that takes an input vector (e.g., ) and output an output vector (e.g., )'s matrix multiplication device MMD operation.

[0342] In step S110, the matrix multiplication device MMD may receive a weight matrix WM. For example, the BCQ circuit 200 may receive a plurality of weights (eg, w 11 tow nm ).

[0343] In step S120, the matrix multiplication device MMD may generate a plurality of quantization scaling coefficients QSC and a plurality of quantization symbol values ​​QSV by performing binary coded quantization on the weight matrix WM. For example, the BCQ circuit 200 may convert each of the plurality of weights into two or more "quantization scaling coefficient (QSC)-quantization symbol value (QSV)" pairs. The BCQ circuit 200 may provide the generated plurality of quantization scaling coefficients QSC and a plurality of quantization symbol values ​​QSV to the matrix multiplier 100. In this case, the plurality of quantization scaling coefficients QSC may be stored in the quantization scaling coefficient buffer 110, and the plurality of quantization symbol values ​​QSV may be stored in the quantization symbol value buffer 140. However, example embodiments are not limited thereto.

[0344] In step S130, the matrix multiplication device MMD may receive an input vector (eg, For example, the matrix multiplier 100 may receive a plurality of input elements (eg, x 11 to x 1n ).

[0345] In some example embodiments, the matrix multiplication device MMD may perform step S130 regardless of the order of steps S110 and S120. For example, the matrix multiplication device MMD may perform step S130 before steps S110 and S120, or may perform step S130 between steps S110 and S120.

[0346] In step S140 , the matrix multiplication device MMD may generate a scaled input vector SCX by scaling the input vector based on a plurality of quantization scaling coefficients QSC. For example, the input vector scaler 120 may scale each of the plurality of input elements based on a plurality of quantization scaling coefficients QSC provided from the quantization scaling coefficient buffer 110 .

[0347] In step S150, the matrix multiplication device MMD may generate an output vector (eg, For example, the matrix multiplier 100 may generate an output element (eg, y) by accumulating elements of the scaled input vector SCX based on the first plurality of quantized symbol values ​​QSVs_1. 11 ). The matrix multiplier 100 may generate another output element (eg, y) by accumulating the elements of the scaled input vector SCX based on the second plurality of quantized symbol values ​​QSVs_2. 12 ). In this manner, the matrix multiplier 100 may generate a plurality of output elements included in an output vector.

[0348] Fig.28 A more detailed map shows Fig. 27 Flow chart of step S150. Figure 1 , Figure 3 and Figures 15 to 28 , step S150 may include steps S151 to S153.

[0349] In step S151, the matrix multiplier 100 may convert the data type of the scaled input vector SCX into a fixed point. For example, the matrix multiplier 100 may generate a fixed-point scaled input vector SCX_fxp based on the scaled input vector SCX. For example, the first data type converter 130 may convert the data type of each of the plurality of scaled input elements SCIE into a fixed point to generate a plurality of fixed-point scaled input elements SCIE_fxp.

[0350] In step S152, the matrix multiplier 100 may generate a fixed-point output vector (eg, SCX_fxp) by accumulating elements of the scaled input vector SCX (eg, the fixed-point scaled input vector SCX_fxp) converted into a fixed-point data type based on a plurality of quantized symbol values ​​QSV. ). For example, the processing element array 150 may generate a fixed-point output vector by accumulating a plurality of fixed-point scaled input elements SCIE_fxp based on a plurality of quantized symbol values ​​QSV. More specifically, the processing element array 150 may generate a fixed-point output element (e.g., y′) by accumulating a plurality of fixed-point scaled input elements SCIE_fxp based on a first plurality of quantized symbol values ​​QSVs_1. 11Similarly, the processing element array 150 may generate a fixed-point output element (eg, y′) by accumulating a plurality of fixed-point scaled input elements SCIE_fxp based on the second plurality of quantized symbol values ​​QSVs_2. 12 ). In this manner, processing element array 150 may compute a fixed-point output vector (e.g., ) includes multiple fixed-point output elements OE_fxp.

[0351] In step S153, the matrix multiplier 100 may output a fixed-point vector (eg ) is converted into a floating point. For example, the second data type converter 160 may receive a plurality of fixed-point output elements OE_fxp included in the fixed-point output vector. The second data type converter 160 may convert the plurality of fixed-point output elements OE_fxp into a plurality of output elements OE, respectively.

[0352] Fig.29 The diagram shows Figure 1 Flow chart of the operation of the matrix multiplication device. Figure 1 , Figure 3 , Figures 15 to 26 and Fig.29 Describes a method that takes an input vector (e.g., ) to generate an output element (e.g., y 11 )'s matrix multiplication device MMD operation.

[0353] In step S210, the matrix multiplication device MMD may receive first to nth weights. For example, the BCQ circuit 200 may receive weights corresponding to a column of the weight matrix WM (eg, w 11 tow n1 ).

[0354] In step S220, the matrix multiplication device MMD may generate first to (n×R)th quantization scaling coefficients QSC and first to (n×R)th quantization symbol values ​​QSV by binary-coding quantization of the first to nth weights. For example, the BCQ circuit 200 may generate R quantization scaling coefficients QSC and R quantization symbol values ​​QSV for each weight.

[0355] In step S230, the matrix multiplication device MMD may receive first to nth input elements. For example, the matrix multiplier 100 may receive a plurality of input elements (eg, x 11 to x 1n ).

[0356] In some example embodiments, the matrix multiplication device MMD may perform step S230 regardless of the order of steps S210 and S220. For example, the matrix multiplication device MMD may perform step S230 before steps S210 and S220, or may perform step S230 between steps S210 and S220.

[0357] In step S240, the matrix multiplication device MMD may generate first to n×R scaled input elements SCIE by scaling the first to nth input elements based on the first to (n×R)th quantization scaling coefficients QSC. For example, the input vector sealer 120 may generate R scaled input element SCIEs for each single input element based on the R quantization scaling coefficients QSC.

[0358] In step S250, the matrix multiplication device MMD may generate an output element (eg, y) by accumulating the first to (n×R)th scaled input elements SCIE based on the first to (n×R)th quantized symbol values ​​QSV. 11 For example, the matrix multiplier 100 may accumulate products (eg, b ) of the first to (n×R)th scaled input elements SCIE and the first to (n×R)th quantized symbol values ​​QSV. 1_1_r1 To b 1_R_rn ) to generate an output element (e.g., y 11 ).

[0359] Fig.30 A more detailed map shows Fig.29 Flow chart of step S250. Figure 1 , Figure 3 , Figures 15 to 26 and Figure 29 to Figure 30 , step S250 may include steps S251 to S253.

[0360] In step S251, the matrix multiplier 100 may convert the data type of each of the first to (n×R) scaled input elements SCIE into a fixed point. For example, the first data type converter 130 may convert the first to (n×R) scaled input elements SCIE into first to (n×R) fixed-point scaled input elements SCIE_fxp, respectively.

[0361] In step S252, the matrix multiplier 100 may generate a fixed-point output element OE_fxp by accumulating the first to (n×R)th fixed-point scaled input elements SCIE_fxp based on the first to (n×R)th quantization symbol values ​​QSV. For example, one processing element PE may sequentially receive the first to (n×R)th quantization symbol values ​​QSV, and may sequentially receive the first to (n×R)th fixed-point scaled input elements SCIE_fxp. The processing element PE may generate a fixed-point output element OE_fxp (e.g., y′) by accumulating the product of the first to (n×R)th quantization symbol values ​​QSV and the first to (n×R)th fixed-point scaled input elements SCIE_fxp. 11 ).

[0362] In step S253, the matrix multiplier 100 may convert the data type of the fixed-point output element OE_fxp into a floating point. For example, the second data type converter 160 may receive the fixed-point output element OE_fxp (eg, y′ 11 ), and can output output elements OE (e.g., y 11 ).

[0363] Fig.31 is a diagram showing the implementation of a systolic array method Fig. 22 A block diagram of an array of processing elements. Figure 1 , Figure 3 , Figures 15 to 22 and Fig.31 , the processing element array 150 may be implemented as Fig.31 The processing element array 250 is provided.

[0364] The processing element array 250 may include a plurality of processing elements PE arranged in row and column directions. The plurality of processing elements PE may operate in a systolic array scheme.

[0365] The processing element array 250 may be implemented to sequentially propagate a plurality of fixed-point scaled input elements SCIE_fxp in a row direction. For example, the first processing element row PER1 may sequentially propagate an input vector (eg, ) in multiple fixed-point scaled input elements SCIE_fxp.

[0366] For a more detailed example, the processing element PE11 may receive one fixed-point scaled input element SCIE_fxp at a first time point. The processing element PE11 may transmit the fixed-point scaled input element SCIE_fxp to the processing element PE12 disposed adjacent to the processing element PE11 in the row direction at a second time point after the first time point. In this way, the processing elements PE included in the first processing element row PER1 may sequentially transmit a plurality of fixed-point scaled input elements SCIE_fxp provided from the first data type converter 130 to adjacent processing elements.

[0367] The processing element array 250 may be implemented to sequentially propagate a plurality of quantized symbol values ​​QSV in a column direction. For example, the first processing element column PEC1 may sequentially propagate a first plurality of quantized symbol values ​​QSVs_1 in a column direction.

[0368] For a more detailed example, the processing element PE11 may receive one quantization symbol value QSV at a first time point. The processing element PE11 may transmit the quantization symbol value QSV to the processing element PE21 disposed adjacent to the processing element PE11 in the column direction at a second time point after the first time point. In this way, the processing elements PE included in the first processing element column PEC1 may sequentially transmit the first plurality of quantization symbol values ​​QSVs_1 provided from the quantization symbol value buffer 140 to the adjacent processing elements.

[0369] Each of the plurality of processing elements PE may be configured as described above. Figures 18 to 24 The described method generates different fixed-point output elements OE_fxp in a similar manner. For a more concise description, a detailed description of the method for each of the multiple processing elements PE to generate the fixed-point output element OE_fxp is omitted.

[0370] The processing element array 250 may be implemented to sequentially propagate the fixed-point output element OE_fxp in the column direction. For example, the fixed-point output element OE_fxp (eg, y′) calculated from the processing element PE11 11 ) can be sequentially propagated in the column direction. In this way, the fixed-point output element OE_fxp can be transmitted to the second data type converter 160. Since the method of propagating the fixed-point output element OE_fxp is similar to the method of propagating the quantization symbol value QSV, its detailed description will be omitted.

[0371] For example, each processing element included in the processing element array 250 may be implemented to receive one or more of the fixed-point output element OE_fxp, the quantized symbol value QSV, and the fixed-point scaled input element SCIE_fxp from an adjacent processing element. Conversely, each processing element included in the processing element array 250 may be implemented to transmit one or more of the fixed-point output element OE_fxp, the quantized symbol value QSV, and the fixed-point scaled input element SCIE_fxp to an adjacent processing element. Fig.32 A more detailed configuration of the processing element PE operating in the systolic array method is described in more detail.

[0372] For a more concise description, Fig.31 200 , each of the fixed-point output element OE_fxp, the quantization symbol value QSV, and the fixed-point scaled input element SCIE_fxp is representatively shown in an embodiment in which each of the fixed-point output element OE_fxp, the quantization symbol value QSV, and the fixed-point scaled input element SCIE_fxp is propagated in a systolic array method, but the scope of the example embodiments is not limited thereto. For example, the processing element array 250 may be implemented to propagate only one or two of the fixed-point output element OE_fxp, the quantization symbol value QSV, and the fixed-point scaled input element SCIE_fxp propagated in a systolic array method.

[0373] In some example embodiments, each processing element included in the processing element array 250 may operate in response to the same control clock signal. In this case, each of the plurality of processing elements may transmit a fixed-point output element OE_fxp, a quantized symbol value QSV, and / or a fixed-point scaled input element SCIE_fxp to another processing element at the same time point. However, example embodiments are not limited thereto.

[0374] Fig.32 A more detailed map shows Fig.31 Figure 1 shows the configuration of the processing elements. Figure 1 , Figure 3 , Figures 15 to 22 and Figure 31 to Figure 32 The processing element PE may include an arithmetic logic unit ALU, an accumulation register REG_ACC, a scaled input element register REG_SCIE and a quantization symbol value register REG_QSV. Fig.25 A detailed description of the configuration and operation of the arithmetic logic unit ALU and the accumulator register REG_ACC is described.

[0375] Below, some example embodiments in which the arithmetic logic unit ALU, the accumulation register REG_ACC, the scaled input element register REG_SCIE, and the quantization symbol value register REG_QSV operate in response to the same control clock signal are representatively described. However, example embodiments are not limited thereto.

[0376] The scaled input element register REG_SCIE can sequentially receive multiple fixed-point scaled input elements SCIE_fxp. The scaled input element register REG_SCIE can receive one fixed-point scaled input element SCIE_fxp, and can transmit the received fixed-point scaled input element SCIE_fxp to the processing element PE adjacent to the scaled input element register REG_SCIE and the first input terminal TI1 after one cycle of the control clock signal has passed.

[0377] The quantization symbol value register REG_QSV can sequentially receive a plurality of quantization symbol values ​​QSV. The quantization symbol value register REG_QSV can receive one quantization symbol value QSV and can transfer the received quantization symbol value QSV to an adjacent processing element and a second input terminal TI2 after one cycle of the control clock signal has passed.

[0378] The accumulation register REG_ACC can store the fixed-point output element OE_fxp provided from the arithmetic logic unit ALU. For example, the accumulation register REG_ACC can be used in the same manner as in the above reference. Fig.25 The fixed-point output element OE_fxp calculated by the arithmetic logic unit ALU is stored in a manner similar to that described.

[0379] The accumulation register REG_ACC may transfer the fixed-point output element OE_fxp to an adjacent processing element and / or the second data type converter 160. For example, the accumulation register REG_ACC may transfer the fixed-point output element OE_fxp to an accumulation register of an adjacent processing element. More specifically, the fixed-point output element OE_fxp (e.g., y′) calculated by the processing element PE11 may be transferred to the accumulation register of the adjacent processing element. h1 ) may be sequentially transmitted through the processing elements PE21-PEh1 to the second data type converter 160. However, example embodiments are not limited thereto.

[0380] To describe it more concisely, Fig.32 An embodiment is described in which each register included in the processing element PE receives and outputs data at each cycle of the control clock signal, but example embodiments are not limited to a specific operation method of the register in response to the control clock signal.

[0381] Fig.33 is a diagram illustrating a method according to some example embodiments Fig.19 FIG. 1 is a diagram of the operation of the input vector scaling circuit. Figure 1 , Figure 3 , Figures 15 to 19 and Fig.33, the first input vector scaling circuit 121 may be implemented as the first input vector scaling circuit 221. However, the scope of the example embodiments is not limited thereto, and the second to h-th input vector scaling circuits 122-12h may operate in a manner similar to the operation of the first input vector scaling circuit 221. Below, the difference between the first input vector scaling circuit 121 and the first input vector scaling circuit 221 will be mainly described.

[0382] The first input vector scaling circuit 221 may repeatedly receive a first input vector (eg, ). For example, the first input vector scaling circuit 221 may repeatedly receive x 11 to x 1n .

[0383] The first input vector scaling circuit 221 may sequentially receive a plurality of quantization scaling coefficients QSC. For example, the first input vector scaling circuit 221 may receive a quantization scaling coefficient (eg, α 1_r1 To α 1_rn ), and then the quantization scaling coefficients (eg, α 2_r1 To α 2_rn ). In this way, the first input vector scaling circuit 221 can sequentially receive all quantized scaling coefficients QSC for calculating the above-mentioned scaled input elements SCIE.

[0384] The first input vector scaling circuit 221 may generate a first scaled input vector based on the order in which the input elements and the plurality of quantized scaling coefficients QSC are received. For example, the first input vector scaling circuit 221 may sequentially calculate a plurality of scaled input elements SCIE by sequentially multiplying the received input elements by the received plurality of quantization scaling coefficients QSC. Because the sequential operation of the first input vector scaling circuit 221 is similar to that of the reference vector scaling circuit 221, Fig. 20 The operation of the first input vector scaling circuit 121 is described herein, so a detailed description thereof will be omitted.

[0385] In some example embodiments, if the input vector scaler 120 is implemented based on the first input vector scaling circuit 221 described above, the processing element array 150 may be implemented in the same manner as described above with reference to Fig.24 For example, the first processing element column PEC1 may receive the quantized symbol values ​​QSV in a different order than previously described. Fig.24 For a more detailed example, the processing element PE11 may sequentially receive the quantization symbol values ​​(eg, b ) arranged in the first column of the first quantization symbol matrix QSM_1. 1_1_r1 To b1_1_rn ), and then the quantization symbol values ​​(eg, b 1_2_r1 To b 1_2_rn ). For example, the order in which the processing element array 150 receives the quantized symbol values ​​QSV may be determined according to the order in which the input vector sealer 120 receives the quantized scaling coefficients QSC.

[0386] Fig.34 is a diagram illustrating a method according to some example embodiments Figure 1 FIG. 1 is a diagram of the operation of the matrix multiplication device. Figures 1 to 3 and Figures 15 to 34 , the matrix multiplication device MMD can receive a full input matrix FXM. The full input matrix FXM can include reference Figures 1 to 34 For example, the full input matrix FXM may include multiple input matrices XM.

[0387] The matrix multiplication device MMD may receive a full weight matrix FWM. The full weight matrix FWM may include reference Figures 1 to 34 The weight matrix WM described. For example, the full weight matrix FWM may include multiple weight matrices WM.

[0388] The BCQ circuit 200 may perform a binary coded quantization operation on each of the plurality of weight matrices WM included in the full weight matrix FWM. For example, the BCQ circuit 200 may generate a plurality of quantized symbol values ​​QSV and a plurality of quantized scaling coefficients QSC from each of the plurality of weight matrices WM.

[0389] The matrix multiplier 100 may receive a plurality of quantization symbol values ​​QSV and a plurality of quantization scaling coefficients QSC. The matrix multiplier 100 may perform matrix multiplication on a full input matrix FXM and a full weight matrix FWM based on the plurality of quantization symbol values ​​QSV and the plurality of quantization scaling coefficients QSC.

[0390] The matrix multiplier 100 can perform matrix multiplication on the full input matrix FXM and the full weight matrix FWM through one of various tiling techniques. For example, the matrix multiplier 100 can calculate the full output matrix FYM by sequentially calculating the products of multiple input matrices XM and multiple weight matrices WM and then combining the calculation results.

[0391] Fig.35 The diagram shows Fig.34 A graph of the full input matrix of . Figures 1 to 3 and Figures 15 to 35, the full input matrix FXM may include a plurality of input matrices XM. For example, the full input matrix FXM may be tiled with a plurality of input matrices XM arranged in the row direction and the column direction. In the following, for a more concise description, the input matrix arranged in the i-th row and the j-th column of the full input matrix FXM will be referred to as "XM_ij".

[0392] In some example embodiments, the above reference Figures 1 to 3 and Figures 15 to 33 The described input matrix XM may be one of a plurality of input matrices XM included in the full input matrix FXM.

[0393] In some example embodiments, the multiple input matrices XM included in the full input matrix FXM may have the same row size and column size. For example, each of the multiple input matrices XM may include n input elements per row. Each of the multiple input matrices XM may include h input elements per column.

[0394] The row size of the full input matrix FXM can be an integer multiple of the row size of each of the multiple input matrices XM. For example, a row of the full input matrix FXM can include N input elements. In this case, "N" can be an integer multiple of "n".

[0395] The column size of the full input matrix FXM may be an integer multiple of the column size of each of the multiple input matrices XM. For example, a column of the full input matrix FXM may include H input elements. In this case, "H" may be an integer multiple of "h".

[0396] Fig.36 The diagram shows Fig.34 A graph of the full weight matrix of . Figures 1 to 3 and Figures 15 to 36 , the full weight matrix FWM may include multiple weight matrices WM. For example, the full weight matrix FWM may be tiled with multiple weight matrices WM arranged in row and column directions. In the following, for a more concise description, the weight matrix arranged in the i-th row and j-th column of the full weight matrix FWM will be referred to as "WM_ij".

[0397] In some example embodiments, the above reference Figures 1 to 3 and Figures 15 to 33 The described weight matrix WM may be one of a plurality of weight matrices WM included in the full input matrix FXM.

[0398] In some example embodiments, each of the multiple weight matrices WM included in the full weight matrix FWM may have the same row size and column size. For example, each of the multiple weight matrices WM may include m weights per row. Each of the multiple weight matrices WM may include n weights per column.

[0399] The row size of the full weight matrix FWM may be an integer multiple of the row size of each of the plurality of weight matrices WM. For example, a row of the full weight matrix FWM may include M weights. In this case, "M" may be an integer multiple of "m".

[0400] The column size of the full weight matrix FWM may be an integer multiple of the column size of each of the plurality of weight matrices WM. For example, one column of the full weight matrix FWM may include N weights. In this case, "N" may be an integer multiple of "n".

[0401] The BCQ circuit 200 may perform a binary coded quantization operation on each of the plurality of weight matrices WM included in the full weight matrix FWM. In this case, the quantization scaling coefficient generated based on the weight matrix WM_11 may be different from the quantization scaling coefficient generated based on the weight matrix WM_12. Similarly, the quantization symbol value generated based on the weight matrix WM_11 may be different from the quantization symbol value generated based on the weight matrix WM_12. The specific method in which the BCQ circuit 200 performs a binary coded quantization operation on each weight matrix WM is similar to the method described above with reference to Figures 15 to 17 The method described herein is omitted in detail.

[0402] Fig.37 The diagram shows Fig.34 A diagram of the full output matrix of . Figures 1 to 3 and Figures 15 to 37 , the full output matrix FYM can correspond to the product of the full input matrix FXM and the full weight matrix FWM.

[0403] The full output matrix FYM may include a plurality of sub-matrices FYM_sub arranged along the row direction and the column direction. Hereinafter, for a more concise description, the sub-matrix arranged in the i-th row and j-th column of the full output matrix FYM will be referred to as "FYM_sub_ij".

[0404] Each of the multiple submatrices FYM_sub can have the same row size and column size. The row size of each of the multiple submatrices FYM_sub can be the same as the row size of the input matrix XM. The column size of each of the multiple submatrices FYM_sub can be the same as the column size of the weight matrix WM. For example, each of the multiple submatrices FYM_sub can include m output elements per row. Each of the multiple submatrices FYM_sub can include h output elements per column.

[0405] The row size of the full output matrix FYM may be the same as the row size of the full weight matrix FWM. For example, the row size of the full output matrix FYM may be "M".

[0406] The column size of the full output matrix FYM may be the same as the column size of the full input matrix FXM. For example, the column size of the full output matrix FYM may be "H".

[0407] The matrix multiplication device MMD can operate the full output matrix FYM through the sub-matrix FYM_sub. For example, the matrix multiplication device MMD can calculate a sub-matrix FYM_sub by adding the products of the tiled multiple input matrices XM and the tiled multiple weight matrices WM.

[0408] For a more detailed example, if N is 3 times of n, the matrix multiplier 100 may calculate the submatrix FYM_sub_11 by sequentially calculating the product of the input matrix XM_11 and the weight matrix WM_11, the product of the input matrix XM_12 and the weight matrix WM_21, and the product of the input matrix XM_13 and the weight matrix WM_31, and then adding the calculated products. In this case, the product of the tiled input matrix and the tiled weight matrix may correspond to the above reference Figures 1 to 3 and Figures 15 to 33 The output matrix YM described. For example, the matrix multiplication device MMD can calculate the first output matrix based on the product of the input matrix XM_11 and the weight matrix WM_11; the second output matrix can be calculated based on the product of the input matrix XM_12 and the weight matrix WM_21; and the third output matrix can be calculated based on the product of the input matrix XM_13 and the weight matrix WM_31. Thereafter, the matrix multiplication device MMD can calculate the submatrix FYM_sub_11 by adding the above-mentioned first to third output matrices. However, example embodiments are not limited thereto.

[0409] For example, the matrix multiplication device MMD can be implemented to accumulate multiple output matrices to calculate a submatrix (e.g., a portion of the full output matrix FYM). For example, the matrix multiplication device MMD can be implemented to temporarily store multiple output matrices in an external volatile memory device (e.g., an SRAM device), and then accumulate the stored multiple output matrices to calculate a submatrix. In this way, the matrix multiplication device MMD can calculate the full output matrix FYM by sequentially calculating multiple submatrices FYM_sub.

[0410] Fig.38 is a block diagram illustrating a matrix multiplication apparatus according to some example embodiments. Fig.38 , the matrix multiplication device MMD may include a matrix multiplier array 1100 , a BCQ circuit 1200 and an output matrix accumulator 3000 .

[0411] The BCQ circuit 1200 may receive a weight matrix WM. The BCQ circuit 1200 may generate a plurality of quantization symbol matrices QSM and a plurality of quantization scaling coefficients QSC by binary-coding quantizing the weight matrix WM at a BCQ resolution “R”. Since the operation of the BCQ circuit 1200 is similar to that described above with reference to Figures 1 to 3 and Figures 15 to 33 The operation of the BCQ circuit 200 is described, so a detailed description thereof will be omitted.

[0412] In the following, for a more concise description, the quantization scaling coefficients corresponding to the first to Rth quantization symbol matrices QSM_1-QSM_R will be referred to as the first to Rth plurality of quantization scaling coefficients QSCs_1-QSCs_R, respectively. For example, the first plurality of quantization scaling coefficients QSCs_1 may refer to Fig.24 Similarly, the second plurality of quantization scaling coefficients QSCs_2 may refer to Fig.24 The quantization scaling factors are shown as dots in FIG.

[0413] The matrix multiplier array 1100 may include first to Rth matrix multipliers 1110 - 11R0 .

[0414] Each of the first to R-th matrix multipliers 1110 - 11R0 may receive an input matrix XM. For example, each of the first to R-th matrix multipliers 1110 - 11R0 may receive the same input matrix XM.

[0415] The first to R-th matrix multipliers 1110-11R0 may receive first to R-th quantization scaling coefficients QSCs_1-QSCs_R, respectively. The first to R-th matrix multipliers 1110-11R0 may receive first to R-th quantization symbol matrices QSM_1-QSM_R, respectively.

[0416] Each of the first to R-th matrix multipliers 1110-11R0 may be configured as described above with reference to Figures 1 to 3 and Figures 15 to 33 The method described above is implemented in a similar manner. For example, the first matrix multiplier 1110 can generate a first sub-output matrix YM_sub_1 by scaling the input matrix XM based on a plurality of quantization scaling coefficients QSCs_1 and then accumulating the scaled input matrix based on the first quantization symbol matrix QSM_1. In this way, the first to R-th matrix multipliers 1110-11R0 can generate the first to R-th sub-output matrices YM_sub_1-YM_sub_R, respectively.

[0417] The output matrix accumulator 1300 may receive the first to Rth sub-output matrices YM_sub_1-YM_sub_R. The output matrix accumulator 1300 may generate an output matrix YM by adding the first to Rth sub-output matrices YM_sub_1-YM_sub_R.

[0418] For example, according to Fig.35 In an embodiment, the output matrix can be calculated in such a way that a plurality of matrix multipliers respectively generate sub-output matrices based on different quantization symbol matrices, and then accumulate the generated sub-output matrices. In this case, even if the weight matrix WM is quantized with high BCQ resolution binary coding, a plurality of matrix multipliers operate in parallel on the sub-output matrices corresponding to different quantization symbol matrices. Therefore, the operating speed of the matrix multiplication device MMD can be improved.

[0419] To describe it more concisely, Fig.38 , the matrix multiplier array 1100 includes a number of matrix multipliers corresponding to the BCQ resolution (e.g., R), but the scope of the example embodiments is not limited thereto. For example, the operation of each matrix multiplier based on multiple quantization symbol matrices may be similar to that described above with reference to Figures 1 to 3 and Figures 15 to 33 Describes the operation.

[0420] Fig.39 is a block diagram showing a neural processing system implemented according to some example embodiments. Fig.39 , the neural processing system 2000 may include a central processing unit 2100, a neural processing unit 2200, a volatile memory device 2300, a non-volatile memory device 2400, and a user interface 2500. The central processing unit 2100, the neural processing unit 2200, the volatile memory device 2300, the non-volatile memory device 2400, and the user interface 2500 may be connected to each other via a bus BUS.

[0421] The central processing unit 2100 may control the overall operation of the neural processing system 2000. For example, the central processing unit 2100 may control each component of the neural processing system 2000 to operate an artificial intelligence model.

[0422] In some example embodiments, the artificial intelligence model executed by the neural processing system 2000 may be one of any type of artificial intelligence model, such as a language model, an image recognition model, an image generation model, a weather analysis model, or the like. For example, the artificial intelligence model executed by the neural processing system 2000 may be one of any type of artificial intelligence model, such as GPT-3, GPT-4, Pangu, GShard, Megatron-LM, or the like. However, example embodiments are not limited thereto.

[0423] In some example embodiments, the artificial intelligence model executed by the neural processing system 2000 may perform reasoning operations and / or training operations. However, example embodiments are not limited thereto.

[0424] Each artificial intelligence model may include multiple processing layers. Each of the multiple processing layers may be implemented to receive layer input data to generate layer output data. In this case, the generated layer output data may be used as layer input data for another processing layer. For example, the layer output data generated from the first processing layer may be used as layer input data for the second processing layer. Fig.40 Describes a more detailed description of the AI ​​model and processing layers.

[0425] Each of the multiple processing layers can convert layer input data into layer output data based on matrix multiplication calculations. For example, each of the multiple processing layers can generate an output matrix corresponding to the layer output data by multiplying the input matrix corresponding to the layer input data by a weight matrix. However, the scope of the example embodiments is not limited to this, and each of the multiple processing layers can generate output data by converting the input matrix corresponding to the layer input data in any manner. For example, each of the multiple processing layers can be implemented to generate layer output data by multiplying the input matrix corresponding to the layer input data by multiple weight matrices in sequence, or converting the input matrix into layer output data based on any conversion parameters. For example, the example embodiments are not limited to a specific manner in which each of the multiple processing layers converts layer input data.

[0426] The neural processing unit 2200 may include a matrix multiplication device 2210. The matrix multiplication device 2210 may perform at least some of the calculations included in the plurality of processing layers. For example, the matrix multiplication device 2210 may perform matrix multiplication calculations included in the plurality of processing layers.

[0427] In some example embodiments, matrix multiplication calculations may take up a majority of the processing load required for neural processing system 2000 to execute each of the multiple processing layers.

[0428] In some example embodiments, the matrix multiplication device 2210 may be implemented as described above with reference to Figures 1 to 38 In this case, the neural processing unit 2200 can perform calculations included in multiple processing layers with a smaller amount of calculations. Therefore, the neural processing system 2000 including the matrix multiplication device MMD according to various example embodiments can drive the artificial intelligence model at a faster speed.

[0429] The volatile memory device 2300 may be used as an operation memory of the neural processing unit 2200. For example, the volatile memory device 2300 may temporarily store data generated during the operation of the neural processing unit 2200.

[0430] In some example embodiments, neural processing unit 2200 may access volatile memory device 2300 to perform calculations included in a plurality of processing layers. For example, neural processing unit 2200 may be implemented to read parameters stored in volatile memory device 2300 to perform calculations of layer input data, or may be implemented to temporarily store intermediate data generated during calculations in volatile memory device 2300.

[0431] In some example embodiments, the computation speed of the neural processing unit 2200 may be higher than the access speed of the neural processing unit 2200 to the volatile memory device 2300. Therefore, due to the communication speed between the neural processing unit 2200 and the volatile memory device 2300, the running speed of the artificial intelligence model may be bottlenecked.

[0432] In some example embodiments, if the matrix multiplication device 2210 is implemented as described above with reference to Figures 1 to 38 In the matrix multiplication device MMD described above, each processing element PE included in the matrix multiplier MMD can calculate an output element. In this case, since the neural processing unit 2200 may not store and read the portion and PSM in the volatile memory device 2300, the number of accesses of the neural processing unit 2200 to the volatile memory device 2300 may be reduced or minimized. Therefore, according to various example embodiments, the bottleneck phenomenon of the running speed of the artificial intelligence model caused by the access of the neural processing unit 2200 to the volatile memory device 2300 may be reduced or minimized.

[0433] In some example embodiments, if the matrix multiplication device 2210 is implemented as described above with reference to Figures 1 to 39 If the matrix multiplication device MMD described above is used, each processing element PE can calculate one output element. In this case, a smaller-sized processing element array 150 (e.g., fewer processing elements PE) can be used to drive the artificial intelligence model. Therefore, according to various example embodiments, the production cost of the neural processing system 2000 for driving the artificial intelligence model can be reduced or minimized.

[0434] In some example embodiments, the volatile memory device 2300 may be implemented by any type of volatile memory such as a dynamic random access memory (DRAM), a static random access memory (SRAM), or the like.

[0435] In some example embodiments, the volatile memory device 2300 may be used as a buffer memory, an operation memory, or a cache memory of the central processing unit 2100. However, example embodiments are not limited thereto.

[0436] Nonvolatile memory device 2400 may store data used for the operation of neural processing system 2000. For example, nonvolatile memory device 2400 may store various types of data, such as parameters for an operating system (OS) of neural processing system 2000, parameters for driving an artificial intelligence model, and the like. However, example embodiments are not limited thereto.

[0437] The central processing unit 2100 may communicate with the user through the user interface 2500. The central processing unit 2100 may provide the model input data provided by the user to the volatile memory device 2300 or the neural processing unit 2200 through the user interface 2500. The central processing unit 2100 may return the model output data generated by the artificial intelligence model based on the model input data to the user through the user interface 2500.

[0438] Fig.40 The diagram shows Fig.39 Block diagram of an artificial intelligence model driven by a neural processing system. Fig.39 and Fig.40 , the neural processing system 2000 can drive the artificial intelligence model AIM.

[0439] The artificial intelligence model AIM may receive model input data MID. The artificial intelligence model AIM may include first to Lth processing layers PL_1 -PL_L.

[0440] The artificial intelligence model AIM can generate model output data MOD by sequentially converting model input data MID through the first to Lth processing layers PL_1-PL_L. For example, the first processing layer PL_1 can receive model input data MID and can generate second layer input data LID_2. The second processing layer PL_2 can receive second layer input data LID_2 and can generate third layer input data LID_3. In this way, the Lth processing layer PL_L can receive Lth layer input data LID_L and can generate model output data MOD.

[0441] Each of the first to Lth processing layers PL_1-PL_L may convert (or transform) received data into data to be output by various types of calculations. For example, a matrix multiplication calculation may be included in the calculation performed by the first processing layer PL_1 to convert the model input data MID into the second layer input data LID_2. Similarly, each of the first to Lth processing layers PL_1-PL_L may have to perform a matrix multiplication calculation to convert the received layer input data. However, the scope of the example embodiments is not limited thereto, and some of the first to Lth processing layers PL_1-PL_L may not perform a matrix multiplication calculation.

[0442] In some example embodiments, the matrix multiplication calculation performed by each of the first to Lth processing layers PL_1 -PL_L may be performed by the matrix multiplication device 2210 .

[0443] In some example embodiments, if the matrix multiplication device 2210 is implemented as described above with reference to Figures 1 to 39 The matrix multiplication device MMD described above can output the result of the matrix multiplication calculation at a faster speed. Therefore, according to various exemplary embodiments, the running speed of the artificial intelligence model AIM can be improved.

[0444] To describe it more concisely, Fig.40 , an embodiment including a plurality of processing layers is representatively described in the example, wherein the artificial intelligence model AIM runs continuously, but the example embodiments are not limited thereto. For example, the artificial intelligence model AIM may further include a processing layer for at least some of the first to Lth processing layers PL_1-PL_L operating in parallel. For example, the example embodiments are not limited to the specific implementation method of the artificial intelligence model AIM.

[0445] Any of the elements and / or functional blocks disclosed above may be included or implemented in a processing circuit (such as hardware including a logic circuit; a hardware / software combination, such as a processor that executes software; or a combination thereof). For example, the processing circuit may more specifically include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a system on a chip (SoC), a programmable logic unit, a microprocessor, an application specific integrated circuit (ASIC), etc. The processing circuit may include electronic elements, such as at least one of a transistor, a resistor, a capacitor, etc. The processing circuit may include electronic components, such as logic gates, including at least one of an AND gate, an OR gate, a NAND gate, a NOT gate, etc.

[0446] Example embodiments may provide matrix multipliers and matrix multiplication methods that may be used in various algorithms, such as various machine learning and / or artificial intelligence algorithms. Machine learning and / or artificial intelligence algorithms may be used to implement various industrial processes and / or various telecommunications processes and / or various transportation processes. For example, matrix multipliers and / or matrix multiplication methods may be used to accelerate applications in one or more of healthcare, customer service, finance, manufacturing, transportation, agriculture, retail, education, energy, human resources, environment, security, entertainment, legal services, space exploration, mining operations, defense operations, and / or government services.

[0447] The above contents are specific embodiments for implementing the present invention. The inventive concept may include not only the above-mentioned exemplary embodiments, but also embodiments that can be simply changed in design or easily modified. In addition, the inventive concept may also include technologies that can be easily modified and implemented using the embodiments. Therefore, the scope of the present disclosure should not be limited to the above-mentioned exemplary embodiments, but should be defined by the following claims and the claims and their equivalents. In addition, the exemplary embodiments are not necessarily mutually exclusive. For example, some exemplary embodiments may include one or more features described with reference to one or more drawings, and may also include one or more other features described with reference to one or more other drawings.

Claims

1. A matrix multiplier, comprising: an input vector sealer configured to generate a first scaled input vector based on the first input vector and a plurality of quantized scaling coefficients; a first data type converter configured to generate a first fixed-point scaled input vector based on the first scaled input vector; an array of processing elements, comprising a first processing element and a second processing element, the first processing element being configured to generate a first fixed-point output element based on the first fixed-point scaled input vector and a first plurality of quantized symbol values, the second processing element being configured to generate a second fixed-point output element based on the first fixed-point scaled input vector and a second plurality of quantized symbol values; as well as A second data type converter is configured to generate a first output element and a second output element by converting a data type of the first fixed-point output element and a data type of the second fixed-point output element, respectively, and is configured to output a first output vector including the first output element and the second output element.

2. The matrix multiplier according to claim 1, wherein: The input vector sealer is further configured to generate a second scaled input vector based on the second input vector and the plurality of quantized scaling coefficients, The first data type converter is further configured to generate a second fixed-point scaled input vector based on the second scaled input vector, The array of processing elements further comprises a third processing element configured to generate a third fixed-point output element based on the second fixed-point scaled input vector and the first plurality of quantized symbol values, and a fourth processing element configured to generate a fourth fixed-point output element based on the second fixed-point scaled input vector and the second plurality of quantized symbol values, and The second data type converter is further configured to generate a third output element and a fourth output element by converting a data type of the third fixed-point output element and a data type of the fourth fixed-point output element, respectively, and output a second output vector including the third output element and the fourth output element.

3. The matrix multiplier according to claim 2, wherein: The first processing element and the second processing element are located in a first processing element row of the processing element array, and the third processing element and the fourth processing element are located in a second processing element row of the processing element array.

4. The matrix multiplier according to claim 2, wherein: The first processing element and the third processing element are located in a first processing element column of the processing element array, and the second processing element and the fourth processing element are located in a second processing element column of the processing element array.

5. The matrix multiplier according to claim 2, wherein: The dimension of the first fixed-point scaled input vector is R times the dimension of the first input vector, where R is an integer greater than or equal to 2, and The dimension of the second fixed-point scaled input vector is R times the dimension of the second input vector.

6. The matrix multiplier according to claim 5, wherein: The dimension of the first input vector, the dimension of the second input vector, the dimension of the first output vector, and the dimension of the second output vector are the same.

7. The matrix multiplier according to claim 2, wherein: The first data type converter comprises: a first exponent extraction circuit configured to extract a first exponent corresponding to a maximum exponent among exponents of each of a first plurality of scaled input elements included in the first scaled input vector; a second exponent extraction circuit configured to extract a second exponent based on a maximum exponent among the exponents of each of a second plurality of scaled input elements included in the second scaled input vector; a first data type conversion circuit configured to generate the first fixed-point scaled input vector by converting a data type of each of the first plurality of scaled input elements to fixed point based on the first index; and A second data type conversion circuit is configured to generate the second fixed-point scaled input vector by converting a data type of each of the second plurality of scaled input elements to fixed point based on the second index.

8. The matrix multiplier according to claim 7, wherein: The second data type converter is further configured as: converting the data types of the first fixed-point output element and the second fixed-point output element to floating point based on the first index, and Based on the second index, data types of the third fixed-point output element and the fourth fixed-point output element are converted to floating points.

9. The matrix multiplier according to claim 8, wherein: The exponent parts of the first output element and the second output element correspond to the first exponent, and The exponent parts of the third output element and the fourth output element correspond to the second exponent.

10. The matrix multiplier according to claim 1, wherein: The first processing element is configured to generate the first fixed-point output element by accumulating products of a first plurality of fixed-point scaled input elements included in the first fixed-point scaled input vector and the first plurality of quantized symbol values, and The second processing element is configured to generate the second fixed-point output element by accumulating products of the first plurality of fixed-point scaled input elements and the second plurality of quantized symbol values ​​included in the second fixed-point scaled input vector.

11. The matrix multiplier according to claim 10, wherein: The first processing element comprises: an accumulator register; and an arithmetic logic unit (ALU) comprising a first input terminal configured to sequentially receive the first plurality of fixed-point scaled input elements, a second input terminal configured to sequentially receive the first plurality of quantized symbol values, and a third input terminal connected to the accumulator register, and The arithmetic logic unit is configured to update the value stored in the accumulation register using a value obtained by adding a product of a scaled input element received via the first input terminal and a quantized symbol value received via the second input terminal to the value received by the third input terminal.

12. A matrix multiplier comprising: an input vector sealer configured to generate a first plurality of scaled input elements based on the first input element and the first plurality of quantized scaling coefficients, and to generate a second plurality of scaled input elements based on the second input element and the second plurality of quantized scaling coefficients; a first data type converter configured to generate a first plurality of fixed-point scaled input elements based on the first plurality of scaled input elements and to generate a second plurality of fixed-point scaled input elements based on the second plurality of scaled input elements; a first processing element configured to generate a first fixed-point output element by accumulating the first plurality of fixed-point scaled input elements and the second plurality of fixed-point scaled input elements based on a plurality of quantized symbol values; as well as The second data type converter is configured to generate a first output element by converting a data type of the first fixed-point output element.

13. The matrix multiplier according to claim 12, wherein: The number of the first plurality of quantized scaling factors, the number of the second plurality of quantized scaling factors, the number of the first plurality of scaled input elements, and the number of the second plurality of scaled input elements are the same.

14. The matrix multiplier according to claim 13, wherein: The first plurality of scaled input elements correspond to values ​​obtained by multiplying each of the first plurality of quantization scaling coefficients by the first input element, and the second plurality of scaled input elements correspond to values ​​obtained by multiplying each of the second plurality of quantization scaling coefficients by the second input element.

15. The matrix multiplier of claim 12, wherein: The data type of each of the first input element and the second input element and each of the first plurality of quantized scaling coefficients and the second plurality of quantized scaling coefficients is floating point, and The data types of the first plurality of fixed-point scaled input elements and the second plurality of fixed-point scaled input elements are fixed-point corresponding to the same position value.

16. The matrix multiplier according to claim 15, wherein: The first data type converter comprises: a first exponent extraction circuit configured to extract a first exponent corresponding to a maximum exponent among the exponents of the first plurality of scaled input elements and the second plurality of scaled input elements; A data type conversion circuit is configured to convert a data type of each of the first plurality of scaled input elements and the second plurality of scaled input elements to a fixed point based on the first index.

17. The matrix multiplier according to claim 16, wherein: The data type conversion circuit is configured to determine an exponent portion of the first output element based on the first exponent.

18. A method for operating a matrix multiplication device, comprising: receiving first to Nth weights from an external device, wherein N is an integer greater than or equal to 2; generating first to N×R quantized symbol values ​​and first to N×R quantized scaling coefficients by binary-coding quantizing the first to N weights, wherein R is an integer greater than or equal to 2; receiving first to Nth input elements from the external device; generating first to N×R-th scaled input elements by scaling the first to N-th input elements based on the first to N×R-th quantization scaling coefficients; and A first output element generated by accumulating the first to N×R-th scaled input elements based on the first to N×R-th quantized symbol values ​​is output.

19. The operating method according to claim 18, wherein: Outputting the first output element includes: Convert the data types of the first to N×R scaled input elements to fixed point; generating a first fixed-point output element by accumulating the first to N×R-th scaled input elements converted into a fixed-point format based on the first to N×R-th quantized symbol values; and The first output element generated by converting the data type of the first fixed-point output element to a floating point is output.

20. The operating method according to claim 18, wherein: The first output element corresponds to a sum of values ​​obtained by respectively multiplying the first to N×R th quantized symbol values ​​by the first to N×R th scaled input elements.

Images