Multiplication and accumulation (MAC) operator and matrix multiplier including the same

Abstract

To provide a multiplication and accumulation (MAC) operator and a matrix multiplier including the same.SOLUTION: A matrix multiplier includes: an input vector scaler configured to generate a scaled input matrix based on a first input matrix and a plurality of scale factors; a first data type converter configured to convert the data type of the scaled input matrix to fixed-point and generate a fixed-point input matrix; a multiplication and accumulation operator array configured to receive the fixed-point input matrix and a plurality of binary vectors, generate a fixed-point output matrix based on the fixed-point input matrix and the plurality of binary vectors, receive the first input matrix and a second input matrix, and generate a first output matrix based on the first input matrix and the second input matrix; and a second data type converter configured to convert the data type of the fixed-point output matrix to a floating point and generate a second output matrix.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The disclosure relates to a multiply and accumulate unit and a matrix multiplier including the same. [Background technology]

[0002] Deep learning, a branch of artificial intelligence, recognizes patterns in complex data and enables sophisticated predictions. Deep learning generally consists of a training stage, in which training data is used to train a neural network model, and an inference stage, in which new data is input into the trained neural network model to obtain output. Most of the operating time of deep learning is spent on matrix multiplication.

[0003] Meanwhile, deep learning can generally be performed using general-purpose processors such as GPUs (Graphics Processing Units) or NPUs (Neural Processing Units).Recently, there has been an accelerating trend to incorporate deep learning technology into mobile and IoT (Internet of Things) application devices, which necessitates technology that can realize the massive computing power required by deep learning in small semiconductors. Summary of the Invention [Problem to be solved by the invention]

[0004] One embodiment provides a multiplication and accumulation (MAC) unit that can be implemented in a small semiconductor, and a matrix multiplier including the MAC unit.

[0005] One embodiment provides a multiplication and accumulation (MAC) operator implemented in a single piece of hardware and configured to perform multiplication operations on multiple input matrices and multiplication operations on an input matrix and a weight matrix, and a matrix multiplier including the MAC operator. [Means for solving the problem]

[0006] According to one embodiment, the matrix multiplier may include an input vector scaler that generates a scaled input matrix based on a first input matrix and a plurality of scale factors; a first data type converter that converts a data type of the scaled input matrix to fixed point and generates a fixed-point input matrix; a multiplier and accumulate operator array that receives the fixed-point input matrix and a plurality of binary vectors and generates a fixed-point output matrix based on the fixed-point input matrix and the plurality of binary vectors, receives the first input matrix and a second input matrix and generates the first output matrix based on the first input matrix and the second input matrix; and a second data type converter that converts a data type of the fixed-point output matrix to floating point and generates a second output matrix.

[0007] According to one embodiment, the MAC calculator may include a first multiplexer that receives first data of a first data type and mantissa data of second data of a second data type different from the first data type, outputs the first data based on a first signal, and outputs mantissa data of the second data based on a second signal different from the first signal; a first AND calculator that receives a plurality of binary values and performs a logical AND operation between a first binary value of the plurality of binary values and the first data; and an adder that adds output data of the first AND calculator and outputs first output data of the first data type.

[0008] According to one embodiment, the matrix multiplier may include: a data buffer that receives a first input vector and a second input vector from an external memory, and receives a plurality of scale factors and a plurality of binary vectors as weight vectors from the external memory; an input parser that receives the plurality of scale factors and the plurality of binary vectors and outputs the plurality of scale factors and the plurality of binary vectors together with a first signal at a first level, and receives the second input vector and outputs the second input vector together with a first signal at a second level different from the first level; an input vector scaler that generates a first scaled input vector based on a multiplication operation of the first input vector and the plurality of scale factors; a first data type converter that receives the first scaled input vector, extracts a first exponent from among the exponents of a first plurality of scaled input elements included in the first scaled input vector, and converts a data type of the first scaled input vector to a fixed point based on the first exponent; and a MAC operator that, upon receiving the first signal at the first level, performs an accumulation operation on the first scaled input vector based on the binary values of the plurality of binary vectors to generate a fixed-point output matrix. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram of a matrix multiplier according to one embodiment. [Figure 2] FIG. 2 is a diagram for explaining the weighting matrix (WM) in FIG. [Figure 3] FIG. 10 is a diagram illustrating floating-point data as a data format. [Figure 4] FIG. 10 is a diagram for explaining a multiplication and accumulation calculator; [Figure 5] 5 is a diagram for explaining the operation method of the mantissa calculation unit in FIG. 4. FIG. [Figure 6] FIG. 2 is a block diagram showing the configuration of a matrix multiplier in FIG. [Figure 7] FIG. 2 is a block diagram of an input vector scaler according to one embodiment. [Figure 8] 8 is a diagram for explaining the operation of the input vector scaling circuit of FIG. 7. FIG. [Figure 9] FIG. 2 is a block diagram of a first data type converter according to one embodiment. [Figure 10] 10 is a diagram for explaining how the first data type converter in FIG. 9 extracts an exponent. FIG. [Figure 11] 11 is a diagram illustrating an operation of the first data type converter of FIG. 10 converting the data type of a received input vector to a fixed-point type based on the extracted exponent. [Figure 12] FIG. 2 is a block diagram of a multiply and accumulate operator array according to one embodiment. [Figure 13] 10A and 10B are diagrams illustrating an operation method of a multiplication and accumulation calculator according to an embodiment; [Figure 14] 10A and 10B are diagrams illustrating an operation method of a multiplication and accumulation calculator according to an embodiment; [Figure 15] 10A and 10B are diagrams illustrating an operation method of a multiplication and accumulation calculator according to an embodiment; [Figure 16] FIG. 2 is a block diagram of a mantissa calculation unit of a multiply and accumulate unit according to one embodiment. [Figure 17] FIG. 2 is a block diagram of a multiply and accumulate unit according to one embodiment. [Figure 18] FIG. 10 is a diagram illustrating the operation of the second data type converter according to an embodiment. [Figure 19] FIG. 1 is a block diagram illustrating a neural processing system according to one embodiment. [Figure 20] 1 is an exemplary diagram illustrating multi-head attention according to an embodiment; [Figure 21] FIG. 21 is an exemplary diagram for explaining the scale inner product attention of FIG. 20. DETAILED DESCRIPTION OF THE INVENTION

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described in detail with reference to the accompanying drawings, so that those skilled in the art can easily understand and practice the present invention. However, as the present invention may be embodied in many different forms, it is not limited to the embodiments set forth herein.

[0011] In the drawings, parts unnecessary for the purpose of clearly explaining the present invention are omitted, and similar parts are designated by similar reference numerals throughout the specification. In the flowcharts described with reference to the drawings, the order of operations may be changed, various operations may be combined, certain operations may be divided, or certain operations may not be performed.

[0012] Furthermore, expressions in the singular can be construed as singular or plural unless explicit expressions such as "one" or "single" are used. Terms including ordinal numbers such as first, second, etc. can be used to describe various components, but the components are not limited by such terms. These terms can be used to distinguish one component from another.

[0013] 1 is a block diagram of a matrix multiplier according to an embodiment. Referring to FIG. 1, a matrix multiplier (MATMUL) 100 may receive a plurality of input matrices (XM, YM). Each of the input matrices (XM, YM) may include a plurality of input vectors. Each of the plurality of input vectors may include a plurality of input elements. For example, a first input matrix (XM) of the plurality of input matrices (XM, YM) may be expressed by the following Equation 1:

number

number

[0014] The second input matrix (YM) of the multiple input matrices (XM, YM) may be expressed by the following Equation 2.

number

[0015] In one embodiment, each of the input elements included in the input matrix (XM, YM) may be floating point data. For example, the data type of each of the input elements included in the input matrix (XM, YM) may be FP16 (16-bit floating point) or FP32 (32-bit floating point). However, the scope of the present disclosure is not limited thereto.

[0016] In one embodiment, the matrix multiplier 100 can perform matrix multiplication on a first input matrix (XM) and a second input matrix (YM). The matrix multiplier 100 can perform a multiplication operation on the first input matrix (XM) and the second input matrix (YM) to generate a first output matrix (ZM). For example, the matrix multiplier 100 can perform floating point multiplication to multiply the input elements of the first input matrix (XM) by the input elements of the second input matrix (YM), and then perform floating point summation to sum up the multiplication results to generate the first output matrix (ZM). A method for performing multiplication on the first input matrix (XM) and the second input matrix (YM) by the matrix multiplier 100 according to one embodiment will be described below with reference to FIGS. 6 and 12 to 14.

[0017] The first output matrix (ZM) may include a plurality of output vectors, each of which may include a plurality of output elements. For example, the first output matrix (ZM) may be expressed by the following Equation 3:

number

[0018] In one embodiment, the matrix multiplier 100 may further receive a weight matrix (WM). The weight matrix (WM) may include multiple weights. For example, the weight matrix (WM) may be expressed as Equation 4 below.

number

[0019] In one embodiment, the weights of the weighting matrix (WM) may be approximated by multiple scale factors (α) and binary values (b). The matrix multiplier 100 can receive multiple scale factors (α) and binary values (b) as the weights of the weighting matrix (WM). A detailed description of the multiple scale factors (α) and binary values (b) will be provided below with reference to FIG. 2.

[0020] In one embodiment, the matrix multiplier 100 may perform matrix multiplication on a first input matrix (XM) and a weighting matrix (WM). For example, the matrix multiplier 100 may be implemented to first multiply input elements of the first input matrix (XM) by a scale factor (α), and then accumulate the products based on a binary value (b) to obtain output elements. A method for multiplying the first input matrix (XM) and the weighting matrix (WM) by the matrix multiplier 100 according to one embodiment will be described below with reference to FIGS. 6 to 17.

[0021] In one embodiment, the matrix multiplier 100 may multiply the first input matrix (XM) by the weight matrix (WM) to generate the second output matrix (PM). The second output matrix (PM) may include a plurality of output vectors. Each of the plurality of output vectors may include a plurality of output elements. For example, the second output matrix (PM) may be expressed by the following Equation 5:

number

[0022] The matrix multiplier 100 according to one embodiment can perform a multiplication operation on a plurality of input matrices (XM, YM) and output a first output matrix (ZM). The matrix multiplier 100 according to one embodiment can perform a multiplication operation on an input matrix (XM) and a weight matrix (WM) and output a second output matrix (PM). The matrix multiplier 100 according to one embodiment is realized by a single piece of hardware and can perform a multiplication operation on a plurality of input matrices and a multiplication operation on an input matrix and a weight matrix. When a neural processing system performing deep learning includes a matrix multiplier according to one embodiment, the area occupied by the matrix multiplier in the neural processing system can be reduced.

[0023] Fig. 2 is a diagram for explaining the weighting matrix (WM) of Fig. 1. As described with reference to Fig. 1, the matrix multiplier 100 can receive multiple scale factors (α) and binary values (b) as weights of the weighting matrix (WM).

[0024] By quantizing the weight matrix (WM), each of the multiple weights can be converted into multiple scale factors (α) and binary values (b). Specifically, a quantization operation may be performed on the weight matrix (WM) to improve the calculation speed of the first input matrix (XM) and the weight matrix (WM). By quantizing the weight matrix (WM), the weights included in the weight matrix (WM) may be approximated into multiple quantum levels (QL). The number of the multiple quantum levels (QL) can be determined by a predetermined quantization resolution. For example, each of the multiple weights may be approximated into 2 R (where R is the quantization resolution) quantum levels (QL). In this case, R Each of the quantum levels (QL) can be determined based on a combination of R scale factors (α) and R binary (b) values.

[0025] In one embodiment, the weight matrix (WM) can include multiple row vectors. Referring to FIG. 2, the weight matrix (WM) includes a first row vector (vector W r1 ) ~ (n)th low vector (vector W rn In one embodiment, the weight matrix (WM) may be quantized by row. Specifically, α k_rj is the (j)th row vector of the weight matrix (WM) (vector W rj ) may represent the (k)th scale factor for the weights included in the first row of the weight matrix (WM). In this case, the scale factor for the weights included in the second row of the weight matrix (WM) may be different from the scale factor for the weights included in the second row of the weight matrix (WM). In one embodiment, each of the multiple scale factors (α) may have the same data type as the weights in the weight matrix (WM). For example, the data type of each of the multiple scale factors (α) may be FP16 or FP32. However, the scope of the present disclosure is not limited thereto.

[0026] In one embodiment, the vector B k_rj is the (j)th low vector (vector W rj )'s (k) scale factor (α k_rj ) can be shown to be the binary vector corresponding to the vector B k_rj is the (j)th low vector (vector W rj )'s (k) scale factor (α k_rj ) and can contain multiple binary values (b). k_rj ) can be realized as a row vector with the same dimension as the number of columns in the weight matrix (WM). For example, the vector B k_rj may be expressed as the following Equation 6:

number

[0027] As described above, each weight in the weight matrix (WM) may be approximated based on a plurality of scale factors (α) and a plurality of binary values (b). Here, the weight matrix (WM) has been described as being quantized row by row, but this is not limiting, and the weight matrix (WM) may also be quantized column by column.

[0028] In one embodiment, the matrix multiplier 100 may receive a plurality of scale factors (α) and a binary value (b) as weights of a weighting matrix (WM). In one embodiment, the matrix multiplier 100 may perform a multiplication operation on a first input matrix (XM) by the plurality of scale factors (α) and the binary value (b) to generate a second output matrix (PM). For example, the matrix multiplier 100 may be implemented to first calculate the products of the elements of the first input matrix (XM) and the scale factors (α), and then calculate the output elements by accumulating the calculated products based on the binary value (b). A specific operation method of the matrix multiplier 100 will be described later with reference to FIGS. 6 to 17.

[0029] 3 and 4 are diagrams for explaining the operation of the multiplication and accumulation unit. Specifically, Fig. 3 is a diagram for explaining floating-point data as a data format, and Fig. 4 is a diagram for explaining the multiplication and accumulation unit.

[0030] As described above, each of the input elements of the input matrix (XM, YM) and the weights of the weight matrix (WM) may be floating-point data.

[0031] Referring to Figure 3, floating-point data can be expressed in various formats depending on the precision, but the most commonly used format is the IEEE 754 standard. The IEEE 754 standard represents a real number by dividing it into a sign, an exponent, and a mantissa. For example, floating-point data may be expressed as in Equation 7 below.

number

[0032] For example, -314.625 can be expressed in the IEEE754 floating point format as follows:

[0033] First, the sign is a negative number, so s becomes 1.

[0034] In relation to the mantissa, the absolute value of the number 314.625 can be expressed in binary as 100111010.101. (2) In this case, if you move the decimal point to the left so that only 1 remains to the left of the decimal point, it can be expressed as follows, which is called the normalized representation method: 100111010.101 (2) =1.00111010101 (2) ×2 8 Here, the mantissa is the part to the right of the decimal point, i.e., 00111010101.

[0035] Regarding the exponent, in the normalized representation of -314.625, the exponent is 8, and adding the bias value of 127 to the exponent 8 gives 135. In this case, converting 135 to binary gives 10000111. (2) This value becomes the exponent.

[0036] Here, single precision is used as an example, but the bits and bias values of the exponent and mantissa may differ for half precision or double precision. In the following explanation, we will assume that the data type of the input elements of the input matrix (XM, YM) and the scale factor (α) of the weight matrix (WM) is FP32.

[0037] FIG. 4 is a diagram for explaining the multiplication and accumulation calculator.

[0038] The multiplication and accumulation unit 400 receives floating-point data as input data and performs multiplication and accumulation operations on the input data. The operation of the multiplication and accumulation unit 400 will now be described in detail.

[0039] 4, the multiplication and accumulation unit 400 receives first input data (IN1[31:0]) and second input data (IN2[31:0]) and performs multiplication and accumulation operations on the input data. In one example, when the first input data (IN1[31:0]) and the second input data (IN2[31:0]) are 32-bit floating-point data, each of the first input data (IN1[31:0]) and the second input data (IN2[31:0]) may include 1-bit sign data (S_IN1

[31] , S_IN2

[31] ), 8-bit exponent data (E_IN1[30:23], E_IN2[30:23]), and 23-bit mantissa data (M_IN1[22:0], M_IN2[22:0]).

[0040] The multiplication and accumulation unit 400 may include a multiplication unit 401 and an accumulation unit 402. The multiplication unit 401 may include a sign calculation unit 410, an exponent calculation unit 420, and a mantissa calculation unit 430, and the accumulation unit 402 may include an adder 450 and an accumulator 460.

[0041] In the multiplication unit 401, the code operation unit 410 may include an exclusive OR (hereinafter, referred to as XOR) operation unit 411. The code operation unit 410 may receive first code data (S_IN1

[31] ) of the first input data (IN1) and second code data (S_IN2

[31] ) of the second input data (IN2). If the first code data (S_IN1

[31] ) and the second code data (S_IN2

[31] ) both have a value of “0” indicating a positive number or a value of “1” indicating a negative number, the XOR operation unit 411 of the code operation unit 410 may output “0” indicating a positive number. On the other hand, when one of the first code data (S_IN1

[31] ) and the second code data (S_IN2

[31] ) has a value of "0" indicating a positive number and the other has a value of "1" indicating a negative number, the XOR operation unit 411 of the code operation unit 410 can output "1" indicating a negative number. The code operation unit 410 can output data generated as a result of the XOR operation as 1-bit code data (OUT

[31] ) of the output data (OUT).

[0042] The exponent calculation unit 420 may include a first exponent adder 421 and a second exponent adder 423. The first exponent adder 421 may receive first exponent data (E_IN1[30:23]) of the first input data (IN1) and second exponent data (E_IN2[30:23]) of the second input data (IN2). The first exponent adder 421 may perform a first addition operation on the first exponent data (E_IN1[30:23]) and the second exponent data (E_IN2[30:23]). The first exponent adder 421 may output first addition result data generated as a result of the first addition operation. The first exponent data (E_IN1[30:23]) and the second exponent data (E_IN2[30:23]) are each added with an exponent bias value, for example, “127.” Since the exponent bias value is doubled by the first addition operation in the first exponent adder 421, it is necessary to subtract the exponent bias value from the first addition result data. Therefore, the second exponent adder 423 receives the first addition result data output from the first exponent adder 421 and can perform a second addition operation to subtract the exponent bias value (Bias), for example, "127", from the first addition result data. The second exponent adder 423 can output the data generated as a result of the second addition operation as 8-bit exponent data (E_OUT[7:0]).

[0043] The mantissa calculation unit (MTS_MUL, 430) may perform a multiplication operation on mantissa data. The mantissa calculation unit 430 may receive first mantissa data (M_IN1[22:0]) of the first input data (IN1) and second mantissa data (M_IN2[22:0]) of the second input data (IN2). The mantissa calculation unit 430 may perform a multiplication operation on the first mantissa data (M_IN1[22:0]) and the second mantissa data (M_IN2[22:0]). The mantissa calculation unit 430 may output data generated as a result of the multiplication operation as 48-bit mantissa data (M_OUT[47:0]). A specific operation method of the mantissa calculation unit 430 will be described later with reference to FIG. 5.

[0044] The normalizer 440 can normalize the exponent data (E_OUT[7:0]) output from the exponent calculation unit 420 and the mantissa data (M_OUT[47:0]) output from the mantissa calculation unit 430. Normalization can refer to shifting the calculated data so that the most significant bit becomes 1. The normalized data (OUT[30:0]) can be output to the accumulator 402 as the result value (OUT) of the multiplier together with the code data (OUT

[31] ) output from the code processing circuit 411.

[0045] In the accumulation unit 402, the adder 450 can perform an accumulation operation by adding the data received from the multiplication unit 401 and the data received from the accumulator 460 together.

[0046] Fig. 5 is a diagram for explaining the operation method of the mantissa operation unit of Fig. 4. Generally, binary multiplication may be performed by bit shift and addition operations. The mantissa operation unit 430 that performs binary multiplication may be realized by a shifter (shfter, 431), a logical product (AND) operation unit 432, and an adder (adder, 433).

[0047] 4, the mantissa calculation unit 430 may receive first mantissa data (M_IN1[22:0]) as a multiplicand and second mantissa data (M_IN2[22:0]) as a multiplicand. Data obtained by ANDing the first mantissa data (M_IN1[22:0]) and the second mantissa data (M_IN2[22:0]) based on their respective bits (M_IN2[0], M_IN2[1], ..., M_IN2

[22] ) may be input to the adder 433. The adder 433 may output mantissa data (M_OUT[47:0]).

[0048] In one embodiment, the least significant bit (M_IN2[0]) of the first mantissa data (M_IN1[22:0]) and the second mantissa data (M_IN2[22:0]) can be input to a first AND operation unit 432_1. The first AND operation unit 432_1 can output the first mantissa data (M_IN1[22:0]) as is or output data with all bits set to "0" based on the least significant bit (M_IN2[0]) of the second mantissa data (M_IN2[22:0]). For example, if the least significant bit (M_IN2[0]) of the second mantissa data (M_IN2[22:0]) is "1", the first logical AND operation unit 432_1 outputs the first mantissa data (M_IN1[22:0]) as is, and if the least significant bit (M_IN2[0]) of the second mantissa data (M_IN2[22:0]) is "0", it can output data with all bits set to "0".

[0049] In one embodiment, the first mantissa data (M_IN1[22:0]) may be left-shifted by one bit by the shifter 431. The first mantissa data (M_IN1[22:0]) left-shifted by the shifter 431 may be input to the logical product operation units 432_2, ..., 432_3 together with each bit (M_IN2[1], ..., M_IN2

[22] ) of the second mantissa data (M_IN2[22:0]).

[0050] In one embodiment, the data output by the AND operation unit 432 may be input to the adder 433. The adder 433 may perform an addition operation on the input data sequentially or all at once. The adder 433 may output 48-bit mantissa data (M_OUT[47:0]) as a multiplication result of the first mantissa data (M_IN1[22:0]) and the second mantissa data (M_IN2[22:0]). Although the mantissa operation unit 430 is illustrated as including one adder here, this is not limited thereto, and the mantissa operation unit 430 may include multiple adders connected to each AND operation unit 432.

[0051] Fig. 6 is a block diagram showing the configuration of the matrix multiplier of Fig. 1. Specifically, the matrix multiplier 100 of Fig. 1 may be realized as a matrix multiplier 600.

[0052] In one embodiment, the matrix multiplier 600 may include a first input matrix buffer (input matrix buffer #1, 610), a second input matrix buffer (input matrix buffer #2, 620), a weight buffer (weight buffer, 630), an input parser (input parser, 640), an input vector scaler (input vector scaler, 650), a first data type converter (data type converter #1, 660), a multiply and accumulate operator array (MAC_ARRAY, 670), and a second data type converter (data type converter #2, 680).

[0053] In one embodiment, first input matrix buffer 610 and second input matrix buffer 620 can store externally provided input matrices. For example, first input matrix buffer 610 and second input matrix buffer 620 can receive a first input matrix (XM) and a second input matrix (YM), respectively, from external memory and provide them to input vector scaler 650 and multiply-and-accumulate array 670.

[0054] In one embodiment, the weight buffer 630 can store multiple scale factors (α) and binary values (b) provided externally. The weight buffer 630 can store multiple scale factors (α) and binary values (b) as weights for a weighting matrix (WM). For example, the weight buffer 630 can receive multiple scale factors (α) and binary values (b) externally and provide them to the input vector scaler 650 and the multiplication and accumulation operator array 670.

[0055] In one embodiment, input parser 640 can receive a second input matrix (YM) from second input matrix buffer 620 or a plurality of scale factors (α) and binary values (b) from weight buffer 630. Input parser 640 can parse the second input matrix (YM) or the plurality of scale factors (α) and binary values (b) and provide them to input vector scaler 650 or multiply-and-accumulate array 670.

[0056] In one embodiment, the second input matrix buffer 620 and the weight buffer 630 may be implemented as a single data buffer. Hereinafter, it is assumed that the input parser 640 receives the second input matrix (YM) from the second input matrix buffer 620 and the multiple scale factors (α) and binary values (b) from the weight buffer 630. However, this is not limiting, and the input parser 640 can receive the second input matrix (YM), the multiple scale factors (α), and the binary values (b) from a single data buffer.

[0057] In one embodiment, upon receiving the second input matrix (YM) from the second input matrix buffer 620, the input parser 640 can provide the second input matrix (YM) to the multiplier and accumulate operator array 670 such that the multiplier and accumulate operators in the multiplier and accumulate operator array 670 perform matrix multiplication on the first input matrix (XM) and the second input matrix (YM).

[0058] In one embodiment, input parser 640 may receive the plurality of scale factors (α) and the binary values (b) from weight buffer 630, provide the plurality of scale factors (α) to input vector scaler 650, and provide the plurality of binary values (b) to multiply and accumulate operator array 670, so as to perform a multiplication operation on the first input matrix (XM) and the plurality of scale factors (α) and the binary values (b). Input vector scaler 650 may perform a multiplication operation on the plurality of scale factors (α) and the first input matrix (XM) and output a scaled input matrix (XM').

[0059] In one embodiment, the multiplier and accumulate array 670 may include multiple multipliers and accumulates. The multiplier and accumulate array 670 receives the scaled input matrix (XM') from the first data type converter 660 as a fixed-point input matrix (XM'_ fxp In one embodiment, input parser 640 may receive a fixed-point input matrix (XM'_ fxp The multiplication and accumulation operator array 670 may output a first level (e.g., high level) control signal (SEL) to the multiplication and accumulation operator array 670 to perform a multiplication operation on the fixed-point input matrix (XM'_xM') based on the binary value (b). fxp ) elements of the fixed-point input matrix (XM'_ fxp ) and binary values (b) to produce a fixed-point output matrix (PM_ fxp ) can be output.

[0060] In one embodiment, the input parser 640 can output a second level (e.g., low level) control signal (SEL) to the multiplier and accumulate operator array 670 so that the multiplier and accumulate operators in the multiplier and accumulate operator array 670 perform matrix multiplication on the first input matrix (XM) and the second input matrix (YM). The multiplier and accumulate operator array 670 can perform multiplication and accumulation operations on the input elements of the first input matrix (XM) and the input elements of the second input matrix (YM) to output a first output matrix (ZM).

[0061] In one embodiment, the input parser 640 can output a control signal (SEL) to the second data type converter 680 so that the second data type converter 680 converts the data format of the output elements of the multiply and accumulate array 670. The second data type converter 680 converts the data format of the fixed-point output matrix (PM_ fxp) can be converted to the second output matrix (PM) in floating-point data format.

[0062] In one embodiment, input vector scaler 650 may receive a first input matrix (XM). For example, input vector scaler 650 may receive multiple input vectors (e.g., vector X1 through vector XM) that include multiple input elements. h ) The input vector scaler 650 may scale the first input matrix (XM) based on a plurality of scale factors (α). For example, the input vector scaler 650 may generate a plurality of scaled input vectors based on the plurality of input vectors. In this case, the plurality of scaled input vectors may be generated based on the plurality of input vectors (vector X1 through vector X2). h ) can be used for vectors X1 to X2. h The scaled input vectors corresponding to the vectors X'1 to X' are respectively h The specific configuration and method of operation of input vector scaler 650 will be described below with reference to FIGS.

[0063] In one embodiment, the first data type converter 660 converts the scaled input vector (vector X'1, ..., vector X' h ) respectively. For example, the first data type converter 660 may extract a first exponent (EXP1) from the first scaled input vector (vector X'1) and a second exponent (EXP2) from the second scaled input vector (vector X'2). The first data type converter 660 may provide the extracted exponents (EXP) to the second data type converter 680.

[0064] In one embodiment, the first data type converter 660 converts multiple scaled input vectors (vector X'1, ..., vector X' h) to a fixed-point data type. For example, the first data type converter 660 may convert the data type of multiple scaled input vectors (vector X'1, ..., vector X' h ) and receives multiple fixed-point input vectors (vector X' 1_fxp ,…,Vector X' h_fxp ) based on the extracted exponent (EXP). h ) can be converted to a fixed-point data type. A specific configuration and operation method for first data type converter 660 will be described later with reference to FIGS.

[0065] In one embodiment, the multiplier and accumulate operator array 670 may receive a first input matrix (XM) from the first input matrix buffer 610 and a second input matrix (YM) from the input parser 640. The multiplier and accumulate operator array 670 may further receive a control signal (SEL) from the input parser 640. The multiplier and accumulate operator array 670 may perform matrix multiplication on the first input matrix (XM) and the second input matrix (YM) based on the second-level control signal (SEL), and output a first output matrix (ZM). Specifically, the multiplier and accumulate operator array 670 may perform multiplication and accumulation operations on the input elements of the first input matrix (XM) and the input elements of the second input matrix (YM) to generate the output elements of the first output matrix (ZM).

[0066] In one embodiment, the multiplier and accumulate array 670 receives multiple fixed-point input vectors (vectors X' 1_fxp ,…,Vector X' h_fxp) and may receive a binary vector (vector B) comprising a plurality of binary values (b) from input parser 640. Multiplier and accumulate array 670 may receive a first level control signal (SEL) from input parser 640. Multiplier and accumulate array 670 may generate a fixed-point input vector (vector X' 1_fxp ,…,Vector X' h_fxp ) elements of a fixed-point input vector (Vector X' 1_fxp ,…,Vector X' h_fxp ) and binary value (b) to produce a fixed-point output vector (vector P 1_fxp , vector P 2_fxp , vector P 3_fxp ,…,vector P n_fxp ) can be generated. A specific configuration and operation method for the multiplication and accumulation operator array 670 will be described later with reference to FIGS.

[0067] In one embodiment, second data type converter 680 can receive multiple exponents (EXP) from first data type converter 660. Second data type converter 680 can receive a fixed-point output vector (vector P 1_fxp , vector P 2_fxp , vector P 3_fxp ,…,vector P n_fxp ) The second data type converter 680 receives a plurality of first level control signals (SEL) and generates a fixed-point output vector (vector P) based on the exponent (EXP). 1_fxp , vector P 2_fxp , vector P 3_fxp ,…,vector P n_fxp ) to floating point. That is, the second data type converter 680 can convert the data type of the second output vector (vector P1, vector P2, vector P3, ..., vector P n) can be generated. In one embodiment, the second data type converter 680 can receive the first output matrix (ZM) from the multiplication and accumulation operator array 670. The second data type converter 680 can receive a second level control signal (SEL) and output the first output matrix (ZM) as output data of the second data type converter 680. A specific configuration and operation method of the second data type converter 680 will be described later with reference to FIG. 18.

[0068] 7 is a block diagram of an input vector scaler according to one embodiment. Input vector scaler 650 according to one embodiment can perform a multiplication operation on a first input matrix (XM) and a plurality of scale factors (α). In one embodiment, input vector scaler 650 can include first through (h)th input vector scaling circuits (651 through 65h).

[0069] In one embodiment, the first through (h)th input vector scaling circuits 651 through 65h can receive different input vectors. For example, the first through (h)th input vector scaling circuits 651 through 65h can receive the first through hth input vectors (i.e., vectors X1 through X2). h ) can be received respectively.

[0070] In one embodiment, each of the first through (h)th input vector scaling circuits 651 through 65h can sequentially receive multiple input elements. For example, the first input vector scaling circuit 651 can receive x 11 ~x 1n The h-th input vector scaling circuit 65h can receive x h1 ~x hn can be received sequentially.

[0071] In one embodiment, each of the first through (h)th input vector scaling circuits 651 through 65h can receive a plurality of scale factors (α) from the input parser 640. The input parser 640 can parse the plurality of scale factors (α) received from the outside and sequentially provide them to the first through (h)th input vector scaling circuits 651 through 65h.

[0072] In one embodiment, the scale factors (α) received by each of the first through (h)th input vector scaling circuits 651 through 65h from the input parser 640 may be identical to one another. For example, the scale factors (α) received by the first input vector scaling circuit 651 may be identical to the scale factors (α) received by the second input vector scaling circuit 652.

[0073] In one embodiment, the order in which the first through (h)th input vector scaling circuits 651 through 65h receive the scale factors (α) may be the same. For example, the scale factor (α) received first by the first input vector scaling circuit 651 may be the same as the scale factor (α) received first by the second input vector scaling circuit 652. Similarly, the scale factor (α) received second by the first input vector scaling circuit 651 may be the same as the scale factor (α) received second by the second input vector scaling circuit 652.

[0074] In one embodiment, each of the first through (h) input vector scaling circuits 651 through 65h can perform a multiplication operation on the input elements and the multiple scale factors (α) based on the order in which the input elements and the multiple scale factors (α) are received. The multiplication operation method of each of the first through (h) input vector scaling circuits 651 through 65h is the same as or similar to the operation method of the multiplication and accumulation unit described in Figure 4, so a detailed description of the operation method of the first through (h) input vector scaling circuits 651 through 65h will be omitted here.

[0075] In one embodiment, the first through (h)th input vector scaling circuits 651 through 65h perform multiplication operations on the input elements and a plurality of scale factors (α) to generate first through (h)th scaled input vectors (vector X'1 through vector X' h For example, the first input vector scaling circuit 651 can generate a vector X'1, and the second input vector scaling circuit 652 can generate a vector X'2.

[0076] 8 is a diagram illustrating the operation of the input vector scaling circuit of FIG. 7. For the sake of simplicity, the operation of the first input vector scaling circuit 651 will be described below. However, the scope of the present disclosure is not limited thereto, and the second through (h)th input vector scaling circuits 652 through 65h may operate in the same or similar manner.

[0077] In one embodiment, the first input vector scaling circuit 651 can receive a first input vector (vector X1). That is, the first input vector scaling circuit 651 can scale x 11 ~x 1n can be received sequentially.

[0078] In one embodiment, the first input vector scaling circuit 651 may receive multiple scale factors (α) sequentially. For example, the first input vector scaling circuit 651 may receive the first row vector (vector W) of the weighting matrix (WM) r1 ) corresponding scale factor (α 1_r1 ~α R_r1 ), then the second low vector (vector W r2 ) corresponding scale factor (α 1_r2 ~α R_r2 In this manner, the first input vector scaling circuit 651 can receive multiple scale factors (α 1_r1 ~α R_rn ) can be received sequentially.

[0079] In one embodiment, the first input vector scaling circuit 651 scales the input elements (x 11 ~x 1n ) and a number of scale factors (α). In one embodiment, the first input vector scaling circuit 651 may perform a multiplication operation on the input elements (x 11 ~x 1n ) and multiple scale factors (α) can be operated on the sign data, exponent data, and mantissa data, respectively.

[0080] In one embodiment, the first input vector scaling circuit 651 scales the input elements (x 11 ~x 1n ) and a plurality of scale factors (α), to output the scaled elements as a first scaled input vector (vector X'1). For example, the first input vector scaling circuit 651 can sequentially multiply the received input elements and a plurality of scale factors (α) to calculate the elements of the plurality of scaled input vectors. Specifically, the first input vector scaling circuit 651 TIFF2025115357000011.tif33 is the first row vector of the weight matrix (WM) (vector W r1 ) corresponding scale factor (α 1_r1 ~α R_r1 ) and multiply each by x 11 Multiple scaled input elements (651_1, x 11 ×α k_r1 ) can be calculated sequentially. After that, the first input vector scaling circuit 651 calculates x 12 the second row vector of the weight matrix (WM) (vector W r2 ) corresponding scale factor (α 1_r2 ~α R_r2 ) and multiplying them respectively, Multiple scaled input elements (651_2, x) corresponding to TIFF2025115357000012.tif33 12 ×αk_r2 ) can be calculated sequentially. In this manner, the first input vector scaling circuit 651 can calculate x 1n Multiple scaled input elements (651_3, x 1n ×α k_rn ) can be calculated sequentially. 11 A plurality of scaled input elements 651_1 corresponding to x' 11 Similarly, x 12 A plurality of scaled input elements 651_2 corresponding to x' 12 Expressed as x 1n A plurality of scaled input elements 651_3 corresponding to x' 1n It can be expressed as:

[0081] In one embodiment, the first input vector scaling circuit 651 can sequentially output a plurality of scaled input elements 651_1, 651_2, . . . , 651_3.

[0082] 9 is a block diagram of a first data type converter 660 according to one embodiment. The first data type converter 660 according to one embodiment converts multiple scaled input vectors (vector X'1 to vector X'2) into a vector X'3. h ) to a fixed point. In one embodiment, the first data type converter 660 may include first to (h)th data type conversion circuits (661 to 66h).

[0083] In one embodiment, the first through (h)-th data type conversion circuits 661 through 66h can receive different input vectors from each other. For example, the first through (h)-th data type conversion circuits 661 through 66h can receive the first through h-th scaled input vectors (vector X'1 through vector X'2). h ) can be received respectively.

[0084] In one embodiment, each of the first through (h)th data type conversion circuits 661 through 66h can extract exponents from multiple input elements included in the scaled input vector. For example, the first data type conversion circuit 661 can extract exponents from the input elements included in the first scaled input vector (i.e., x' 11 ~x' 1n ), and the second data type conversion circuit 662 can extract the first exponent (EXP1) from the input element (i.e., x' 21 ~x' 2n ) can extract the second exponent (EXP2). In this manner, the first to (h)-th data type conversion circuits 661 to 66h can extract the first to h-th exponents (EXP1 to EXPh), respectively. Each of the first to (h)-th data type conversion circuits 661 to 66h can provide the extracted exponent to the second data type converter 680.

[0085] In one embodiment, the first to (h)-th data type conversion circuits 661 to 66h respectively convert the scaled input vectors (vector X'1 to vector X'h) based on the extracted exponents (EXP1 to EXPh). h The first to (h)-th data type conversion circuits 661 to 66h can convert the data type of the first to (h)-th fixed-point input vectors (vector X' 1_fxp ~vector X' h_fxp For example, the first data type conversion circuit 661 may output the input elements (x' 11 ~x' 1n ) can be converted into a fixed-point format based on a first exponent (EXP1). The operations of the first to (h) data type conversion circuits 661 to 66h will be described in more detail below with reference to FIGS. 10 and 11.

[0086] 10 is a diagram for explaining how the first data type converter in FIG. 9 extracts an exponent. For the sake of simplicity, the following description will representatively explain the operation of the first data type conversion circuit 661 extracting an exponent. However, the scope of the present disclosure is not limited thereto.

[0087] 10, the first data type conversion circuit 661 may receive multiple scaled input elements. For example, the first data type conversion circuit 661 may receive x' 11 ~x' 1n can be received.

[0088] In one embodiment, the data type of each of the multiple input elements may be floating point. 11 ~x' 1n Each may include sign data (S), exponent data (EXP), and mantissa data (MTS).

[0089] In one embodiment, the first data type conversion circuit 661 may determine the largest value among the exponent values (EXP) of the received input elements. In this case, the first data type conversion circuit 661 may determine the exponent of the determined input element as the first exponent (EXP1). That is, the first data type conversion circuit 661 may determine the exponent of x' 11 ~x' 1n The largest exponent among the exponents can be extracted.

[0090] 10 exemplarily illustrates an example in which the first data type conversion circuit 661 extracts the largest value among the values of the exponent parts of the multiple input elements, but the scope of the present disclosure is not limited thereto. For example, the first data type conversion circuit 661 may be implemented to extract the smallest value among the values of the exponent parts of the multiple input elements.

[0091] 11 is a diagram illustrating an operation of converting the data type of a received input vector to a fixed point based on the exponent extracted by the first data type converter of FIG. 10. For the sake of simplicity, the operation of the first data type conversion circuit 661 extracting the exponent will be representatively described below. However, the scope of the present disclosure is not limited thereto.

[0092] 11, the first data type conversion circuit 661 can receive multiple input elements. For example, the first data type conversion circuit 661 can receive x' 11 ~x' 1n can be received.

[0093] In one embodiment, the first data type conversion circuit 661 can extract a first exponent (EXP1). The first data type conversion circuit 661 can convert the data type of each scaled input element to fixed point based on the first exponent (EXP1). That is, the first data type conversion circuit 661 can convert x' 11 ~x' 1n x' respectively 11_fxp ~x' 1n_fxp However, for the sake of simplicity, we will use the input element "x' 11 " to the fixed-point input element "x' 11_fxp The operation of the first data type conversion circuit 661 that converts

[0094] In one embodiment, the first data type conversion circuit 661 converts a first exponent (EXP1) and x' 11 The input element "x'" scaled by the difference in exponents of 11 The mantissa of x' (hereinafter referred to as the first mantissa (MTSa)) can be shifted (sft1) in the LSB (least significant bit) direction. For example, the first exponent (EXP1) and x' 11If the difference between the exponent (EXP) values is "4", the first data type conversion circuit 661 can insert "4" "0" bits into the most significant bit (MSB) position of the first mantissa (MTSa).

[0095] In one embodiment, the first data type conversion circuit 661 converts the fixed-point input element “x′” based on the shifted first mantissa (MTSa). 11_fxp " (hereinafter referred to as the second mantissa (MTSb)). For example, the first data type conversion circuit 661 may cut off the least significant bits of the shifted first mantissa (MTSa) according to the number of bits of the second mantissa (MTSb). Alternatively, the first data type conversion circuit 661 may determine the least significant bits of the shifted first mantissa (MTSPa) based on various types of rounding algorithms such as "nearest even rounding" according to the number of bits of the second mantissa (MTSb). However, the scope of the present disclosure is not limited thereto.

[0096] In one embodiment, the number of bits of the first mantissa (MTSa) may be 23-bits, although the scope of the present disclosure is not limited thereto.

[0097] In one embodiment, the number of bits of the second mantissa (MTSb) may be 7 bits, but the scope of the present disclosure is not limited thereto.

[0098] In one embodiment, each of the plurality of fixed-point input elements may be an integer. For example, the data type of each of the plurality of fixed-point input elements may be INT8 (8-bit integer). However, the scope of the present disclosure is not limited in this respect.

[0099] 12 is a block diagram of a multiplier and accumulate unit array according to one embodiment. In one embodiment, the multiplier and accumulate unit array 670 may include a plurality of multiplier and accumulate units (MACs) arranged in rows and columns. For simplicity, the following description assumes that the plurality of MACs are arranged along (h) rows and (m) columns.

[0100] In one embodiment, the multiplication and accumulation operator array 670 can include first through (h) rows of multiplication and accumulation operators (MR1 through MRh). Each of the first through (h) rows of multiplication and accumulation operators (MR1 through MRh) can include a plurality of multiplication and accumulation operators (MACs). For example, the first row of multiplication and accumulation operators (MR1) can include a plurality of multiplication and accumulation operators 671 through 67m. In one embodiment, each of the first through (h) rows of multiplication and accumulation operators (MR1 through MRh) can calculate the input vectors (vector X1 through vector X2) of the first input matrix (XM). h ) and fixed-point input matrix (XM'_ fxp ) fixed-point input vector (vector X' 1_fxp ,…,Vector X' h_fxp In one embodiment, different multiplication and accumulation rows may receive different input vectors (vector X1 to vector X2). h ) and different fixed-point input vectors (vector X' 1_fxp ,…,Vector X' h_fxp In one embodiment, the multipliers and accumulaters included in the same multiplier and accumulater row can receive the same input vectors (vector X1 to vector X2). h ) and the same fixed-point input vector (vector X' 1_fxp ,…,Vector X' h_fxp For example, each of the multiplication and accumulation units 671 to 67m included in the first multiplication and accumulation unit row (MR1) receives a first input vector (vector X1) and a first fixed-point input vector (vector X' 1_fxp) can be received.

[0101] In one embodiment, the multiplier and accumulate operator array 670 may include first through (m)-th columns of multipliers and accumulate operators (MC1 through MCm). Each of the first through (m)-th columns of multipliers and accumulate operators (MC1 through MCm) may include a plurality of multipliers and accumulate operators (MACs). In one embodiment, each of the first through (m)-th columns of multipliers and accumulate operators (MC1 through MCm) may include a plurality of input vectors (vector Y1 through vector Y m ) and the binary value (b) of a binary vector (vector B). In one embodiment, different columns of multipliers and accumulate operators can receive different input vectors (vector Y1 to vector Y m ) and the binary vector (vector B) with different binary values (b1, b2, …, b m In one embodiment, the multipliers and accumulaters (MACs) included in the same multiplier and accumulater column can receive the same input vectors (vector Y1 to vector Y m ) and binary vector (vector B) with the same binary values (b1,b2,…,b m For example, a multiplier and accumulate unit (MAC) included in a first multiplier and accumulate unit column (MC1) may receive a first input vector (vector Y1) and a first binary value (b1) of a binary vector (vector B).

[0102] In one embodiment, each multiplier and accumulate unit (MAC) can perform a multiplication and accumulation operation on input elements. For example, multiplier and accumulate unit 671 can perform a multiplication and accumulation operation on elements of a first input vector (vector X1) of a first input matrix (XM) and elements of a first input vector (vector Y1) of a second input matrix (YM) to generate a first element (z 11 ) can be output. A specific operation method of such a multiplication and accumulation calculator will be described with reference to FIGS.

[0103] In one embodiment, each multiplier and accumulate (MAC) can perform a multiplication operation on a fixed-point input element and a binary value. For example, multiplier and accumulate 671 can perform a multiplication operation on a binary vector (vector B 1_r1 , vector B 2_r1 ,…,Vector B R_rn ) based on the first binary value (b1) of the fixed-point input vector (vector X' 1_fxp ) input elements to perform a multiplication operation on the fixed-point input elements and binary values to produce the first fixed-point output element (p 11_fxp ) can be output. A specific operation method of such a multiplication and accumulation calculator will be described with reference to FIGS.

[0104] 13 and 14 are diagrams illustrating an operation method of the multiplication and accumulation unit according to one embodiment. Specifically, the multiplication and accumulation unit performs multiplication and accumulation on a first input matrix (XM) and a second input matrix (YM) to output a first output matrix (ZM).

[0105] In one embodiment, the multiplier and accumulate operator is a first output element (z 11 ), a multiplication and accumulation operation can be performed on the input elements included in the first row 1311 of the first input matrix (XM) and the input elements included in the first column 1412 of the second input matrix (YM). In addition, the multiplication and accumulation operation can be performed on the second output elements (z 12 ), a multiplication and accumulation operation can be performed on the input elements contained in the first row 1311 of the first input matrix (XM) and the input elements contained in the second column 1422 of the second input matrix (YM).

[0106] 14, in one embodiment, each of the plurality of multipliers and accumulate operators 671, 672, ..., 67m included in the first multiplier and accumulate operator row (MR1) may receive a first input vector (vector X1) of the first input matrix (XM). For example, each of the plurality of multipliers and accumulate operators 671, 672, ..., 67m may receive an element (x 11 ~x 1n ) can be received sequentially.

[0107] In one embodiment, each of the multiple multiplication and accumulation operators 671, 672, ..., 67m included in the first multiplication and accumulation operator row (MR1) receives different input vectors (vector Y1 to vector Y m For example, the first multiplier and accumulate operator 671 may receive a plurality of input elements (y 11 ~y n1 ) in sequence, and the second multiplier and accumulate operator 672 sequentially receives the input elements (y 12 ~y n2 ) can be received sequentially.

[0108] In one embodiment, each of the multipliers and accumulaters may perform multiplication and accumulation operations on the input elements of the first input matrix (XM) and the input elements of the second input matrix (YM) based on the order in which they are received. For example, the multipliers and accumulaters may perform multiplication and accumulation operations on the first output element (z 11 ) can be calculated.

number

[0109] 15 is a diagram illustrating the operation of a multiplication and accumulation unit according to one embodiment. Specifically, a fixed-point input matrix (XM'_ fxp ) and binary values (b). For the sake of simplicity, the fixed-point output matrix (PM_ fxp ) first output element (p 11_fxp ) will be representatively described. However, the scope of the present disclosure is not limited thereto.

[0110] As discussed above in connection with FIG. 2, the weight matrix (WM) is made up of a number of raw vectors (vector W r1 , …, vector W rn ), and by performing a quantization operation on the weighting matrix (WM), each raw vector may be expressed as a sum of products of multiple scale coefficients (α) and the corresponding binary vector (vector B). Also, the binary vector (vector B) may be realized as a raw vector having the same dimension as the number of columns of the weighting matrix (WM) (see Equation 6).

[0111] In one embodiment, the first multiplier and accumulate unit 671 multiplies the first fixed-point input vector (vector X' 1_fxp ) and binary vector (vector B 1_r1 , vector B 2_r1 , …, Vector B R_rn The first multiplier and accumulate operator 671 may receive a first binary value (b1) of the first fixed-point input vector (vector X' 1_fxp ) and binary vector (vector B 1_r1 , vector B 2_r1 , …, Vector B R_rn), the fixed-point output matrix (PM_ fxp ) first output element (p 11_fxp Specifically, the first multiplier and accumulator 671 can calculate the first output element (p 11_fxp ) can be calculated.

number

[0112] In one embodiment, the fixed-point input vector (vector X' 1_fxp ,…,Vector X' h_fxp ) and binary vector (vector B 1_r1 , vector B 2_r1 , …, Vector B R_rn The binary value (b) of (a) can be input to a mantissa calculation unit in a multiply and accumulate unit (MAC), which will be described in more detail below with reference to FIG.

[0113] 16 is a block diagram of a mantissa calculation unit of a multiplier and accumulate unit according to one embodiment. As described above with reference to FIGS. 4 and 5, the multiplier and accumulate unit may include a sign calculation unit, an exponent calculation unit, and a mantissa calculation unit. The mantissa calculation unit of the multiplier and accumulate unit according to one embodiment performs multiplication and accumulation operations on the mantissa data of the input elements of the first input matrix (XM) and the second input matrix (YM), and performs multiplication and accumulation operations on the mantissa data of the first fixed-point input vector (vector X' 1_fxp ) and a binary value (b). For the sake of simplicity, the operation of the mantissa calculation unit 671_1 of the first multiplication and accumulation unit 671 will be representatively described below. However, the scope of the present disclosure is not limited thereto, and the mantissa calculation units of other multiplication and accumulation units may operate in the same or similar manner.

[0114] In one embodiment, the first multiplier and accumulate unit 671 multiplies the first fixed-point input vector (vector X' 1_fxp ) and binary vector (vector B 1_r1 , vector B 2_r1 , …, Vector B R_rn ) from the first data type converter 660. 1_fxp ) and receives a binary vector (vector B 1_r1 , vector B 2_r1 , …, Vector B R_rn ) can receive a first binary value (b1) of the first fixed-point input vector (vector X' 1_fxp ) and binary vector (vector B 1_r1 , vector B 2_r1 , …, Vector B R_rn ) can be input to the mantissa calculation unit 671_1 of the first multiplication and accumulation calculator 671.

[0115] In one embodiment, the mantissa calculation unit 671_1 may include a shifter 672, a multiplexer 673, a logical AND calculation unit 674, and an adder 675.

[0116] In one embodiment, the multiplexer 673 receives the mantissa data (mts(x)) of the first input element. 11 )) and the first fixed-point input vector (vector X' 1_fxp ) input elements. The multiplexer 673 receives the mantissa data (mts(x)) of the first input element from the first input matrix buffer 610. 11 )) from the first data type converter 660 and converts the first fixed-point input vector (vector X' 1_fxp ) input elements. Multiplexer 673 receives the first fixed-point input vector (vector X' 1_fxp ) input element as output data. For example, the first multiplexer 673_1 outputs the mantissa data (mts(x 11 )) and the first fixed-point input vector (vector X' 1_fxp ) input element ((x 11 ×α 1_r1 ) fxp ) and based on the first level control signal (SEL), (x 11 ×α 1_r1 ) fxp can be output as output data.

[0117] In one embodiment, the first AND operation unit 674_1 performs AND on the output data ((x 11 ×α 1_r1 ) fxp ) and the first binary vector (vector B 1_r1 ) first binary value (b 1_1_r1 The first AND operation unit 674_1 can receive the output data ((x 11 ×α 1_r1 ) fxp) from the input parser 640 and receives a first binary vector (vector B 1_r1 ) first binary value (b 1_1_r1 ) can be received. The first AND operation unit 674_1 can output the result of the AND operation. For example, the first AND operation unit 674_1 can receive the first binary value (b 1_1_r1 ) is "1", the output data ((x 11 ×α 1_r1 ) fxp ) is output as is, and the first binary value (b 1_1_r1 ) is "0", data with all bits set to "0" can be output.

[0118] In one embodiment, the shifter 672 shifts the mantissa data (mts(x)) of the first input element. 11 )) can be shifted left by one bit, and the shifted data can be provided as input data to the second multiplexer 673_2.

[0119] In one embodiment, the second multiplexer 673_2 receives the shifted mantissa data (mts(x 11 )) and the first fixed-point input vector (vector X' 1_fxp ) input element ((x 11 ×α 2_r1 ) fxp ) and based on the first level control signal (SEL), (x 11 ×α 2_r1 ) fxp The second AND operation unit 674_2 can output the output data ((x 11 ×α 2_r1 ) fxp ) and a second binary vector (vector B 2_r1 ) first binary value (b 1_2_r1 ) and outputs the logical AND result thereof. In the above-described manner, the multiplexer 673 receives the first fixed-point input vector (vector X' 1_fxp ), and the logical product operation unit 674 outputs the input elements of the first fixed-point input vector (vector X'1_fxp ) input elements and binary vector (vector B 1_r1 , vector B 2_r1 , …, Vector B R_rn ) to the first binary value (b1).

[0120] In one embodiment, the adder 675 may receive the logical product result of the logical product operation unit 674. The adder 675 sums the logical product results of the logical product operation unit 674 and outputs a first fixed-point output element (p 11_fxp ) can be output.

[0121] In one embodiment, the first multiplexer 673_1 receives the mantissa data (mts(x 11 )) and the first fixed-point input vector (vector X' 1_fxp ) input elements. The multiplexer 673 selects the mantissa data (mts(x)) of the first input element based on the second level control signal (SEL). 11 In one embodiment, the logical AND operation unit 674 receives the mantissa data (mts(x 11 )) of the second input element from the input parser 640, and 11 For example, the first AND operation unit 674_1 can receive the mantissa data (mts(x 11 )) and the least significant bit of the mantissa data of the second input element (mts(y 11 )[0]) as input data and output the logical product result thereof as output data. In one embodiment, the adder 675 adds the logical product results of the logical product operation unit 674 to obtain the mantissa data (mts(z 11 )) can be output.

[0122] As described above, the multiplication and accumulation unit according to one embodiment can perform multiplication and accumulation operations on the mantissa data of the input elements, and can perform accumulation operations on the input elements of a fixed-point input vector based on binary values. Here, the mantissa data (mts(y 11 )) and the first fixed-point input vector (vector X' 1_fxp ) and the number of input elements in the binary vector (vector B 1_r1 , vector B 2_r1 , …, Vector B R_rn ) are assumed to have the same number of first binary values (b1), but this is not limited thereto. 1_fxp ) and the number of input elements in the binary vector (vector B 1_r1 , vector B 2_r1 , …, Vector B R_rn ) is the number of first binary values (b1) of the second input matrix (YM) 11 )) bits.

[0123] 17 is a block diagram of a multiplier and accumulate unit according to one embodiment. For the sake of simplicity, the configuration and operation of the first multiplier and accumulate unit 671 will be described below. However, the scope of the present disclosure is not limited thereto, and other multiplier and accumulate units may operate in the same or similar manner.

[0124] In one embodiment, the multiply and accumulate operator 671 may receive a first input vector (vector X1) and a second input vector (vector Y1). The data type of the first input vector (vector X1) and the second input vector (vector Y1) may be floating-point data. The multiply and accumulate operator 671 performs a multiplication operation on the first input vector (vector X1) and the second input vector (vector Y1) and generates a first output element (z 11 ) can be generated.

[0125] In one embodiment, the multiply and accumulate operator 671 receives a first fixed-point input vector (vector X' 1_fxp ) and the first binary value (b1) of the binary vector (vector B). 1_fxp The data type of the first fixed-point input vector (vector X' 1_fxp ) and the first binary value (b1) of the binary vector (Vector B), and outputs the first fixed-point output element (p 11_fxp ) can be output.

[0126] In one embodiment, the multiplication and accumulation unit 671 may include a multiplication unit 671_4 and an accumulation unit 671_5. The multiplication unit 671_4 may include a sign calculation unit 671_3, an exponent calculation unit 671_2, and a mantissa calculation unit 671_1.

[0127] In one embodiment, the sign calculation unit 671_3 receives the sign data of the elements of the first input vector (vector X1) and the sign data of the elements of the second input vector (vector Y1), and outputs the first output element (z 11 ) code data can be output.

[0128] In one embodiment, the exponent calculation unit 671_2 receives exponent data of a plurality of elements of a first input vector (vector X1) and exponent data of a plurality of elements of a second input vector (vector Y1), and calculates a first output element (z 11 ) exponent data can be output.

[0129] In one embodiment, the mantissa calculation unit 671_1 receives mantissa data of a plurality of elements of a first input vector (vector X1) and mantissa data of a plurality of elements of a second input vector (vector Y1), and calculates a first output element (z 11) can be output. In addition, the mantissa calculation unit 671_3 can output the mantissa data of the first fixed-point input vector (vector X' 1_fxp ) and the first binary value (b1) of the binary vector (Vector B), and outputs the first fixed-point output element (p 11_fxp ) can be output.

[0130] In one embodiment, the accumulator 671_5 performs an accumulation operation by adding the data received from the multiplier 671_4, and outputs the first output element (z 11 ) can be output.

[0131] In one embodiment, the multiplication and accumulation unit 671 may further include a multiplexer 671_6. The multiplexer 671_6 outputs the first fixed-point output element (p 11_fxp ) and outputs the first output element (z 11 ) can be received. The multiplexer 671_6 selects the first fixed-point output element (p 11_fxp ) or outputs the first output element (z 11 ) can be output.

[0132] 18 is a diagram illustrating the operation of the second data type converter according to an embodiment. For the sake of simplicity, the first fixed-point output element (p 11_fxp ) will be representatively described. However, the scope of the present disclosure is not limited thereto. For example, the second data type converter 680 may be configured to operate on any fixed-point output element (p ij_fxp ) could work in a similar way.

[0133] Referring also to FIG. 6, in one embodiment, the second data type converter 680 converts the first output matrix (ZM) and the fixed-point output vector (vector P) from the multiply and accumulate operator array 670. 1_fxp, vector P 2_fxp ,…,vector P n_fxp ) can be received. The second data type converter 680 outputs the first output matrix (ZM) as output data as is, or converts it into a fixed-point output vector (vector P 1_fxp , vector P 2_fxp ,…,vector P n_fxp ) data type to floating point and convert the converted output vectors (Vector P1, Vector P2, …, Vector P n ) as output data. For example, upon receiving the second level control signal (SEL) from the input parser 640, the second data type converter 680 can directly output the first output matrix (ZM) received from the multiplication and accumulation operator array 670 as output data. Alternatively, upon receiving the first level control signal (SEL) from the input parser 640, the second data type converter 680 can directly output the first output matrix (ZM) received from the multiplication and accumulation operator array 670 as output data. Alternatively, upon receiving the first level control signal (SEL) from the input parser 640, the second data type converter 680 can directly output the fixed-point output vector (vector P 1_fxp , vector P 2_fxp ,…,vector P n_fxp ) data type to floating point and convert the converted output vectors (Vector P1, Vector P2, …, Vector P n ) can be output as output data.

[0134] Referring to FIG. 18, the first fixed-point output element (p 11_fxp ) can include exponent data (S) and mantissa data (MTS). The second data type converter 680 outputs a first fixed-point output element (p 11_fxp ) data type to floating point and output the first output element (p 11 Specifically, the second data type converter 680 can output a first fixed-point output element (p 11_fxp For example, the second data type converter 680 may add exponent data (EXP) to the first fixed-point output element (p11_fxp ) to the first output element (p 11 ) can be determined as the exponent (EXP) of the first fixed-point output element (p 11_fxp ) is the first fixed-point output vector (vector P 1_fxp ), the first fixed-point output element (p 11_fxp ) can be determined as the first exponent data (EXP1).

[0135] In one embodiment, the second data type converter 680 outputs a first fixed-point output element (p 11_fxp ) mantissa data and the first output element (p 11 ) according to the difference in the number of bits of the mantissa data of the first output element (p 11 ) can be added to the LSB position of the mantissa data, but the scope of the present disclosure is not limited thereto.

[0136] According to one embodiment, the second data type converter 680 can convert the data type of the fixed-point output elements to floating-point in the manner described above and output a second output matrix (PM) including the output elements as output data.

[0137] FIG. 19 is a block diagram illustrating a neural processing system according to an embodiment. The neural processing system 200 can infer information contained in input data by training (or learning) a neural network or analyzing input data using the neural network. The neural processing system 200 can determine a situation based on the inferred information or control the configuration of an electronic device in which the neural processing system 200 is installed. For example, the neural processing system 200 can be applied to smartphones, tablet devices, smart TVs, augmented reality (AR) devices, Internet of Things (IoT) devices, autonomous vehicles, robotics, medical devices, drones, advanced driver assistance systems (ADAS), video display devices, measuring devices, and the like, which perform voice recognition, video recognition, and video classification using neural networks. The neural processing system 200 can also be installed in various types of electronic devices.

[0138] 19, the neural processing system 200 may include a central processing unit (110), a neural processing unit (120), a volatile memory device (130), a nonvolatile memory device (140), and a user interface (150). In one embodiment, some or all of the components of the neural processing system 200 (e.g., the central processing unit 110, the neural processing unit 120, the volatile memory device 130, and the nonvolatile memory device 140) may be formed on a single semiconductor chip. For example, the neural processing system 200 may be implemented as a system-on-chip (SoC). The central processing unit 110, the neural processing unit 120, the volatile memory device 130, the non-volatile memory device 140, and the user interface 150 can be connected to each other via a bus (BUS).

[0139] The central processing unit 110 can control the operations of the neural processing system 200. The central processing unit 110 can include one processor core (single core) or multiple processor cores (multi-core). The central processing unit 110 can process or execute programs and / or data stored in storage areas such as the volatile memory device 130 and the non-volatile memory device 140.

[0140] The central processing unit 110 can execute an application program (application) and control the neural processing unit 120 to perform neural network-based tasks required by the execution of the application program. The neural network can include at least one of various types of neural network models, such as a convolution neural network (CNN), a region with convolution neural network (R-CNN), a region proposal network (RPN), a recurrent neural network (RNN), a stacking-based deep neural network (S-DNN), a state-space dynamic neural network (S-SDNN), a deconvolution network, a deep belief network (DBN), a restricted Boltzmann machine (RBM), a fully convolutional network, a long short-term memory (LSTM) network, a classification network, and a transformer.

[0141] A neural network model may include multiple layers. Each of the multiple layers may be implemented to receive input data and generate output data. The generated output data may then be used as input data for another layer. Each of the multiple layers may transform input data into output data through neural network operations. For example, each of the multiple layers may perform an inner product of an input matrix corresponding to the input data of the layer and a weight matrix to generate an output matrix corresponding to the output data of the layer. However, the scope of the present disclosure is not limited thereto, and each of the multiple layers may generate output data by converting an input matrix corresponding to the input data of the layer in any manner. For example, each of the multiple layers may be implemented to generate output data of the layer by sequentially multiplying an input matrix corresponding to the input data of the layer by multiple weight matrices, or to generate output data of the layer by transforming an input matrix based on a transformation parameter.

[0142] The neural processing unit 120 can access the volatile memory device 130 to perform neural network operations. For example, the neural processing unit 120 can read parameters stored in the volatile memory device 130 to perform operations on layer input data, and can temporarily store intermediate data generated during operations in the volatile memory device 130.

[0143] To perform neural network operations, the neural processing unit 120 can access the nonvolatile memory device 140. For example, the neural processing unit 120 can read operation parameters (e.g., weight values, bias values, etc.) and input data (e.g., input feature maps) for the neural network stored in the nonvolatile memory device 140 to perform the neural network operations.

[0144] In one embodiment, the neural processing unit 120 may include a matrix multiplier (121; MATMUL) for performing neural network operations. For example, the matrix multiplier 121 may perform an inner product operation on the input matrix and weight matrix of each layer. The matrix multiplier 121 may perform an inner product operation on the input matrix and weight matrix of each layer, or may include a multiplication and accumulation (MAC) operator for performing an inner product operation on the input matrix. The matrix multiplier 121 may perform multiplication and accumulation operations on input data.

[0145] In one embodiment, the volatile memory device 130 can be used as a buffer memory, working memory, or cache memory for the central processing unit 110. Volatile memory includes dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FeRAM), etc. However, the scope of the present disclosure is not limited thereto.

[0146] The non-volatile memory device 140 can store data for the operation of the neural processing system 200. For example, the non-volatile memory device 140 can store an operating system (OS) of the neural processing system 200, operation parameters (e.g., weight values, bias values, etc.) for the neural network, parameters for quantization of the neural network (e.g., scale factors, bias values, etc.), input data (e.g., input feature maps), and output data (e.g., output feature maps). The operation parameters, quantization parameters, input data, and output data may be floating-point data or integer data. The non-volatile memory device 140 may include a read-only memory (ROM), a programmable read-only memory (PROM), an electrically programmable read-only memory (EPROM), an electrically erasable and programmable read-only memory (EEPROM), a flash memory, a phase-change random access memory (PRAM), a magnetic random access memory (MRAM), a resistive random access memory (RRAM), a ferroelectric random access memory (FRAM), etc. However, the scope of the present disclosure is not limited thereto.

[0147] The central processing unit 110 can communicate with a user via a user interface 150. The central processing unit 110 can provide input data provided by the user via the user interface 150 to the volatile memory device 130 or the neural processing unit 120. The central processing unit 110 can return output data generated by the artificial intelligence model based on the input data to the user via the user interface 150.

[0148] FIG. 20 is an exemplary diagram illustrating multi-head attention according to an embodiment.

[0149] The multi-head attention 2000 is a mechanism used in neural network models such as transformer models and automatic speech recognition (ASR) models, and has a structure in which a scaled dot-product attention (2120) structure is superimposed. In one embodiment, the multi-head attention 2000 can perform self-attention regarding embedding of input data. Embedding can refer to the result of converting natural language used by humans into a vector form that can be understood by machines, or the series of processes involved.

[0150] When processing sequence data such as natural language, long-term dependencies can be a problem. Self-attention is used to resolve these long-term dependencies and can measure the relationships between words in a sentence. Specifically, the relationship value between each word and other words can be calculated, and this value is called the attention score. Words with a high degree of relationship can have a high attention score. A table containing attention scores is called an attention map. Multihead Attention 2000 can create multiple attention maps to check attention for various feature values.

[0151] In the multi-head attention 2000, the input of the scale dot product attention layer 2120 includes a query (Q), a key (K), and a value (V). For example, when searching for the meaning of a specific word through an English dictionary, the specific word corresponds to the query, the word registered in the dictionary corresponds to the key, and the meaning of the key corresponds to the value.

[0152] The multi-head attention 2000 can reduce the dimensionality of the value (V), key (K), and query (Q) through a first linear operation layer 2110, perform self-attention through h scale dot-product attention layers 2120, and perform linear operations through concatenation 2130 and a second linear operation layer 2140 on the attention results.

[0153] In one embodiment, the linear operations performed by the first linear operation layer 2110 and the second linear operation layer 2140 may be referred to as matrix dot product operations. For example, the first linear operation layer 2110 may perform a dot product operation to multiply an embedding vector of input data by a specific weight in order to reduce the dimensions of the value (V), key (K), and query (Q) derived from the embedding information of the input data, and the second linear operation layer 2140 may perform a dot product operation to multiply a concatenated output matrix by a specific weight.

[0154] The first linear operation layer 2110 and the second linear operation layer 2140 can use a matrix multiplier (100 in FIG. 1) according to one embodiment to perform a matrix dot product operation. The matrix multiplier 100 according to one embodiment can receive an input matrix and a weight matrix to be input to the first linear operation layer 2110 and the second linear operation layer 2140 as input data, and perform a matrix dot product operation thereon.

[0155] FIG. 21 is an exemplary diagram for explaining the scale inner product attention of FIG.

[0156] The matrix multiplication layer 2111 of the scaled dot product attention 2120 can perform a dot product operation between the key (K) and the query (Q) and output an attention score indicating the degree of similarity. The scaling layer 2113 can perform scaling to adjust the magnitude of the attention score output from the first matrix multiplication layer 2111. The mask layer 2115 can prevent attention to erroneous connections by masking. The softmax operation layer 2117 calculates weights based on the attention scores, and the second matrix multiplication layer 2119 can perform a dot product operation on the weights output from the softmax operation layer 2117 and the value (V).

[0157] The first matrix multiplication layer 2111 and the second matrix multiplication layer 2119 may use a matrix multiplier 100 according to one embodiment. For example, the first matrix multiplication layer 2111 may receive a key (K) vector and a query (Q) vector as input data, and the second matrix multiplication layer 2119 may receive a value (V) vector and a weight matrix as input data. In one embodiment, the input data vectors of the first matrix multiplication layer 2111 and the second matrix multiplication layer 2119 may be floating-point data. The matrix multiplier 100 according to one embodiment may perform a matrix inner product operation on the key (K) vector and the query (Q) vector, which are input data of the first matrix multiplication layer 2111, and perform a matrix inner product operation on the value (V) vector and a weight matrix, which are input data of the second matrix multiplication layer 2119.

[0158] Although the embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements made by those skilled in the art using the basic concept of the present invention defined in the following claims also fall within the scope of the present invention. [Explanation of symbols]

[0159] 100, 600 matrix multiplier 610 First input matrix buffer 620 Second input matrix buffer 630 Wait Buffers 640 Input Parser 650 Input Vector Scaler 660 First Data Type Converter 670 Accumulator Array 680 Second Data Type Converter

Claims

1. an input vector scaler that generates a scaled input matrix based on a first input matrix and a plurality of scale factors; a first data type converter that converts the data type of the scaled input matrix to a fixed point to generate a fixed point input matrix; a multiplier and accumulate operator array that receives the fixed-point input matrix and a plurality of binary vectors and generates a fixed-point output matrix based on the fixed-point input matrix and the plurality of binary vectors, and receives the first input matrix and a second input matrix and generates a first output matrix based on the first input matrix and the second input matrix; a second data type converter that converts the data type of the fixed-point output matrix to a floating-point data type to generate a second output matrix; A matrix multiplier containing

2. an input parser that receives the second input matrix, the plurality of scale factors as a weight matrix, and the plurality of binary vectors from an external device, sequentially provides the plurality of scale factors to the input vector scaler, sequentially provides the second input matrix and the plurality of binary vectors to the multiplication and accumulation operator array, and provides a first signal that controls the multiplication and accumulation operator array and the second data type converter; The matrix multiplier of claim 1 further comprising:

3. The multiplication and accumulation array comprises: a first multiplier and accumulate unit that receives a first input vector of the first input matrix and a first input vector of the second input matrix, and receives a first fixed-point input vector of the fixed-point input matrix and a first binary value of the plurality of binary vectors; The matrix multiplier of claim 2 .

4. The first multiplier and accumulate unit performing a multiplication operation on first binary values of the plurality of binary vectors and the first fixed-point input vector based on a first signal at a first level to generate first fixed-point output elements of the fixed-point output matrix; performing a multiplication operation on a first input vector of the first input matrix and a first input vector of the second input matrix based on a first signal of a second level different from the first level, to generate a first output element of the first output matrix; The matrix multiplier of claim 3 .

5. The first data type converter: a first data type conversion circuit for receiving a first scaled input vector of the scaled input matrix; 3. The matrix multiplier of claim 2, wherein the first data type conversion circuit extracts a first exponent that is the largest among the exponents of each of a first plurality of scaled input elements included in the first scaled input vector, and converts the data type of each of the first plurality of scaled input elements to fixed point based on the first exponent.

6. The second data type converter:

6. The matrix multiplier of claim 5, further comprising: a first fixed-point output vector of the fixed-point output matrix and a first signal of the first level; and a first exponent for converting a data type of each of a first plurality of fixed-point output elements included in the first fixed-point output vector to floating point.

7. a first multiplexer that receives first data of a first data type and mantissa data of second data of a second data type different from the first data type, and outputs the first data based on a first signal and outputs the mantissa data of the second data based on a second signal different from the first signal; a first AND operation unit that receives a plurality of binary values and performs a logical AND operation between a first binary value among the plurality of binary values and the first data; an adder that adds output data of the first logical product operation unit and outputs first output data of the first data type; A MAC calculator including:

8. the first AND operation unit further receives mantissa data of third data of the second data type, and performs a logical AND operation on a first bit value of the mantissa data of the third data and the mantissa data of the second data; 8. The MAC computing unit according to claim 7, wherein the adder outputs mantissa data of the second output data of the second data type.

9. a second multiplexer that receives the first output data and the second output data, outputs the first output data based on the first signal, and outputs the second output data based on the second signal; The MAC calculator of claim 8 further comprising:

10. a data buffer that receives a first input vector and a second input vector from an external memory, and receives a plurality of scale factors and a plurality of binary vectors as weight vectors from the external memory; an input parser that receives the plurality of scale factors and the plurality of binary vectors and outputs the plurality of scale factors and the plurality of binary vectors together with a first signal at a first level, and receives the second input vector and outputs the second input vector together with a first signal at a second level different from the first level; an input vector scaler that generates a first scaled input vector based on the first input vector and a multiplication operation on the plurality of scale factors; a first data type converter that receives the first scaled input vector, extracts a first exponent from among the exponents of each of a first plurality of scaled input elements included in the first scaled input vector, and converts a data type of the first scaled input vector to a fixed point based on the first exponent; a MAC calculator that, upon receiving the first level first signal, accumulates the first scaled input vector based on the binary values of the plurality of binary vectors to generate a fixed-point output matrix; A matrix multiplier containing

Images