Hardware accelerator for performing computations of deep neural networks and electronic device comprising the same

Abstract

A hardware accelerator comprising: a processing core comprising a plurality of multipliers configured to perform one-dimensional (1D) subword parallelization between a sign and mantissa of a first tensor and a sign and mantissa of a second tensor, a first processing device configured to operate in a two-dimensional (2D) mode of operation in which a result of the computation of the plurality of multipliers is output, and a second processing device configured to operate in a three-dimensional (3D) mode of operation in which the result of the computation of the plurality of multipliers is accumulated in a channel direction before outputting a result of the accumulated computation.

Description

[0001] This application is based on and claims priority to Korean Patent Application No. 10-2022-0008090, filed on January 19, 2022, and Korean Patent Application No. 10-2022-0186268, filed on December 27, 2022, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference. TECHNICAL FIELD

[0002] The disclosure relates to a hardware accelerator for performing computation of a deep neural network and an electronic device including the same, and more particularly, to a hardware accelerator supporting learning and inference operations with various precisions by using block floating point and operating with high efficiency at each stage of a learning operation, and an electronic device including the same. BACKGROUND

[0003] Deep learning is developing very rapidly due to high-performance computing systems and growing open-source datasets. In addition, as the precision is improved, deep learning technology is being used in many applications such as computer vision, language modeling, or autonomous driving.

[0004] In order to use deep learning in applications, a process called training is required. In deep learning, training refers to a process of updating weights of a deep neural network (DNN) through a certain dataset. The better the weights are updated, the better the DNN can perform a given task.

[0005] Training includes a forward pass step, a backward pass step, a weight update step, etc. The forward pass step is a process of calculating a loss in the training process, and the backward pass step is a process of calculating a gradient of a loss function. The gradient is generally obtained through the chain rule and is propagated to all layers constituting the DNN in a direction opposite to that of the forward pass step. The weight update step is a process of updating weights constituting the DNN by subtracting a value obtained by multiplying the gradient of the loss function of the weights by a learning rate from the current weights.

[0006] Such a training process requires a considerable amount of computation, and thus takes a large amount of time when performed on a central processing unit (CPU). A graphics processing unit (GPU) is more suitable for parallel processing, and thus consumes less time than the CPU, but shows low utilization due to its structural characteristics.

[0007] Recently, many dedicated hardware accelerators have been proposed to overcome the disadvantages of CPUs and GPUs. However, the accelerators in the prior art support only a certain precision or exhibit high efficiency only for a certain training step (e.g., a forward pass step and a backward pass step). SUMMARY

[0008] Additional aspects will be set forth in part in the description which follows, and in part will be apparent from the description, or can be learned by practice of the presented embodiments of the disclosure.

[0009] According to an aspect of the disclosure, a hardware accelerator includes a plurality of multipliers that perform 1D subword parallelization between a sign and a mantissa of a first tensor and a sign and a mantissa of a second tensor. The hardware accelerator can include a first processing device that operates in a 2D operation mode to output a result of a computation of the plurality of multipliers. The hardware accelerator can include a second processing device that operates in a 3D operation mode to accumulate the result of the computation of the plurality of multipliers in a channel direction and output a result of the accumulated computation.

[0010] According to another aspect of the disclosure, an electronic device includes a hardware accelerator that performs 1D subword parallelization between a sign and a mantissa of a first tensor and a sign and a mantissa of a second tensor by using a plurality of multipliers and performs processing between a shared exponent of the first tensor and a shared exponent of the second tensor by using a shared exponent processor. The electronic device can include a processor configured to execute at least one instruction to control the hardware accelerator based on deep neural network information including at least one of a number of layers in a deep neural network, a type of layer, a shape of a tensor, a dimension of a tensor, an operation mode, a bit precision, a type of batch normalization, a type of a pooling layer, and a type of a rectified linear unit (ReLU) function. The electronic device can include a memory that stores the at least one instruction and the deep neural network. BRIEF DESCRIPTION OF DRAWINGS

[0011] The above and other aspects, features, and advantages of certain embodiments of the disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0012] Figure 1 is a diagram for describing a deep learning operation environment in the related art;

[0013] Figure 2 is a block diagram illustrating a configuration of an electronic device according to an embodiment;

[0014] Figure 3 is a diagram for describing a block floating point (BFP) according to an embodiment;

[0015] Figure 4 is a diagram for describing a main operation of a process of training a deep neural network according to an embodiment;

[0016] Figure 5is a diagram for describing a two-dimensional (2D) operation mode and a three-dimensional (3D) operation mode according to an embodiment;

[0017] Figure 6 is a diagram for describing a configuration of a hardware accelerator according to an embodiment;

[0018] Figure 7 is a diagram for describing Figure 6 a configuration of a sub-core illustrated in FIG. 1;

[0019] Figure 8 is a diagram for describing Figure 7 a detailed configuration of a processing unit illustrated in FIG. 1;

[0020] Figure 9 is a diagram for describing Figure 8 a detailed configuration of a multiplier illustrated in FIG. 1;

[0021] Figure 10 is a diagram for describing an operation of an electronic device according to an embodiment;

[0022] Figure 11 is a diagram for describing Figure 6 a detailed configuration of a rectified linear unit (ReLU)-pool unit of FIG. 1;

[0023] Figure 12 is a diagram for describing Figure 6 a detailed configuration of a kernel output buffer of FIG. 1;

[0024] Figure 13 is a diagram for describing Figure 6 a detailed configuration of a first-in first-out (FIFO) of FIG. 1;

[0025] Figure 14 is a diagram for describing Figure 6 a detailed configuration of a weight update unit of FIG. 1;

[0026] Figure 15 is a diagram for describing Figure 6 a detailed configuration of an FP2BFP converter of FIG. 1;

[0027] Figure 16 is a diagram for describing Figure 6 a detailed configuration of a quantization unit of FIG. 1;

[0028] Figure 17 is a diagram for exemplarily describing how an input tensor is mapped to a processing kernel in a Conv3 layer;

[0029] Figure 18 is a diagram for exemplarily describing how a weight tensor is mapped to a processing kernel in a Conv3 layer;

[0030] Figure 19 is a diagram for describing an operation of a sub-core according to a layer type of a deep neural network according to an embodiment;

[0031] Figure 20 is a diagram for describing an example of a mapping method in a 2D operation mode according to an embodiment;

[0032] Figure 21 is a diagram for describing an operation of an electronic device according to an embodiment;

[0033] Figure 22 is a diagram for describing a detailed configuration of a sharing exponent processor of Figure 6 ; and

[0034] Figure 23 is a block diagram exemplarily illustrating an electronic device according to an embodiment. DETAILED DESCRIPTION

[0035] Reference will now be made in detail embodiments, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout. In this regard, the present embodiments can have different forms and should not be construed as being limited to the descriptions set forth herein. Accordingly, the embodiments are merely described below, by referring to the drawings, to explain aspects of this description. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Expressions such as "at least one of," when preceding a list of elements, modify the entire list of elements and do not modify the individual elements of the list.

[0036] Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings. In the information (data) transmission process performed in the present disclosure, encryption / decryption can be applied as necessary, and the expressions describing the information (data) transmission process in the present disclosure and claims should be interpreted to include the case of encryption / decryption, although not mentioned. In the present disclosure, expressions such as "transmitting (transmitting) from A to B" or "receiving A from B" can include transmitting (transmitting) or receiving with another medium therebetween, and not only mean directly transmitting or receiving from A to B.

[0037] In describing the present disclosure, it is understood that the order of each operation is not limited unless the preceding step should be performed logically and chronologically before the subsequent operation. That is, except for the exceptional case as described above, even when the process described as the subsequent operation is performed before the process described as the preceding operation, it does not affect the nature of the present disclosure, and the claims should not be considered to limit the order of operations. Also, in the specification, "A or B" indicates not only either one of A and B selectively, but also is defined to include both A and B. In addition, as used herein, the term "include" can have a meaning including other elements than the listed elements.

[0038] In the specification, only the essential elements necessary for the description of the present disclosure are described, and elements irrelevant to the gist of the present disclosure can not be mentioned. In addition, it should not be interpreted as an exclusive meaning including only the mentioned elements, but as a non-exclusive meaning including other elements.

[0039] In the present disclosure, the term "value" is defined to include not only a scalar value but also a concept of a vector.

[0040] In the present disclosure, "convK layer" means a convolution layer in which the weight kernel size is KxK. For example, the weight kernel size of "conv3 layer" is 3x3, the weight kernel size of "conv5 layer" is 5x5, and the weight kernel size of "conv7 layer" is 7x7.

[0041] In the present disclosure, the term "one-dimensional (1D) subword parallelization" refers to an operation in which a series of subwords are input in the form of a 1D array into a series of multipliers, and then an arithmetic operation is performed thereon in parallel. For example, assuming that 16-bit data of each of a first tensor and a second tensor includes four 4-bit subwords, and the 4-bit subwords are input into four multipliers. The four 4-bit subwords of the first tensor can be input into the four multipliers in the form of a 1D array, respectively, and one of the four 4-bit subwords of the second tensor can be input into all of the four multipliers. In this case, it can be expressed that 1D subword parallelization is performed on the first tensor. Next, the four 4-bit subwords of the second tensor can be input into the four multipliers in the form of a 1D array, respectively, and one of the four 4-bit subwords of the first tensor can be input into all of the four multipliers. In this case, it can be expressed that 1D subword parallelization is performed on the second tensor. Since 1D subword parallelization is performed on each of the first tensor and the second tensor, 2D subword parallelization can be implemented.

[0042] The components described herein with reference to the term "unit", "module", "block", "…or block", etc. and the functional blocks shown in the drawings can be implemented as software, hardware, or a combination thereof. For example, the software can be machine code, firmware, embedded code, or application software. For example, the hardware can include a circuit, an electronic circuit, an integrated circuit, a processor, a computer, an integrated circuit, an integrated circuit core, a pressure sensor, an inertial sensor, a micro electro mechanical system (MEMS) device, a passive element, or a combination thereof.

[0043] Each operation of the present disclosure described below can be implemented as a computer operation through a known encoding method of the operation or calculation and / or an encoding appropriately considered for the present disclosure.

[0044] The specific equations described below are described as examples among many possible alternatives and should not be interpreted as the scope of the present disclosure being limited to the equations in the present disclosure.

[0045] Hereinafter, various embodiments will be described in detail with reference to the accompanying drawings.

[0046] Figure 1 is a diagram for describing a deep learning operation environment in the related art.

[0047] In the related art, a central processing unit (CPU) or a graphic processing unit (GPU) is utilized to train a deep neural network. However, the training of the deep neural network requires a large amount of calculation, and the CPU has a small number of arithmetic and logic units (ALUs), and thus requires a large amount of time to perform the training. In addition, the GPU has a large number of ALUs, but has low applicability, and thus it is difficult to apply to various training calculations.

[0048] Therefore, in the present disclosure, a hardware accelerator is described, which is highly applicable and has a large number of ALUs, and thus is faster and more flexible than a method using a GPU or a method using a CPU.

[0049] In addition, as Figure 1 shown in the related art, a calculation is performed by using one precision, i.e., FP32, and in the case of using FP32 for all data, a large-capacity memory 120 is required, and a high communication cost occurs in the process of transmitting and receiving data.

[0050] It is difficult to train a deep neural network on a mobile device having low resources in a case where a large-capacity memory and a high communication cost are required. Also, various steps are included in the training process, and the precision required for each step can be different from each other. In the related art, only one precision is used in the training process, and thus the training process takes a long time, whereas, in the present disclosure, a faster training can be performed by adaptively changing the precision according to the precision required for each step of the training process. Detailed configurations and operations of the hardware accelerator 200 according to the present disclosure will be described below.

[0051] Figure 2 is a block diagram illustrating a configuration of an electronic device 100 according to an embodiment.

[0052] Referring to Figure 2 , the electronic device 100 can include a communication device 110, a memory 120, a display 130, a manipulation input device 140, and a processor 150. Such an electronic device can be a mobile device such as a smartphone, and a device such as a personal computer (PC), a laptop computer, or a server.

[0053] The communication device 110 is provided to connect the electronic device 100 with an external device (not shown) through a local area network (LAN) and an Internet network or through a universal serial bus (USB) port or a wireless communication (e.g., Wi-Fi 802.11a / b / g / n, near field communication (NFC), Bluetooth) port. The communication device 110 can also be referred to as a transceiver.

[0054] The communication device 110 can receive a deep neural network to be trained and / or a data set for training the deep neural network.

[0055] In an embodiment, the communication device 110 can transmit the trained deep neural network to the outside, or can transmit a result value obtained by applying data provided from the outside to the deep neural network to the deep neural network.

[0056] In an embodiment, the communication device 110 can receive a parameter required to train the deep neural network, such as a size (or precision) of a mantissa to be used in the training, from the outside. Meanwhile, in an embodiment, various parameters can be directly input from a user through the manipulation input device 140 described below.

[0057] The memory 120 can store at least one instruction related to the electronic device 100. In detail, various programs (or software) for operating the electronic device 100 according to various embodiments can be stored in the memory 120.

[0058] In an embodiment, the memory 120 can be implemented in various forms such as a random access memory (RAM), a read only memory (ROM), a flash memory, a hard disk drive (HDD), an external memory, or a memory card, but is not limited thereto.

[0059] In an embodiment, the memory 120 can store a deep neural network required for machine learning or deep learning. Here, the deep neural network can be a deep learning network, but is not limited thereto, and various models can be applied as long as the models can update their internal weights based on a data set.

[0060] The display 130 displays a user interface window for receiving a selection of a function supported by the electronic device 100. In detail, the display 130 can display a user interface window for receiving a selection of various functions provided by the electronic device 100. The display 130 can be a monitor such as a liquid crystal display (LCD) or an organic light emitting diode (OLED) display, and can be implemented as a touch screen capable of simultaneously performing the function of the manipulation input device 140 to be described below.

[0061] The display 130 can display a message requesting input of various parameters to be applied to a training process. In an embodiment, the parameters can be directly input by a user or can be automatically selected according to characteristics of a deep neural network and a data set.

[0062] The manipulation input device 140 can receive a selection of a function of the electronic device 100 and a control command for the function from a user.

[0063] The processor 150 controls the overall operation of the electronic device 100. In detail, the processor 150 can control the overall operation of the electronic device 100 by executing at least one instruction stored in the memory 120. The processor 150 can be a single device such as a CPU or an application specific integrated circuit (ASIC), or can include a plurality of devices such as a CPU or a GPU.

[0064] When a command to train a deep neural network is input, the processor 150 can perform a training operation on the deep neural network by using an input data set. In this process, the processor 150 can perform the training operation by using an internal dedicated hardware accelerator. The hardware accelerator performs a calculation by using a block floating point (BFP) method, and can perform the calculation at various precisions.

[0065] To this end, the processor 150 can convert data constituting a training data set into data having the same exponent size and a sign and a mantissa of a preset size. Here, the preset size can be 4 bits, 8 bits, or 16 bits.

[0066] In this case, the hardware accelerator can perform arithmetic operations with different precisions during various processes of the deep neural network. In detail, the processor 150 can perform a training operation such that data is converted to have a first size (e.g., 4 bits or 8 bits) in at least one of a forward pass step and a backward pass step, and is converted to have a second size (e.g., 16 bits) larger than the first size in a weight update process. Also, in the weight update process, a loss gradient graph can be divided into blocks of a preset size, and a calculation operation can be performed in units of blocks into which the loss gradient graph is divided. Hereinafter, a detailed operation of the hardware accelerator will be described with reference to FIGS. 2 to 6. Figure 6 A detailed operation of the hardware accelerator is described.

[0067] In an embodiment, the processor 150 can generate a control signal for controlling the hardware accelerator, and provide the generated control signal to the hardware accelerator. Here, the control signal can include information such as a network type, a number of layers, a data dimension, a rectified linear unit (ReLU), or a pooling option.

[0068] As described above, the electronic device 100 according to the disclosure performs training or inference by using a dedicated hardware accelerator for training or inference of a deep neural network, and thus can quickly perform a training or inference operation. In addition, by performing a training process using a precision suitable for each step of the training process rather than using a fixed precision, the training can be flexibly and highly accurately performed.

[0069] Meanwhile, although the hardware accelerator is described above as a component within the processor 150, in an embodiment, the hardware accelerator can be a component separate from the processor 150. In addition, although the above description is made with reference to a process of training a deep neural network, in an embodiment, the operation can also be applied to an inference process using the trained deep neural network. Figure 2 It is illustrated and described that the operation is applied only to a process of training a deep neural network, but in an embodiment, the operation can also be applied to an inference process using the trained deep neural network.

[0070] Figure 3 is a diagram for describing a BFP according to an embodiment.

[0071] In the disclosure, a calculation is performed by using a BFP method. Before describing the BFP method, a general method of representing a floating point number will be described.

[0072] A floating point number can have various forms depending on a precision, but the most commonly used is FP32 20, which is a format specified by the Institute of Electrical and Electronics Engineers (IEEE). The FP32 20 represents a real number having a sign, an exponent, and a mantissa. For example, a floating point number can be expressed as Equation 1 below.

[0073] "Equation 1"

[0074]

[0075] Here, for a real number, x i denotes a real number, s i denotes a sign, m i denotes a mantissa, and e i denotes an exponent.

[0076] Meanwhile, in the FP32 20, the sign is indicated with 1 bit, the mantissa is indicated with 23 bits, and the exponent is indicated with 8 bits, so that the floating point number is indicated with a total of 32 bits.

[0077] However, in the case of processing data having 32 bits for all values, many resources can be wasted in the processing. Therefore, in the present disclosure, the concept of BFP is introduced to represent various pieces of data having the same exponent.

[0078] In detail, the BFP 30 is a special form of the above floating point representation, and each number shares the exponent 32 of the highest number in N number blocks. That is, as shown on the right side of Figure 3 “Equation 2”

[0079]

[0080] Here, es denotes a shared exponent of a block, and is

[0081] In an embodiment, each value consists of only a sign and a mantissa, as shown in Equation 3 below. “Equation 3”

[0082]

[0083] By using the BFP as described above, not only can the data storage space be greatly reduced, but also the real number calculation can be performed only with integer arithmetic operations.

[0084] In addition, various sizes of the sign and the mantissa are supported according to the precision. For example, the sizes of the sign and the mantissa can be 4 bits, 8 bits, or 16 bits. In an embodiment, the sizes of the sign and the mantissa can be differently applied for each step in the training process. For example, 4-bit signs and mantissas can be used for feature map calculation, and 8-bit or 16-bit signs and mantissas can be used for local gradient calculation or weight update.

[0085]

[0086] ​Thus, this disclosure supports mantissas of various sizes, and therefore models generated according to this disclosure can be executed on accelerators that support CPUs / GPUs with the same precision using the same number of exponent bits (e.g., bfloat 16). Furthermore, 4-bit, 8-bit, and 16-bit integer arithmetic operations can be performed by controlling the exponent processing program. This will be referenced below. Figure 6 Describe it.

[0087] Figure 4 This is a diagram illustrating the main operations used to describe the process of training a deep neural network according to an embodiment. (Reference) Figure 4 The training process includes, for example: Figure 4 The three main computational operations are shown.

[0088] The first operation is the training loss calculation operation 410, which can also be called the forward pass. The training loss calculation operation is performed on the set W0, ..., W1 with weight kernels. Co-1 Input features Figure X This involves performing a convolution operation to compute the output feature map Y. Input features Figure X The number of channels can be C i And the number of channels in the output feature map Y can be C. o The set of weighted kernels W0, ..., W Co-1 The quantity can be C o The width and height of the output feature map Y can be W and H, respectively. In this embodiment, the input feature map... Figure X The width and height of the input feature map Y, as well as the width W and height H of the output feature map Y, can be different from each other. In this embodiment, the input feature map Y... Figure X The width and height of the input feature map Y, as well as the width W and height H of the output feature map Y, can be the same as each other. In this case, the set of weighted kernels W0, ..., W... can be used. Co-1 Input features Figure X Zero padding is performed during convolution operations. When performing training loss calculation, the total loss for mini-batches can be calculated. In this disclosure, the input feature map may also be referred to as the input tensor, and the weight kernel may also be referred to as the weight tensor.

[0089] The second operation is the local gradient operation 420, which can also be called backpropagation. The local gradient operation propagates the loss to each layer in the network. In this operation, a set of kernels with transposed weights is executed. The convolution operation is performed, and the local gradients in each layer (e.g., the I-th layer) are input. The output of this process is the local gradient in the layer (e.g., the (I-1)th layer).

[0090] The final operation is a weight gradient calculation operation 430, which can also be referred to as a weight update. In the weight gradient calculation operation, a convolution operation is performed on the local gradient using the input map in each layer, and the weight gradient can be used to update the weight. For example, for a channel pair (c, k) including the c-th channel C i and the k-th channel C o , a convolution operation can be performed on the local gradient G Y [k] and the input feature Figure X [c] corresponding to the channel pair (c, k), and the output in this process can be the weight gradient ΔW ck .

[0091] As can be seen, this deep learning process is performed by calculating the loss and transferring the calculated loss, and the convolution operation is the main operation in each step.

[0092] Meanwhile, in the above operations, values are not accumulated and used in the processing of the weight update operation, but are accumulated and used in the forward pass and backward pass processing.

[0093] However, in the case of calculating accumulated values in all processes, a delay occurs in the calculation or efficiency is reduced, and thus, in the present disclosure, the calculation values are processed differently for each step of the deep neural network calculation. This will be described below with reference to Figure 5 .

[0094] Figure 5 is a diagram for describing a two-dimensional (2D) operation mode and a three-dimensional (3D) operation mode according to an embodiment.

[0095] The 3D operation mode 510 requires an operation of outputting an output feature map 514 by accumulating the partial output feature maps 513 obtained by performing a convolution operation (511*512) in the channel direction. The 2D operation mode 520 does not require an operation of accumulating the output feature maps 523 obtained by performing a convolution operation (521*522) in the channel direction. The operations in each mode are as shown in Table 1 below.

[0096] "Table 1"

[0097]

[0098] Referring to Table 1, in the case where weight gradient calculation, depth-wise (DW) convolution, dilated convolution, or up-convolution is performed when training a deep neural network, the hardware accelerator can operate in a 2D operation mode. Also, in the case where (general) convolution, point-wise convolution, or fully connected layer calculation in which a calculation result is accumulated is performed when training a deep neural network, the hardware accelerator can operate in a 3D operation mode. As described above, in the present disclosure, whether the 3D operation mode or the 2D operation mode is required in each calculation step of a deep neural network is determined, and accumulation of parallelization results is selectively performed and then output according to the result of the determination. Hereinafter, this will be described with reference to Figure 6 A detailed configuration of such an operation is described.

[0099] Meanwhile, the present disclosure aims to support various precisions in the process of training or inferring a deep neural network. Previously, an architecture having various precisions has been proposed. However, in the existing method, the utilization rate of a plurality of calculation cores (specifically, multiplication-accumulation (MAC) units) varies according to a change in precision. For example, BitFusion is an architecture supporting various precisions including 16-bit (size of a sign and a mantissa), 8-bit, and 4-bit precisions, and in the case of operating with 8-bit precision, the utilization rate is reduced by about 13.8% compared to the case of operating with 16-bit precision, and in the case of operating with 4-bit precision, the utilization rate is further reduced by about 22% compared to the case of operating with 16-bit precision.

[0100] In this regard, the present disclosure aims to support various precisions while operating a plurality of calculation cores with a high utilization rate at each precision.

[0101] Hereinafter, this will be described with reference to Figure 6 A detailed hardware configuration for achieving this purpose is described.

[0102] Figure 6 is a diagram for describing a configuration of a hardware accelerator according to an embodiment.

[0103] Referring to Figure 6 , the hardware accelerator 200 includes an accelerator core and a plurality of function blocks 251 to 267. The hardware accelerator 200 can be implemented as a hardware component such as an ASIC.

[0104] The accelerator core can include a processing core 210, a first processing device 230, a second processing device 220, and a core output buffer 240. The first processing device 230 and the second processing device 220 can also be referred to as reduction units as shown in Figure 6 .

[0105] The processing core 210 can perform a convolution operation or a general matrix multiplication (GEMM) operation. In detail, the processing core 210 can be hierarchically configured with a plurality of multipliers (or multiplication units) capable of performing 1D subword parallelization. The processing core 210 can include a plurality of multipliers that perform 1D subword parallelization between a sign and a mantissa of a first tensor and a sign and a mantissa of a second tensor. For ease of description, the following description is made assuming that the first tensor is an input tensor and the second tensor is a weight tensor, but the present disclosure is not limited thereto.

[0106] In an embodiment, a size of a shared exponent of the first tensor and a size of a shared exponent of the second tensor can be 8 bits. A size of a sign and a mantissa of the first tensor or a size of a sign and a mantissa of the second tensor can be one of 4 bits, 8 bits, and 16 bits. Based on the size of the sign and the mantissa of the first tensor or the size of the sign and the mantissa of the second tensor, the first tensor and the second tensor can be mapped to the processing core 210.

[0107] In an embodiment, the size of the sign and the mantissa of the first tensor or the size of the sign and the mantissa of the second tensor can be determined based on a forward pass step, a backward pass step, or a weight update step of training of the deep neural network.

[0108] In an embodiment, the processing core 210 is composed of only integer multipliers and adders, and has a hierarchical structure, e.g., multiplier -> processing element (PE) (or processing engine) -> processing unit (PU) -> sub-core -> (processing) core. The processing core 210 can include a plurality of sub-cores, each of which can include a plurality of PUs, and each of which can include a plurality of PEs. Each PE can include a plurality of multipliers. Although Figure 6 It is illustrated that the processing core 210 includes six sub-cores, each of which includes four PUs, each of which includes four PEs (or processing engines), and each of which includes nine multipliers, but this is an example, and the present disclosure is not limited thereto. For ease of description, an example of the configuration, function, and operation of the processing core 210 illustrated in FIG. 13 will be described. Figure 6

[0109] In an embodiment, a first group of the plurality of multipliers can perform a multiplication operation between a first subword of a series of subwords of a first value included in a first tensor (e.g., an input tensor) and a second subword of a series of subwords of a second value included in a second tensor (e.g., a weight tensor). A second group of the plurality of multipliers can perform a multiplication operation between a second subword of the series of subwords of the first value and the series of subwords of the second value.

[0110] ​Meanwhile, the hardware accelerator 200 can support various forms of data types with shared exponent or without any exponent. That is, the processor can support a first data type of fixed point type, a second data type with only integer, a third data type with sign and integer, and a fourth data type of real number type with shared exponent (i.e., BFP).

[0111] In an embodiment, the processing core 210 can perform a calculation by using only a significand (i.e., sign and mantissa), and an exponent can be processed by the shared exponent processor 205. For example, the shared exponent processor can process a shared exponent of a first tensor (e.g., an input tensor) and a shared exponent of a second tensor (e.g., a weight tensor). Reference will be made to Figure 22 A detailed configuration and operation of the shared exponent processor 205 are described.

[0112] In a calculation process, the mantissas of the input tensor and the weight tensor can be provided after being mapped to the above-described sub-core. For example, in a case where the size of the mantissa is 8 bits instead of 16 bits, the number of input channels mapped to the processing core 210 can be twice the number of input channels in a case where the size of the mantissa is 16 bits. For example, in a case of 4 bits, 4 times more input channels than in a case of 16 bits can be mapped. In an embodiment, as the size of the weight kernel increases, the number of input channels can be proportionally reduced.

[0113] For example, in a calculation of a Conv3 layer of a deep neural network, in a case where the size of the sign and the mantissa of a first tensor (e.g., an input tensor) is 16 bits, the first tensor corresponding to one input channel of the Conv3 layer can be broadcast to four PUs constituting one sub-core.

[0114] In a case where the size of the sign and the mantissa of the first tensor is 8 bits, the first tensor corresponding to two input channels of the Conv3 layer can be broadcast to four PUs constituting one sub-core.

[0115] In a case where the size of the sign and the mantissa of the first tensor is 4 bits, the first tensor corresponding to four input channels of the Conv3 layer can be broadcast to four PUs constituting one sub-core. Reference will be made to Figure 17 The operation of the processing core 210 according to an embodiment is described in detail.

[0116] In a case where the deep neural network includes a convolution layer having a weight kernel size larger than that of the Conv3 layer, a plurality of clustered sub-cores can process a single channel or a plurality of channels. For example, in a case of a Conv5 layer of a deep neural network, three sub-cores can process a single channel or a plurality of channels. In a case of a Conv7 layer of a deep neural network, six sub-cores can process a single channel or a plurality of channels. The number of channels processed according to data precision is the same as the above-described Conv3.

[0117] As such, since the number of channels to be mapped varies according to the size of the sign and mantissa and the size of the weight kernel, operations can be performed in various combinations. Detailed operations according to various combinations will be described below.

[0118] Although an example in which the processing kernel includes six sub-kernels and each sub-kernel includes four PUs is shown and described, in an embodiment, the number of sub-kernels and PUs of each sub-kernel can be configured to have adaptive values according to the size of the mantissa supported and the number of channels to be processed at the same time. Detailed configurations of the sub-kernels constituting the processing kernel will be described below with reference to FIG. 3. Figure 7 Detailed configurations of the sub-kernels constituting the processing kernel will be described below.

[0119] The first processing device 230 is configured to output the output graphs of the processing kernel 210 without accumulating them in the channel direction when operating in the 2D operation mode. That is, the first processing device 230 can operate in the 2D operation mode in which the calculation results of the plurality of multipliers are output without accumulation in the channel direction. The first processing device 230 can include six 4-way adder trees 231, six bit truncators 233, a selective 6-way adder tree 235, an arithmetic converter 236, and an accumulator 237. However, the present disclosure is not limited to the illustrated example, and for example, in the case where the processing kernel 210 includes i sub-kernels and each sub-kernel includes j PUs, the first processing device 230 can be understood to include i j-way adder trees, i bit truncators, and a selective i-way adder tree.

[0120] Each of the six 4-way adder trees 231 can sum the outputs of the four PUs within one sub-kernel. Each of the six 4-way adder trees 231 corresponds to one of the six sub-kernels and can sum the outputs of the four PUs included in each of the six sub-kernels.

[0121] Each of the six bit truncators 233 can round the output results of the corresponding 4-way adder tree 231 to have a preset number of bits. However, the present disclosure is not limited thereto, and each of the six bit truncators 233 can round the output results of the corresponding 4-way adder tree 231 up or down.

[0122] In an embodiment, the selective 6-way adder tree 235 can selectively sum the outputs of the plurality of 4-way adder trees 231. In detail, the selective 6-way adder tree 235 can receive the outputs of the bit truncators 233 and selectively accumulate or separately output the output results.

[0123] The arithmetic converter 236 can convert the BFP into FP32. The arithmetic converter 236 can include at least one of a data type converter, a leading zero counter, a barrel shifter, and a normalizer. According to an embodiment, training accuracy can be maintained by performing batch normalization sensitive to precision and data format using a value obtained by converting into FP32. The arithmetic converter 236 can output 32-bit floating point data (i.e., FP32 part and data) based on the exponent operation result output by the shared exponent processor 205 and the sign and mantissa operation result output by the selective 6-way adder tree 235.

[0124] In an embodiment, the accumulator 237 can accumulate the value obtained by converting into FP32. The accumulator 237 can also be referred to as an FP32 adder as Figure 6 indicated. The accumulated value can be stored in a register (or buffer). The accumulator 237 can add the value obtained by converting into FP32 to the accumulated value from the register (or buffer). That is, the accumulator 237 can accumulate the partial sum psum, which is the value obtained by converting into FP32.

[0125] The second processing device 220 is configured to accumulate and output the output map of the processing core 210 in the channel direction when operating in the 3D operation mode. That is, the second processing device 220 can operate in the 3D operation mode in which the calculation results of a plurality of multipliers are accumulated in the channel direction and then output. The second processing device 220 can include four 6-way adder trees 221, six arithmetic converters 223, six accumulators 225, and a selective 4-way adder tree 227. However, the present disclosure is not limited to the illustrated example, and for example, in the case where the processing core 210 includes i sub-cores and each sub-core includes j PUs, the second processing device 220 can be understood to include j i-way adder trees, j arithmetic converters, j accumulators, and a selective j-way adder tree.

[0126] Each of the plurality of 6-way adder trees 221 sums the outputs of the PUs corresponding to each other in different sub-cores. For example, each sub-core can include first to fourth PUs. In this case, the first 6-way adder tree can sum the outputs of the first processing units of the sub-cores, the second 6-way adder tree can sum the outputs of the second PUs of the sub-cores, the third 6-way adder tree can sum the outputs of the third PUs of the sub-cores, and the fourth 6-way adder tree can sum the outputs of the fourth PUs of the sub-cores. In detail, because the sub-cores have different input channels, it is necessary to perform a summing operation in the channel direction when operating in the 3D operation mode. For this purpose, the 6-way adder tree 221 can receive the outputs from the corresponding PUs within the plurality of sub-cores and then perform a summing operation. Through this process, the summing operation can be performed in the channel direction.

[0127] The plurality of arithmetic converters 223 can convert the BFP into FP32. The plurality of arithmetic converters 223 can include at least one of a data type converter, a leading zero counter, a barrel shifter, and a normalizer. According to an embodiment, the plurality of arithmetic converters 223 performs batch normalization sensitive to precision and data format using values obtained by conversion into FP32, and thus, training precision can be maintained. The plurality of arithmetic converters 223 can output 32-bit floating point data (i.e., FP32 part and data) based on the exponent operation result output by the shared exponent processor 205 and the sign and mantissa operation result output by the plurality of 6-way adder trees 221.

[0128] In an embodiment, the accumulators 225 can accumulate values obtained by conversion into FP32. The accumulators 225 can also be referred to as FP32 adders, as shown in Figure 6 The accumulated values can be stored in a register (or buffer). The accumulators 225 can add values obtained by conversion into FP32 to the accumulated values from the register (or buffer). That is, the accumulators 225 can accumulate partial sums psum, which are values obtained by conversion into FP32.

[0129] The selective 4-way adder tree 227 selectively sums the outputs of the respective accumulators 225 according to a precision mode.

[0130] The core output buffer 240 selectively outputs the output of the first processing device 230 or the second processing device 220. In detail, the core output buffer 240 can output the output value of the first processing device 230 in the weight update step, and output the output value of the second processing device 220 in the forward pass and backward pass steps.

[0131] In an embodiment, the form (word number) of data output from the first processing device 230 or the second processing device 220 can depend on the operation mode, precision, and the size of the weight kernel. However, because it is inefficient for a subsequent module to perform an adaptive operation according to each size, the core output buffer 240 can convert the output data input thereto to have the same size, and then output the converted result. The detailed configuration and operation of the core output buffer 240 will be described below with reference to Figure 12

[0132] The FSM block 251 can receive a control signal from the processor 150 and optimize the received control signal according to the operation state of the processing core. In an embodiment, the optimized control signal can be distributed to each component through the control signal distributor 252. The detailed operation of the FSM block 251 and the control signal distributor 252 will be described below with reference to Figure 10

[0133] ​​Input buffer 253 can be used as a receiver to receive input feature maps. Input buffer 253 can send the input feature maps to processing core 210. Weight buffer 254 can be used as a receiver to receive weight kernels. Weight buffer 254 can send weight kernels to processing core 210. Output buffer 255 can receive data output from the accelerator core. Output buffer 255 can send data output from the MAC operator to the outside.

[0134] Meanwhile, deep learning mainly consists of DNN layers and non-DNN layers. Past deep learning accelerators were designed to accelerate only the computation of DNN layers because the computational load required in DNN layers is quite large.

[0135] In recent years, the internal structure of deep learning networks has changed, and the amount of computation required in non-DNN layers has increased. Therefore, high-speed computation is required not only for DNN layers but also for non-DNN layers.

[0136] Therefore, in this disclosure, a non-DNN layer accelerator (or additional accelerator, multiple computation modules, and multiple functional blocks) is used. This non-DNN layer accelerator may include a batch normalization unit 261, a ReLU pooling unit 262, a masking unit 263, a FIFO 264, a weight update unit 265, an FP2BFP converter 266, and a quantization unit 267. The batch normalization unit 261, ReLU pooling unit 262, masking unit 263, FIFO 264, weight update unit 265, FP2BFP converter 266, and quantization unit 267 may also be referred to as a batch normalization circuit, a ReLU pooling circuit, a masking circuit, a FIFO circuit, a weight update circuit, an FP2BFP converter circuit, and a quantization circuit, respectively.

[0137] Such non-DNN layer accelerators can perform computations for non-DNN layers. In other words, deep learning DNN layers can handle the computations in the aforementioned accelerator cores, and other non-DNN layers can perform computations by using the aforementioned multiple computation modules.

[0138] At the same time, although reference Figure 6 A non-DNN layer accelerator is described as being arranged outside the accelerator core; however, in some embodiments, the non-DNN layer accelerator can also be arranged inside the accelerator core. Furthermore, in some embodiments, the operation of the aforementioned multiple operation modules can be stopped, and only the GEMM operation can be performed.

[0139] The batch normalization unit 261 performs batch normalization. The batch normalization unit 261 can perform batch normalization based on the output of the kernel output buffer 240. In detail, batch normalization is a processing method for finding a weight parameter with faster convergence, and makes the training process more stable by reducing internal covariant shift. In an embodiment, in order to update the batch normalization parameter (e.g., perform mean or variance), it is generally required to read each input tensor three times from the memory, but in the present disclosure, range batch normalization can be used. Accordingly, the number of memory accesses can be reduced by half compared to the prior art method.

[0140] In general, a nonlinear activation function and a selective pooling layer are arranged after the batch normalization unit 261. However, because there can be no pooling layer between the batch normalization layer and the convolution layer, the flexible ReLU pool unit 262 is used in the present disclosure. The ReLU pool unit 262 can perform ReLU function values and pooling values based on the output of the batch normalization unit 261. The detailed configuration and operation of the ReLU pool unit 262 will be described below with reference to FIG. 4. Figure 11 The detailed configuration and operation of the ReLU pool unit 262 are described.

[0141] The masking unit 263 can be used in the backward passing process to minimize energy consumption of accessing unnecessary feature maps (Fmaps). For example, the output value of the backward passing of the ReLU layer has a value of 0 or 1. For example, when the input value of the forward passing is a positive number, the output value can have a value of "1", and when the input value of the forward passing is a negative number, the output value can have a value of "0". In the present disclosure, in the case where both the ReLU and the pooling layer exist, data can be more efficiently stored by fusing the outputs of the ReLU and the pooling layer in the backward passing rather than storing them separately.

[0142] The FIFO 264 can store and output the output data of the ReLU pool unit. The detailed configuration and operation of the FIFO 264 will be described below with reference to FIG. 5. Figure 13 The detailed configuration and operation of the FIFO 264 are described.

[0143] The weight update unit 265 can update the weights of the deep neural network. In detail, the weight update unit 265 can receive the input of the weight gradient and the learning rate, and update each weight element in the deep neural network according to the input weight gradient and the learning rate. The weight update unit 265 can be connected to the kernel output buffer 240 and the FIFO 264. The detailed configuration and operation of the weight update unit 265 will be described below with reference to FIG. 6. Figure 14 The detailed configuration and operation of the weight update unit 265 are described.

[0144] The FP2BFP converter 266 converts the data type of the output value and outputs the conversion result. Specifically, the MAC operation in the accelerator core is basically performed in the BFP manner. Accordingly, the FP2BFP converter 266 can convert the floating point type data (i.e., the output of the FIFO 264) into the type of the BFP 30. The detailed configuration and operation of the FP2BFP converter 266 will be described below with reference to Figure 15 The detailed configuration and operation of the FP2BFP converter 266 are described.

[0145] The quantization unit 267 quantizes the input value (i.e., the output of the FP2BFP converter 266) according to a predefined precision and outputs the quantized value. In detail, in the present disclosure, BFP24, BFP16, and BFP12 are supported. The BFP24, BFP16, and BFP12 have effective lengths of 16 bits, 8 bits, and 4 bits, respectively, and thus the input value can be rounded to fit each effective length. The detailed configuration and operation of the quantization unit 267 will be described below with reference to Figure 16 The detailed configuration and operation of the quantization unit 267 are described.

[0146] Meanwhile, the processing core is included in the processor as shown and described in Figure 6 , thus each of the above-described components can be implemented as a higher concept. That is, the above-described accelerator core can be implemented as a device such as an electronic device, and the processing core 210 can be implemented as a processor.

[0147] Figure 7 is a diagram for describing the configuration of the sub-core shown in Figure 6 , and Figure 8 is a diagram for describing the detailed configuration of the processing unit shown in Figure 7 . For ease of description, it is assumed that the precision of the "sign and mantissa" of the input tensor X and the weight tensor W is 16 bits, but the present disclosure is not limited thereto. It is assumed that the size of the weight tensor W as a weight kernel is 3x3, but the present disclosure is not limited thereto.

[0148] Referring to Figure 7 and Figure 8 , the sub-core 710 can include a plurality of PUs (e.g., PU3, PU2, PU1, and PU0). As in the illustrated example, the number of PUs can be four, but the present disclosure is not limited thereto. In an embodiment, each PU can include a plurality of PEs.

[0149] For example, each PU can include four PEs 810, 820, 830, and 840. Figure 8 An example is shown assuming that the PU 800 is referred to as PU0 in Figure 7 . The outputs of the PEs 810, 820, 830, and 840 can be summed by a 4-way adder tree 850. The 4-way adder tree 850 can output a sum value having a bit width of PU0.

[0150] In an embodiment, one PE can include nine multipliers, a 9-way adder tree, and selective shift logic. The 9-way adder tree can sum outputs of the nine multipliers. The selective shift logic (or can be referred to as selective shift logic circuit) can shift bits corresponding to the sum of the values by a predetermined number of bits before passing the sum of the values to the 4-way adder tree 850. In an embodiment, the multipliers constituting the PE can be Baugh-Wooley multipliers.

[0151] In an embodiment, the nine multipliers are clustered into one cluster in each PE. Such a PE can perform subword parallelization on the input tensor X.

[0152] In detail, when an input 16-bit input tensor X is inputted, the input tensor X is mapped to each PE in units of 4-bit subwords. For example, each of the 16-bit input tensor X elements X 0：8 may include 4-bit subwords x0, x1, x2, and x3. Each of the 4-bit subwords x0, x1, x2, and x3 can be mapped to one of the four PEs included in each of the PUs PU0, PU1, PU2, and PU3.

[0153] In an embodiment, the weight tensor W can also be applied to the four PUs within the same subkernel in parallel. For example, in the case of a 16-bit weight tensor W, only the first 4-bit subword w3 of each of the 16-bit weight tensor W elements W 0：8 may be mapped to the fourth PU PU3, and the other subwords are mapped to the other PUs. That is, the other subwords w2, w1, and w0 can be passed to the third PU PU2, the second PU PU1, and the first PU PU0, respectively. Outputs of the first to fourth PUs PU0, PU1, PU2, and PU3 can be selectively summed by the selective adder tree 720.

[0154] By hierarchically using the subword parallelization as described above, the accumulated partial sums can be doubled when a 2-fold reduction in precision occurs for each of X and W.

[0155] The operation as described above is for the case where the mantissa is 4 bits, and in the case where the mantissa is 8 bits and 16 bits or the precision of the weights is different from each other, some of the above-described PEs and PUs can operate in coordination with each other. The operation will be described below with reference to Figures 17 to 20 .

[0156] Referring to Figures 6 to 8 , the subkernel can perform 1D subword parallelization on the weight tensor W, and the PU can perform 1D subword parallelization on the input tensor X.

[0157] Figure 9 is used to describe Figure 8A detailed configuration of the illustrated multiplier. For ease of description, it is assumed that the size of the weight tensor W as a weight kernel is 3x3 or is convenient for description, but the present disclosure is not limited thereto.

[0158] Referring to Figure 9 , the PE according to the present disclosure includes at least one multiplier. Such a multiplier supports signed and unsigned operations. One global sign bit can be used to indicate whether an input or weight tensor is signed or unsigned. As a multiplier of the present disclosure, a 5-bit multiplier is used. In detail, in the present disclosure, various mantissa sizes are supported, and the smallest unit of the supported mantissa has a 4-bit size, and thus, a multiplier capable of simultaneously processing a global sign bit and one mantissa is used.

[0159] Referring to Figure 9 and Figure 8 , the nine multipliers included in the PE 810 can perform signed or unsigned operations by using a global sign bit sign_x of an input tensor and a global sign bit sign_w of a weight tensor. Figure 8 The precision of the input tensor and the weight tensor is shown to be 16 bits, but in the present disclosure, a plurality of precisions (for example, 4 bits, 8 bits, and 16 bits) are supported for each of the input tensor and the weight tensor.

[0160] The nine multipliers included in the PE 810 can perform multiplication by using 4-bit subwords x03, x13, …, x83 having a first series of bits (for example, [15:12]) of a 3x3 input tensor and 4-bit subwords w00, w10, …, w80 having a fourth series of bits (for example, [3:0]) of a 3x3 weight tensor as operands. For example, the multipliers can perform multiplication by using x03 and w00 as operands, multiplication by using x13 and w10 as operands, and similarly, multiplication by using x83 and w80 as operands.

[0161] The nine multipliers included in PE 820 can perform multiplication by using 4-bit sub-words x02, x12,..., x82 of a second series of bits (e.g., [11:8]) of the 3x3 input tensor and 4-bit sub-words w00, w10,..., w80 of a fourth series of bits (e.g., [3:0]) of the 3x3 weight tensor as operands. Similarly, the nine multipliers included in PE 830 can perform multiplication by using 4-bit sub-words x01, x11,..., x81 of a third series of bits (e.g., [7:4]) of the 3x3 input tensor and 4-bit sub-words w00, w10,..., w80 of the fourth series of bits (e.g., [3:0]) of the 3x3 weight tensor as operands. Similarly, the nine multipliers included in PE 840 can perform multiplication by using 4-bit sub-words x00, x10,..., x80 of a fourth series of bits (e.g., [3:0]) of the 3x3 input tensor and 4-bit sub-words w00, w10,..., w80 of the fourth series of bits (e.g., [3:0]) of the 3x3 weight tensor as operands.

[0162] Referring back to Figure 9 , the multiplier 900 can include a first multiplexer (MUX) 910, a 5b x 5b multiplier kernel 920, a first register 930, a second register 940, and a second MUX 950. The multiplier 900 can support a fixed dataflow that enables reuse of data.

[0163] The first MUX 910 can receive a previous weight tensor through a first input terminal and a current weight tensor through a second input terminal. The first MUX 910 can output the current weight tensor or the previous weight tensor in response to a keep signal. For example, the first MUX 910 can output the previous weight tensor in response to a logic high keep signal. In contrast, the first MUX 910 can output the current weight tensor in response to a logic low keep signal. However, the present disclosure is not limited to the illustrated example, and the relationship between the logic value of the keep signal and the output of the first MUX 910 can be reversed. The first MUX 910 can include a plurality of switches or logic elements that turn on / off in response to a plurality of signals.

[0164] The multiplier kernel 920 can perform a multiplication operation by using a 5-bit input tensor and a 5-bit (current or previous) weight tensor as operands. The multiplier kernel 920 can output a multiplication result value. The output value of the multiplier kernel 920 can be stored in the first register 930.

[0165] The weight tensor output by the first MUX 910 can be stored in the second register 940. The weight tensor stored in the second register 940 can be transferred to the first input terminal of the first MUX 910.

[0166] The second MUX 950 can receive the weight tensor stored in the second register 940 through a first input terminal and receive the weight tensor output from the first MUX 910 through a second input terminal. The second MUX 950 can output the weight tensor stored in the second register 940 or the weight tensor output by the first MUX 910 in response to the bypass signal bypass. For example, the second MUX 950 can output the weight tensor output by the first MUX 910 in response to the logic high bypass signal bypass. In contrast, the second MUX 950 can output the weight tensor stored in the second register 940 in response to the logic low bypass signal bypass. However, the present disclosure is not limited to the illustrated example, and the relationship between the logic value of the bypass signal bypass and the output of the second MUX 950 can be reversed. The second MUX 950 can include a plurality of switches or logic elements that turn on / off in response to a plurality of signals.

[0167] The weight tensor output by the second MUX 950 can be transferred to a PE (e.g., 810) next to the PE (e.g., 820) including the multiplier 900. Figure 8 Figure 8

[0168] According to an embodiment, the weight tensor stored in the second register 940 can be maintained by a feedback loop formed by the keep signal. According to an embodiment, the number of cycles for loading data can be reduced by one cycle and two cycles at 8-bit precision and 16-bit precision, respectively. As the number of cycles for loading data is reduced, the number of times of extracting data from a memory can be reduced.

[0169] Figure 10 FIG. 1 is a diagram for describing an operation of an electronic device 100 according to an embodiment.

[0170] Referring to Figure 2 and Figure 6 and Figure 10 The electronic device 100 includes a processor 150 (e.g., a host CPU), an FSM block 251, a control signal distributor 252, and a hardware accelerator 200.

[0171] The processor 150 can identify a network type, a number of layers, a data dimension, a precision, a ReLU, and a pooling option by analyzing a network for training, a data set, etc. In an embodiment, the processor 150 can generate a control signal for training.

[0172] The FSM block 251 checks an operation state of the hardware accelerator 200. In an embodiment, the FSM block 251 receives a control signal and optimizes the control signal based on the checked operation state of the hardware accelerator 200. ​​

[0173] FSM block 251 can provide optimized control signals to each component in hardware accelerator 200 by using control signal distributor 252.

[0174] Figure 11 It is used to describe Figure 6 A diagram showing the detailed configuration of the ReLU pool unit 262. Figure 6 The configuration, function, and operation of ReLU pool unit 262 can correspond to the configuration, function, and operation of ReLU pool unit 1100. In detail, Figure 11 This is a diagram showing the configuration of the reconfigurable ReLU pool unit 1100.

[0175] refer to Figure 11 The ReLU pooling unit 1100 can be reconfigured to output appropriate results for various cases of activation functions and pooling layers.

[0176] For example, the activation function in ReLU pooling unit 1100 can provide ReLU and ReLU-α. In an embodiment, ReLU pooling unit 1100 can allow no pooling, max pooling, local average pooling, and global average pooling by controlling the “out_sel” signal of the pooling layer.

[0177] In an embodiment, the ReLU pool unit 1100 may include a first MUX 1110, ReLU logic 1120, ReLU-α logic 1130, a second MUX 1140, a third MUX 1150, max pooling logic 1160, average pooling logic 1170, and a fourth MUX 1180.

[0178] refer to Figure 6 as well as Figure 11 The first MUX 1110 can receive the output of the bulk normalization unit 261 via its input terminals. In response to the activation function selection signal act_sel, the first MUX 1110 can output the output of the bulk normalization unit 261 to ReLU logic 1120 or ReLU-α logic 1130. For example, the first MUX 1110 can output the output of the bulk normalization unit 261 to ReLU-α logic 1130 in response to a logic high activation function selection signal act_sel. Conversely, the first MUX 1110 can output the output of the bulk normalization unit 261 to ReLU logic 1120 in response to a logic low activation function selection signal act_sel. However, the invention is not limited to the illustrated example, and the relationship between the logic value of the activation function selection signal act_sel and the output of the first MUX 1110 can be reversed. The first MUX 1110 may include multiple switches or logic elements that are turned on / off in response to multiple signals.

[0179] The ReLU logic 1120 can output a ReLU function value based on the output of the batch normalization unit 261. The ReLU logic 1120 can include a plurality of logic elements for implementing a ReLU function that outputs 0 when an input value is less than 0, and outputs the input value as is when the input value is greater than or equal to 0.

[0180] The ReLU-α logic 1130 can output a ReLU-α function value based on the output of the batch normalization unit 261. The ReLU-α logic 1130 can include a plurality of logic elements for implementing a ReLU-α function that outputs 0 when an input value is less than 0, outputs the input value as is when the input value is greater than or equal to 0 and less than α, and outputs α when the input value is greater than or equal to α. α can be a predefined value, and can be a training parameter. According to an embodiment, by using the ReLU-α function, it is possible to prevent an output value from being too large, and to improve the accuracy of a quantized neural network.

[0181] Although Figure 11 Although the ReLU pool unit 1100 is shown to include only the ReLU logic 1120 and the ReLU-α logic 1130, the ReLU pool unit 1100 can also include logic for implementing any activation function (e.g., sigmoid, tanh, leaky ReLU, parametric ReLU (PReLU), exponential linear unit (ELU), scaled exponential linear unit (SELU), etc.), and at least one of the ReLU logic 1120 and the ReLU-α logic 1130 can be omitted.

[0182] The second MUX 1140 can receive the output of the ReLU logic 1120 through an input terminal. In response to the pooling selection signal pool_sel, the second MUX 1140 can output the output of the ReLU logic 1120 to the max pooling logic 1160 or the average pooling logic 1170. For example, in response to a logic high pooling selection signal pool_sel, the second MUX 1140 can output the output of the ReLU logic 1120 to the average pooling logic 1170. In contrast, the second MUX 1140 can output the output of the ReLU logic 1120 to the max pooling logic 1160 in response to a logic low pooling selection signal pool_sel. However, the present disclosure is not limited to the illustrated example, and the relationship between the logic value of the pooling selection signal pool_sel and the output of the second MUX 1140 can be reversed. The second MUX 1140 can include a plurality of switches or logic elements that turn on / off in response to a plurality of signals.

[0183] The third MUX 1150 can receive the output of the ReLU-α logic 1130 through an input terminal. In response to a pooling selection signal pool_sel, the third MUX 1150 can output the output of the ReLU-α logic 1130 to the max pooling logic 1160 or the average pooling logic 1170. For example, the third MUX 1150 can output the output of the ReLU-α logic 1130 to the average pooling logic 1170 in response to a logically high pooling selection signal pool_sel. Conversely, the third MUX 1150 can output the output of the ReLU-α logic 1130 to the max pooling logic 1160 in response to a logically low pooling selection signal pool_sel. However, the disclosure is not limited to the illustrated example, and the relationship between the logical value of the pooling selection signal pool_sel and the output of the third MUX 1150 can be reversed. The third MUX 1150 can include a plurality of switches or logic elements that turn on / off in response to a plurality of signals.

[0184] The max pooling logic 1160 can output a max pooled value based on the output of the ReLU logic 1120 or the output of the ReLU-α logic 1130. The max pooling logic 1160 can include a plurality of logic elements for implementing a max pooling layer that outputs a maximum value within a predefined region among a series of input values (e.g., an MxN mapping).

[0185] The average pooling logic 1170 can output an average pooled value based on the output of the ReLU logic 1120 or the output of the ReLU-α logic 1130. The average pooling logic 1170 can include a plurality of logic elements for implementing an average pooling layer that outputs an average value within a predefined region among a series of input values (e.g., an MxN mapping).

[0186] Although Figure 11 Although the ReLU pooling unit 1100 is illustrated as including only the max pooling logic 1160 and the average pooling logic 1170, the ReLU pooling unit 1100 can also include logic for implementing any pooling layer or subsampling layer, and at least one of the max pooling logic 1160 and the average pooling logic 1170 can be omitted.

[0187] The fourth MUX 1180 can receive the output of the ReLU logic 1120 through a first input terminal, the output of the max pooling logic 1160 through a second input terminal, the output of the average pooling logic 1170 through a third input terminal, and the output of the ReLU-α logic 1130 through a fourth input terminal. In response to an output selection signal out_sel, the fourth MUX 1180 can output the output of the ReLU logic 1120, the output of the max pooling logic 1160, the output of the average pooling logic 1170, or the output of the ReLU-α logic 1130. That is, the ReLU pooling unit 1100 can output the output of the ReLU logic 1120, the output of the max pooling logic 1160, the output of the average pooling logic 1170, or the output of the ReLU-α logic 1130 based on the output of the batch normalization unit 261.

[0188] Figure 12 is a diagram for describing a detailed configuration of the core output buffer 240. Figure 6 The configuration, function, and operation of the core output buffer 240 of Figure 6 may correspond to the configuration, function, and operation of the core output buffer 1200.

[0189] Referring to Figure 12 , the core output buffer 1200 can include 12 flip-flops (FFs) 1210. When all of the FFs 1210 are filled with data from the first processing device 230 (see Figure 6 ) or the second processing device 220 (see Figure 6 ), the core output buffer 1200 can output the data to the output buffer 255 (see Figure 6 ).

[0190] The core output buffer 1200 can output data in response to a stretched clock signal "Stretched CLK". The clock divider 242 can divide the clock signal CLK by n. Here, n is a natural number greater than or equal to 2. The clock divider 242 can divide the clock signal CLK by a preset division ratio. The clock divider 242 can include first to fourth clock dividers 242_1, 242_2, 242_3, and 242_4. For example, the first clock divider 242_1 can divide the clock signal CLK by 2. For example, the second clock divider 242_2 can divide the clock signal CLK by 3. For example, the third clock divider 242_3 can divide the clock signal CLK by 6. For example, the fourth clock divider 242_4 can divide the clock signal CLK by 12. However, the present disclosure is not limited to Figure 12The contents shown in FIG. 18, and the clock divider 242 can divide the clock signal CLK by various division ratios. The MUX 241 can receive the clock signal obtained by the division and output a stretched clock signal "Stretched CLK". The MUX 241 can output the stretched clock signal "Stretched CLK" in response to a control signal.

[0191] According to embodiments, because the number of inputs to be filled into the FF varies depending on the situation, the point in time at which the FF is completely filled with data can differ from each other. Therefore, because data needs to be output at the right time, the clock divider 242 can be used to output data stored in the FF at the right time.

[0192] Figure 13 is a diagram for describing a detailed configuration of the FIFO 264 of Figure 6 . Figure 6 The configuration, function, and operation of the FIFO 264 of

[0193] Referring to Figure 13 , the FIFO 1300 can include a MUX 1310, a flip-flop circuit 1320, and a concatenator 1330. The FIFO 1300 can output data bits in multiples of 18, for proper operation of subsequent logic (e.g., the FP2BFP converter 266 of Figure 6 ) that operates in multiples of 18 bits.

[0194] The MUX 1310 can receive 64-bit or 144-bit input data through an input terminal. In response to a control signal, the MUX 1310 can output the input data through a path (i.e., a first path) in which the data is transferred to the converter 1330 via the flip-flop circuit 1320 or a path (i.e., a second path) in which the data is directly transferred to the converter 1330. In embodiments, the control signal can be generated by a mode generator (not shown). Because the point in time at which the FIFO 1300 outputs data varies depending on the mode (e.g., a 2D or 3D mode), the point in time is controlled by using the mode generator.

[0195] In an embodiment, the FIFO 1300 can acquire data to the flip-flop circuit 1320, and then acquire the last data through another path. For example, in the case of a pool size of 2, data is transferred to the flip-flop circuit 1320 for 8 cycles. Thereafter, the last data can be directly transferred to the concatenator 1330 through another path in the next cycle. The concatenator 1330 can concatenate the data transferred through the first path and the second path. Because the time point at which the FIFO 1300 outputs data varies according to a mode (for example, a 2D or 3D mode), the time point is controlled by using a mode generator (not shown).

[0196] Figure 14 is a diagram for describing a detailed configuration of the weight update unit 265 of Figure 6 . Figure 6 The configuration, function, and operation of the weight update unit 265 of

[0197] Referring to Figure 14 , the weight update unit 1400 can include an element-wise multiplication unit 1410 and an element-wise subtraction unit 1420.

[0198] The element-wise multiplication unit 1410 can include six multipliers. For example, the six multipliers can be FP32 multipliers. In detail, the element-wise multiplication unit 1410 can receive inputs of six weight gradients and a learning rate α, and calculate an update amount of weights.

[0199] The element-wise subtraction unit 1420 can include six adders / subtractors. In detail, the element-wise subtraction unit 1420 can receive inputs of outputs of the element-wise multiplication unit 1410 and weights W l , and perform a weight update operation. For example, the element-wise subtraction unit 1420 can output updated weights l by subtracting the outputs of the element-wise multiplication unit 1410 from the weights W .

[0200] Figure 15 is a diagram for describing a detailed configuration of the FP2BFP converter 266 of Figure 6 . Figure 6 The configuration, function, and operation of the FP2BFP converter 266 of

[0201] Referring to Figure 15 , the FP2BFP converter 1500 can include an extractor 1510, a comparator 1520, a subtractor 1530, and a normalizer 1540.

[0202] The extractor 1510 finds a maximum value among the input values.

[0203] The comparator 1520 can compare the maximum values extracted by the extractor 1510 with each other to find a maximum exponent in a block unit (i.e., an exponent of a block tensor). The size of the block unit can vary according to a BFP format (e.g., FB12, FB16, or FB24) and / or a type of layer (e.g., CONV1 / FC, CONV3, CONV5, or CONV7).

[0204] The subtracter 1530 can include a plurality of subtracters, and each of the plurality of subtracters can correspond to an input value (e.g., 18 input values), respectively. The subtracter 1530 can subtract an exponent of each input value from the maximum exponent. That is, the subtracter 1530 can receive the maximum exponent extracted by the comparator 1520 and an exponent of a value in the block tensor, and then calculate an exponent in the BFP format.

[0205] The normalizer 1540 can perform normalization based on the exponent calculated by the subtracter 1530. Here, the normalization refers to converting a mantissa into a format of "1.xxx...". In detail, in order to calculate the mantissa at the time of conversion into the BFP format, the normalizer 1540 can adjust a significant value (i.e., a mantissa) in response to the previously calculated exponent by performing a shift significant digit operation using a barrel shifter.

[0206] Figure 16 is a diagram for describing a detailed configuration of the quantization unit 267. Figure 6

[0207] Referring to Figure 16 , the quantization unit 267 can quantize the input values according to a size of a mantissa used. For example, in the case of inputting a 25-bit value, the mantissa according to the disclosure is 4 bits, 8 bits, or 16 bits, and thus, the quantization unit 267 can perform rounding according to the size of the precision (i.e., the mantissa) to be currently used. For example, the maximum size of the mantissa to be used is 16 bits, and thus, the quantization unit 267 can perform quantization to adapt to each precision (1610) by performing a rounding operation to leave only 15 bits (i.e., performing rounding at the 16th bit, and truncating below the rounded digit to leave only 15 bits), performing an additional rounding operation to leave only 8 bits in the case of 8 bits, and performing a rounding operation to additionally leave only 4 bits in the case of 4 bits. In an embodiment, the quantization unit 267 can output one of three outputs (1620).

[0208] Figure 17 is a diagram for exemplarily describing how to map an input tensor to the processing core 210 in a Conv3 layer (see Figure 6 ). ​

[0209] refer to Figure 17 In this disclosure, each of the three precisions (i.e., the input tensor, the weight tensor, and the weight gradient) uses an 8-bit shared exponent, and the sign and mantissa sizes of the three precisions are 4 bits, 8 bits, and 16 bits, respectively. Hereinafter, the precision with an 8-bit shared exponent and a 4-bit sign and mantissa size is referred to as FB12, and the other two are referred to as FB16 and FB24, respectively.

[0210] As described above, with three types of mantissa sizes and three types of tensors (e.g., 8-bit input, 4-bit weights, and 16-bit gradients), 27 combinations are possible. The processing kernel 210 according to this disclosure will be described (see...). Figure 6 Operations in such combinations.

[0211] First, we will describe the case of FB24 (i.e., 16-bit sign and mantissa) used for Conv3 layers, such as input activation (forward pass) or local gradient (backward pass).

[0212] refer to Figure 17 In 1710, in 16-bit mode, each element constituting the input feature map (i.e., the input tensor) X comprises four 4-bit subwords, and each 4-bit subword is mapped to a multiplier (e.g., a 4b×4b multiplier) in the corresponding PEs PE0, PE1, PE2, and PE3. For example, the 4-bit subword x[15:12] can be mapped to the fourth PE PE3. For example, the 4-bit subword x[11:8] can be mapped to the third PE PE2. For example, the 4-bit subword x[7:4] can be mapped to the second PE PE1. For example, the 4-bit subword x[3:0] can be mapped to the first PE PE0. In a single cycle, the input features Figure X A single input channel can be broadcast to all PUs: PU0, PU1, PU2, and PU3.

[0213] refer to Figure 17 The 1720 (with a sign and mantissa size of 8 bits) constitutes the input feature in 8-bit mode. Figure X Each element consists of two 4-bit subwords. In this case, the input features Figure X The two input channels can be broadcast to PU0, PU1, PU2, and PU3. Specifically, the two 4-bit subwords x of the first input channel... (0) [7:4] and x (0) [3:0] and the two 4-bit subwords x of the second input channel (1) [7:4] and x (1)[3:0] are mapped to multipliers (e.g., 4b x 4b multipliers) in the respective PEs PE0, PE1, PE2, and PE3. For example, 4-bit subword x (0) [3:0] can be mapped to the third PE PE2. For example, 4-bit subword x (1) [7:4] can be mapped to the second PE PE1. For example, 4-bit subword x (1) [3:0] can be mapped to the first PE PE0. That is, each of the four PUs PU0, PU1, PU2, and PU3 constituting one sub-core can process data corresponding to two input channels of the Conv3 layer in parallel.

[0214] Referring to Figure 17 1730 (in the case where the size of the symbol and mantissa is 4 bits), in the 4-bit mode, each element constituting the input feature Figure X includes one 4-bit subword. In this case, the four input channels of the input feature Figure X can be broadcast to the PUs PU0, PU1, PU2, and PU3. In detail, the 4-bit subword x (0) [3:0] of the first input channel, the 4-bit subword x (1) [3:0] of the second input channel, the 4-bit subword x (2) [3:0] of the third input channel, and the 4-bit subword x (3) [3:0] of the fourth input channel are mapped to multipliers (e.g., 4b x 4b multipliers) in the respective PEs PE0, PE1, PE2, and PE3. For example, 4-bit subword x (0) [3:0] can be mapped to the fourth PE PE3. For example, 4-bit subword x (1) [3:0] can be mapped to the third PE PE2. For example, 4-bit subword x (2) [3:0] can be mapped to the second PE PE1. For example, 4-bit subword x (3) [3:0] can be mapped to the first PE PE0. That is, each of the four PUs PU0, PU1, PU2, and PU3 constituting one sub-core can process data corresponding to four input channels of the Conv3 layer in parallel.

[0215] Figure 18 is a diagram for exemplarily describing how a weight tensor is mapped to the processing core 210 (see Figure 6 ) in the Conv3 layer.

[0216] Referring to Figure 18of 1810, in 16-bit mode (i.e., the size of the sign and mantissa of the second tensor (e.g., weight tensor) is 16 bits), with one output channel C out The corresponding weight tensor can be distributed to the four PUs constituting one sub-kernel.

[0217] In detail, each of the elements W0, W1, …, W8 constituting the weight tensor W includes four 4-bit sub-words, and the 4-bit sub-words can be mapped to the corresponding PUs PU0, PU1, PU2, and PU3, respectively. For example, the first 4-bit sub-word w[15:12] of each of the elements W0, W1, …, W8 of the weight tensor W can be mapped to the fourth PU PU3. For example, the second 4-bit sub-word w[11:8] of each of the elements W0, W1, …, W8 of the weight tensor W can be mapped to the third PU PU2. For example, the third 4-bit sub-word w[7:4] of each of the elements W0, W1, …, W8 of the weight tensor W can be mapped to the second PU PU1. For example, the fourth 4-bit sub-word w[3:0] of each of the elements W0, W1, …, W8 of the weight tensor W can be mapped to the first PU PU0. That is, the total number of bits of the weight tensor W of a single channel consisting of 16-bit elements is 144, and 36 bits can be allocated to each of the PUs PU3, PU2, PU1, and PU0.

[0218] Reference Figure 18 of 1820, in 8-bit mode (i.e., the size of the sign and mantissa of the second tensor (e.g., weight tensor) is 8 bits), with two output channels C out The corresponding second tensor can be distributed to the four PUs constituting one sub-kernel.

[0219] In detail, the four PUs PU3, PU2, PU1, and PU0 can be divided into two clusters and used. For example, a first cluster can include a first PU PU0 and a second PU PU1, and a second cluster can include a third PU PU2 and a fourth PU PU3. Each of the elements W0, W1, …, W8 constituting the weight tensor W includes two 4-bit sub-words, and each 4-bit sub-word can be mapped to the corresponding cluster. The two output channels C out of the weight tensor W can be mapped (or distributed) to the first cluster or the second cluster. For example, the two 4-bit sub-words w (0) [7:4] and w (0) [3:0] of the first output channel can be mapped to the second cluster. For example, the two 4-bit sub-words w (1) [7:4] and w (1)[3:0] can be mapped to the first cluster. Accordingly, each of the first PU PU0 and the second PU PU1 included in the first cluster can provide a partial sum of the second output channel, and each of the third PU PU2 and the fourth PU PU3 included in the second cluster can provide a partial sum of the first output channel. That is, the total number of bits of the weight tensor W having two output channels composed of 8-bit elements is 144, and 36 bits can be allocated to each of the PUs PU3, PU2, PU1, and PU0.

[0220] Referring to Figure 18 , in a 4-bit mode (i.e., the sign and the mantissa of the second tensor (e.g., weight tensor) are 4 bits in size), the second tensor corresponding to four output channels C out of the Conv3 layer can be distributed to four PUs constituting one sub-kernel.

[0221] In detail, each of the four PUs PU3, PU2, PU1, and PU0 can correspond to one output channel. Each of the elements W0, W1,..., W8 constituting the weight tensor W includes a single 4-bit sub-word, and the single 4-bit sub-word can correspond to one of the four output channels C out . For example, the 4-bit sub-word w (0) [3:0] can be mapped to the fourth PU PU3. For example, the 4-bit sub-word w (1) [3:0] can be mapped to the third PU PU2. For example, the 4-bit sub-word w (2) [3:0] can be mapped to the second PU PU1. For example, the 4-bit sub-word w (3) [3:0] can be mapped to the first PU PU0. That is, the total number of bits of the weight tensor W having four output channels composed of 4-bit elements is 144, and 36 bits can be allocated to each of the PUs PU3, PU2, PU1, and PU0.

[0222] Figure 19 is a diagram for describing the operation of a sub-kernel according to a layer type of a deep neural network according to an embodiment.

[0223] The hardware accelerator 200 (see Figure 6 ) according to the disclosure is designed in a layer structure, and can be managed differently according to a layer type. That is, the above-described sub-kernel or PE can be operated by clustering according to a layer type of a deep neural network. Hereinafter, it is assumed that the precision of an input tensor and a weight tensor is FB16 (i.e., the sign and the mantissa are 8 bits in size).

[0224] Referring toFigure 19 In Conv1 or a fully connected layer, only on input channel C... in The partial sums are accumulated along the dimension of the input tensor. Therefore, the input elements (i.e., one of the elements of the input tensor) and their corresponding weights can be mapped to sub-kernels, PEs, and multipliers. For example, multiple input channels (e.g., 18 input channels) can be mapped to each sub-kernel. For instance, the first 18 input channels 0 to 17 can be mapped to the first sub-kernel, and the next 18 input channels 17 to 35 can be mapped to the second sub-kernel.

[0225] refer to Figure 19 In 1920, Conv3 layer and Figure 19 The difference between the Conv1 or fully connected layers shown in 1910 lies in the method of mapping operands to multipliers. In a Conv1 or fully connected layer, tensors (i.e., operands) are mapped to the input channel C. in In the Convk layer (e.g., the Conv3 layer), the tensor (i.e., the operand) can be mapped to multipliers in the width and height (W / H) dimensions of the feature map. Here, k can be a natural number greater than 1.

[0226] When using kernels with larger weights (e.g., Conv5 layers, Conv7 layers, etc.), multiple sub-kernels can be clustered. (See reference) Figure 19 For example, in the Conv5 layer, three sub-cores can be clustered. For example, in the Conv7 layer, six sub-cores can be clustered. According to an embodiment, in the Conv5 layer, three PEs, each including nine multipliers, are activated to perform 25 (5×5) multiplication operations, and thus, a core utilization of 93% (i.e., 5×5 / 9×3) can be achieved. According to an embodiment, in the Conv7 layer, six PEs, each including nine multipliers, are activated to perform 49 (7×7) multiplication operations, and thus, a core utilization of 91% (i.e., 7×7 / 6×9) can be achieved.

[0227] The foregoing has described a mapping method for 3D operations that performs accumulation using a first processing device. However, accumulation does not occur in 2D operations, and because the hardware accelerator according to this disclosure has a separate processing device for 2D operations, core utilization can be maximized. This will be referenced below. Figure 20 Describe it.

[0228] Figure 20 This is an illustration used to describe an example of a mapping method in a 2D operation mode according to an embodiment. A 3D operation refers to an operation that outputs an output feature map by performing convolution and accumulating a portion of the output feature map as the result of the convolution in the channel direction. A 2D operation refers to an operation that performs convolution without accumulating the output feature map having the channel direction.

[0229] The DW convolution layer is a good example of a 2D operation (which can be referred to as 2D computation or 2D processing). Reference will be made to Figure 20 A method of mapping the DW Conv3 layer to sub-cores will be described.

[0230] In the weight update step in the training process, the computation is performed in a 2D manner, and because the computation is sensitive to the reduction of precision, it is preferable to maintain the precision to 8 bits or 16 bits for each tensor.

[0231] Reference will be made to Figure 6 and Figure 20 The outputs of the PUs PU0, PU1, PU2, and PU3 for each of the sub-cores Subcore0,..., Subcore5 can be accumulated by the 4-way adder tree 231 of the first processing device 230. For example, in the DW Conv3, the outputs from each of the sub-cores Subcore0,..., Subcore5 can correspond to one output channel C out .

[0232] In the case of using the DW Conv5 or the DW Conv7, a plurality of sub-cores can be clustered and then used. For example, in the DW Conv5, three sub-cores can be clustered and then used, and in the DW Conv7, six sub-cores can be clustered and then used. The method of clustering the sub-cores according to the size of the weight kernel can be similar to the method described with reference to 1930. Figure 19

[0233] Figure 21 is a diagram for describing an operation of an electronic device according to an embodiment.

[0234] Reference will be made to Figure 2 and Figure 21 The electronic device 100 can obtain parameters of a deep neural network. For example, the electronic device 100 can receive an input of parameters of a target network (i.e., a deep neural network) to be trained from a user. For example, the parameters of the deep neural network can be at least one parameter corresponding to the target network, such as a block size, a precision, or a number of epochs.

[0235] The electronic device 100 can set a bit precision and a block size for each tensor (e.g., an input tensor or a weight tensor) based on the parameters. According to an embodiment, the electronic device 100 can control a set value of a weight gradient based on the parameters.

[0236] ​After the setting, the electronic device 100 can train the target network (i.e., the deep neural network) and output a training result (e.g., accuracy) according to the training. For example, it can be checked whether the training performance is good according to the conditions of the hardware accelerator (e.g., block size, precision, and mapping method).

[0237] Figure 22 is a diagram for describing a detailed configuration of a shared exponent processor of Figure 6 .

[0238] A multiplier unit proposed in BitFusion uses a multiplier capable of performing multiplication in units of 2b, and exhibits a high utilization rate of the multiplier for various precisions (2b, 4b, and 8b).

[0239] However, the multiplier unit supports only signed and unsigned integer data types. Because deep learning training needs to be performed with high precision, the multiplier unit cannot be used for deep learning training.

[0240] The hardware accelerator according to an embodiment aims to support deep learning inference as well as training with high efficiency, and thus supports a wide range of precision operations including high precision.

[0241] Meanwhile, the hardware accelerator processes significant figures (sign and mantissa) and exponents separately in order to efficiently process MAC operations. For example, the accelerator core can include a processing core 2210 and a shared exponent processor 2220. According to an embodiment, in the MAC operation, the significant figures can be processed by the processing core 2210, and the exponents can be processed by the shared exponent processor 2220.

[0242] Figure 22 The processing core 2210 of Figure 6 may correspond to one of Subcore0, Subcore1,..., Subcore5 of . The processing core 2210 can include 144 multipliers supporting a plurality of precisions (e.g., 4b, 8b, and 16b). However, the number of multipliers is only an example, and can be a number suitable for a system in an implementation.

[0243] The shared exponent processor 2220 is a module for an exponent operation. The shared exponent processor 2220 can include a multiplier unit performing an exponent operation. For example, the shared exponent processor 2220 can include a first unsigned adder 2221 and a second unsigned adder 2222. The first unsigned adder 2221 can sum two input exponents. The second unsigned adder 2222 can output a value obtained by subtracting (or adding) a bias from (or to) the output of the first unsigned adder 2221.

[0244] In the case of processing operations between BFP data types (e.g., FB12, FB16, FB24, etc.) or fixed point types, processing of exponents can be performed by turning on the adder unit, and in the case of processing integer type data, processing of data types such as INT4, INT8, or INT16 can be performed by turning off the adder unit.

[0245] Meanwhile, because the present disclosure supports BFP data types, only one exponent processing module is required per 144 multipliers.

[0246] Figure 23 is an exemplary block diagram illustrating an electronic device 2300 according to an embodiment.

[0247] Referring to Figure 23 , the electronic device 2300 can include a hardware accelerator 2310, a processor 2320, a memory 2330, and an input / output interface 2340. However, components of the electronic device 2300 are not limited to the above-described examples, and the electronic device 2300 can include more or less components than the above-described components. In an embodiment, at least some of the hardware accelerator 2310, the processor 2320, the memory 2330, and the input / output interface 2340 can be implemented as a single chip, and the processor 2320 can include one or more processors.

[0248] The configuration, functions, and operations of the hardware accelerator 2310 can correspond to the configuration, functions, and operations of the hardware accelerator 200 described with reference to Figure 6 . Accordingly, the description provided above with reference to Figure 6 will be omitted.

[0249] The hardware accelerator 2310 can perform 1D subword parallelization between the sign and mantissa of a first tensor (e.g., an input tensor) and the sign and mantissa of a second tensor (e.g., a weight tensor) by using a plurality of multipliers.

[0250] The hardware accelerator 2310 can perform processing between the shared exponent of the first tensor and the shared exponent of the second tensor by using a shared exponent processor.

[0251] The hardware accelerator 2310 can perform training or inference of a deep neural network under the control of the processor 2320. The hardware accelerator 2310 can read data (e.g., a first tensor and a second tensor) stored in the memory 2330 to perform computation. The result of the computation of the hardware accelerator 2310 can be stored in the memory 2330.

[0252] In an embodiment, the hardware accelerator 2310 can operate in a 2D operation mode in which the calculation results of the plurality of multipliers are outputted without being accumulated in the channel direction, or the hardware accelerator 2310 can operate in a 3D operation mode in which the calculation results of the plurality of multipliers are accumulated in the channel direction and the result of the accumulated calculation results is outputted.

[0253] The processor 2320 is a component configured to control a series of processes so that the electronic device 2300 operates according to the embodiments described above with reference to FIGS. 1 to 23, and can include one or more processors. In this case, the one or more processors can be a general-purpose processor such as a CPU, an application processor (AP), or a digital signal processor (DSP). Figures 1 to 22

[0254] The processor 2320 can write data in or read data stored in the memory 2330, and specifically, can execute a program stored in the memory 2330 to process data according to a predefined operation rule or an artificial intelligence model. In an embodiment, the processor 2320 can control the hardware accelerator based on deep neural network information including at least one of the number of layers in the deep neural network, the type of layers, the shape of a tensor, the dimension of a tensor, an operation mode, a bit precision, the type of batch normalization, the type of a pooling layer, and the type of a ReLU function.

[0255] In an embodiment, the processor 2320 can obtain at least one of a bit precision and a block size of a deep neural network based on a user input. The processor 2320 can set at least one of a bit precision and a block size of a first tensor and a second tensor based on the obtained at least one of a bit precision and a block size. The processor 2320 can control the hardware accelerator 2310 to train the deep neural network based on the setting.

[0256] The hardware accelerator 2310 and the processor 2320 can perform the operations described above with reference to the embodiments, and unless otherwise specified, the above-described operations performed by the electronic device 2300 in an embodiment can be considered to be performed by the hardware accelerator 2310 or the processor 2320.

[0257] ​The memory 2330 is a component for storing various programs or data, and can include a storage medium such as a ROM, a RAM, a hard disk, a compact disk ROM (CD-ROM), or a digital versatile disk (DVD), or a combination of storage media. The memory 2330 can not be separately provided, but can be included in the processor 2320 or the hardware accelerator 2310. The memory 2330 can include a volatile memory, a non-volatile memory, or a combination of a volatile memory and a non-volatile memory. A program for the hardware accelerator 2310 or the processor 2320 to perform an operation can be stored in the memory 2330. The memory 2330 can provide data stored therein to the hardware accelerator 2310 or the processor 2320 according to a request of the hardware accelerator 2310 or the processor 2320. In an embodiment, the memory 2330 can store at least one instruction to be executed by the processor 2320. The memory 2330 can store a deep neural network. The memory 2330 can store parameters or hyperparameters for training the deep neural network or causing the deep neural network to perform inference.

[0258] The input / output interface 2340 can include an input interface (for example, a touch screen, a hard button, or a microphone) to receive a control command or information from a user and an output interface (for example, a display panel or a speaker) to display a result of an execution operation or a state of the electronic device 2300 according to a user's control. According to an embodiment, the input / output interface 2340 can receive a user input corresponding to at least one of a bit precision and a block size of a deep neural network.

[0259] In an embodiment, the hardware accelerator can include a plurality of multipliers to perform 1D subword parallelization between a sign and a mantissa of a first tensor and a sign and a mantissa of a second tensor. The hardware accelerator can include a first processing device to operate in a 2D operation mode to output a calculation result of the plurality of multipliers. The hardware accelerator can include a second processing device to operate in a 3D operation mode to accumulate the calculation result of the plurality of multipliers in a channel direction and output a result of the accumulated calculation.

[0260] In an embodiment, a first group of the plurality of multipliers of the hardware accelerator can perform a multiplication operation between a first subword of a series of subwords of a first value included in a first tensor and a first subword of a series of subwords of a second value included in a second tensor. A second group of the plurality of multipliers of the hardware accelerator can perform a multiplication operation between a second subword of the series of subwords of the first value and a second subword of the series of subwords of the second value.

[0261] In embodiments, in the computation of a deep neural network, when performing weight gradient computation, DW convolution, dilation convolution, or up convolution, the hardware accelerator can operate in a 2D operation mode in which the computation result is not accumulated in the channel direction. In the computation of a deep neural network, when performing convolution, pointwise convolution, or fully connected layer computation, the hardware accelerator can operate in a 3D operation mode in which the computation result is accumulated in the channel direction.

[0262] In embodiments, the processing core can include six sub-cores. Each sub-core can include 4 PUs. Each of the PUs can include four PE. Each PE can include nine multipliers.

[0263] In embodiments, in the computation of a Conv3 layer of a deep neural network, in the case where the size of the sign and mantissa of a first tensor is 16 bits, the first tensor corresponding to one input channel of the Conv3 layer can be broadcasted to the four PUs constituting one sub-core. In the computation of a Conv3 layer of a deep neural network, in the case where the size of the sign and mantissa of a first tensor is 8 bits, the first tensor corresponding to two input channels of the Conv3 layer can be broadcasted to the four PUs constituting one sub-core. In the computation of a Conv3 layer of a deep neural network, in the case where the size of the sign and mantissa of a first tensor is 4 bits, the first tensor corresponding to four input channels of the Conv3 layer can be broadcasted to the four PUs constituting one sub-core.

[0264] In embodiments, in the computation of a Conv3 layer of a deep neural network, in the case where the size of the sign and mantissa of a second tensor is 16 bits, the second tensor corresponding to one output channel of the Conv3 layer can be distributed to the four PUs constituting one sub-core. In the computation of a Conv3 layer of a deep neural network, in the case where the size of the sign and mantissa of a second tensor is 8 bits, the second tensor corresponding to two output channels of the Conv3 layer can be distributed to the four PUs constituting one sub-core. In the computation of a Conv3 layer of a deep neural network, in the case where the size of the sign and mantissa of a second tensor is 4 bits, the second tensor corresponding to four output channels of the Conv3 layer can be distributed to the four PUs constituting one sub-core.

[0265] In an embodiment, the first processing device can include six 4-way adder trees that sum outputs of four PUs included in each of the six sub-kernels. The first processing device can include six bit truncators that round each of the outputs of the 4-way adder trees to have a preset number of bits. The first processing device can include a selective 6-way adder tree that selectively sums outputs of the bit truncators. The first processing device can include an arithmetic converter that outputs FP32 partial sum data based on an output of the selective 6-way adder tree and an output of a shared exponent processor. The first processing device can include an accumulator that accumulates the FP32 partial sum data.

[0266] In an embodiment, the second processing device can include four 6-way adder trees that sum outputs of PUs corresponding to each other in different sub-kernels. The second processing unit can include four arithmetic converters that output FP32 partial sum data based on outputs of the 6-way adder trees and an output of a shared exponent processor. The second processing unit can include four accumulators that accumulate the FP32 partial sum data. The second processing unit can include a selective 4-way adder tree that selectively sums outputs of the accumulators.

[0267] In an embodiment, each of the PUs can include nine multipliers that perform multiplication operations. Each of the PUs can include a 9-way adder tree that sums outputs of the nine multipliers. Each of the PUs can include selective shift logic circuitry that shifts an output of the 9-way adder tree by a preset number of bits.

[0268] In an embodiment, the hardware accelerator can include a shared exponent processor that processes a shared exponent of the first tensor and a shared exponent of the second tensor.

[0269] In an embodiment, the hardware accelerator can perform a computation by using a plurality of data types corresponding to a control signal. In an embodiment, the data types can include a first data type of a fixed point type, a second data type having only an integer, a third data type having a sign and an integer, and a fourth data type of a real number type of a shared exponent.

[0270] In an embodiment, a size of the shared exponent of the first tensor and a size of the shared exponent of the second tensor can be 8 bits. A size of the sign and the mantissa of the first tensor or a size of the sign and the mantissa of the second tensor can be one of 4 bits, 8 bits, and 16 bits. The first tensor and the second tensor can be mapped to the processing core based on the size of the sign and the mantissa of the first tensor or the size of the sign and the mantissa of the second tensor.

[0271] In an embodiment, the size of the sign and the mantissa of the first tensor or the size of the sign and the mantissa of the second tensor can be determined based on a forward pass step, a backward pass step, or a weight update step of a training of a deep neural network.

[0272] In an embodiment, the hardware accelerator can include a kernel output buffer that outputs the output value of the first processing device in the weight update step and outputs the output value of the second processing device in the forward pass step and the backward pass step.

[0273] In an embodiment, the hardware accelerator can include a batch normalization circuit that performs batch normalization based on the output of the kernel output buffer. The hardware accelerator can include a ReLU pool circuit that performs a ReLU function value and a pooling value based on the output of the batch normalization circuit. The hardware accelerator can include a FIFO circuit that stores and outputs the output of the ReLU pool circuit. The hardware accelerator can include an FP2BFP converter circuit that converts the data type of the output of the FIFO circuit in a floating point form to a BFP form. The hardware accelerator can include a quantization circuit that quantizes the output of the FP2BFP converter according to a predefined precision.

[0274] In an embodiment, each of the plurality of multipliers can include a first MUX that receives a previous second tensor through a first input terminal, receives a current second tensor through a second input terminal, and outputs the current second tensor or the first tensor in response to a keep signal. Each of the plurality of multipliers can include a multiplier kernel that performs a multiplication operation by using a 5-bit first tensor and a 5-bit second tensor as operands. Each of the plurality of multipliers can include a first register for storing an output of the multiplier kernel. Each of the plurality of multipliers can include a second register for storing an output of the first MUX. Each of the plurality of multipliers can include a second MUX that receives a value stored in the second register through a first input terminal, receives an output of the first MUX through a second input terminal, and outputs the value stored in the second register or the output of the first MUX in response to a bypass signal.

[0275] In an embodiment, the value stored in the second register can be maintained by a feedback loop generated by the keep signal.

[0276] In an embodiment, the electronic device can include a hardware accelerator that performs 1D subword parallelization between a sign and a mantissa of a first tensor and a sign and a mantissa of a second tensor by using a plurality of multipliers, and performs processing between a shared exponent of the first tensor and a shared exponent of the second tensor by using a shared exponent processor. The electronic device can include a processor configured to execute at least one instruction to control the hardware accelerator based on deep neural network information including at least one of a number of layers in a deep neural network, a type of layer, a shape of a tensor, a dimension of a tensor, an operation mode, a bit precision, a type of batch normalization, a type of a pooling layer, and a type of a ReLU function. The electronic device can include a memory storing at least one instruction and a deep neural network.

[0277] In an embodiment, the processor can execute at least one instruction to obtain at least one of a bit precision and a block size of a deep neural network based on a user input. The processor can execute at least one instruction to set at least one of a bit precision and a block size of a first tensor and a second tensor based on the obtained at least one of a bit precision and a block size. The processor can execute at least one instruction to control the hardware accelerator to train the deep neural network based on the setting.

[0278] In an embodiment, the hardware accelerator can operate in a 2D operation mode in which a calculation result of the plurality of multipliers is output without accumulation in a channel direction, or the hardware accelerator can operate in a 3D operation mode in which a calculation result of the plurality of multipliers is accumulated in the channel direction and a result of the accumulated calculation result is output.

[0279] Hereinafter, main features reflected in the design of the hardware accelerator using the simulator as described above will be described.

[0280] [Hardware efficiency]

[0281] Area and power are important design considerations as they affect performance and energy consumption. In the present disclosure, various methods for reducing area and energy consumption are applied even in the process of designing the hardware accelerator.

[0282] First, for the PE, bypass is supported on the multiplier to reduce the number of on-chip memory accesses for weight and input bandwidth. Because the accelerator of the present disclosure is used for training, not for inference, many memory accesses occur.

[0283] Meanwhile, a general MAC device fetches data required for calculation from an on-chip buffer in one cycle. In the case of a low-precision mode, a large amount of data needs to be loaded into the MAC device in one cycle, and thus, a large bandwidth is required between the on-chip buffer and the MAC device. To solve this problem, the MAC device in the above-described accelerator supports bypass.

[0284] Second, hardware efficiency is improved. Here, hardware efficiency refers to efficiency with respect to area and power consumption. In the present disclosure, the following methods are used to improve hardware efficiency: 1) reducing the number of shifters, 2) replacing floating point operators with integer operators, and 3) performing operand isolation.

[0285] Among MAC operators supporting various precisions, there is a 2D subword parallelization operator as a solution having high utilization and high hardware efficiency for all supported precisions. The operator has a large number of shifters. In the present disclosure, the operator is divided into 1D subword parallelization operators, and 1D subword parallelization operators having the same shifter are clustered. The clustered operators share shifters, and thus the number of shifters of the subword parallelization operators is significantly reduced. For reference, in the present disclosure, a group obtained by clustering is referred to as a PE. This reduction in the number of shifters reduces the area of the operator.

[0286] Meanwhile, since the processing core in the hardware accelerator proposed in the present disclosure supports BFP, all units in the sub-core include an integer operator. The integer operator has a simpler structure than the floating point type operator, and thus has a smaller area and lower power consumption. By replacing a large number of floating point operators with integer type operators, the hardware accelerator has high logic density and low power consumption.

[0287] Meanwhile, all modules selectively used in the present disclosure are designed with operand isolation involved. For example, when the first processing device is operating, the second processing device does not need to operate. Similarly, when the second processing device is operating, the first processing device does not need to operate. Placing devices not operating in the present disclosure into an idle state makes it possible to reduce dynamic power consumption occurring in the devices.

[0288] Third, high utilization. In the prior art, calculation is performed without distinguishing between 2D operation mode and 3D operation mode. In the case of 2D operation mode, unlike 3D operation mode, the calculation result is not accumulated in the channel direction, and thus the amount of output result generated is much larger than that in 3D operation mode. However, due to the limited bandwidth of the device, only a part of the output result is output. For this reason, the accelerator of the prior art uses only a part of the operator in 2D operation mode. To solve the problem of low utilization of the prior art device, the present disclosure has a first processing device and a second processing device. The first processing device is dedicated to 2D operation mode, and the second processing device is dedicated to 3D operation mode. The first processing device has a high input bandwidth so that many output result values generated by the operator can be transferred to the device. Values sent in this way are finally output by rounding to the nearest type of quantization. By configuring the first processing device in this way, the operator can have high utilization in the limited bandwidth.

[0289] Fourth, segmentation of loss gradient map. In the related art, low utilization is realized in the weight update step. In detail, in the related art, only the forward pass of DNN training is focused, i.e., it is optimized for the calculation between a large feature map and a small feature map. However, the weight update is a calculation between a large feature map and a large feature map. In this regard, the accelerator of the related art has low utilization in the weight update step. To solve such a problem, in the present disclosure, one feature map is segmented into small-sized feature maps in the weight update step to have high utilization as in the forward pass.

[0290] Fifth, in the present disclosure, the calculation is performed with different sizes of precision for each step, so the calculation can be efficiently performed. For example, in the update process, high-precision calculation needs to be performed, but in other processes, performance is not greatly affected even if low-precision operation is performed. Therefore, by performing fast calculation with low precision for operations other than the weight update process, and performing calculation with high precision in the weight update process, the calculation operation can be efficiently performed without reducing the training performance throughout the training process.

[0291] Meanwhile, the processing method according to the above-described various embodiments can be implemented in the form of program codes for performing each operation, stored in a recording medium, and then distributed. In this case, a device loaded with the recording medium can perform the above-described processing operation.

[0292] Such a recording medium can be various types of computer-readable media such as a ROM, a RAM, a memory chip, a memory card, an external hard disk drive, a hard disk drive, a CD, a DVD, a magnetic disk, or a magnetic tape.

[0293] The present disclosure has been described with reference to the accompanying drawings, but the scope of the present disclosure is intended to be determined by the appended claims, and is not intended to be interpreted as being limited to the above-described embodiments and / or drawings. Furthermore, it should be clearly understood that changes, alterations, and modifications of the present disclosure described in the claims obvious to one skilled in the art are also included in the scope of the present disclosure.

[0294] It should be understood that the embodiments described herein should be considered in a descriptive sense only and not for purposes of limitation. Descriptions of features or aspects within each embodiment should typically be considered as being applicable to other similar features or aspects in other embodiments. While one or more embodiments have been described with reference to the figures, it will be understood by those of ordinary skill in the art that various changes in form and details can be made therein without departing from the spirit and scope of the present disclosure as defined by the following claims.

Claims

1. A hardware accelerator for performing computations on a deep neural network, the hardware accelerator comprising: The processing core includes multiple multipliers configured to perform one-dimensional subword parallelization between the sign and mantissa of a first tensor and the sign and mantissa of a second tensor. The first processing device is configured to operate in a two-dimensional operation mode, wherein the calculation results of multiple multipliers are output in the two-dimensional operation mode; as well as The second processing device is configured to operate in a three-dimensional operation mode, in which the calculation results of multiple multipliers are accumulated in the channel direction, and then the result of accumulating the calculation results is output.

2. The hardware accelerator as described in claim 1, wherein, The first group of the plurality of multipliers performs a multiplication operation between a series of subwords containing the first value in the first tensor and the first subword among a series of subwords containing the second value in the second tensor, and The second group of the plurality of multipliers performs a multiplication operation between the series of subwords of the first value and the second subword of the series of subwords of the second value.

3. The hardware accelerator as described in claim 1, wherein, In deep neural network computation, hardware accelerators operate in a two-dimensional operation mode when performing weight gradient calculations, depthwise DW convolutions, dilated convolutions, or up convolutions, where the computation results are not accumulated in the channel direction. They also operate in a three-dimensional operation mode when performing convolutions, pointwise convolutions, or fully connected layer computations, where the computation results are accumulated in the channel direction.

4. The hardware accelerator as described in claim 1, wherein, The processing core comprises six sub-cores. Each of the sub-cores includes four processing units. Each of the processing units includes four processing elements, and Each of the processing elements includes nine multipliers.

5. The hardware accelerator as described in claim 4, wherein, In the case of computation in the Conv3 layer of a deep neural network, based on the fact that the sign and mantissa of the first tensor are 16 bits in size, the first tensor corresponding to one input channel of the Conv3 layer is broadcast to the four processing units that make up a sub-kernel. Given that the sign and mantissa of the first tensor are 8 bits in size, the first tensor, corresponding to the two input channels of the Conv3 layer, is broadcast to the four processing units that constitute a sub-core. Based on the fact that the sign and mantissa of the first tensor are 4 bits in size, the first tensor, corresponding to the four input channels of the Conv3 layer, is broadcast to the four processing units that constitute a sub-core.

6. The hardware accelerator as described in claim 4, wherein, In the case of computation in the Conv3 layer of a deep neural network, based on the 16-bit size of the sign and mantissa of the second tensor, the second tensor corresponding to one output channel of the Conv3 layer is distributed to the four processing units that constitute a sub-kernel. Based on the fact that the sign and mantissa of the second tensor are 8 bits in size, the second tensor corresponding to the two output channels of the Conv3 layer is allocated to the four processing units that constitute a sub-core, and Based on the fact that the sign and mantissa of the second tensor are 4 bits in size, the second tensor corresponding to the four output channels of the Conv3 layer is allocated to the four processing units that constitute a sub-core.

7. The hardware accelerator as described in claim 4, wherein, The first processing device includes: Six 4-way adder trees are configured to sum the outputs of four processing units included in each of the six sub-cores; A six-bit truncate is configured to round each of the outputs of a 4-way adder tree to have a preset number of bits. A selective 6-way adder tree is configured to selectively sum the outputs of bit-truncate ... An arithmetic converter is configured to output FP32 portions and data based on the output of a selective 6-way adder tree and the output of a shared exponent processor; and The accumulator is configured to accumulate FP32 portions and data.

8. The hardware accelerator as described in claim 4, wherein, The second processing device includes: Four 6-way adder trees, each configured to sum the outputs of corresponding processing units in different subcores; Four arithmetic converters are configured to output FP32 portions and data based on the output of a 6-way adder tree and the output of a shared exponent processor; Four accumulators are configured to accumulate FP32 portions and data; and A selective 4-way adder tree is configured to selectively sum the outputs of the accumulators.

9. The hardware accelerator as claimed in claim 4, wherein, Each of the processing units includes: Nine multipliers are configured to perform multiplication operations; A 9-way adder tree is configured to sum the outputs of nine multipliers; and The selective shift logic circuit is configured to shift the output of a 9-way adder tree by a preset number of bits.

10. The hardware accelerator of claim 1, further comprising a shared index processor configured to process the shared index of the first tensor and the shared index of the second tensor.

11. The hardware accelerator as claimed in claim 1, wherein, The processing core is configured to perform calculations using multiple data types corresponding to control signals, and Data types include a first data type with fixed-point format, a second data type with only integers, a third data type with both signed and integers, and a fourth data type with a shared exponent in real number format.

12. The hardware accelerator as claimed in claim 1, wherein, The size of the sharing exponent of the first tensor and the size of the sharing exponent of the second tensor are both 8 bits. The sign and mantissa of the first tensor, or the sign and mantissa of the second tensor, are one of 4 bits, 8 bits, and 16 bits, and Map the first tensor and the second tensor to the processing kernel based on the sign and mantissa of the first tensor or the sign and mantissa of the second tensor.

13. The hardware accelerator of claim 12, wherein, The sign and mantissa of the first tensor or the sign and mantissa of the second tensor are determined based on the forward pass step, backward pass step, or weight update step of training the deep neural network.

14. The hardware accelerator of claim 1 further includes a core output buffer configured to output the output value of the first processing device in the weight update step, and to output the output value of the second processing device in the forward pass step and the backward pass step.

15. The hardware accelerator of claim 14, further comprising: The batch normalization circuit is configured to perform batch normalization based on the output of the core output buffer. The modified linear unit ReLU pooling circuit is configured to perform ReLU function values ​​and pooling values ​​based on the output of the batch normalization circuit; The first-in-first-out (FIFO) circuit is configured to store and output the output of the ReLU pool circuit. An FP2BFP converter circuit is configured to convert the floating-point data type of the output of a FIFO circuit to a block floating-point format. as well as The quantization circuit is configured to quantize the output of the FP2BFP converter circuit according to a predefined precision.

16. The hardware accelerator of claim 1, wherein, Each of the plurality of multipliers includes: The first MUX is configured to receive a previous second tensor via a first input terminal, receive a current second tensor via a second input terminal, and output the current second tensor or the first tensor in response to a hold signal; The multiplier core is configured to perform multiplication operations using a 5-bit first tensor and a 5-bit second tensor as operands; The first register is configured to store the output of the multiplier core; The second register is configured to store the output of the first MUX; and The second MUX is configured to receive a value stored in the second register via the first input terminal, receive the output of the first MUX via the second input terminal, and output the value stored in the second register or the output of the first MUX in response to a bypass signal.

17. The hardware accelerator of claim 16, wherein, The value stored in the second register is maintained by a feedback loop generated by the hold signal.

18. An electronic device for performing training and inference of a deep neural network, the electronic device comprising: The hardware accelerator is configured to perform one-dimensional subword parallelization between the sign and mantissa of the first tensor and the sign and mantissa of the second tensor using multiple multipliers, and to perform processing between the shared exponent of the first tensor and the shared exponent of the second tensor using a shared exponent processor. The processor is configured to execute at least one instruction to control a hardware accelerator based on deep neural network information, the deep neural network information including at least one of the following: the number of layers in the deep neural network, the type of layers, the shape of the tensor, the dimension of the tensor, the operating mode, the bit precision, the type of batch normalization, the type of pooling layer, and the type of the modified linear unit ReLU function. as well as The memory stores the at least one instruction and the deep neural network.

19. The electronic device of claim 18, wherein, The at least one processor is further configured to execute the at least one instruction to obtain at least one of the bit precision and block size of the deep neural network based on user input, set at least one of the bit precision and block size of a first tensor and a second tensor based on the obtained bit precision and block size, and control a hardware accelerator to train the deep neural network based on at least one of the set bit precision and block size of the first tensor and the second tensor.

20. The electronic device of claim 18, wherein, The hardware accelerator is also configured to operate in a two-dimensional operation mode, in which it outputs the calculation results of multiple multipliers without accumulating them in the channel direction, or to operate in a three-dimensional operation mode, in which it accumulates the calculation results of multiple multipliers in the channel direction and outputs the result of accumulating the calculation results.

Images