Arithmetic unit, related device and method

Abstract

The present disclosure provides an arithmetic unit, related devices and methods. The arithmetic unit includes: a weight buffer for storing row vectors taken from a sparse post-weight matrix of M×Kα, where M and K are the number of rows and columns of the pre-sparse weight matrix respectively, and α is a sparse coefficient; an activation buffer for storing a K×N activation matrix; an index strobe for storing strobe indices corresponding to the row vectors and strobbing rows of the activation matrix according to the strobe indices to obtain a strobed post-activation matrix of Kα×N; and a dot product arithmetic unit for multiplying the row vectors by the strobed post-activation matrix. The present disclosure implements a hardware operation mode of a DNN, which can make full use of the structural sparsity of the DNN, thereby improving the inference efficiency, and has a small register file bandwidth occupancy and weak timing constraints.

Description

Technical Field

[0001] The present disclosure relates to the field of chips, and more particularly, to an arithmetic unit, related devices, and methods. Background Art

[0002] Deep neural networks (DNNs) are widely used in the field of artificial intelligence. To reduce the computational intensity and memory overhead of DNNs, a large number of network pruning techniques have been proposed in previous studies to remove redundant connections in DNNs. A deep neural network has multiple layers of nodes, and each node is connected to the nodes in the previous layer, which is equivalent to receiving the output of the nodes in the previous layer as input and performing operations (such as multiplication, convolution, etc.) with the weight matrix of the node. The pruning technique is equivalent to changing a large number of nodes in the weight matrix of the node to 0 to increase its sparsity. For the weights of 0, there is no need to participate in the operation, thereby reducing the computational intensity and memory overhead. Among all pruning methods, structured pruning is a pruning technique that is promising to be effectively implemented on hardware. Through structured pruning, the resulting structural sparsity usually presents a fixed pattern and distribution. For example, 50% of the weights in the weight matrix are 0, which is quite friendly to hardware.

[0003] However, the above pruning techniques focus on the software implementation of DNNs. When a DNN needs to be deployed on hardware for execution, there are currently two hardware deployment methods. One is to implement the DNN using a traditional tensor engine or core dedicated to dense DNN inference, and this method cannot fully utilize the structural sparsity of the DNN. The other is a tensor engine design method based on weight reuse, which has a large register file bandwidth occupancy and strong timing constraints. Summary of the Invention

[0004] In view of this, the present disclosure aims to implement a hardware operation mode of a DNN, which can fully utilize the structural sparsity of the DNN, thereby improving the inference efficiency, and has a small register file bandwidth occupancy and weak timing constraints.

[0005] According to one aspect of the present disclosure, there is provided an arithmetic unit, including:

[0006] A weight buffer for storing row vectors taken from a sparse post-weight matrix of M×Kα, where M and K are the number of rows and columns of the pre-sparse weight matrix respectively, and α is a sparse coefficient;

[0007] An activation buffer for storing a K×N activation matrix;

[0008] An index strobe for storing strobe indices corresponding to the row vectors and strobbing rows of the activation matrix according to the strobe indices to obtain a strobed activation matrix of Kα×N;

[0009] A dot product arithmetic unit for multiplying the row vectors by the strobed activation matrix.

[0010] Optionally, there are L weight buffers for storing L row vectors sequentially taken from the sparse post-weight matrix respectively; there are L index gating units for storing gating indices corresponding to the L row vectors respectively; and there are L·N dot product operation units for multiplying the L row vectors with N column vectors of the gated excitation matrix respectively.

[0011] Optionally, L = 2.

[0012] Optionally, α = 50%.

[0013] Optionally, the dot product operation unit sets M / L clock cycles for the multiplication of the weight matrix and the excitation matrix, where M is an integer multiple of L; in a single clock cycle, the row vectors in the weight buffer are multiplied with N column vectors of the gated excitation matrix through the N dot product operation units corresponding to the row vectors respectively.

[0014] Optionally, the row vectors are loaded into the weight buffer in the first predetermined number of cycles before the clock cycle when the dot product operation unit performs the multiplication operation.

[0015] Optionally, the first predetermined number is 1.

[0016] Optionally, the gating indices are loaded into the index gating unit in the second predetermined number of cycles before the clock cycle when the gated excitation matrix selected according to the gating indices is multiplied by the dot product operation unit.

[0017] Optionally, the second predetermined number is 1.

[0018] Optionally, the excitation matrix is loaded into the excitation buffer in the third predetermined number of cycles before the clock cycle when the first gated excitation matrix obtained by gating the excitation matrix is multiplied by the dot product operation unit, and is held for M / L clock cycles.

[0019] Optionally, the third predetermined number is M / L + 1.

[0020] Optionally, the operation unit further includes:

[0021] An alternative excitation buffer for storing the next excitation matrix, and once the excitation matrix is unloaded from the excitation buffer, loading the excitation buffer with the next excitation matrix stored in the alternative excitation buffer.

[0022] Optionally, the excitation buffer includes a first excitation buffer and a second excitation buffer that are ping-pong buffers, and are respectively used to store the current excitation matrix and the next excitation matrix. The first excitation buffer and the second excitation buffer are both connected to the index strobe. After the dot product operation unit completes the multiplication of each row vector of the sparse post-weight matrix and the strobed current excitation matrix, the index strobe switches to strobe the second excitation buffer.

[0023] According to one aspect of the present disclosure, an acceleration unit is further provided, including:

[0024] An instruction cache for caching instructions to be executed;

[0025] The arithmetic unit as described above;

[0026] An sequencer for fetching the instruction to be executed from the instruction buffer, and if the instruction to be executed is a multiplication instruction of the sparse post-weight matrix and the excitation matrix, allocating the multiplication instruction to the arithmetic unit;

[0027] On-chip memory for storing the sparse post-weight matrix and the excitation matrix respectively for the weight buffer and the excitation buffer to load.

[0028] According to one aspect of the present disclosure, a computing device is further provided, including:

[0029] A memory for storing instructions to be executed;

[0030] A processing unit for loading the instruction to be executed, and if the instruction to be executed is an instruction executed by the acceleration unit, sending the instruction to be executed to the acceleration unit for execution;

[0031] The acceleration unit as described above.

[0032] According to one aspect of the present disclosure, a system-on-chip is further provided, including the acceleration unit as described above.

[0033] According to one aspect of the present disclosure, a data center is further provided, including the computing device as described above.

[0034] According to one aspect of the present disclosure, a method for running a deep neural network is further provided, including:

[0035] Taking out row vectors from an M×Kα sparse post-weight matrix and putting them into a weight buffer, where M and K are the number of rows and columns of the pre-sparse weight matrix respectively, and α is a sparsity coefficient;

[0036] Putting a K×N excitation matrix into the excitation buffer;

[0037] Put the strobe index corresponding to the row vector into the index strobe;

[0038] Through the index strobe, strobe the excitation matrix according to the strobe index to obtain a Kα×N strobed excitation matrix;

[0039] Multiply the row vector by the strobed excitation matrix through the dot product operation unit.

[0040] Optionally, there are L weight buffers; taking the row vector from the sparse post-weight matrix of M×Kα and putting it into the weight buffer includes: sequentially taking out L row vectors from the sparse post-weight matrix and putting them into the L weight buffers respectively.

[0041] Optionally, there are L index strobing devices; putting the strobe index corresponding to the row vector into the index strobe includes: putting the strobe indexes corresponding to the L row vectors into the L index strobing devices.

[0042] Optionally, there are L·N dot product operation units. Multiplying the row vector by the strobed excitation matrix through the dot product operation unit includes: multiplying the L row vectors by the N column vectors of the strobed excitation matrix respectively through the L·N dot product operation units.

[0043] Optionally, multiplying the row vector by the strobed excitation matrix includes: setting M / L clock cycles for the multiplication of the weight matrix and the excitation matrix, where M is an integer multiple of L; in a single clock cycle, the row vectors in the weight buffer are multiplied by the N column vectors of the strobed excitation matrix respectively through the N dot product operation units corresponding to the row vectors.

[0044] Optionally, taking the row vector from the sparse post-weight matrix of M×Kα and putting it into the weight buffer includes: putting the taken row vector into the weight buffer in the first predetermined number of cycles before the clock cycle when the row vector is multiplied by the dot product operation unit.

[0045] Optionally, putting the strobe index corresponding to the row vector into the index strobe includes: putting the strobe index into the index strobe in the second predetermined number of cycles before the clock cycle when the strobed excitation matrix selected according to the strobe index is multiplied by the dot product operation unit.

[0046] Optionally, putting the K×N excitation matrix into the excitation buffer includes: putting the excitation matrix into the excitation buffer in the third predetermined number of cycles before the clock cycle when the first strobed excitation matrix obtained by strobing the excitation matrix is multiplied by the dot product operation unit, and maintaining it for M / L clock cycles.

[0047] Compared with implementing a DNN using a traditional operation unit or core dedicated to dense DNN inference, the weight buffer of the embodiments of the present disclosure does not need to store the sparse pre-weight matrix of M×K, but only needs to store the sparse post-weight matrix of M×Kα. Therefore, the required storage bandwidth is reduced, and only the sparse post-weight matrix is stored, avoiding the participation of the zero-weight part in the operation, making full use of the structural sparsity of the DNN and thus improving the inference efficiency. Compared with the operation unit based on weight reuse, it has been proven in practice that when it adopts specific matrix parameters (number of rows or columns), it can minimize the occupied register file bandwidth, and it does not have the ping-pong buffer design in the operation unit based on weight reuse and is not subject to timing constraints. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Through the description of the embodiments of the present disclosure with reference to the following drawings, the above and other objects, features, and advantages of the present disclosure will become clearer. In the drawings:

[0049] Figure 1 is a structural diagram of a data center to which an embodiment of the present disclosure is applied;

[0050] Figure 2 is an internal structural diagram of a server in the data center according to an embodiment of the present disclosure;

[0051] Figure 3 is an internal structural diagram of a processing unit and an acceleration unit inside the server according to an embodiment of the present disclosure;

[0052] Figure 4 is an internal structural diagram of an acceleration unit core according to an embodiment of the present disclosure;

[0053] Figure 5A is an example of a slicing scheme for the excitation matrix and weight matrix of a tensor engine for dense DNN inference;

[0054] Figure 5B is an example of the internal structure of a tensor engine for dense DNN inference;

[0055] Figure 5C is Figure 5B an example of the operating timing of each unit of;

[0056] Figure 6A is an example of a slicing scheme for the excitation matrix and weight matrix of a tensor engine for weight-reused sparse DNN inference;

[0057] Figure 6B is an example of the internal structure of a tensor engine for weight-reused sparse DNN inference;

[0058] Figure 6C is Figure 6B an example of the operating timing of each unit of;

[0059] Figure 7A An example of a slicing scheme for the activation matrix and weight matrix of a tensor engine that incentivizes reuse in sparse DNN inference according to an embodiment of the present disclosure;

[0060] Figure 7B An example of the internal structure of a tensor engine that incentivizes reuse in sparse DNN inference according to an embodiment of the present disclosure;

[0061] Figure 7C is Figure 7B An example of the operating timing of each unit of

[0062] Figure 8A A slicing scheme in a broad sense for the activation matrix and weight matrix of a tensor engine that incentivizes reuse in sparse DNN inference according to an embodiment of the present disclosure;

[0063] Figure 8B An internal structure diagram in a broad sense of a tensor engine that incentivizes reuse in sparse DNN inference according to an embodiment of the present disclosure;

[0064] Figure 8C is Figure 8B An example of the operating timing of each unit of

[0065] Figure 9 A comparison of the occupied bandwidth and other consumption parameters between activation-reuse sparse DNN inference and weight-reuse sparse DNN inference under different matrix scaling schemes;

[0066] Figure 10 A flowchart of a method for running a deep neural network according to an embodiment of the present disclosure. Detailed implementation manners

[0067] The following describes the present disclosure based on embodiments, but the present disclosure is not limited to these embodiments. In the following detailed description of the present disclosure, some specific details are described in detail. Those skilled in the art can fully understand the present disclosure without the description of these details. In order to avoid obscuring the essence of the present disclosure, well-known methods, processes, and procedures are not described in detail. Additionally, the drawings are not necessarily drawn to scale.

[0068] The following terms are used herein.

[0069] Deep Learning Network (DNN): The deep learning network is a new research direction in the field of machine learning. It introduces machine learning to make it closer to the original goal - artificial intelligence (AI). The deep learning network learns the internal laws and representation levels of sample data. The information obtained during these learning processes is very helpful for the interpretation of data such as text, images, and sounds. Its ultimate goal is to enable machines to have the ability to analyze and learn like humans, and be able to recognize data such as text, images, and sounds.

[0070] Weight Matrix: A deep neural network has multiple layers of nodes. Each node is connected to the nodes in the previous layer, which is equivalent to receiving the output of the nodes in the previous layer as input and processing these inputs to produce an output to the nodes in the next layer. Since its input includes the outputs of multiple nodes in the previous layer and its output includes different outputs for multiple nodes in the next layer, its input and output are not single values but are presented in the form of vectors or matrices (receiving the outputs of multiple nodes in the previous layer at a single time point can be represented as a vector, and since the outputs of multiple nodes in the previous layer are different at different time points, it can be represented as a matrix). Its output can also be represented in the form of vectors or matrices (at a single time point, the output to multiple nodes in the next layer can be represented as a vector, and since these outputs are different at different time points, it can be represented as a matrix). For each input from the nodes in the previous layer, it can be considered that the input has undergone a process of multiplication by the corresponding weight and an offset by the offset amount, and finally the corresponding output is obtained. Since the input and output are in the form of vectors or matrices, the corresponding weights can form a matrix, which is called the weight matrix. The input vector or matrix is multiplied by the weight matrix, and there may be an offset by the offset matrix to obtain the output vector or matrix.

[0071] Activation Matrix: As introduced and discussed in the previous weight matrix, if the input of a node is in the form of a matrix, it is the activation matrix. The processing of the node can be regarded as the multiplication of the activation matrix and the weight matrix.

[0072] Pruning: Make a judgment on the weight elements in the weight matrix. If a weight has little impact on the operation result of the deep learning network, it can be removed during the operation, that is, make the weight become 0. In this way, in the weight matrix, only the non-zero weights participate in the operation, thus reducing the computational intensity and memory overhead.

[0073] Structured Pruning: Pruning that makes the weight distribution in the pruned weight matrix present a fixed pattern. For example, through pruning, 50% of the weights in each row of the weight matrix are made to become 0, or 20% of the weights in each column are made to become 0. Since the structural sparsity produced by structured pruning presents a fixed pattern and distribution, it is very friendly to hardware.

[0074] Sparse coefficient: The ratio of the unpruned weights in the weight matrix after pruning to all the weights, or the ratio of the unpruned weights in each row or column of the weight matrix after pruning to the weights in that row or column. For example, a sparsity of 50% means that half of the weights in each row of the weight matrix become 0.

[0075] Weight matrix before sparsification: The weight matrix before pruning the weights of the weight matrix (where a part of the weights become 0).

[0076] Weight matrix after sparsification: The weight matrix after pruning the weights of the weight matrix (where a part of the weights become 0).

[0077] Strobe index: An index indicating the strobed object. Generally, 1 is used to indicate that the object is strobed and participates in the operation, and 0 is generally used to indicate that the object is not strobed and does not participate in the operation. The object can be a row in the weight matrix or a weight, etc.

[0078] Acceleration unit: In view of the low efficiency of traditional processing units in some specialized fields (such as processing images, performing various operations of deep learning networks, etc.), a processing unit designed to improve the data processing speed in these specialized fields. The acceleration unit is also called an artificial intelligence (AI) processing unit, including a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and dedicated intelligent acceleration hardware (such as a neural network processor NPU, a hardware accelerator, etc.).

[0079] Processing unit: A unit that performs traditional processing (processing other than complex operations such as image processing and fully connected operations in various deep learning networks) in the servers of a data center. In addition, the processing unit also undertakes the scheduling function for the acceleration unit and itself, and allocates the tasks that need to be undertaken to the acceleration unit and itself. The processing unit can take various forms such as a processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc.

[0080] Operation unit: A unit that executes DNN inference in the embodiments of the present disclosure, which can be a tensor engine, but is not limited to a tensor engine.

[0081] Data center

[0082] A data center is a specific network of devices for global collaboration, used to transfer, accelerate, display, compute, and store data information on the Internet network infrastructure. In future development, data centers will also become assets for enterprise competition. With the widespread application of data centers, artificial intelligence and other technologies are increasingly applied to data centers. As an important technology in artificial intelligence, deep learning has been widely applied to big data analysis and computing in data centers.

[0083] In traditional large data centers, the network structure is usually as Figure 1 shown, that is, the hierarchical inter-networking model. This model consists of the following parts:

[0084] Servers 140: Each server 140 is a processing and storage entity in the data center, and the processing and storage of a large amount of data in the data center are completed by these servers 140.

[0085] Access switches 130: Access switches 130 are used to connect servers 140 to the data center. One access switch 130 connects multiple servers 140. Access switches 130 are usually located at the top of the rack, so they are also called Top of Rack switches, and they are physically connected to the servers.

[0086] Aggregation switches 120: Each aggregation switch 120 connects multiple access switches 130 and provides other services, such as firewalls, intrusion detection, network analysis, etc.

[0087] Core switches 110: Core switches 110 provide high-speed forwarding for packets entering and leaving the data center and provide connectivity for aggregation switches 120. The network of the entire data center is divided into an L3 layer routing network and an L2 layer routing network. Core switches 110 usually provide a flexible L3 layer routing network for the network of the entire data center.

[0088] Normally, the aggregation switch 120 is the demarcation point between the L2 and L3 layer routing networks. Below the aggregation switch 120 is the L2 network, and above is the L3 network. Each group of aggregation switches manages a Point Of Delivery (POD), and each POD is an independent VLAN network. Servers can migrate within a POD without modifying the IP address and default gateway because one POD corresponds to one L2 broadcast domain.

[0089] The Spanning Tree Protocol (STP) is usually used between the aggregation switch 120 and the access switch 130. STP makes only one aggregation layer switch 120 available for a VLAN network, and other aggregation switches 120 are used only when a failure occurs. That is to say, at the level of the aggregation switch 120, horizontal expansion cannot be achieved because even if multiple aggregation switches 120 are added, only one is still working.

[0090] In the deployment and operation of a deep learning network, the deep learning network to be deployed can be assigned to an ingress server 140 in the data center, and the ingress server 140 distributes the deployment instructions of the entire deep learning network to multiple different servers 140 for deployment. When the deep learning network is to be run for inference, the inference instruction is sent to the ingress server 140, and the ingress server 140 distributes it to the corresponding server 140 for running according to the distribution of the deep learning network on multiple servers 140.

[0091] Server

[0092] Since the server 140 is the real processing device in the data center, Figure 2 The internal structural block diagram of a server 140 is shown. The server 140 includes a memory 210, a processing unit cluster 270, and an acceleration unit cluster 280 connected by a bus. The processing unit cluster 270 includes multiple processing units 220. The acceleration unit cluster 280 includes multiple acceleration units 230. The acceleration unit 230 is a processing unit designed to improve the data processing speed in a specific application field. The acceleration unit is an artificial intelligence (AI) processing unit, including a central processing unit (CPU), a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), and a dedicated intelligent acceleration hardware (e.g., a neural network processor NPU, a hardware accelerator, etc.).

[0093] The architecture design of the traditional processing unit 220 is such that the control unit and the storage unit occupy a large part of the space in the architecture, while the computing unit occupies insufficient space. Therefore, it is very effective in logical control, but not efficient enough in large-scale parallel computing. As a result, various specialized acceleration units 230 have been developed to more effectively improve the operation speed for calculations with different functions and in different fields. For example, it can more effectively improve the operation speed for deep learning networks. Such an acceleration unit 230 for deep learning networks adopts an architecture of data-driven parallel computing and is a processing unit used to process a large number of operations (such as fully connected operations, pooling, etc.) of each deep learning network node. Since the data and intermediate results in a large number of operations (such as fully connected operations, pooling, etc.) of each deep learning network node are closely related throughout the calculation process and are often used, with the existing processing unit architecture, due to the small memory capacity inside the core of the processing unit, it is necessary to frequently access the off-chip memory in large amounts, resulting in low processing efficiency. By using such an acceleration unit dedicated to accelerating the operation processing speed of deep learning networks, since each core has on-chip memory with a storage capacity suitable for deep learning network calculations, it can avoid frequently accessing the off-chip memory, thus greatly improving the processing efficiency and computing performance.

[0094] The acceleration unit 230 is to be scheduled by the processing unit 220. As Figure 2 shown, a part of various deep learning networks, including the weight matrices of some nodes of these models, etc., are stored in the memory 210. When needed, a part of these deep learning networks is Figure 2 deployed by one of the processing units 220 in the memory 210 to an acceleration unit 230. That is, the processing unit 220 can send the addresses of some parameters (such as the weight matrices of each node) in the deep learning network in the form of instructions to the acceleration unit 230. When the acceleration unit 230 actually uses the deep learning network for calculation, it will directly address these parameters in the memory 210 according to the addresses of these parameters (such as the weight matrix) in the memory 210 and temporarily store them in its on-chip memory. When the acceleration unit 230 actually uses the deep learning network for calculation, the processing unit 220 will also send the activation matrix of the model to the acceleration unit 230 in the form of instructions and temporarily store it in the on-chip memory of the acceleration unit 230. In this way, the acceleration unit 230 can perform inference calculations based on these input activation matrices and the weight matrices in the model.

[0095] Internal structure of the processing unit and the acceleration unit

[0096] Next, in combination with Figure 3 the internal structure diagrams of the processing unit 220 and the acceleration unit 230, it will be specifically described how the processing unit 220 schedules the acceleration unit 230 and itself to work.

[0097] As shown Figure 3 in FIG., the processing unit 220 includes a plurality of processor cores 222 and a cache 221 shared by the plurality of processor cores 222. Each processor core 222 includes an instruction fetch unit 203, an instruction decoding unit 224, an instruction issuing unit 225, and an instruction execution unit 226.

[0098] The instruction fetch unit 223 is used to transfer the instruction to be executed from the memory 210 to an instruction register (which can be one of the registers in the register file 229 shown Figure 3 for storing instructions), and receive the next instruction fetch address or calculate the next instruction fetch address according to an instruction fetch algorithm. The instruction fetch algorithm includes, for example: incrementing or decrementing the address according to the instruction length.

[0099] After the instruction is fetched, the processing unit 220 enters the instruction decoding stage. The instruction decoding unit 224 decodes the fetched instruction according to a predetermined instruction format to obtain the operand acquisition information required for the fetched instruction, so as to prepare for the operation of the instruction execution unit 225. The operand acquisition information points to, for example, an immediate number, a register, or other software / hardware that can provide source operands.

[0100] The instruction issuing unit 225 is located between the instruction decoding unit 224 and the instruction execution unit 226, and is used for instruction scheduling and control to efficiently allocate each instruction to different instruction execution units 226, making it possible to perform parallel operations on multiple instructions.

[0101] After the instruction issuing unit 225 issues the instruction to the instruction execution unit 226, the instruction execution unit 226 starts to execute the instruction. However, if the instruction execution unit 226 determines that the instruction should be executed by an acceleration unit, it forwards the instruction to the corresponding acceleration unit for execution. For example, if the operation involved in the instruction has been previously deployed by the processing unit 220 on the acceleration unit 230 for execution, such as an inference instruction of a deep learning network, the instruction execution unit 226 no longer executes the instruction, but sends the instruction to the acceleration unit 230 through the bus, and the acceleration unit 230 executes the instruction.

[0102] The acceleration unit 230 internally includes a plurality of cores 236( Figure 3 4 cores are shown in FIG., but those skilled in the art should understand that the acceleration unit 230 may also include other numbers of cores 236), a command processor 237, a direct memory access mechanism 235, and a bus channel 231.

[0103] The bus channel 231 is a channel for instructions to enter and exit the acceleration unit 230 from the bus.

[0104] The Direct Memory Access (DMA) mechanism 235 is a function provided by some computer bus architectures that enables data to be directly written from an attached device to the memory on the computer motherboard. This method greatly improves the efficiency of data access compared to the way that all data transfers between devices have to go through the processing unit. Because of such a mechanism, the cores of the acceleration unit 230 can directly access the memory 210 to read parameters (such as the weight matrix of each node) in the deep learning network, etc., greatly improving the data access efficiency.

[0105] The command processor 237 distributes the instructions sent from the processing unit 220 to the acceleration unit 230 to the cores 236 for execution. The instruction execution unit 226 sends the sequence of instructions to be executed that need to be executed by the acceleration unit 230 to the acceleration unit 230. After entering from the bus channel 231, this sequence of instructions to be executed is cached in the command processor 237. The command processor 237 selects the core 236 and distributes the instruction sequence to it for execution. In addition, the command processor 237 is also responsible for the synchronization operation between the cores 236.

[0106] In the embodiment of the present disclosure, when the instruction fetch unit 223 fetches an instruction from the memory 210 and, after being decoded by the instruction decoding unit 224, it is found that the instruction is a deployment instruction of the deep learning network, this instruction is sent to the instruction execution unit 226, and the instruction execution unit 226 deploys this deployment instruction to the acceleration unit 230, that is, sends the addresses of the parameters (such as the weight matrix of each node) in this part of the deep learning network in the memory 210 to the acceleration unit 230 in the form of an instruction, and the acceleration unit 230 temporarily stores them in its on-chip memory.

[0107] In the embodiment of the present disclosure, when the instruction fetch unit 223 fetches an instruction from the memory 210 and, after being decoded by the instruction decoding unit 224, it is found that the instruction is an actual inference instruction of the deep learning network, the excitation matrix that is the input of this deep learning network is sent to the acceleration unit 230 in the form of an instruction and temporarily stored in the on-chip memory of the acceleration unit 230. In this way, the acceleration unit 230 can perform operations based on these input excitation matrices and the parameters (such as the weight matrix) found according to the addresses of the parameters (such as the weight matrix) temporarily stored in the on-chip memory to obtain the inference result.

[0108] Acceleration unit core

[0109] Figure 4 It is the internal structure diagram of the acceleration unit core according to an embodiment of the present disclosure.

[0110] In one embodiment, as Figure 4As shown, the acceleration unit core 236 includes a tensor engine 310 , a pooling engine 320 , a memory copy engine 330 , a sequencer 350 , an instruction cache 340 , an on-chip memory 360 , and a constant buffer 370 .

[0111] The instruction sequence assigned by the command processor 237 to the acceleration unit core 236 first enters the instruction cache 340 cache. Then, the sequencer 350 fetches instructions from the instruction cache 340 in a first-in, first-out order, and assigns them to the tensor engine 310 or the pooling engine 320 for execution according to the nature of the instructions. The tensor engine 310 is responsible for processing related operations such as convolution, array multiplication, and fully connected operations in the deep learning network. The embodiment of the present disclosure is mainly executed by the tensor engine 310. The pooling engine 320 is responsible for processing pooling operations in the deep learning network. The memory copy engine 330 is a unit that specializes in processing data copying. The data copy here includes copying some data from the on-chip memory 360 to the memory shared by each core 236, or the on-chip memory 360 of other cores 236, because the on-chip memory 360 may overflow. The sequencer 350 decides to assign the instruction to the tensor engine 310, the pooling engine 320, or the memory copy engine 330 according to the nature of the operation, such as convolution, array multiplication, fully connected operation, pooling, or data copy.

[0112] The on-chip memory 360 is an in-core memory that stores the weight matrix in the deep learning network, as well as the excitation matrix and various intermediate results input when the deep learning network is actually used. The constant buffer 370 is a buffer that stores other constant parameters in the deep learning network (for example, hyperparameters in the neural network model) in addition to the weight matrix. As described above, when the processing unit 220 pre-configures the deep learning network in the acceleration unit 230, the processing unit 220 sends the address of the parameters in the deep learning network in the memory 210 to the acceleration unit 230 in the form of instructions. These parameters include the weight matrix of the node and other parameters (such as hyperparameters). For the weight matrix, the acceleration unit 230 takes it out from the corresponding position of the memory 210 during the actual deep learning network inference and places it in the on-chip memory 360. For other parameters, the acceleration unit 230 takes it out from the corresponding position of the memory 210 during the actual deep learning network inference and places it in the constant buffer 370. In addition, when the instruction to actually start inference is assigned by the command processor 237 to the core 236 for execution, the activation matrix in the instruction (the input to the deep learning network) is also stored in the on-chip memory 360. In addition, after the tensor engine 310 and the pooling engine 320 perform convolution, full connection, or pooling operations, the various intermediate results obtained are also stored in the on-chip memory 360.

[0113] Background of the present disclosure

[0114] As described above, when deploying and running a DNN for inference, in order to reduce the computational intensity and memory overhead of the DNN, previous research has proposed network pruning techniques, that is, changing a large number of nodes in the weight matrix of the nodes to 0 to increase its sparsity. For the weights of 0, there is no need to participate in the operation, thus reducing the computational intensity and memory overhead. Especially for structured pruning, a fixed ratio of weights in the weight matrix after pruning becomes 0, which is quite friendly to hardware. In theory, this feature can be utilized to reduce the amount of computation in the hardware when the DNN is deployed to the hardware.

[0115] However, in practice, currently when a DNN is deployed to a hardware for execution, there are mainly two methods. One is to implement the DNN using a traditional tensor engine or core dedicated to dense DNN inference, and the other is a tensor engine design method based on weight reuse. Before discussing the embodiments of the present disclosure in detail, it is necessary to discuss these two solutions to better understand the embodiments of the present disclosure.

[0116] Dense DNN inference scheme

[0117] Figure 5A It is an example of the slicing scheme of the excitation matrix and the weight matrix of the tensor engine 310 for dense DNN inference. Figure 5B It is the structural diagram of the tensor engine 310 for dense DNN inference. As Figure 5B shown, the tensor engine 310 for dense DNN inference includes a weight buffer 312, an excitation buffer 311, and a dot product operation unit 313.

[0118] The weight buffer 312 is used to cache the weights moved into the tensor engine 310 from Figure 4 the on-chip memory 360. During the process of the processing unit 220 pre-configuring the deep learning network in the acceleration unit 230, the processing unit 220 sends the addresses of parameters such as the weight matrix in the memory 210 to the acceleration unit 230 in the form of instructions. When the acceleration unit 230 actually performs deep learning network inference, according to these addresses, the weight matrix is taken out from the corresponding position in the memory 210 and placed in the on-chip memory 360. When the tensor engine 310 needs weights during operation, the required part of the weights in the weight matrix is placed in the weight buffer 312 for caching, and then the dot product operation unit 313 uses these weights for operation.

[0119] The excitation buffer 311 is used to cache the excitations moved into the tensor engine 310 from the on-chip memory 360. When the instruction for actual inference starts to be assigned to the core 236 by the command processor 237 for execution, the excitation matrix in the instruction is stored in the on-chip memory 360. In addition, when the tensor engine 310 starts to perform operations, the corresponding excitations in the excitation matrix are moved into the excitation buffer 311 for caching, and then the dot product operation unit 313 uses these excitations to perform multiplication operations with the weights in the weight buffer 312.

[0120] Due to the large sizes of the excitation matrix and the weight matrix, the storage and computational overhead required for multiplying the two matrices as a whole is too large. Therefore, a method of partitioning the excitation matrix and the weight matrix is adopted. When multiplying the excitation matrix and the weight matrix, the excitation matrix and the weight matrix are partitioned separately. The number of columns of the partitioned excitation matrix should be equal to the number of rows of the partitioned weight matrix to ensure correct multiplication of the matrix partitions. As Figure 5A shown, a 16×16 excitation matrix can be divided into 16 2×8 excitation matrix partitions, and a 16×16 weight matrix can be divided into 2 8×16 weight matrix partitions. In this way, the number of columns of the excitation matrix partitions and the number of rows of the weight matrix partitions are both 8, and the excitation matrix partitions can be multiplied by the weight matrix partitions to obtain 2×16 result matrix partitions. The result matrix partitions obtained by multiplying each divided excitation matrix partition by the corresponding weight matrix partition are superimposed to obtain the final result matrix.

[0121] When each 2×8 excitation matrix partition enters the tensor engine 310, it is divided into two rows (i.e., two 1×8 row vectors) and enters Figure 5B the excitation buffers 0 and 1 respectively. The two 8×16 weight matrix partitions enter Figure 5B the weight buffers 0 and 1 respectively. The weight buffers 0 and 1 are ping-pong buffers, and they work alternately. Only one buffer works at a certain moment. The row vectors stored in the excitation buffers 0 and 1 either multiply with the 8×16 weight matrix partition in the weight buffer 0 or multiply with the 8×16 weight matrix partition in the weight buffer 1, and cannot multiply with the two weight matrix partitions separately. For example, Figure 5A the two row vectors obtained by dividing the excitation matrix partition 0 are both only allowed to multiply with the weight matrix partition 0, because according to the matrix multiplication rule, when multiplying the excitation matrix and the weight matrix, they can only multiply with the weight matrix partition 0. When the row vectors stored in the excitation buffer 0 or 1 multiply with the 8×16 weight matrix partition stored in the weight buffer 0 or 1, the 8×16 weight matrix partition is divided into 16 columns, and then the 1×8 row vector stored in the excitation buffer 0 or 1 multiplies with each divided column (i.e., each 8×1 column vector) in 16 dot product operation units 313 respectively. Each dot product operation unit 313 completes the multiplication of each column and summarizes to obtain the multiplication result of the row vector stored in the excitation buffer 0 or 1 and the 8×16 weight matrix partition stored in the weight buffer 0 or 1, that is, the multiplication result of the 2×8 excitation matrix partition and the 8×16 weight matrix partition.

[0122] Figure 5C shows Figure 5BTiming diagram of the structural operation of the shown tensor engine 310. Since the excitation matrix is divided into 16 excitation matrix slices, and each excitation matrix slice needs to be multiplied by the corresponding weight matrix slice to obtain the final multiplication result, the entire matrix multiplication is divided into 16 clock cycles. The excitation buffer 311 loads different excitation matrix slices 0 - 15 at each clock cycle 0 - 15 respectively, and it needs to load the corresponding excitation matrix slice at least one clock cycle before the operation of the excitation matrix slice in the dot product operation unit 313 to ensure that the dot product operation unit 313 can obtain the corresponding excitation matrix slice during the operation. The weight buffer 312 does not need to load the weight every clock cycle because the weight can be reused after being loaded once. For example, Figure 5A The excitation matrix slices 0 - 3 of need to be multiplied by the weight matrix slice 0 according to the multiplication rule of the excitation matrix and the weight matrix. Therefore, the weight buffer 0 can load the weight matrix slice 0 at least 4 clock cycles before the dot product operation unit 313 needs to use the weight matrix slice 0 and hold it for at least 4 clock cycles. In this way, in clock cycles 0 - 3, the dot product operation unit 313 performs the multiplication operations of the excitation matrix slices 0 - 3 and the weight matrix slice 0 respectively. Figure 5A The excitation matrix slices 4 - 7 of need to be multiplied by the weight matrix slice 1 according to the multiplication rule of the excitation matrix and the weight matrix. Therefore, the weight buffer 1 can load the weight matrix slice 1 at least 4 clock cycles before the dot product operation unit 313 needs to use the weight matrix slice 1 and hold it for at least 4 clock cycles. In this way, in clock cycles 4 - 7, the dot product operation unit 313 performs the multiplication operations of the excitation matrix slices 4 - 7 and the weight matrix slice 1 respectively.

[0123] In Figure 5B 's example, the storage capacity requirement for the excitation buffer 311 is to accommodate at least 2 × 8 weight data. Due to the ping - pong buffer mechanism, the storage capacity requirement for the weight buffer 312 is to accommodate at least (8 × 16) × 2 weight data. Each weight data occupies 16 bits. Therefore, the bandwidth requirement caused by the excitation is (2 × 8) × 16 = 256 bits / clock cycle, and the bandwidth requirement caused by the weight is (8 × 16 / 4) × 16 = 512 bits / clock cycle. The reason for dividing by 4 is that since the weight buffer 312 does not need to update the data every clock cycle, it can be updated once every 4 clock cycles, as Figure 5C shown.

[0124] The disadvantage of the dense DNN inference scheme is that it cannot fully utilize the structural sparsity of the DNN. Because many weights in the pruned weight matrix become 0, they still need to participate in the operation of the dot product operation unit 313, without any savings, and both the bandwidth occupation and the computing resource consumption are relatively large.

[0125] Weight reuse sparse DNN inference scheme

[0126] Figure 6A It is an example of a slicing scheme for the excitation matrix and weight matrix of the tensor engine 310 for weighted reuse sparse DNN inference. [[ID= It is a structural diagram of the tensor engine 310 for weighted reuse sparse DNN inference. As ​ shown, the tensor engine 310 for weighted reuse sparse DNN inference includes a weight buffer 312, an excitation buffer 311, an index gater 314, and a dot product operation unit 313.

[0127] The weight buffer 312 is used to cache the weights of the sparse post-weight matrix that are moved from the ​ on-chip memory 360 into the tensor engine 310. For the weights pruned from the weight matrix, since they have become 0 and do not participate in the operation of the dot product operation unit 313, bandwidth and computing resources are saved. Therefore, the weight buffer 312 only needs to cache the unpruned weights in the weight matrix. During the process of the processing unit 220 pre-configuring the deep learning network in the acceleration unit 230, the processing unit 220 sends the address of the sparse post-weight matrix in the memory 210 to the acceleration unit 230 in the form of an instruction. When the acceleration unit 230 performs actual deep learning network inference, according to these addresses, the sparse post-weight matrix is retrieved from the corresponding position in the memory 210 and placed in the on-chip memory 360. When the tensor engine 310 needs weights during operation, the required weights in the sparse post-weight matrix are placed in the weight buffer 312 for caching, and then the dot product operation unit 313 uses these weights for operation.

[0128] The excitation buffer 311 is used to cache the excitations that are moved from the on-chip memory 360 into the tensor engine 310. When the instruction for actual inference start is assigned to the core 236 by the command processor 237 for execution, the excitation matrix in the instruction is stored in the on-chip memory 360. Additionally, when the tensor engine 310 starts to perform operations, the corresponding excitations in the excitation matrix are moved into the excitation buffer 311 for caching, and then the dot product operation unit 313 uses these excitations to perform multiplication operations with the weights in the weight buffer 312. Since the weight buffer 312 stores the weights of the sparse post-weight matrix, and the excitation buffer 311 stores the excitation rows in the entire excitation matrix. Not every excitation element in these excitation rows will participate in the operation of the dot product operation unit 313. Only those excitation elements corresponding to the unpruned weights will be input into the dot product operation unit 313 to participate in the operation. Therefore, an index gater 314 is set between the excitation buffer 311 and the dot product operation unit 313, which gates the excitation elements corresponding to the unpruned weights in the excitation rows according to the stored gating index. In one example, the gating index corresponding to the unpruned weights can be set to 1, and the gating index corresponding to the pruned weights can be set to 0.

[0129] Due to the large size of the excitation matrix, multiplying it with the weight matrix requires excessive storage and computational overhead. Therefore, a method of fragmenting the excitation matrix is adopted. As ​ shown, a 16×16 excitation matrix can be divided into 8 excitation matrix fragments of 2×16. Assuming a sparsity coefficient of 50%, the 16×16 weight matrix becomes an 8×16 sparsified excitation matrix after pruning. Since the number of columns of the excitation matrix fragment is 16 and the number of rows of the sparsified weight matrix is 8, the excitation matrix fragment and the sparsified weight matrix cannot be directly multiplied. However, since 50% of the weights in the weight matrix have been pruned to 0 and do not need to be multiplied with the corresponding excitation, in fact, only half of the 16 columns of the excitation matrix fragment, that is, 8 columns, need to be multiplied with the sparsified weight matrix, and the sparsified weight matrix has 8 rows, so multiplication can be achieved. For a 2×16 excitation matrix fragment, when entering the tensor engine 310, it is divided into two rows (i.e., two 1×16 row vectors) and enters ​ the excitation buffers 0 and 1 respectively. Only 8 of the 16 excitation elements of this row vector need to be multiplied with the 8×16 sparsified excitation matrix. Therefore, through the selection of the 8:16 index gater 314, only 8 excitation elements can enter the corresponding dot product operation unit 313 and be multiplied with the corresponding column of the 16 columns of the 8×16 sparsified excitation matrix. Which 8 of the 16 excitation elements of the row vector are selected is determined by the selection index corresponding to this row vector stored in the index gater 314. Different row vectors have different corresponding selection indexes, and different excitation elements are selected.

[0130] The 8×16 sparsified weight matrix enters ​ the weight buffer 0, and the next 8×16 sparsified weight matrix enters ​ the weight buffer 1. The weight buffers 0 and 1 are ping-pong buffers. After all the operations involving the sparsified weight matrix stored in the weight buffer 0 are completed, the weight buffer 1 starts to work. Only one weight buffer is working at a certain moment. The 1×16 row vector stored in the excitation buffer 311, after being selected by the index gater 314, actually becomes a 1×8 row vector, so that it can be multiplied with the 8×16 sparsified excitation matrix stored in the weight buffer 0. Each dot product operation unit 313 completes the multiplication of the 1×8 row vector with each column vector of the 8×16 sparsified excitation matrix, and summarizes to obtain the multiplication with the entire 8×16 sparsified excitation matrix stored in the weight buffer 0.

[0131] ​ shows ​Timing diagram of the structure operation of the shown tensor engine 310. Since the excitation matrix is divided into 8 excitation matrix slices, and each excitation matrix slice needs to be multiplied by the sparse post-weight matrix to obtain the final multiplication result, the entire matrix multiplication is divided into 8 clock cycles. The index strobe 314 needs to load different strobe indexes respectively in each clock cycle from 0 to 7, and it needs to load the corresponding strobe index at least one clock cycle before the operation of the excitation matrix slice in the dot product operation unit 313, so as to ensure that the dot product operation unit 313 can select a 1×8 row vector according to the strobe index during the operation, so as to multiply with each column vector of the 8×16 sparse post-excitation matrix. The excitation buffer 311 loads different excitation matrix slices 0-7 respectively in each clock cycle from 0 to 7, and it needs to load the corresponding excitation matrix slice in an even earlier clock cycle before the corresponding strobe of the index strobe 314, so as to ensure that the index strobe 314 can strobe the excitation matrix slice in time without delaying the operation of the dot product operation unit 313. The weight buffer 312 does not need to load the weight once per clock cycle because the weight can be used repeatedly after being loaded once. The weight buffer 0 can load the sparse post-excitation matrix at one time, and then the dot product operation unit 313 uses this sparse post-excitation matrix in clock cycles 0-7. Therefore, it can be loaded in the first 8 clock cycles before the index strobe 314 strobes according to the strobe index, but ​ it is loaded in the first 4 clock cycles before the index strobe 314 strobes according to the strobe index. As long as it can be loaded before the index strobe 314 strobes, it is okay.

[0132] In ​ the example, the storage capacity requirement for the excitation buffer 311 is to accommodate at least 2×16 weight data. Due to the ping-pong buffer mechanism, the storage capacity requirement for the weight buffer 312 is to accommodate at least (8×16)×2 weight data. Each weight data occupies 16 bits. Therefore, the bandwidth requirement caused by excitation is (2×16)×16 = 512 bits / clock cycle, and the bandwidth requirement caused by weight is (8×16 / 4 + the number of bytes of the strobe index)×16. Since the number of bytes of the strobe index can be ignored, the result is approximately equal to 512 bits / clock cycle. The reason for dividing by 4 is that the weight buffer 312 does not need to update the data once per clock cycle, and it can be updated once every 4 clock cycles, as ​ shown.

[0133] The disadvantage of the weight reuse sparse DNN inference scheme is that due to the ping-pong buffer design of the weight buffer, the timing requirements are relatively strict, and the register file bandwidth requirement is relatively high.

[0134] ​

[0135] The excitation reuse sparse DNN inference according to an embodiment of the present disclosure is performed by an arithmetic unit. The arithmetic unit may be, but is not limited to, the tensor engine 310. Hereinafter, the tensor engine 310 is taken as an example. ​ is a structural diagram of the tensor engine 310 for excitation reuse sparse DNN inference according to an embodiment of the present disclosure. As ​ shown, the tensor engine 310 for excitation reuse sparse DNN inference includes a weight buffer 312, an excitation buffer 311, an index gater 314, and a dot product arithmetic unit 313.

[0136] ​ is a slicing scheme diagram of the weight matrix and the excitation matrix of the tensor engine 310 for excitation reuse sparse DNN inference according to an embodiment of the present disclosure. Different from ​ the excitation matrix slicing in the weight reuse scheme, in the embodiment of the present disclosure, the sparse weight matrix is sliced, and every L rows in the sparse weight matrix are used as a slice, where L is ​ the number of the weight buffers 312 in ​ Each weight buffer 312 places a row vector sequentially taken from the sparse weight matrix. L weight buffers 312 can place L row vectors taken from the sparse weight matrix. Assume that the sparse weight matrix has M rows, and M is a positive integer multiple of L. Then the sparse weight matrix is divided into M / L slices, namely slice 0,..., M / L - 1. The pre-sparse weight matrix has K columns, and the sparsity coefficient is α. Then the sparse weight matrix has Kα columns. Therefore, ​ each of the L weight buffers 312 in ​ stores a 1×Kα row vector. ​ is a specific example of

[0137] The weight buffer 312 caches the data taken from ​The row vector of 1×Kα in the sparse post-weight matrix in the on-chip memory 360 is moved into the tensor engine 310. During the process of the processing unit 220 pre-configuring the deep learning network in the acceleration unit 230, the processing unit 220 sends the address of the sparse post-weight matrix in the memory 210 to the acceleration unit 230 in the form of an instruction. When the acceleration unit 230 performs actual deep learning network inference, according to these addresses, the sparse post-weight matrix is fetched from the corresponding position in the memory 210 and placed in the on-chip memory 360. When the tensor engine 310 needs weights during operation, L rows are sequentially fetched from the sparse post-weight matrix and placed in each of the L weight buffers 312, one row per buffer, for the dot product operation unit 313 to use these weights for calculation. In ​ In a specific example, the tensor engine 310 sequentially fetches 2 rows from a 16×8 sparse post-weight matrix and places them in weight buffers 0 and 1 respectively. Each weight buffer contains a 1×8 row vector.

[0138] The activation buffer 311 is used to cache the activation matrix moved from the on-chip memory 360 into the tensor engine 310. When the instruction for actual inference start is assigned by the command processor 237 to the core 236 for execution, the activation matrix in the instruction is stored in the on-chip memory 360. The activation matrix is a K×N activation matrix. When the tensor engine 310 starts to perform operations, the K×N activation matrix is moved into the activation buffer 311 for caching. Since the row vectors stored in the weight buffer 312 are 1×Kα and the activation matrix is K×N, they cannot be directly multiplied according to the rules of matrix multiplication. It is necessary to select Kα rows from the K rows of the K×N activation matrix cached in the activation buffer 0 according to the gating index 440 to form a gated activation matrix of Kα×N, so that the 1×Kα row vector can be multiplied by it. That is, not every row in the K×N activation matrix will participate in the matrix multiplication operation. Because some weights are pruned after the weight matrix is pruned, only the activation rows corresponding to the unpruned weights need to participate in the multiplication operation, and the activation rows corresponding to the pruned weights do not need to participate in the multiplication operation, thus reducing the computational load of the dot product operation unit 313. The weight matrix has M rows, and there are also M gating indexes 440, namely gating index 0, …… M-1. The index gater 314 stores the gating indexes corresponding to each row vector in the weight matrix and gates the rows of the activation matrix according to the gating indexes to obtain a gated activation matrix of Kα×N. For each row vector in the weight matrix, a different gated activation matrix of Kα×N is gated out, such as ​As shown. The dot product operation units 0, …… N-1 fetch a 1×Kα row vector of the sparse weight matrix from the weight buffer 312, and the corresponding index strobe 314 also strobes out the corresponding Kα×N strobed excitation matrix from the K×N excitation matrix cached in the excitation buffer 0, so that the 1×Kα row vector can be multiplied by the corresponding Kα×N strobed excitation matrix. The N dot product operation units 313 are respectively responsible for multiplying the 1×Kα row vector by one of the corresponding columns in the N columns of the Kα×N strobed excitation matrix, so as to obtain the multiplication result of the 1×Kα row vector and the entire Kα×N strobed excitation matrix after multiplying the N columns respectively. In ​ In a specific example, K = N = 16. For each 1×8 row vector stored in the weight buffer 312, the index strobe 314 strobes out 8 rows from the 16 rows of the 16×16 excitation matrix cached in the excitation buffer 0 according to the strobe index 440, becoming an 8×16 strobed excitation matrix. In this way, the 1×8 row vector stored in the weight buffer 312 can be multiplied by the 8×16 strobed excitation matrix.

[0139] Each index strobe 314 corresponds to a weight buffer 312, so there are also L index strobes 314. Each index strobe 314 stores the strobe index of the row vector in the corresponding weight buffer 312, and according to this strobe index, it strobes out the corresponding Kα×N strobed excitation matrix from the K×N excitation matrix cached in the excitation buffer 0. Each index strobe 314 can work in parallel to improve the operation efficiency. Since each weight buffer 312 requires N dot product operation units 313 to complete the corresponding multiplication operation, and there are L weight buffers 312 in total, there are L·N dot product operation units. In ​ In the specific example shown, there are 2 weight buffers 312, and there are also 2 corresponding index strobes 314. The dot product operation units 313 need to be 2×16 = 32.

[0140] ​ The excitation buffer 0 stores the current K×N excitation matrix used in the multiplication of the weight matrix and the current K×N excitation matrix, while the alternative excitation buffer 1 stores the next K×N excitation matrix of the current K×N excitation matrix. Once the dot product operation unit 313 completes the relevant multiplication operation of the current K×N excitation matrix, the current K×N excitation matrix is unloaded from the excitation buffer 0, and the next K×N excitation matrix stored in the excitation buffer 1 is loaded into the excitation buffer 0. In ​ In the specific example, the excitation buffer 0 stores the current 16×16 excitation matrix, and the excitation buffer 1 stores the next 16×16 excitation matrix.

[0141] ​ is shown ​Timing diagram of the operation of the structure of the tensor engine 310 shown. Since the post-sparse weight matrix is divided into M / L weight matrix slices, and each weight matrix slice needs to be multiplied by the excitation matrix to obtain the final multiplication result, the entire matrix multiplication is divided into M / L clock cycles. The index strobe 314 needs to load different strobe indices at each clock cycle 0, ……, M / L - 1 respectively, and it needs to load the corresponding strobe index at least a second predetermined number of clock cycles before the dot product operation unit 313 performs the operation on the weight matrix slice, so as to ensure that the dot product operation unit 313 has selected a Kα×N post-strobe excitation matrix from the excitation matrix according to the corresponding strobe index when it operates. Otherwise, the dot product operation unit 313 has to wait for the index strobe 314 to complete the strobe before it can start the operation. In one embodiment, the second predetermined number can be 1. The weight buffer 312 also needs to load different weight matrix slices 0, ……, M / L - 1 at each clock cycle 0, ……, M / L - 1 respectively, and it also needs to load the corresponding weight matrix slice at least a first predetermined number of clock cycles before the dot product operation unit 313 performs the operation on the weight matrix slice, so as to ensure that the dot product operation unit 313 has obtained the slice before operating on it. In one embodiment, the first predetermined number can be 1. The excitation buffer 311 does not need to load the excitation matrix every clock cycle because each row of the weight matrix needs to be multiplied by the same excitation matrix. The excitation buffer 311 can load the excitation matrix at one time, and then the dot product operation unit 313 uses the post-sparse excitation matrix in clock cycles 0, ……, M / L - 1. Therefore, it can be loaded in the third predetermined number of clock cycles before the index strobe 314 performs the strobe according to the strobe index and remains for M / L clock cycles, as ​ shown. In one embodiment, the third predetermined number is M / L + 1. In the specific example of 7C, since there are 8 post-sparse weight matrix slices, it is divided into 8 clock cycles. The index strobe 314 needs to load the corresponding strobe index in the previous clock cycle before the dot product operation unit 313 performs the operation on the weight matrix slice, so as to ensure that the dot product operation unit 313 has selected an 8×16 post-strobe excitation matrix from the excitation matrix according to the corresponding strobe index when it operates. The weight buffer 312 also needs to load the corresponding weight matrix slice in the previous clock cycle before the dot product operation unit 313 performs the operation on the weight matrix slice, so as to ensure that the dot product operation unit 313 has obtained the slice before operating on it. The excitation buffer 311 can be loaded 8 clock cycles before the index strobe 314 performs the strobe according to the strobe index and remains for 8 clock cycles, as ​ shown.

[0142] In ​In the example, the storage capacity requirement for the weight buffer 312 is to accommodate at least 2×8 weight data. The storage capacity requirement for the activation buffer 311 is to accommodate at least (16×16)×2 weight data. Each weight data occupies 16 bits. Therefore, the bandwidth requirement caused by the weights is (2×8 + number of strobe index bytes)×16. Since the number of strobe index bytes can be ignored, the result is approximately equal to 256 bits / cycle. The bandwidth requirement caused by the activation is (16×16 / 8)×2 = 512 bits / cycle. The reason for dividing by 8 is that the activation buffer 311 does not need to update data every cycle and can be updated once every 8 cycles, as ​ shown.

[0143] The bandwidth occupation caused by the activation matrix and the bandwidth occupation caused by the weight matrix in the three schemes in the case of M = N = K = 16 and α = 50% are listed above. ​ The table shows the bandwidth occupation of the weight reuse scheme and the activation reuse scheme of the present disclosure embodiment when M and N are increased respectively. The data in the second and third rows of the table are the bandwidth occupation of the weight reuse scheme when M is increased to 32 and N is increased to 32 respectively. The data in the fourth and fifth rows of the table are the bandwidth occupation of the activation reuse scheme of the present disclosure embodiment when M is increased to 32 and N is increased to 32 respectively. For the weight reuse scheme, when M is increased to 32, the sum of the bandwidth occupation caused by the activation, the bandwidth occupation of the weight engine, and the bandwidth occupation caused by the strobe index is 800 bits / cycle. Adding the bandwidth occupation of other aspects of 1024, the total bandwidth occupation is 1824; when N is increased to 32, the sum of the bandwidth occupation caused by the activation, the bandwidth occupation of the weight engine, and the bandwidth occupation caused by the strobe index is 656 bits / cycle. Adding the bandwidth occupation of other aspects of 1024, the total bandwidth occupation is 1680. For the activation reuse scheme of the present disclosure embodiment, when M is increased to 32, the sum of the bandwidth occupation caused by the activation, the bandwidth occupation of the weight engine, and the bandwidth occupation caused by the strobe index is 544 bits / cycle. Adding the bandwidth occupation of other aspects of 1024, the total bandwidth occupation is 1568; when N is increased to 32, the sum of the bandwidth occupation caused by the activation, the bandwidth occupation of the weight engine, and the bandwidth occupation caused by the strobe index is 800 bits / cycle. Adding the bandwidth occupation of other aspects of 1024, the total bandwidth occupation is 1824. Therefore, when M is increased, adopting the activation reuse scheme of the present disclosure embodiment can minimize the occupied register file bandwidth, and it does not have the ping-pong buffer design in the tensor engine based on weight reuse and is not subject to timing constraints.

[0144] Although it is mentioned in the above embodiments that the embodiments of the present disclosure do not require a ping-pong buffer design and are not subject to timing constraints, a ping-pong buffer design can also be adopted. After adopting the ping-pong buffer, instead of waiting for the operation of one excitation matrix to be completed before loading the next excitation matrix, the ping-pong buffer can be used to load both the current excitation matrix and the next excitation matrix. After the dot product operation of the current excitation matrix is completed, it is directly switched to the next excitation matrix for dot product multiplication, which is more efficient than the scheme of completing the operation of one excitation matrix and then loading the next excitation matrix. In this embodiment, the excitation buffer includes a first excitation buffer and a second excitation buffer (not shown), and these two excitation buffers are designed as ping-pong buffers. The first excitation buffer and the second excitation buffer are respectively used to store the current excitation matrix and the next excitation matrix. The first excitation buffer and the second excitation buffer are both connected to the index strobe 314. When the dot product operation unit 313 completes the multiplication of each row vector of the sparse post-weight matrix and the gated current excitation matrix, the index strobe 314 is switched to gate the second excitation buffer, and the dot product operation unit 313 starts to perform the multiplication of each row vector of the sparse post-weight matrix and the gated next excitation matrix.

[0145] ​

[0146] As ​ shown, according to an embodiment of the present disclosure, a method for running a deep neural network is further provided, including:

[0147] Step 610, taking out row vectors from the M×Kα sparse post-weight matrix and putting them into the weight buffer, where M and K are the number of rows and columns of the pre-sparse weight matrix respectively, and α is the sparse coefficient;

[0148] Step 620, putting the K×N excitation matrix into the excitation buffer;

[0149] Step 630, putting the gated index corresponding to the row vector into the index strobe;

[0150] Step 640, gating the excitation matrix according to the gated index through the index strobe to obtain a Kα×N gated excitation matrix;

[0151] Step 650, multiplying the row vector and the gated excitation matrix through the dot product operation unit.

[0152] Optionally, there are L weight buffers; step 610 includes: sequentially taking out L row vectors from the sparse post-weight matrix and putting them into the L weight buffers respectively.

[0153] Optionally, there are L index gaters; step 630 includes: putting the gating indices corresponding to the L row vectors into the L index gaters.

[0154] Optionally, there are L·N dot product operation units; step 650 includes: multiplying the L row vectors with the N column vectors of the gated excitation matrix respectively through the L·N dot product operation units.

[0155] Optionally, step 650 includes: setting M / L clock cycles for the multiplication of the weight matrix and the excitation matrix, where M is an integer multiple of L; in a single clock cycle, the row vectors in the weight buffer are multiplied with the N column vectors of the gated excitation matrix respectively through the N dot product operation units corresponding to each row vector.

[0156] Optionally, step 610 includes: putting the fetched row vectors into the weight buffer in the first predetermined number of clock cycles before the clock cycle when the dot product operation units perform multiplication operations on the row vectors.

[0157] Optionally, step 630 includes: putting the gating indices into the index gaters in the second predetermined number of clock cycles before the clock cycle when the dot product operation units perform multiplication operations on the gated excitation matrix gated out according to the gating indices.

[0158] Optionally, step 620 includes: putting the excitation matrix into the excitation buffer in the third predetermined number of clock cycles before the clock cycle when the dot product operation units perform multiplication operations on the first gated excitation matrix obtained by gating the excitation matrix, and keeping it for M / L clock cycles.

[0159] Since the implementation details of the above process have been described in detail in the description of the foregoing device embodiments, they will not be elaborated here.

[0160] ​

[0161] Verified by experiments, the embodiments of the present disclosure double the efficiency and reduce the bandwidth occupancy by 10%-40% when the tensor engine performs multiplication of the sparse post-weight matrix and the excitation matrix, greatly improving the performance of the tensor engine and having good market prospects.

[0162] It should be understood that the embodiments in this specification are all described in a progressive manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the method embodiments, since they are basically similar to the methods described in the device and system embodiments, the description is relatively simple, and for the relevant parts, reference can be made to the partial descriptions of other embodiments.

[0163] It should be understood that the above description has been made of specific embodiments of this specification. Other embodiments are within the scope of the claims. In some cases, the acts or steps recited in the claims may be performed in a different order than in the embodiments and still achieve the desired result. Additionally, the processes depicted in the figures need not be in the particular order shown or sequential order to achieve the desired result. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0164] It should be understood that an element described herein in the singular or shown only in one of the figures is not to be construed as limiting the quantity of that element to one. Additionally, a module or element described or shown herein as separate may be combined into a single module or element, and a module or element described or shown herein as a single one may be split into multiple modules or elements.

[0165] It should also be understood that the terminology and expressions employed herein are for the purpose of description only and that one or more embodiments of this specification should not be limited to these terminology and expressions. The use of these terminology and expressions does not exclude any equivalent features of the illustration and description (or parts thereof), and it should be recognized that various modifications that may exist should also be included within the scope of the claims. Other modifications, variations and substitutions may also exist. Accordingly, the claims should be regarded as covering all such equivalents.

Claims

1. An arithmetic unit, comprising: A weight buffer for storing row vectors taken from a sparse post-weight matrix of M×Kα, where M and K are the number of rows and columns of the pre-sparse weight matrix respectively, and α is a sparsity coefficient; An activation buffer for storing a K×N activation matrix; An index gater for storing gating indices corresponding to the row vectors and gating rows of the activation matrix according to the gating indices to obtain a gated post-activation matrix of Kα×N; A dot product arithmetic unit for multiplying the row vectors by the gated post-activation matrix; The activation buffer includes a first activation buffer and a second activation buffer that act as ping-pong buffers, which are respectively used to store the current activation matrix and the next activation matrix. The first activation buffer and the second activation buffer are both connected to the index gater. When the dot product arithmetic unit completes the multiplication of each row vector of the sparse post-weight matrix by the gated current activation matrix, the index gater switches to gate the second activation buffer.

2. The arithmetic unit according to claim 1, wherein There are L weight buffers, which are respectively used to store L row vectors sequentially taken from the sparse post-weight matrix; there are L index gaters, which are respectively used to store gating indices corresponding to the L row vectors; there are L·N dot product arithmetic units, which are respectively used to multiply the L row vectors by N column vectors of the gated post-activation matrix.

3. The arithmetic unit according to claim 2, wherein, L＝2。 4. The arithmetic unit according to claim 1, wherein, α＝50％。 5. The arithmetic unit according to claim 2, wherein, The dot product arithmetic unit sets M / L clock cycles for the multiplication of the weight matrix and the activation matrix, where M is an integer multiple of L; in a single clock cycle, the row vectors in the weight buffer are respectively multiplied by N column vectors of the gated post-activation matrix through N dot product arithmetic units corresponding to each row vector.

6. The arithmetic unit according to claim 2, wherein, The row vectors are loaded into the weight buffer in the first predetermined number of clock cycles before the clock cycle in which the dot product arithmetic unit performs the multiplication operation.

7. The arithmetic unit according to claim 6, wherein, The first predetermined number is 1.

8. The arithmetic unit according to claim 2, wherein, The gating indices are loaded into the index gater in the second predetermined number of clock cycles before the clock cycle in which the gated post-activation matrix gated according to the gating indices is multiplied by the dot product arithmetic unit.

9. The arithmetic unit according to claim 8, wherein The second predetermined number is 1.

10. The arithmetic unit according to claim 2, wherein, The activation matrix is loaded into the activation buffer in the third predetermined number of clock cycles before the clock cycle in which the first gated post-activation matrix gated from the activation matrix is multiplied by the dot product arithmetic unit, and is held for M / L clock cycles.

11. The arithmetic unit according to claim 10, wherein, The third predetermined number is M / L + 1.

12. The arithmetic unit according to claim 10, further comprising: An alternative activation buffer for storing the next activation matrix, and once the activation matrix is unloaded from the activation buffer, the next activation matrix stored in the alternative activation buffer is used to load the activation buffer.

13. An acceleration unit, comprising: An instruction cache for caching instructions to be executed; The arithmetic unit according to any one of claims 1-12; A sequencer for fetching the instruction to be executed from the instruction buffer and, if the instruction to be executed is a multiplication instruction of the sparse post-weight matrix and the excitation matrix, allocating the multiplication instruction to the arithmetic unit; On-chip memory for storing the sparse post-weight matrix and the excitation matrix respectively for loading by the weight buffer and the excitation buffer.

14. A computing device, comprising: A memory for storing instructions to be executed; A processing unit for loading the instruction to be executed and, if the instruction to be executed is an instruction executed by the acceleration unit, sending the instruction to be executed to the acceleration unit for execution; The acceleration unit according to claim 13.

15. A system-on-chip comprising the acceleration unit according to claim 13.

16. A data center comprising the computing device according to claim 14.

17. A method for running a deep neural network, comprising: Fetching row vectors from an M×Kα sparse post-weight matrix and putting them into a weight buffer, where M and K are respectively the number of rows and columns of the pre-sparse weight matrix, and α is a sparsity coefficient; Putting a K×N excitation matrix into the excitation buffer; Putting the gating index corresponding to the row vector into the index gater; Gating the excitation matrix according to the gating index through the index gater to obtain a gated excitation matrix of Kα×N; Multiplying the row vector by the gated excitation matrix through a dot product arithmetic unit; The excitation buffer includes a first excitation buffer and a second excitation buffer that serve as ping-pong buffers for storing the current excitation matrix and the next excitation matrix respectively, where both the first excitation buffer and the second excitation buffer are connected to the index gater. When the dot product arithmetic unit completes the multiplication of each row vector of the sparse post-weight matrix by the gated current excitation matrix, the index gater switches to gate the second excitation buffer.

18. The method according to claim 17, wherein, There are L weight buffers; fetching row vectors from an M×Kα sparse post-weight matrix and putting them into the weight buffer includes: sequentially fetching L row vectors from the sparse post-weight matrix and putting them into the L weight buffers respectively.

19. The method according to claim 18, wherein There are L index gaters; putting the gating index corresponding to the row vector into the index gater includes: putting the gating indexes corresponding to the L row vectors into the L index gaters.

20. The method according to claim 18, wherein There are L·N dot product arithmetic units. Multiplying the row vector by the gated excitation matrix through the dot product arithmetic unit includes: multiplying each of the L row vectors by the N column vectors of the gated excitation matrix through the L·N dot product arithmetic units respectively.

21. The method according to claim 20, wherein Multiplying the row vector by the gated excitation matrix includes: Setting M / L clock cycles for the multiplication of the weight matrix and the excitation matrix, where M is an integer multiple of L; in a single clock cycle, the row vectors in the weight buffer are respectively multiplied by the N column vectors of the gated excitation matrix through the N dot product arithmetic units corresponding to each row vector.

22. The method according to claim 21, wherein, Taking a row vector from the sparse post-weight matrix of M×Kα and putting it into the weight buffer includes: putting the taken row vector into the weight buffer in the first predetermined number of clock cycles before the clock cycle in which the dot product operation unit performs the multiplication operation.

23. The method according to claim 21, wherein Putting the strobe index corresponding to the row vector into the index strobe includes: putting the strobe index into the index strobe in the second predetermined number of clock cycles before the clock cycle in which the dot product operation unit performs the multiplication operation on the strobed post-excitation matrix strobed according to the strobe index.

24. The method according to claim 21, wherein, Putting the excitation matrix of K×N into the excitation buffer includes: Putting the excitation matrix into the excitation buffer in the third predetermined number of clock cycles before the clock cycle in which the dot product operation unit performs the multiplication operation on the first strobed post-excitation matrix obtained by strobbing the excitation matrix, and maintaining it for M / L clock cycles.

Images