Hardware accelerators for performing computational tasks

Abstract

The present disclosure relates to a hardware accelerator for performing a computational task consisting of a set of operations. The hardware accelerator includes a controller and a set of computational units. Each computational unit of the set of computational units is configured to receive input data for an operation of the set of operations and perform the operation, the input data being represented by a different bit length associated with each computational unit. The controller is configured to receive the input data represented by a particular bit length of the bit lengths and select one of the set of computational units that can provide a valid result, the selected computational unit being associated with a bit length less than or equal to the particular bit length.

Description

[Technical Field]

[0001] The present invention relates to the field of digital computer systems, and more particularly to hardware accelerators. [Background technology]

[0002] Hardware acceleration allows the use of specially crafted computer hardware to perform some functions more efficiently than is possible with software running on a general-purpose central processing unit (CPU). However, there is a demand for improving the logic utilization of hardware accelerators. Summary of the Invention

[0003] Various embodiments provide a hardware accelerator, a method for a hardware accelerator, and a computer program product for a hardware accelerator. The embodiments of the present invention may be freely combined if they are not mutually exclusive.

[0004] In one aspect, the present invention relates to a hardware accelerator for performing a computational task consisting of a set of operations, the hardware accelerator including a controller and a set of computational units, each computational unit of the set of computational units configured to receive input data for an operation of the set of operations and perform the operation, the input data being represented by a different bit length associated with each computational unit, the set of computational units being associated with the different set of bit lengths, the controller configured to receive input data represented by a particular bit length of the set of bit lengths and select a computational unit of the set of computational units that can provide a valid result, the selected computational unit being associated with a bit length less than or equal to the particular bit length, and an output of the selected computational unit being provided as a result of the operation.

[0005] In another aspect, the invention relates to a method for performing a computational task consisting of a set of operations, the method including: providing a hardware accelerator including a controller and a set of computation units; receiving input data for an operation of the set of operations and performing the operation at each computation unit of the set of computation units, the input data being represented by a different bit length associated with each computation unit; receiving, at the controller, the input data being represented by a particular bit length of the bit lengths associated with the set of computation units; selecting, by the controller, a computation unit of the set of computation units that can provide a valid result, the selected computation unit being associated with a bit length less than or equal to the particular bit length; and providing an output of the selected computation unit as a result of the operation.

[0006] In another aspect, the present invention relates to a computer program product including a computer readable storage medium having computer readable program code embodied therein, the computer readable program code configured to perform a method.

[0007] In the following, embodiments of the invention will be described in more detail, by way of example only, with reference to the drawings, in which: FIG. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram of a hardware acceleration system according to an embodiment of the present invention. [Figure 2] FIG. 1 is a block diagram of a hardware accelerator according to an embodiment of the present invention. [Figure 3] FIG. 1 is a block diagram of a hardware accelerator according to an embodiment of the present invention. [Figure 4] FIG. 2 is a block diagram of a controller according to an embodiment of the present invention. [Figure 5]FIG. 2 is a diagram illustrating an execution flow of a given data operation according to an embodiment of the present invention. [Figure 6] 1 is a flowchart illustrating a method for performing a computational task using a hardware accelerator according to an embodiment of the present invention. [Figure 7] 1 is a flowchart illustrating a method for creating a replication unit according to an embodiment of the present invention. [Figure 8] 1 is a flowchart of a method for executing a computational task consisting of a set of operations according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] The description of various embodiments of the present invention is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. It will be apparent to those skilled in the art that many modifications and variations are possible without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best explain the principles of the embodiments, practical applications or technical improvements to technology found in the market, or to enable those skilled in the art to understand the embodiments described herein.

[0010] The computational task may be any task consisting of operations that can be performed by a hardware accelerator. The set of operations may be basic calculations that can be performed by a hardware accelerator. Each operation in the set of operations may include another operation, such as a multiplication operation, an addition operation, or an Open Neural Network Exchange (ONNX) operation or a Neural Network Exchange Format (NNEF) operation. The computational task may be, for example, training or inference of a neural network. The set of operations may be tensor operations, such as block matrix operations or multidimensional convolutions.

[0011] A hardware accelerator may be configured to perform or implement at least a portion of a computational task, e.g., at least a portion of the computational task may be one operation in a series of operations. A hardware accelerator may be an integrated circuit configurable to perform data operations. A hardware accelerator may include computer hardware specially tailored to perform some functions more efficiently than is possible with software running on a general-purpose central processing unit (CPU). The computer hardware may include, for example, a graphics processing unit (GPU), a field programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). A computational unit may include, for example, a two-dimensional systolic array. A systolic array may include a number of cells organized into columns and rows. A systolic array may have more rows than columns, more columns than rows, or an equal number of columns and rows. A systolic array may be hardwired for specific operations, such as "multiply-accumulate" to perform massively parallel integration, convolution, correlation, matrix multiplication, or data sorting tasks. A computational unit may be configured to perform logic functions, such as multiply-accumulate. The computational units may be, for example, logic elements of an FPGA. The logic elements may include, for example, a three-input lookup table (3-LUT) and a D flip-flop. A bitstream file may be used to configure the FPGA to determine, for example, the logic functions that can be performed by the set of computational units.

[0012] The set of computation units may be configured to perform the same operation simultaneously. For example, the set of computation units may begin performing the operation simultaneously. The smaller the bit length representing the input data, the faster the operation based on the input data can be performed. A computation unit associated with a particular bit length may be referred to as a first computation unit. The particular bit length may, for example, be the largest bit length among the set of bit lengths. The selected computation unit may be a computation unit that can perform the data operation faster than the first computation and provide a valid result of the data operation. The valid result of the data operation may be a result of the data operation obtained from a valid representation of the input data. For example, the number 15 may be validly represented in a 5-bit or 4-bit representation, but not validly represented in a 3-bit representation, because the least significant 3 bits of a correct representation (e.g., 4-bit) of the value 15 may only represent a maximum value of 7.

[0013] A controller (referred to as a first controller) may be configured to receive input data for each operation in a set of operations and select a computation unit that can provide a valid result for the operation. In another example, a hardware accelerator may include a second controller configured to receive input data for a subset of the set of operations, while the first controller is configured to receive input data for the remaining subset of the set of operations. Like the first controller, the second controller is configured to receive input data represented by a particular bit length in a set of bit lengths and select a computation unit in the set of computation units that can provide a valid result for the operation associated with or assigned to the second controller.

[0014] The present invention can speed up calculations performed by a hardware accelerator. To this end, the present invention includes precision estimation to provide results more quickly if the precision of the partial products allows without compromising accuracy. Precision estimation means that the computation unit reads only the number of least significant bits (LSBs) of the input data and therefore infers that the ignored most significant bits (MSBs) are 0. Thus, the present invention can provide parallel processing units with different precision capabilities. The selected computation unit can have a lower number of bits, which means a smaller memory footprint, a more efficient arithmetic unit, lower latency, and higher memory bandwidth. The advantage of using a lower precision format can be found in the efficiency of multiply-and-accumulate operations, for example, in deep learning inference or training. Therefore, the hardware accelerator can enable an inference system that is competitive with fast and efficient matrix multipliers.

[0015] According to one embodiment, input data is received simultaneously at a set of computational units and at a controller. This embodiment causes the computational units to begin execution simultaneously (speculatively). In parallel, the controller can determine or select which of the computational units can provide a result faster, if the precision of the partial products allows it without loss of accuracy.

[0016] The input data may include, for example, operands of a data operation. If the data operation is a multiplication operation, the operands of the input data may be two operands. The controller may be configured to select a computation unit, for example, by calculating the maximum number of leading zeros present in both operands of the input data. The calculated number of leading zeros indicates the maximum bit length that can be used to quickly perform the operation without losing precision. For example, if the calculated number of leading zeros is L and the maximum bit length handled by the controller is n bits, the selected computation unit may be associated with a bit length k bits, where k≧nL and k≦n. This may be advantageous because it may be implemented using a simplified design of a comparator (e.g., only logic gates may be used in the controller). Therefore, the area and latency overhead of the controller may be small. The controller may be able to quickly determine which computation unit can produce a valid result without overflowing.

[0017] According to one embodiment, the controller is configured to stop the execution of operations by the non-selected compute units. The controller may be configured to generate control signals or commands to the set of compute units and other units of the hardware accelerator. For example, the set of compute units includes N compute units, where N≧2. The controller is configured to stop the execution of operations by the non-selected N−1 compute units. For example, the controller may make a decision to forward the result of the selected compute unit, forcing all remaining speculative executions to cancel their calculations.

[0018] According to one embodiment, the set of computing units includes a first computing unit associated with the largest bit length in the set of bit lengths, and each computing unit in the set of computing units different from the first computing unit is a replica unit of the first computing unit. Replicating a computing unit refers to determining one or more copies of the first computing unit. The replica units are copies of the first computing unit.

[0019] The first computation unit may be associated with a bit length of n bits, and each of the remaining N-1 computation units may be a different bit length of Nj bits, where j has a value ranging from 1 to n-1.

[0020] According to one embodiment, the maximum bit length in the set of bit lengths is n bits, and each calculation unit of the set of calculation units associated with a bit length k bits less than n bits is configured to read k least significant bits (LSBs) of the received input data. This embodiment can enable each of the calculation units to be provided with input from the same loaded data. That is, instead of converting all parameter values ​​from high precision to low precision, the same input data can be loaded and further used as input to the set of calculation units. This may allow the precision to be inferred by assuming that the ignored (unread) MSB bits are 0.

[0021] According to one embodiment, the hardware accelerator comprises an FPGA, a GPU, an ASIC, a neuromorphic device, or a bit-addressable device.

[0022] An FPGA is an integrated circuit designed to be configurable by a customer or designer after manufacture. The configuration of an FPGA can typically be specified using a hardware description language (HDL). An FPGA may contain an array of programmable logic blocks and a hierarchy of reconfigurable interconnects for wiring the blocks. The logic blocks can be configured to perform complex combinatorial functions. Each compute unit in a set of compute units may, for example, contain a subarray of the array of programmable logic blocks.

[0023] In one example, an FPGA may be provided, and the FPGA may include one computational unit (referred to as an initial or first computational unit) associated with a maximum bit length (e.g., 8 bits). For example, the FPGA may be configured by an initial streaming file that enables the first computational unit of the FPGA. Furthermore, one or more copies of the first computational unit may be created. The one or more copies may be replication units. This may be performed, for example, by creating a bitstream file to configure the FPGA to enable the replication units in addition to the first computational unit. In a first example, the created bitstream file may enable partial configuration of the FPGA to provide the FPGA with multiple computational units, i.e., the first computational unit and its copies. That is, the bitstream file may enable reconfiguration of only a sub-region of the FPGA that covers the replication units. The FPGA may further be configured to include a controller. The bitstream file may be created automatically, for example.

[0024] According to one embodiment, the set of compute units includes a minimum number of compute units such that the logic utilization of the FPGA is higher than a predefined threshold. For example, the logic utilization of the FPGA may be higher than 90%. Thus, the silicon area of ​​the FPGA may be optimally utilized.

[0025] According to one embodiment, the number of computation units in the set of computation units is the number of bits of the largest bit length in the set of bit lengths. For example, if the largest bit length is 8 bits, the set of computation units includes 8 computation units. In one example, the computation units in the set are associated with bit lengths of 8 bits, 7 bits, 6 bits, 5 bits, 4 bits, 3 bits, 2 bits, and 1 bit, respectively. In this case, the step size of the controller is 1. This may be advantageous because it covers all possible representations. In another example, the controller may have a step size of 2 so that the set computation units are associated with bit lengths of 8 bits, 6 bits, 4 bits, and 2 bits. This may be advantageous because it can reduce the area occupied by the controller. In one example, the step size of the controller may be defined by a user. In another example, the step size may be determined by profiling input data and based on their statistical distribution, the step size of the controller may be determined. For example, if 80% of the computations involve numbers in the format XXX0000X, the step size may be defined accordingly.

[0026] According to one embodiment, the computational task is one of deep neural network training, deep neural network inference, matrix vector multiplication, and matrix matrix multiplication.

[0027] According to one embodiment, input data for the operation includes two operands for multiplication. The controller includes a cascade of logic gates for determining a maximum number of leading zeros present in the operands of the input data, the maximum number of leading zeros being indicative of the bit length of the selected computation unit. The maximum number of leading zeros may be associated with a bit length greater than or equal to the bit length of the selected computation unit. This embodiment may provide a simplified design of the controller (e.g., only logic gates may be used) and may reduce the area and latency overhead of the controller.

[0028] For example, the set of calculation units includes a calculation unit associated with 8 bits and a calculation unit associated with 3 bits and 5 bits. If the selected bits are 2 bits, a calculation unit associated with 3 bits or 5 bits may be selected, preferably the smallest of the two. If the selected bits are 4 bits, a 5-bit calculation unit is selected because it is the only calculation unit whose bit length is greater than 4 bits and less than 8 bits.

[0029] According to one embodiment, the controller and the set of computational units are configured to operate in a pipeline order, which may enable high throughput data processing.

[0030] According to one embodiment, the selected computation unit is associated with a minimum bit length that is less than or equal to a particular bit length.

[0031] A computation unit has a bit length associated with it which means that the computation unit is capable of performing operations using input data operands or values ​​represented by said bit length.

[0032] According to one embodiment, the particular bit length is the largest bit length in a set of bit lengths.

[0033] 1 is a block diagram of a hardware acceleration system 100 according to an embodiment of the present invention. The hardware acceleration system 100 may be a heterogeneous processing system. For example, the hardware acceleration system 100 may include different types of processing devices, such as a CPU, a GPU, an FPGA, an ASIC, etc.

[0034] 1, for ease of explanation, a simplified configuration is shown in which a hardware accelerator 102 is coupled to a host computer 101 to speed up data processing compared to when data is processed by software. However, the configuration is not limited to this, and other configurations of the hardware acceleration system 100 may also be used. For example, a configuration including multiple hardware accelerators coupled to a host computer may be used, or a configuration including only a hardware accelerator may be used.

[0035] The hardware accelerator 102 may be connected to the host computer 101 via a PCI Express bus or an Ethernet connection. The host computer 101 may be configured to perform initial boot procedures, basic I / O functions, etc. The host computer 101 may include a host processor 103, such as a CPU. The host processor 103 may be configured to access data stored on an external disk 108 and data stored in a local host cache memory. The host processor 103 may also communicate data via a network interface 109. The hardware accelerator 102 has an accelerator architecture optimized to accelerate the execution of specific classes of computing functions with respect to the host computer architecture. Such accelerated computing functions include, for example, vector processing and floating-point operations.

[0036] The hardware acceleration system 100 may be configured to implement applications such as inference of a trained neural network, for example, the hardware acceleration system 100 may be an FPGA-based neural network accelerator system.

[0037] The host computer 101 and the hardware accelerator 102 are adapted to communicate data with each other. This data communication may occur via a connection such as a PCIe bus or an Ethernet connection. In another embodiment, the hardware accelerator 102 may be part of the host computer 101. The hardware accelerator 102 and the host processor 103 may share the same package or the same die. In this case, the communication link between the hardware accelerator 102 and the host processor 103 may be any commonly used in-package or on-chip communication bus (e.g., AXI, Wishbone, etc.). The hardware accelerator 102 may read input data from global memory and perform computations. The input data may be received via a network interface as a stream of network phits (the network interface may stream in and out fixed-size data). The output may be written back to global memory or sent via the network interface as a stream of network phits.

[0038] For example, the hardware accelerator 102 may include an FPGA. The host processor 103 and the FPGA may be configured to execute or implement a computational task. The host processor 103 may be coupled to a memory 105 storing a compiled software application 107. The compiled software application 107 includes function calls. The FPGA coupled to the hardware processor may include compiled user functions. The compiled user functions may be executable in response to one of the function calls. The function calls may include messages of variables that need to be processed by the user functions. The hardware accelerator may be coupled to memory, such as an off-chip memory 111 and an on-chip memory 112 for storing intermediate computation results of the hardware accelerator.

[0039] The hardware accelerator 102 may be configured to perform a computational task consisting of a set of operations. For example, the hardware accelerator includes a controller and a set of computational units, the set of computational units including at least a first computational unit and a second computational unit. The first computational unit is configured to receive input data of an operation of the set of operations and perform the operation, the input data being represented by a first bit length. The second computational unit is configured to receive the input data represented by a second bit length and perform the operation. The controller is configured to receive input data represented by a maximum bit length among the first and second bit lengths and select one of the set of computational units that can first provide a valid result, and the output of the selected computational unit is provided as the result of the operation. In one example, the first bit length is n bits, the second bit length is k bits, where k is less than n, and the second computational unit is configured to read k least significant bits (LSBs) of the received input data.

[0040] FIG. 2 is a block diagram of a hardware accelerator 202, such as an FPGA, according to one embodiment of the present invention.

[0041] The hardware accelerator 202 may be configured to perform a computational task. The hardware accelerator 202 may be configured to process tensors and may therefore be referred to as a tensor coprocessor. A tensor may be a multidimensional array having a uniform type. A zeroth-order tensor may be a scalar. An individual value of a tensor may be a zeroth-order tensor. A first-order tensor is a vector. The computational task may include a set of operations 201A-201D, which may be, for example, tensor accelerator functions. Each operation in the set of operations may include, for example, one or more ONNX operators or one or more NNEF operators. Each operation in the set of operations 201A-201D may be performed based on input data. For example, if the operation is a multiplication operation, the input data may be two operands of the multiplication. The hardware accelerator 202 may be configured to perform the set of operations 201A-201D using multiple precisions. As shown in FIG. 2, hardware accelerator 202 may be configured to perform a set of operations 201A-201D (e.g., tensor accelerator functions #1-4) using different fixed numbers of bits to represent input data. For example, two different numbers of bits may be used to represent input data, as indicated by precision A and precision B in FIG. 2. For example, precision A may refer to an 8-bit representation, and precision B may refer to a 3-bit representation. To that end, hardware accelerator 202 may include computation units 210A-210D for performing operations 201A-201D, respectively, using precision A, and replica units 212A-212D for performing operations 201A-201D, respectively, using precision B. Replicated units 212A-212D may each replicate computation units 210A-210D. 0 A~21 0 D. Thus, for each operation (eg 201A of the set of operations 201A-201D), there is a respective pair of units comprising a computation unit 210A and a replica unit 212A that can perform said operation.

[0042] Each of the computation units 210A-210D and replication units 212A-212D is configured to receive input data for the associated operation 201A-201D and to perform the operation, where the input data is represented in a distinct bit length (or bit width) associated with that unit. For example, computation unit 210A may be configured to receive input data in precision A of the associated operation 201A and to perform operation 201A. Replication unit 212D may be configured to receive input data in precision B of the associated operation 201D and to perform operation 201D. In this manner, the replication units may use words having a lower bit width to represent values.

[0043] Each pair of units may be connected to a controller 215 (also referred to as a speculative precision controller (SPC)). For each pair of units, the controller 215 is configured to receive input data represented by the largest bit length and select one of the units of the pair that can first provide a valid result, and the output of the selected unit is provided as the result of the operation. For example, for a pair of units including a calculation unit 210A and a replication unit 212A, the controller 215 is configured to receive input data of the operation 201A, where the received input data has a precision A corresponding to the largest bit length of 8 bits. The controller 215 may select one of the two units 210A and 212A of the pair that can first / early provide a valid result. The replication unit 212A uses data with lower precision and can therefore execute the operation 201A faster than the calculation unit 210A. Therefore, the controller 215 may determine whether the result provided by the replication unit 212A is a valid result. To do so, the controller may, for example, calculate the number of leading zeros on the input data operands of data operation 201A (e.g., as described with reference to FIG. 4). As soon as the controller makes its decision, the results of each unit are forwarded and the remaining speculative execution may be forced to cancel the calculation. Hardware accelerator 202 further includes on-chip memory 214 (e.g., tensor on-chip memory) for storing intermediate processing results and / or for providing each computation unit with the data it needs on each cycle.

[0044] The choice of the number of replication units and the precision they should represent can be an optimization problem that trades off efficiency and silicon area and can depend on the problem data, so a general optimizer can be employed (the optimizer can be guided by simulation and data profiling).

[0045] 3 is a block diagram of a hardware accelerator 302 according to one embodiment of the present invention. The hardware accelerator 302 may be configured to compute a matrix-matrix multiplication A*B of matrices A and B to obtain an output matrix C. The matrix-matrix multiplication A*B may include a set of operations, where each operation in the set of operations may require two operands of matrices A and B, respectively.

[0046] The hardware accelerator 302 includes a set of computational units 310.1-310.8. Each of the computational units 310.1-310.8 may be, for example, a General Matrix Multiply (GEMM) core of an FPGA. Each of the eight computational units 310.1-310.8 is associated with a respective bit length. Computational unit 310.1 is associated with a bit length of 1 bit. Computational unit 310.2 is associated with a bit length of 2 bits. Computational unit 310.3 is associated with a bit length of 3 bits. Computational unit 310.4 is associated with a bit length of 4 bits. Computational unit 310.5 is associated with a bit length of 5 bits. Computational unit 310.6 is associated with a bit length of 6 bits. Computational unit 310.7 is associated with a bit length of 7 bits. Computational unit 310.8 is associated with a bit length of 8 bits. The expression "a computation unit U is associated with a bit length lb" means that the computation unit U is configured to receive or process input data represented by the bit length lb.

[0047] Each of the computation units 310.1-310.8 is connected to a pair of input registers that store two input operands according to the bit length of the computation unit. Computation unit 310.8 is connected to a pair of 8-bit input registers (referred to as tensor registers 8-bit). Computation unit 310.7 is connected to a pair of 7-bit input registers (referred to as tensor registers 7-bit). Computation unit 310.6 is connected to a pair of 6-bit input registers (referred to as tensor registers 6-bit). Computation unit 310.5 is connected to a pair of 5-bit input registers (referred to as tensor registers 5-bit). Computation unit 310.4 is connected to a pair of 4-bit input registers (referred to as tensor registers 4-bit). Computation unit 310.3 is connected to a pair of 3-bit input registers (referred to as tensor registers 3-bit). Computation unit 310.2 is connected to a pair of 2-bit input registers (referred to as tensor registers 2-bit). The computation unit 310.1 is connected to a pair of 1-bit input registers (called tensor register 1-bit).

[0048] Each of the computation units 310.1-310.8 is configured to perform an operation (e.g., multiplication) on two operands stored in a pair of input registers to which the computation unit is connected, and to output the result of the operation to an output register to which the computation unit is connected. Computation unit 310.8 is connected to a 16-bit output register (referred to as a tensor register 16-bit). Computation unit 310.7 is connected to a 14-bit output register (referred to as a tensor register 14-bit). Computation unit 310.6 is connected to a 12-bit output register (referred to as a tensor register 12-bit). Computation unit 310.5 is connected to a 10-bit output register (referred to as a tensor register 10-bit). Computation unit 310.4 is connected to an 8-bit output register (referred to as a tensor register 8-bit). Computation unit 310.3 is connected to a 6-bit output register (referred to as a tensor register 6-bit). The calculation unit 310.2 is connected to a 4-bit output register (called tensor register 4 bit), and the calculation unit 310.1 is connected to a 1-bit output register (called tensor register 1 bit).

[0049] Hardware accelerator 302 further includes a controller 315 and a selector 317. Controller 315, also referred to as a speculative-precision controller, is coupled to the set of computation units 310.1-310.N and to selector 317. Selector 317 is configured to receive data in an output register. Although shown as separate components, controller 315 and selector 317 may form an integrated part in alternative embodiments.

[0050] Input data for each operation in the set of operations may be obtained from matrices A and B using direct memory access (DMA), which allows the input data to be loaded into DMA loads 321 and 322. For example, the two operands of each operation may be loaded into two DMA loads 321 and 322, respectively, before being stored in a pair of input registers having the largest bit length and provided as inputs to controller 315. As shown in FIG. 3, the two operands of a given operation (e.g., OP1) may be represented in 8-bit representation and initially stored in a pair of input 8-bit registers. In one example, the input registers of the remaining computation units 310.1 through 310.7 may have the same size as the number of bits of their associated bit lengths; for example, computation unit 310.7 may have a 7-bit wide register, etc. The 7 LSB bits of the two operands stored in the pair of input 8-bit registers are transferred to the pair of input 7-bit registers. The 6 LSB bits of the two operands stored in the pair of input 7-bit registers are transferred to the pair of input 6-bit registers. The 5 LSB bits of the two operands stored in the pair of 5-bit input registers are transferred to the pair of 5-bit input registers. The 4 LSB bits of the two operands stored in the pair of 5-bit input registers are transferred to the pair of 4-bit input registers. The 3 LSB bits of the two operands stored in the pair of 4-bit input registers are transferred to the pair of 3-bit input registers. The 2 LSB bits of the two operands stored in the pair of 3-bit input registers are transferred to the pair of 2-bit input registers. The 1 LSB bit of the two operands stored in the pair of 2-bit input registers is transferred to the pair of 1-bit input registers.

[0051] Each of computation units 310.1 through 310.8 may perform a given operation OP1 using two operands stored in a respective pair of input registers. Controller 315 may select one of computation units 310.1 through 310.N as the computation unit that can provide a valid result of operation OP1 the fastest. The selected result may be provided by controller 315 as an input to selector 317. Selector 317 may use the input to select one of the results received from the output registers. The result of operation OP1 may be provided by selector 317 to DMA store 323 for storage as part of result matrix C. In another embodiment, the input registers of computation units 310.1 through 310.8 may have the same size (e.g., 8 bits), and the input register of each of the remaining computation units 310.1 through 310.7 is associated with logic, such as shown by component 330, that selects the least significant bit associated with the computation unit. Component 330 describes, for example, the input registers of computation unit 310.6.

[0052] 4 is a block diagram of a controller 415 according to one embodiment of the present invention. The controller 415 is configured to receive two input operands 421 and 422 of a matrix-matrix multiplication A*B operation. The two operands are represented in 8-bit representation. For example, the first operand 421 includes 8 bits ordered from b0 to b7, where b0 is the LSB of the 8 bits. The second operand 422 includes 8 bits ordered from c0 to c7, where c0 is the LSB of the 8 bits.

[0053] Each pair of bits (b7,c7), (b6,c6), (b5,c5), (b4,c4), (b3,c3), (b2,c2) and (b1,c1) of the pair is 5may be input to each of the OR gates 401.1 to 401.7. The output of the pair of OR gates 401.7 and 401.6 may be connected to OR gate 403.6. The output of the pair of OR gates 403.6 and 401.5 may be connected to OR gate 403.5. The output of the pair of OR gates 403.5 and 401.4 may be connected to OR gate 403.4. The output of the pair of OR gates 403.4 and 401.3 may be connected to OR gate 403.3. The output of the pair of OR gates 403.3 and 401.2 may be connected to OR gate 403.2. The output of the pair of OR gates 403.2 and 401.1 may be connected to OR gate 403.1.

[0054] Each of OR gates 401.7 and 403.1-403.6 may be connected to a respective NOT gate 405.1-405.7. The output of each of NOT gates 405.1-405.7 may represent one of a 1-bit, 2-bit, ..., 7-bit representation, respectively. An 8-bit representation is associated with the output of NOT gate 405.8, which receives a value "1" as an input. Controller 415 further includes selector 417 configured to receive the outputs of NOT gates 405.1-405.8 and parse them from MSB to LSB to select the final signal / output having a value "1." For example, if the selected signal corresponds to a 3-bit representation, this indicates that controller 415 may select a calculation unit associated with the 3-bit representation to provide the calculation result.

[0055] 5 is a diagram illustrating the execution flow of the GIB data operation according to an embodiment of the present invention. In this example, the set of calculation units includes an 8-bit GEMM and a 3-bit GEMM.

[0056] The diagram in FIG. 5 illustrates data paths associated with components involved in computing a data operation. Data path 501 indicates the time when input operands are ready at DMA loads 321 and 322. As shown by data paths 502 and 504, respectively, the 8-bit GEMM and the 3-bit GEMM begin data operations at the same time, t0. Data paths 503 and 505 indicate the time when the 8-bit GEMM and the 3-bit GEMM are ready, respectively. Data path 506 indicates that selector 317 selects the 3-bit GEMM at time t1, before either the 8-bit GEMM or the 3-bit GEMM is ready. Thus, controller 315 may become ready at the same time the selected 3-bit GEMM becomes ready. This is shown by data path 507. The result of the data operation may be provided as an output of controller 315 at time t2 and shown on data path 508. As shown in FIG. 5, the present invention may be able to obtain the time difference between time t2 and time t3, when the 8-bit GEMM becomes ready.

[0057] FIG. 6 is a flowchart of a method for performing a computational task using a hardware accelerator according to one embodiment of the present invention. The computational task may be neural network inference. The hardware accelerator may be an FPGA tensor accelerator. The hardware accelerator may include a first computation unit configured to perform operations of the computational task in full precision (e.g., 8 bits). For illustrative purposes, the method may be implemented in the hardware acceleration system 100 illustrated above in FIGS. 1-2, but is not limited to this implementation.

[0058] In step 601, a neural network application may be provided. For example, the architecture and parameters of the neural network may be provided. The parameters may be the weights, biases, and input / output data types of the neural network. In step 602, code for the neural network application may be received. The code may be in a programming language such as Tensorflow, Keras, or PyTorch. In step 603, the received code may be adapted to be executed by a hardware accelerator. For example, the code may be reduced to an intermediate representation (IR) level that can be executed by the hardware accelerator. In step 604, replication units may be created from the first computation unit according to different precisions. In step 605, the parameters of the neural network may be transferred to the replication unit so that processing of the application can be initiated on the first computation unit and the replication unit. In step 606, it may be determined whether the controller has found a replication unit that can provide a computation result. If it is determined in step 606 that the controller has not found a replication unit that can provide a computation result, the result of the first computation unit may be used in step 608. If it is determined in step 606 that the controller has found a replication unit that can provide the computation result, then in step 607 the result of the found replication unit may be used.

[0059] Figure 7 is a flowchart of a method for creating a replication unit according to one embodiment of the present invention. The method of Figure 7 may be performed, for example, by an automatic generator 700 using an FPGA 710. The FPGA 710 may include a controller and a first computation unit capable of performing data operations with full precision (e.g., 8 bits).

[0060] A data operation (e.g., of a neural network) may be defined in step 701. The data operation may be, for example, a tensor accelerator function. In step 703, an empirical model may be created. The empirical model may define how FPGA resources in step 704 for performing the data operation can scale with different precisions, e.g., how FPGA resources required to perform the data operation using a 6-bit representation can be obtained from FPGA resources used to perform the data operation using an 8-bit representation. As shown in FIG. 7, step 703 may be optional because the FPGA resources provided in step 704 may be sufficient to generate a bitstream file. The profiles of the neural network and input data set provided in step 705 may be used to determine a speculative granularity in step 706. The speculative granularity may indicate bit representations that can be used to perform the data operation (in addition to the full 8-bit precision); e.g., 2-bit and 6-bit representations may be determined in step 706. The automatic generator 700 may receive as input the data operations, empirical models, speculative granularity, and FPGA resources to generate a bitstream file capable of creating replication units in the FPGA 710 according to the speculative granularity. The bitstream file may configure the FPGA 710 to perform the data operations according to the present invention. As shown in FIG. 7, different tools may be used to execute the method of FIG. 7. For example, steps 700, 703, and 705 may be implemented in a software tool such as Python. The FPGA resources and speculative granularity may be described in a constraint file such as a JSON file. The tensor accelerator function of step 701 may be implemented in a high-level language (e.g., C, C++, SystemC, OpenCL, Chisel, Python, etc.), an HDL language (e.g., VHDL, Verilog, etc.), or any form of semiconductor intellectual property core (e.g., softcore, hardcore, encrypted netlist, etc.).

[0061] FIG. 8 is a flowchart of a method for executing a computational task consisting of a set of operations according to one embodiment of the present invention. The computational task may be decomposed into a set of operations, such that a result of the set of operations represents a result of the computational task. In step 801, a hardware accelerator may be provided, including a controller and a set of computation units. Each computation unit of the set of computation units may receive input data for an operation of the set of operations and initiate the operation to be executed in step 803, the input data being expressed in a different bit length associated with each computation unit. In step 805, the controller may receive input data expressed in a specific bit length of the bit lengths. The specific bit length of the bit lengths may be, for example, a maximum bit length among the bit lengths. In step 807, the controller may select a computation unit of the set of computation units associated with a bit length less than or equal to the specific bit length that can provide a valid result. The controller and each of the set of computation units may receive the input data simultaneously and may initiate their respective execution simultaneously. Steps 803-807 may be performed simultaneously. The output of the selected computation unit may be provided in step 809 as a result of the operation. Steps 803-809 may be repeated for each operation in the set of operations, for example, the controller may or may not select the same computation unit in different iterations.

[0062] The input data of each operation in the set of operations may be of integer type, float type, or hybrid float type. In the case of float type input, the hardware accelerator may further include a unit configured to perform float-to-integer casting of the input data before input to the calculation unit (which is an integer type unit). In the case of hybrid float type, each calculation unit in the set of calculation units may include an integer element that processes the mantissa of the input operand and an x-bit exponent element that provides an exponent that can be shared between the input operands. This allows the operation to be performed quickly with integer arithmetic, and the final result to share the exponent of the input operand. In this way, the user processes I / O data in floating point, but the hardware is operating with integers.

[0063] The input data of each operation in the set of operations may be represented by a number of bits, Nb. The number of bits representing the input data of the first operation to be executed in the set of operations may be N1. If Nb for each operation in the remaining operations in the set of operations is less than or equal to N1, the selected computation unit of the first operation may be used to execute the remaining operations in the set of operations; e.g., the method (except for steps 805 and 807) may be repeated for each further operation in the set of operations using the selected computation unit. If Nb for any operation in the set of operations may be greater than N1, the method may be repeated so that a computation unit is selected for execution of each operation.

[0064] The present invention may be a system, method, or computer program product, or combination thereof, integrated at any possible level of technical detail. The computer program product may include a computer-readable storage medium having stored thereon computer-readable program instructions for causing a processor to carry out aspects of the present invention.

[0065] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. The computer-readable storage medium may be, by way of example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or a suitable combination thereof. More specific examples of computer-readable storage media include portable computer diskettes, hard disks, RAM, ROM, EPROM (or flash memory), SRAM, CD-ROMs, DVDs, memory sticks, floppy disks, mechanically encoded devices having instructions recorded on punch cards or ridge-in-groove structures, or the like, and suitable combinations thereof. Computer-readable storage devices, as used herein, should not be construed as ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.

[0066] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device via a network (e.g., the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof). The network may be comprised of copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface of each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on a computer-readable storage medium within the respective computing / processing device.

[0067] Computer-readable program instructions for carrying out operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or object-oriented programming languages ​​such as Smalltalk, C++, etc., and procedural programming languages ​​such as the "C" programming language and similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, as a standalone software package, or partially on the user's computer. Alternatively, the computer may be executed partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the computer-readable program instructions in order to carry out aspects of the present invention.

[0068] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0069] These computer-readable program instructions can be provided to a processor of a computer or other programmable data processing apparatus to create a machine, such that the instructions, when executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions can also be stored in a computer-readable storage medium connectable to a computer, programmable data processing apparatus, or other device, or combination thereof, that functions in a particular way, such that the computer-readable program instructions stored therein form one of a product comprising instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0070] Computer-readable program instructions, such as instructions to perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams on a computer, other programmable apparatus, or other device, can also be loaded into a computer, other programmable data processing apparatus, or other device to perform a series of operational steps on the computer, other programmable apparatus, or other device to produce a computer-implemented process.

[0071] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of executable implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, which constitute one or more executable instructions for implementing the specified logical function(s). In some alternative embodiments, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may actually be accomplished as a single step, executed concurrently, substantially concurrently, partially, or fully in a time-overlapping manner, or the blocks may be executed in the reverse order depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, can be implemented by a special-purpose hardware-based system that performs the specified functions or operations or executes a combination of special-purpose hardware and computer instructions.

Claims

1. 1. A hardware accelerator for performing a computational task comprising a set of operations, said hardware accelerator comprising: a controller; and a set of computing units, each computing unit of the set of computing units configured to receive input data for an operation of the set of operations and to perform the operation, the input data being represented by a different bit length associated with each computing unit, the set of computing units being associated with a set of bit lengths; the controller is configured to receive the input data represented by a particular bit length of the set of bit lengths and select a computing unit of the set of computing units associated with a bit length less than or equal to the particular bit length to provide a valid result, the set of computing units including a replica unit of the selected computing unit configured to receive input data represented by the second bit length different from the particular bit length of the input data to perform the same operation on input data represented by the second bit length, the replica unit being provided as a result of the operation; Hardware accelerator.

2. The hardware accelerator of claim 1 , wherein the controller is configured to stop non-selected compute units from performing the operations.

3. The hardware accelerator of claim 1 , wherein the input data is received simultaneously at the set of compute units and the controller.

4. The hardware accelerator of claim 1 , wherein the selected computation unit is associated with a minimum bit length that is less than or equal to the particular bit length.

5. The hardware accelerator of claim 1 , wherein the particular bit length is a maximum bit length in the set of bit lengths.

6. 2. The hardware accelerator of claim 1, wherein the set of computing units includes a first computing unit associated with the particular bit length, and each computing unit in the set of computing units different from the first computing unit is a replica unit of the first computing unit.

7. 2. The hardware accelerator of claim 1, wherein the particular bit length of the set of bit lengths is n bits, and each computation unit of the set of computation units associated with a bit length k bits is configured to read k least significant bits (LSBs) of the received input data, where k bits is less than n bits.

8. 10. The hardware accelerator of claim 1, wherein the hardware accelerator is selected from the group consisting of a field programmable gate array (FPGA), a graphics processing unit (GPU), and an application specific integrated circuit (ASIC).

9. 9. The hardware accelerator of claim 8, wherein the set of compute units includes a minimum number of compute units such that logic utilization of the FPGA is above a predefined threshold.

10. The hardware accelerator of claim 1 , wherein the number of computational units in the set of computational units is the number of bits of the largest bit length in the set of bit lengths.

11. 10. The hardware accelerator of claim 1, wherein the computational task comprises one of deep neural network training, deep neural network inference, matrix vector multiplication, and matrix matrix multiplication.

12. the input data of the operation includes two operands, and the controller includes a logic gate that determines a maximum number of leading zeros present in the operands of the input data, the maximum number of leading zeros indicating a bit length of the selected computation unit. The hardware accelerator of claim 1 .

13. 1. A method for performing a computational task comprising a set of operations, the method comprising: providing a hardware accelerator including a controller and a set of computational units; receiving, at each computing unit of the set of computing units, input data for operations of the set of operations and initiating the operations, the input data being represented by different bit lengths associated with each computing unit, the set of computing units being associated with a set of bit lengths; receiving, at the controller, the input data represented by a particular bit length of the set of bit lengths; selecting, by the controller, a computing unit of the set of computing units to provide a valid result, the selected computing unit being associated with a bit length less than or equal to the particular bit length, the set of computing units including a replica unit of the selected computing unit configured to receive input data represented in a second bit length different from the particular bit length of the input data to perform the same operation on the input data represented in the second bit length; providing an output of the selected computation unit as a result of the operation; A method comprising:

14. Providing the hardware accelerator comprises: providing the hardware accelerator including the controller and a first calculation unit configured to receive input data for the operation and to perform the operation, the input data being represented by a bit length of n bits; creating one or more replica units of the first computation unit, each of the replica units configured to receive the input data of the operation and perform the operation, the input data being represented with a bit length k bits, where k is less than n, k<n; the set of computing units including the first computing unit and the created replica unit; 14. The method of claim 13, comprising:

15. The hardware accelerator includes a field programmable gate array (FPGA), and creating each of the replicated units includes: generating a bitstream file; programming the FPGA according to the bitstream file so that a portion of the FPGA is configured as the replication unit; 15. The method of claim 14, comprising:

16. 16. The method of claim 15, which is performed automatically.

17. 14. The method of claim 13, further comprising: for each operation of the set of operations, repeating: receiving input data for an operation in the set of operations; selecting a computation unit of the set of computation units for providing the valid result; and providing an output of the selected computation unit.

18. One or more non-transitory computer-readable storage media, wherein program instructions stored on the one or more computer-readable storage media, when executed by a hardware accelerator including a controller and a set of computing units, cause the hardware accelerator to: causing each computation of the set of computation units to receive input data for the computation of the set of operations and initiate the computation, the input data being represented by a different bit length associated with each computation unit, the set of computation units being associated with a set of bit lengths; receiving, at the controller, the input data represented by a particular bit length of the set of bit lengths; causing the controller to select a computing unit of the set of computing units to provide a valid result, the selected computing unit being associated with a bit length less than or equal to the particular bit length, the set of computing units including a replica unit of the selected computing unit configured to receive input data represented in a second bit length different from the particular bit length of the input data to perform the same operation on the input data represented in the second bit length; causing an output of the selected computing unit to be provided as a result of the operation; One or more non-transitory computer-readable storage media.

Images