Logarithmic notation processor for inner product calculation

Abstract

A method and apparatus are described that uses a multidimensional logarithmic notation for hardware acceleration of dot product calculations. These methods and apparatus can be used in any device requiring a low-power, low-area, and high-speed dot product calculation unit, such as for training and inference calculations of deep neural networks on edge devices. In a particular embodiment, neural network training is performed using a multidimensional logarithmic data representation to obtain a set of weight coefficients for the neural network. Given the determined weight coefficients, a second base is optimized for the multidimensional logarithmic data representation. This optimized representation can be used to perform inference using the neural network.

Description

[Technical Field]

[0001] This disclosure generally relates to methods and apparatus for data representation to improve the computational efficiency of artificial intelligence (AI) accelerators.

[0002] (Reference to related applications) This application claims priority to U.S. Patent Application No. 63 / 109136, filed November 3, 2020, entitled “Multi-Dimensional Logarithmic Number System Processor for Inner Product Computations.” This application claims the benefit under Section 119 of U.S. Patent Application No. 63 / 109136, filed November 3, 2020, entitled “Multi-Dimensional Logarithmic Number System Processor for Inner Product Computations,” which is incorporated herein by reference in its entirety for any purpose. [Background technology]

[0003] Deep neural networks, such as convolutional neural networks (CNNs), can be used for image and video recognition and classification, as well as other artificial intelligence (AI) applications such as recommendation engines, natural language processing, and medical image analysis. The neural networks used in these applications are experiencing increasing computational complexity and higher power consumption required for training and inference. Running neural networks on mobile or embedded platforms, in particular, presents a significant challenge due to hardware and power constraints. Edge devices (devices that enable local devices or networks to connect to the edge of the internet, interfacing with consumer or commercial products (e.g., robots, drones, surveillance equipment, augmented reality (AR) products, virtual reality (VR) products, autonomous vehicles, smartphones, wearable devices, etc.)) are constrained by their size and available power. Therefore, solutions are needed to enable more efficient operation of neural networks on such edge devices. Some of these efforts focus on improving the efficiency of hardware design. Others aim to increase the efficiency of machine learning models. However, given the increasing computational complexity of neural networks, improvements in hardware design and modeling alone may not provide a sufficient solution.

[0004] Therefore, there is growing interest in data representation methods to improve the computational efficiency of operations performed by neural networks. For example, low-precision computation and / or compression can be used when it is practical to accept some loss of precision in exchange for improved efficiency. However, some low-precision computation methods provide only small improvements in computational efficiency and / or lead to poor or even invalid results in training and inference. There is a need for data representation methods and devices that can be used to improve the computational efficiency, such as the inner product calculations used in convolutional neural networks, while achieving acceptable output. [Overview of the project]

[0005] Generally, this specification describes methods and apparatus that incorporate the use of data representations based on multidimensional logarithmic notation to accelerate dot product calculations in neural networks such as convolutional neural networks (CNNs) in hardware.

[0006] One aspect of the present invention provides a method for training and inferring a deep neural network. The method includes the steps of: receiving a set of training data; representing the set of training data in a multidimensional logarithmic number system (MDLNS), wherein the MDLNS representation uses a first exponent associated with a first base and a second exponent associated with a second base; training a deep neural network on the set of training data using a default first base and a default second base to determine a set of weight coefficients for the neural network; optimizing a second base for the multidimensional logarithmic data representation with respect to a default first base based on the determined set of weight coefficients for the neural network; and performing deep neural network inference on a network input set using the optimized multidimensional logarithmic data representation to obtain a network output set.

[0007] In some embodiments, optimizing a second base for a multidimensional logarithmic data representation involves determining the optimal second base that minimizes the mean squared error (MSE). A mixed-integer global optimization procedure may be performed to optimize the possible range of the second base and its associated second exponent.

[0008] In some embodiments, the default first base is 2. In some embodiments, the default second base is 2. ωand ω = (1 + sqrt(5)) / 2. MDLNS may optionally use one or more additional exponents (e.g., a third exponent, a fourth exponent, etc.), and these exponents are each associated with one or more corresponding additional bases (e.g., a third base, a fourth base, etc.). In some embodiments, performing training of a deep neural network on a set of training data may include using a default third base, and the default second base is

Number

Number

[0009] In some embodiments, the exponents of the bases are integer values. In some embodiments, the first exponent and the second exponent have opposite polarities. In some embodiments, the first exponent and the second exponent are decimal values. In some embodiments, the default second base is

Number

[0010] Another aspect of the present invention provides a hardware accelerator that can be employed on an edge device to perform the method described herein. The hardware accelerator includes a multidimensional logarithmic notation (MDLNS) converter connected to the memory of the computing device and the cache of the hardware accelerator, a plurality of processing units arranged in an array of rows of first numbers and columns of second numbers, which collectively form a processing core, and a microcontroller connected to the processing core and the MDLNS converter. The MDLNS converter may be configured to create an MDLNS representation of a dataset received from the memory of the computing device and to store the MDLNS representation in the cache of the hardware accelerator. The MDLNS representation is binary bottom The first index and non-binary related to bottom A second index related to the same may be used.

[0011] In some embodiments, the hardware accelerator processing unit includes a first adder operating at the binary base and a second adder operating at the non-binary base. The processing unit may optionally include an aggregation adder connected to the first and second adders. The aggregation adder has multiple aggregation channels, each aggregation channel corresponding to a unique combination of pairs (N,M) determined by the number of bits in the first exponent and the number of bits in the second exponent. The aggregation adder may optionally have two additional adders operating in parallel to aggregate the unique (N,M) pairs of exponents. N+M It may also include individual up counters.

[0012] In some embodiments, the processing units of the processing core are configured as a systolic array of matrix-vector multiplication units. In some embodiments, the second base is 2 ωTherefore, ω = (1 + sqrt(5)) / 2. In some embodiments, the hardware accelerator includes multiple processing tiles connected to other processing tiles by a chip-on network. Each of the processing tiles may include multiple processing cores as described above.

[0013] The hardware accelerators described herein are used in computing devices such as edge computing devices and can perform deep neural network inference by incorporating the use of logarithmic data representation to improve computational efficiency and reduce power consumption.

[0014] Further aspects of the present invention will become apparent from the following description.

[0015] Features and advantages of embodiments of the present invention will become apparent from the following detailed description, which will be illustrated with reference to the accompanying drawings. [Brief explanation of the drawing]

[0016] [Figure 1] Figure 1 shows a geometric interpretation of a multidimensional logarithmic representation. [Figure 2] Figure 2 schematically illustrates an exemplary embodiment of an edge computing device that can be used to run deep neural networks using hardware accelerators to improve computational efficiency. [Figure 3] Figure 3 is a block diagram of an exemplary embodiment of the hardware accelerator shown in Figure 2. [Figure 3A] Figure 3A schematically shows the processing unit of the hardware accelerator of Figure 3 according to an exemplary embodiment. [Figure 3B] Figure 3B schematically shows an exemplary aggregate adder that may be included in some embodiments of the processing unit shown in Figure 3A. [Figure 4]Figure 4 shows a method according to one embodiment that incorporates a multidimensional logarithmic data representation for training and inference of a deep neural network. [Modes for carrying out the invention]

[0017] The following description and embodiments are provided for illustrative purposes to illustrate specific embodiments of the principles of the present invention. These examples are provided for illustrative purposes only, and not to limit, these principles and the present invention.

[0018] This specification describes methods and apparatus that incorporate the use of data representations based on multidimensional logarithmic notation to accelerate dot product calculations in neural networks, such as convolutional neural networks (CNNs), in hardware. Applications of these methods and apparatus include computation for training and inference of neural networks. However, any device requiring a low-power, low-area, and high-speed dot product calculation unit can benefit from the methods and apparatus described herein. Embodiments of the present invention can be incorporated into accelerators that can be used for computer vision, artificial intelligence (AI) applications, image compression, speech recognition, machine learning, or other applications in edge devices (e.g., robots, drones, surveillance equipment, augmented reality (AR) products, virtual reality (VR) products, autonomous vehicles, smartphones, wearable devices, etc.).

[0019] The classical one-dimensional logarithmic number system (LNS) has various applications in low-power, low-precision digital signal and image processing. LNS is used in the field of digital filters (finite impulse response (FIR), infinite impulse response (HR), adaptive filters, etc.) and in signal transformations. A mechanical version of LNS is the well-known slide rule.

[0020] LNS can be summarized as follows: LNS converts multiplication and division into addition and subtraction. Addition and subtraction are performed by a lookup table (LUT) and additional addition. A significant drawback is the size of the LUT, which tends to increase exponentially depending on the dynamic range of the calculation. As a result, the use of LNS is generally limited to applications requiring low precision (e.g., 8-16 bit dynamic range). Because the multiplier is removed from the overall inner product architecture, low-power implementations are generally achieved, which is a desirable feature for mobile applications.

[0021] Low-precision computation is advantageous for accelerating calculations related to applications such as machine learning (e.g., deep learning, continuous learning with or without weights updated during training and after deployment, reinforcement learning, etc.), artificial intelligence, 3D imaging, and AR / VR simulation. Exploring logarithmic representation as an alternative to standard floating-point representation has yielded promising results in these applications, including significant power consumption reductions. In contrast, the use of floating-point representation significantly slows down calculations and increases power consumption because it provides an unnecessarily large dynamic range for calculations in such applications.

[0022] The multidimensional logarithmic number system (MDLNS) can be seen as an extension of LNS to two dimensions. In MDLNS, a real number x is given by x = s * D1 a *D2 b It can be encoded as follows: where (D1,D2) are a pair of multiplicatively independent bases (which may be real or complex) and (a,b) are a pair of integers. Also, if x is positive, s=1, and if x is negative, s=-1. A simple geometric interpretation of this representation is a two-dimensional slide rule 10, as shown in Figure 1.

[0023] Table 1 below outlines some of the main differences between LNS and MDLNS.

Table 1

[0024] Note that when the second base D2 is selected as the numerical value 1, LNS is a partial case of MDLNNS. In applications where LNS provides attractive practical performance, MDLNNS can be adapted to provide the same advantages as LNS.

[0025] As can be seen in Table 1, MDLNNS contains features not similar to LNS. Therefore, MDLNNS can be utilized by more efficiently using such functions in order to provide computational advantages over LNS. For example, MDLNNS exponentially speeds up the conversion from logarithmic form to binary form. In LNS, the conversion from logarithmic form to binary form is achieved by using a large LUT or by using a dedicated circuit implementing the function f(x)=2 x . In MDLNNS, when the powers of the second base D2 are stored in floating-point format for all possible values of the exponent exp (e.g., D2 exp = 1.ddddd*2 eeeee ), the conversion can be performed quite fast.

[0026] One of the differences between MDLNNS and classical LNS (or floating-point arithmetic) is the existence of a non-trivial approximation of 1 (the multiplicative identity), e.g., a number of the form 2 exp1 * D2 exp2 that is very close to 1. The following example shows how these approximations of 1 can be advantageously used to prevent computational overflow.

[0027] In one example, the bases of a particular MDLNNS are D1 = 2 and D2 = 3. In this example, good approximations of 1 include numbers such as: 2 8 * 3 -5 , 2 19 * 3 -12 , 2 84 * 3 -53And so on. For example, 2 a *3 b (Here, a 2 +b 2 The logarithm (where >0, a and b are integers) can be arbitrarily approached to 1 because 2 and 3 are multiplicatively independent (i.e., log23 is an irrational number), provided there is no restriction on the bit size of the pair of exponents (a, b).

[0028] As another example, let x = (180, -115) 2 This is calculated using 9-bit fixed-point binary arithmetic. The real value of x is approximately 0.207231. 2 Direct calculation of (360, -230) in 9-bit signed fixed-point arithmetic results in an overflow error. However, the excellent approximation of 1 provided by MDLNS offers an optimization option that mitigates the overflow problem. Notably, this optimization option is not analogous to either one-dimensional logarithmic notation (1DLNS) or floating-point binary arithmetic. If x is multiplied by a number encoded as (-84, 53) in two-dimensional logarithmic notation (2DLNS) (i.e., a number very close to 1), the error associated with this scaling becomes very small, and the size of the exponent can be reduced to (96, -62). This allows the squaring operation to be safely performed within the dynamic range of 9-bit fixed-point, and the final answer becomes (192, -164). Exemplarily, by reducing the size of the numbers used to within the overflow-free range, a vast number of computational options can be provided.

[0029] Standard calculation procedures, such as standard multiplication, can also be performed with 2DLNS. For example, using 2DLNS with bases D1=2 and D2=2.0228, we can multiply 41 by 109. With this base choice, 41 is encoded as (-17,22) and 109 as (21,-14). Adding the exponents component by component gives the pair (4,8). To obtain the number encoded with this pair, we raise D2 to a power (i.e., 1. ddddd*2 eeeeeA small LUT can be used that includes (encoded as shown above). In this example, D2 8 =1.0001100001....*2 8 Multiplying by 24 gives 1.0001100001...*212, which corresponds to 4485 in decimal. The correct product when multiplying 41 by 109 is 4469 in decimal.

[0030] Aspects of the present invention relate to a system and method for providing a multidimensional logarithmic data representation for performing calculations using MDLNNS such as the 2DLNS described above. Exemplarily, using a multidimensional logarithmic representation of data can improve the computational efficiency of applications such as image compression, image recognition or speech recognition, and machine learning performed by edge computing devices.

[0031] Figure 2 shows a computing device 12 according to an exemplary embodiment. The computing device 12 may be a computing device operating at the edge of the internet. For the purposes of this specification, the computing device 12 may be referred to as an “edge computing device.” However, it should be understood that other types of computing devices are interchangeable with the “edge computing device” within the scope of the present invention.

[0032] As shown in Figure 2, the edge computing device 12 includes a processor 14, a network interface 16 (e.g., supporting Ethernet and / or Wi-Fi connectivity to the Internet), memory 18, and one or more hardware accelerators 20 that can be adapted to perform application-specific computations. The edge computing device 12 may be connected to the Internet via the cloud or cloud server 100. As shown in Figure 2, various local devices 2 or local networks of interest (e.g., commercial robots or drones, military robots or drones, on-orbit data centers, satellites, surveillance equipment, augmented reality (AR) products, virtual reality (VR) products, personal computing devices such as smartphones, wearable devices, autonomous vehicles, or any other wireless electronic devices) may be connected to the edge computing device 12.

[0033] The edge computing device 12 is typically positioned relatively close to the local device 2 to reduce latency associated with data transmission between the local device 2 and the edge computing device 12. For example, the edge computing device 12 may be installed on a wind turbine to receive and process data collected from the wind turbine's local sensor 2. In another example, the edge computing device 12 may be installed on a traffic light to receive and process data transmitted from an autonomous vehicle 2. In some embodiments, the edge computing device 12 is physically located on and / or forms part of the local device 2 (i.e., the local device 2 may include the edge computing device 12).

[0034] The edge computing device 12 may, in some cases, be required to run deep neural networks. For example, the edge computing device 12 may employ deep neural networks to run AI applications such as image or video recognition, speech recognition, recommendation engines, natural language processing, and medical image analysis. In such cases, the edge computing device 12 may be configured to assign computational tasks related to running the neural network to one or more of its hardware accelerators 20. In some embodiments, the edge computing device 12 includes hardware accelerators 20 that are programmable or otherwise custom-designed to perform matrix-vector multiplication and / or dot product calculations. Although not required, the hardware accelerators 20 typically incorporate a tile-based architecture. Exemplaryly, the hardware accelerators 20 can perform such calculations in a way that is computationally more efficient than using a conventional central processing unit (CPU) or graphical processing unit (GPU).

[0035] Figure 3 is a block diagram of an exemplary embodiment of the hardware accelerator 20. The hardware accelerator 20 includes a plurality of processing elements 22 arranged in a parallel architecture. The processing elements 22 may be arranged in an array of any suitable dimension (e.g., 2, 3, etc.). For example, the processing elements 22 may be arranged in a two-dimensional array having m rows and n columns, as shown in Figure 3. The number of rows m can be any suitable number, including any number in the range from 2 to 16 (e.g., 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, or 16). The number of columns n can be any suitable number, including any number in the range from 2 to 16 (e.g., 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, or 16). In some embodiments, the processing elements 22 are arranged in a two-dimensional array having the same number of m rows and n columns (i.e., the processing elements 22 may be arranged in a square array).

[0036] In this specification, for the sake of ease of explanation, an assembly of processing elements 22 arranged in an array configuration (as described above, for example) may be referred to as a processing core 40. In the exemplary embodiment of Figure 3, a single processing core 40 of the hardware accelerator is shown for illustrative purposes and for brevity. Generally, the hardware accelerator 20 may include any appropriate number of processing cores 40. Each processing core 40 may have the same or a different number of processing elements 22 as the other processing cores 40 of the hardware accelerator 20.

[0037] In this specification, for ease of explanation, an assembly of processing cores 40 may be referred to as a processing tile. Each processing tile includes a suitable number of processing cores 40, for example, any number in the range of 2 to 16 (e.g., 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, or 16). For example, in one exemplary embodiment, a single processing tile of the hardware accelerator 20 may include eight processing cores 40, and each processing core 40 may include 64 processing elements 22 arranged in an 8x8 square array. The hardware accelerator 20 may include any suitable number of processing tiles depending on the processing power required by the edge computing device 12. In many applications, the edge computing device 12 includes a hardware accelerator 20 having 64 to 512 processing tiles.

[0038] The processing core 40 and the processing units 22 contained therein are controlled by one or more microcontrollers 24 of the hardware accelerator 20. The microcontrollers 24 may be implemented using one or more of the following: specially designed hardware, configurable hardware, a programmable data processor configured by having software or firmware executable on a data processor, and a specific-purpose data processor specifically programmed, configured, or built to control the processing units 22 in accordance with the methods described herein.

[0039] In some embodiments, the microcontroller 24 is a reduced instruction set computer (RISC) microcontroller. In such embodiments, the microcontroller 24 may include one or more of data memory, instruction memory, program counter, registers, control circuits, and input / output devices.

[0040] In some embodiments, each processing core 40 is controlled by its own microcontroller 24. In other embodiments, a single microcontroller 24 of the hardware accelerator 20 controls two or more processing cores 40. For example, all processing cores 40 forming a processing tile of the hardware accelerator 20 may be controlled by a single microcontroller 24.

[0041] The microcontroller 24 communicates with the processing unit 22 and the data memory 30 of the hardware accelerator 20 to execute computation tasks assigned to the hardware accelerator 20 (for example, tasks assigned by the central processor 14 of the edge computing device 12). For example, the microcontroller 24 may be configured to provide load instructions to load data stored in the memory 30 into the processing unit 22. The load instructions may be executed in a clock cycle defined by the local clock 26 of the hardware accelerator 20. Once the data is loaded into the processing unit 22, arithmetic instructions (for example, addition, subtraction, multiplication, division) provided by the microcontroller 24 may be executed on the data loaded into the processing unit 22 in the next clock cycle.

[0042] Unlike conventional computer architectures (such as the von Neumann architecture) that require the output data of the processing unit to be stored in memory immediately after the arithmetic operations are performed, the architecture of the processing core 40 and the processing unit 22 contained within it allows a series of arithmetic operations to be performed before the final data is output to and stored in memory.

[0043] In the example shown in Figure 3, data can be loaded from memory 30 to the first processing unit 22A of the processing core 40 during the first clock cycle of the local clock 26. The first processing unit 22A can then perform a first arithmetic operation on the loaded data during the second clock cycle of the local clock 26. After performing the first arithmetic operation, the output data of the processing unit 22A is transferred directly to the second processing unit 22B of the processing core 40, instead of returning to memory 30. The second processing unit 22B can then immediately perform a second arithmetic operation on the transferred data during the next (i.e., third) clock cycle of the local clock 26, without needing to load the output data of the first processing unit 22A from memory 30. This process can be repeated until the last processing unit 22C of the processing core 40 performs the final arithmetic operation on the data transferred thereto and stores the final output data of the processing core 40 back into memory 30. This reduces the number of load and store operations required by the processing core 40, thereby improving the computational efficiency of the processing core 40 compared to conventional processors.

[0044] In some embodiments, the memory 30 is implemented using static random access memory (SRAM) or other suitable storage technology to facilitate simultaneous load and store operations. That is, the memory 30 may be implemented using storage technology that allows one or more data sets to be loaded into one or more processing units 22 in the same clock cycle as one or more other data sets (i.e., from one or more other processing units) are stored in the memory 30. For example, the memory 30 may be implemented using 8T SRAM. Optionally, the memory 30 may be pitch-matched to the operating speed of the processing units 22.

[0045] Exemplary, the architecture of the hardware accelerator 20 is globally asynchronous but locally synchronous, so that each processing core 40 can operate independently of the others. In embodiments where each processing core 40 includes its own local clock 26, each processing core 40 can be accelerated or decelerated as needed by, for example, a microcontroller 24. In other embodiments, a processing tile may include a single clock 26 that synchronizes the processing of the processing cores 40 contained therein. The architecture of the hardware accelerator 20 avoids the need for a global clock tree, which can consume a large amount of dynamic energy and a large area.

[0046] In some embodiments, different processing tiles of the hardware accelerator 20 are connected to each other by a network-on-chip (NoC) 50. The NoC 50 may be dataflow reconfigurable to increase flexibility while keeping the power consumption of the hardware accelerator relatively low.

[0047] In some embodiments, the processing unit 22 is a multiplication unit designed or configured to perform multiplication operations on input data. In the example shown in Figure 3, each processing unit 22 may be configured to multiply a first data received via a first input by a second data received via a second input and output the product to two or more processing units 22 located downstream of the processing unit 22 pipeline (or memory 30 if the processing unit is the last processing unit 22C in the pipeline). Exemplary, the multiplication operation may be performed on data that represents numbers in MDLNNS to improve the computational efficiency associated with performing the multiplication operation by utilizing the tile-based architecture of the hardware accelerator 20. For example, the hardware accelerator 20 may include, or can otherwise interface with, an MDLNNS converter 40 that converts data that represents numbers in binary (e.g., corresponding to numbers stored in the main memory 18 of the edge computing device 12) to data that represents the same numbers in MDLNNS. The MDLNS converter 40 can also convert data that represents numbers in MDLNS back to data that represents numbers in binary.

[0048] Figure 3A is a schematic diagram illustrating an exemplary embodiment of the processing unit 22. In the example shown in Figure 3A, the processing unit 22 processes a first numerical value (x=s) expressed in MDLNS. x *D1 ax *D2 bx ) and the second numerical value expressed in the same MDLNS (y=s y *D1 ay *D2 byThe MDLNS converter is configured or designed to multiply a number by a number. As shown in Figure 3A, the MDLNS converter 40 converts the binary data representation of a number stored in the main memory 18 of the edge computing device 12 into an MDLNS data representation of the same number (e.g., "x", "y"). The MDLNS numbers are then stored in the cache 30 of the hardware accelerator 20. The processing unit 22 can access two or more MDLNS numbers (e.g., "x", "y") and operate (e.g., via the microcontroller 24) to perform multiplication on two or more MDLNS numbers.

[0049] In the exemplary embodiment shown in Figure 3A, the processing unit 22 includes a first base adder 62 and a second base adder 64. The first base adder 62 is configured to add the exponent of the first base D1 of the first number (i.e., “ax”) to the exponent of the first base D1 of the second number (i.e., “ay”). The second base adder 64 is configured to add the exponent of the second base D2 of the first number (i.e., “bx”) to the exponent of the second base D2 of the second number (i.e., “by”). Exemplarily, the bases D1 and D2 may be any suitable numbers and may be optimized according to the specific application of the hardware accelerator 20. The bases D1 and D2 may be optimized, for example, using the methods described in detail below. The adders 62 and 64 can be configured according to the optimized bases D1 and D2. For example, the first adder 62 may be a binary adder when D1=2. As another example, the second adder 64 may be a ternary adder when D2=3.

[0050] As shown in Figure 3A, the first adder 62 generates the sum of the exponent of the first base D1 of the first number and the exponent of the first base D1 of the second number (i.e., "ax + ay"). The second adder 64 generates the sum of the exponent of the second base D2 of the first number and the exponent of the second base D2 of the second number (i.e., "bx + by"). As mentioned above, the product of x and y is in MDLNS D1 a1+a2 *D2 b1+b2This can be expressed as follows. By supplying the output 66 of the first adder 62 and the output 68 of the second adder 64 to the MDLNS converter 40, the product of x and y can be converted back into a binary data representation.

[0051] In some cases, it is desirable to aggregate the product by multiplying a large set of numbers together. Numerous multiplication operations may be necessary, for example, when performing dot product calculations, matrix multiplication, and / or types of calculations commonly found in machine learning and AI applications. In such cases, the processing unit 22 may include an aggregate adder 70 configured to add up a large number of MDNLS numbers and calculate their sum.

[0052] Figure 3B schematically shows an exemplary embodiment of a processing unit 22 including an aggregate adder 70. In the example shown in Figure 3B, the accelerator cache 30 stores a large amount of data representing MDNLS numbers. For each number represented in MDNLS, the data may include N bits corresponding to the exponent of a first base D1 and M bits corresponding to the exponent of a second base D2. In these types of data, the exponent of D2 is 2 M There are 2 possible different values, each of which is derived from the first base D1. N It has a number of possible multiplicative factors.

[0053] The aggregate adder 70 is a 2-pair (ordered pair of 2 components) of (N,M) N+M Each of the different combinations includes a separate and dedicated aggregation channel. In some embodiments, the aggregation adder 70 includes 2 N+MThe system includes n parallel up counters, each up counter configured to aggregate unique (N,M) combinations of values. Each up counter may be a simple digital counter containing multiple D-flops. The D-flops may be connected such that, for each up counter, the clock input of the D-flop at position n (i.e., F(n)) is connected to the output of the D-flop at position (n-1), where n = 0, 1, ..., U, and U is the number of bits in the counter. In some embodiments, the clock of the first D-flop at n = 0 is a master clock (e.g., clock 26 of the processing core 40) operating at a desired clock speed "f" determined by the architecture of the hardware accelerator 20.

[0054] The output of the upcounter is a channeled partial sum and must be converted (e.g., by the MDLNS converter 40) to a number system (e.g., fixed-point) that is recognized by the edge computing device 12. In situations where P MDNLS numbers must be summed, the final output of the processing unit 22 is calculated only after P clock cycles of clock 26. The upcounter value is D1 every P cycles. M *D2 N Therefore, it is scaled. The upcounter value may, in some cases, then be summed by a fixed-point adder of the hardware accelerator 20. The fixed-point adder may, in some cases, be implemented as part of the MDLNS converter 40.

[0055] Exemplary, a high-precision fixed-point addition and final reconstruction step (FRS), implemented by a fixed-point adder to map the upconverter value to fixed-point, can be performed at a reduced rate (e.g., a rate of f / P Hz). For a typical machine learning structure such as a convolutional neural network (CNN), P can be a number in the range of 10,000 to 100,000 or more. For such applications, the FRS step can be implemented in software, for example, using a suitable embedded processor core of the edge computing device 12. In some embodiments, the hardware accelerator 20 includes a set of progressive downsampling integrators configured to convert the MDLNS upcounter value to fixed-point via the FRS. The set of progressive downsampling integrators can be operated in conjunction with the processing unit 22 to determine an appropriate trade-off between the speed, power, and chip area of ​​the hardware accelerator 20.

[0056] In some embodiments, the aggregate adder 70 includes, in addition to or instead of, some or all of the up counters, a fixed-point adder having a barrel shifter. Such a fixed-point adder may be, for example, 2 N It may also be implemented to compute the number of terms, thereby reducing the number of aggregate channels to 2 N+M From 2 M This reduces it to [a certain level].

[0057] In some embodiments, the processing core 40 and the processing units 22 contained therein are configured as systolic array matrix-vector multiplication units connected to a single accumulator. In such embodiments, each processing unit 22 is or functions as an MDLNS multiplication unit capable of operating to compute partial results of dot product calculations and / or matrix multiplications. Exemplarily, by configuring the processing units 22 as a systolic array, an ordered data flow can be provided, and / or the processing core 40 can be enabled to utilize properties such as weight stationary and / or output stationary to increase the efficiency and / or throughput of the hardware accelerator 20.

[0058] In some embodiments, the hardware accelerator 20 includes a nonlinearity and reduction unit for handling activation functions and / or pooling. An activation function is a nonlinear function in a neural network, such as a convolutional network (CNN), that determines how the result of matrix multiplication (i.e., a weighted sum) is translated into an output. Examples of activation functions include, but are not limited to, the ReLU activation function, the Leaky ReLU activation function, the sigmoid activation function, the soft plus activation function, or other differentiable nonlinear functions. The nonlinearity and reduction unit may be designed or configured to apply an appropriate activation function to the results of matrix multiplications performed by the processing core 40 (e.g., “AX+b” (matrix-matrix), “Ax+b” (matrix-vector)).

[0059] The following describes how a hardware accelerator 20 can be used to improve the computational efficiency of deep neural network operations, with reference to Figure 4.

[0060] Figure 4 shows a method 200 according to one embodiment in which the above-described concepts are incorporated into the data representation used to perform training and inference of a deep neural network. Method 200 may also be implemented using one or more of the above-described hardware accelerators 20 to utilize their tile-based architecture, and these hardware accelerators 20 may be designed or configured to perform some of the computations required to train the deep neural network in an efficient manner.

[0061] Method 200 begins in block 202 by accepting a set of data inputs (training data) 201 and representing the set of data inputs 201 in an MDLNS with a first base and a second base. After representing the set of data in an MDLNS, block 202 proceeds by training a deep neural network on the inputs 201. As mentioned above, a real number x is represented in the MDLNS as x = s * D1 a *D2 b , can be encoded as follows: where (D1,D2) are two multiplicatively independent bases and (a,b) is a pair of integers. Block 202 may include encoding or representing a real number x using any one of several different possible MDLNS. For example, block 202 may include encoding or representing a real number x using any one of the following: they have base (2,2 ω ) using 2DLNS (where ω is the golden ratio), 3DLNS using base (2, D2, D3) (where,

number

number

number

[0062] Training in block 202 may be performed by inputting numerical values, represented in either the 2DLNS or MDLNS described above, into the hardware accelerator 20. Exemplaryly, in block 202, the 2DLNS with a golden ratio base may be used to provide the initial encoding of the weight coefficients and to perform the dot product calculation (i.e., using the hardware accelerator 20 by leveraging the tile-based architecture of the processing core 40). This results in the computational efficiency improvements described above (e.g., exponentially faster conversion from logarithmic to binary form, and reduction of exponential size). As a result of training in block 202, an initial set 203 of weight coefficients for the deep neural network is determined.

[0063] Method 200 proceeds to block 204 after determining an initial set 203 of weight coefficients for a deep neural network. In block 204, an optimization procedure is applied using the initial set 203 of coefficients to determine the optimal second base 205 in a 2DLNS or MDLNS. The optimization procedure in block 204 may optionally include determining the second base 205 that yields the smallest mean squared error for a fixed first base (e.g., the first base is 2 in a particular embodiment). In one embodiment, a mixed-integer global optimization procedure is used to find the optimal base and exponent values ​​under a given dynamic range of exponents. The mixed-integer global optimization procedure can be performed using a mixed-integer optimization algorithm for MDLNS with integer exponents that are real and a non-binary base.

[0064] Finally, method 200 proceeds to block 206. In block 206, inference is performed on the set of inputs 207 using the newly determined optimal second base 205 for 2DLNS. For example, weight coefficients may be represented in 2DLNS using the optimal second base. The inference computation yields the set of outputs 208. The use of 2DLNS with the optimal second base for inference computation by the hardware accelerator 20 can result in improved computational efficiency (e.g., exponentially faster conversion from logarithmic to binary form, and reduced exponential size), which enables inference to be performed on edge devices (limited by size or power) or other devices requiring low-power, low-area, and / or high-speed dot product calculation units. The conversion from binary representation to logarithmic representation can be obtained via a pre-computed lookup table (LUT). For example, one LUT (containing 256 words or 2K, respectively) can be used for an 8-bit or 12-bit dynamic range.

[0065] In addition to the exemplary embodiments described above, the present invention will be illustrated by the following examples, which are provided to aid in understanding the invention and should not be construed as limiting in any way the scope of the invention as defined in the subsequent claims.

[0066] (example) The following examples in the field of digital hearing aids aim to demonstrate a) the importance of the choice of the second base in MDLNS, and b) the importance of the number of digits achieved by the exponential reduction of the exponential size based on the use of a two-digit MDLNS approximation.

[0067] Consider a 53-tap FIR filter with the following coefficients (coefficients 28-53 are mirrors of 1-26 to ensure linear phase). This filter is used in digital hearing aids, and its purpose is to ensure a stopband attenuation of 80 dB or more. [Table 2] [Table 3]

[0068] Table 3 above clearly demonstrates that MDLNS enables a significant reduction in exponential size. In fact, using ideal (infinite-precision) coefficients, a stopband attenuation of -85.362 dB is achieved. With a single-digit MDLNS and a nine-digit exponent, an optimal base of x = 0.7278946656 achieves a stopband attenuation of -80.315 dB. For a two-digit MDLNS, using only three bits for the exponent and an optimal base of x = 0.735254518 achieves a stopband attenuation of -81.562 dB. Thus, the combination of optimizing the second base and using a two-digit representation avoids the main drawback of classical LNS, namely the need for large LUTs to perform (for LNS) difficult arithmetic operations. The role of precisely selecting the second base is highlighted by the comparison with using a random (unoptimized) second base.

[0069] Embodiments of the present invention incorporate a two-dimensional logarithmic notation (2DLNS), in which case the selection of the second base D2 is made by considering an optimal-on-average base selection. First, let's consider some particularly bad choices for D2 (assuming the first base D1=2). For example, let D2=sqrt(2). In this case, even powers of D2 are perfect powers of 2, and therefore we will have many numbers with very poor approximations. If we consider other bases (both non-binary), such as (19,83), we may notice phenomena that should be avoided. In fact, 19 -3 * 83 2 = 1.00437... Therefore, this particular pair is also considered bad. If 19 a *83 bIf we consider numbers of the form (where a and b are integers), they form clusters. Numbers within a cluster provide a very good approximation, while numbers outside a cluster provide a poor approximation unless we use extremely large exponents (a and b). Therefore, the theoretical constraint that bases are multiplicatively independent is necessary, but not sufficient, for selecting bases to obtain optimal computational performance.

[0070] Therefore, a "good" second base is one such that log2(D) (the logarithm of D with base 2) is a badly approximable irrational number. The number known to be the "worst" with respect to rational approximation is the golden ratio (ω=(1+sqrt(5) / 2=1.618...)), so a very good (universal) second base is D=2 ω =2 1.618... =3.069... For convenience, we can also consider a base where D=D / 2. In fact, this base works well, and (for example) if an adaptive digital filter is implemented with 2DLNS, the base (2,2 ω Using ) seems computationally optimal, and if you want good performance (in an average sense), you can choose this. This can be called the mean-optimal base selection. If the first base is not 2, the general rule for selecting the mean-optimal second base is D2=D1 ω That is the case.

[0071] Experiments were conducted by the inventors to find the optimal second base for the matrix multiplication task tested on MDLNS (specifically, 2DLNS in this example). For image understanding applications, the main computation is Wx + b, where W is a (non-square) matrix and x and b are vectors. Wx + b follows a Gaussian distribution law located between -2 and +2.

[0072] Based on the above, the interval [-2,2] is divided into 256 equally spaced intervals, and each of the 256 numbers is approximated to be of the form 2^a*D^b. The mean squared error (MSE) with respect to D is minimized assuming that all numbers in this interval have weights provided by a Gaussian distribution. In supervised training, the input is provided to a deep neural network, and the network output is compared to the target output. The error is measured as the difference between the target output and the network output. It is desirable to minimize the mean of the sum of these errors (mean squared error). For all specific exponents and a fixed first base (fixed to 2 in this example), the optimal second base that minimizes the MSE is derived. Table 4 below shows the results. [Table 4]

[0073] The optimal second base must lie within the interval [1 / sqrt(2),sqrt(2)]. This explains the values ​​for the optimal second base seen in Table 4 above, calculated with five decimal places of precision. As shown in Table 4, the mean squared error decreases as a function of the number of bits in the non-binary exponent.

[0074] Table 5 below shows (a) the second base (which varies for each given bit size) that is particularly optimized with the first base 2, (b) the 2DLNS with base (2,3), and (c) the mean-optimal base (2,2) with respect to the mean squared error of the input data approximation. ω This shows a comparison of numerically obtained data between 2DLNS. The data is assumed to follow a Gaussian distribution located between [-2, 2]. [Table 5]

[0075] From Table 5 above, several important conclusions can be drawn. In particular, when using an optimized second base, the mean squared error is improved by more than double at every step because it adds another bit to the exponent. Mean optimal base (2,2ω If you choose ), the error is slightly more than twice as large, and you can get a very robust reduction. This base choice is almost always better than 2DLNS with base (2,3), with one exception (6-bit exponent).

[0076] In digital signal processing, there are fundamental differences in the use of MDLNS in FIR / IRR filters and adaptive filters. The above discussion emphasizes the importance of carefully selecting the second base to ensure a highly efficient MDLNS inner product architecture, including very small exponents, very small LUTs, and very small adders. On the other hand, randomly selecting the second base significantly degrades the performance of FIR / IR architectures.

[0077] In the case of adaptive filters, the filter coefficients change iteratively according to the adaptive filtering algorithm, so there is no way to select the optimal second base. Therefore, as mentioned above, mean-optimal techniques can be used to select the second base.

[0078] The concepts described above can also be applied to inner product calculations using data representations with three-dimensional logarithmic notation (3DLNS). To select the most optimal base on average, it is necessary to search for "pairs of poorly approximated real numbers." While there is little research in this area, some explicit estimates for pairs of irrational numbers that cannot be well approximated as rational numbers are discussed in TW Cusick, *The two-dimensional Diophantine approximation constant - II*, *Pacific Journal of Mathematics*, vol.105, pp.53-67, 1983. Cusick's results were translated by Keith Briggs as *Some explicitly badly approximable pairs*.<arxiv.org / pdf / math / 0211143.pdf> This was extended on October 25, 2018, and the pairs that he considered particularly difficult to approximate as rational numbers are (α,α 2) is the case. Here, a very good candidate for α is known to be α = cos(2*pi / 7). Therefore, (2,2 α ,2 α^2 The triplet of bases )=(2,1.18671,1.469117) is a practical and good choice as the average optimal base for 3DLNS applications.

[0079] The values ​​in the table below were obtained using mixed-integer optimization techniques (integer exponents, real bases for non-binary variables). [Table 6] TIFF0007843282000013.tif201170TIFF0007843282000014.tif63170

[0080] The examples and corresponding figures used herein are for illustrative purposes only. Different constructs and terminology may be used without departing from the principles expressed herein.

[0081] While the present invention has been described with reference to certain specific embodiments, various modifications thereto will be apparent to those skilled in the art that do not depart from the scope of the invention. The claims should not be limited by the exemplary embodiments described in the examples, but rather should be given the broadest interpretation consistent with the description as a whole. For example, various features are described herein as being present in “some embodiments.” Such features are not essential and are not necessarily present in all embodiments. Embodiments of the invention may include zero, any one, or any combination of two or more such features. This is limited only to the extent that some of such features are incompatible with others of such features in the sense that it is impossible for those skilled in the art to construct a practical embodiment combining such incompatible features. Accordingly, statements that “some embodiments” have feature A and “some embodiments” have feature B should be interpreted as an explicit indication that the inventors also envision embodiments combining feature A and feature B (unless otherwise stated in the description or if feature A and feature B are fundamentally incompatible).

Claims

1. A method for training and inferring from a deep neural network, (a) A step of receiving a set of training data, (b) A step of representing the training data set in multidimensional logarithmic notation (MDLNS), wherein the MDLNS representation uses a first exponent associated with a first base and a second exponent associated with a second base, (c) Training a deep neural network on a set of training data using a default first base and a default second base, and determining a set of weight coefficients for the neural network, (d) Based on the determined set of neural network weight coefficients, the step of optimizing a second base for multidimensional logarithmic data representation with respect to a predetermined first base, (e) To obtain a network output set, perform deep neural network inference on the network input set using the optimized multidimensional logarithmic data representation determined in step (d), A method that includes this.

2. The method according to claim 1, characterized in that optimizing the second base for multidimensional logarithmic data representation includes determining the optimal second base that minimizes the mean squared error (MSE).

3. The method according to 1 or 2, further comprising the step of performing a mixed-integer global optimization procedure to optimize the range of a second base and a second exponent associated with the second base.

4. The method according to any one of claims 1 to 3, characterized in that the default first base is 2.

5. The default second base is 2 ω The method according to 4, characterized in that ω = (1 + sqrt(5)) / 2.

6. The method according to any one of claims 1 to 4, characterized in that the MDLNS uses one or more additional indices, each of which is associated with one or more corresponding additional bases.

7. Training a deep neural network on a set of training data involves using a default third base, where the default second base is: [Math 1] And the default third base is, [Math 2] The method according to 6, characterized in that it is the same.

8. The exponent is an integer value, and the default second base is, [Math 3] The method according to 6, characterized in that it is selected from a group consisting of the following.

9. The method according to 6, characterized in that the first exponent and the second exponent are opposite in polarity.

10. The method according to 6, characterized in that the first and second indices are decimal values.

11. The method according to any one of claims 6 to 10, comprising the step of optimizing at least one of one or more additional bases for a multidimensional logarithmic data representation.

12. A hardware accelerator configured to perform the method described in any one of claims 1 to 11.

13. A hardware accelerator for performing dot product calculations allocated from the processor of a computing device, A multidimensional logarithmic notation (MDLNS) converter connected to the memory and hardware accelerator cache of a computing device, A plurality of processing units arranged in an array of rows of a first number and columns of a second number, which collectively form a processing core, Includes a processing core and a microcontroller connected to an MDLNS converter, The MDLNS converter is configured to create an MDLNS representation of a dataset received from the memory of a computing device and to store the MDLNS representation in the cache of a hardware accelerator, the hardware accelerator using a first exponent related to the binary base and a second exponent related to the non-binary base.

14. The hardware accelerator according to claim 13, characterized in that the processing unit includes a first adder operating at the bottom of the binary and a second adder operating at the bottom of the non-binary.

15. The hardware accelerator according to claim 14, wherein the processing unit includes an aggregation adder connected to the first adder and the second adder, the aggregation adder having a plurality of aggregation channels, each aggregation channel corresponding to a unique combination of pairs (N, M) defined by N bits of the first exponent and M bits of the second exponent.

16. The aforementioned aggregating adder operates in parallel to aggregate unique (N, M) pairs of exponents. Ｎ＋Ｍ The hardware accelerator according to claim 15, characterized by including an up counter.

17. The hardware accelerator according to any one of claims 13 to 16, characterized in that the processing unit of the processing core is configured as a systolic array of matrix-vector multiplication units.

18. The base of the aforementioned non-binary is 2 ω The hardware accelerator according to any one of claims 13 to 17, characterized in that ω = (1 + sqrt(5)) / 2.

19. The hardware accelerator according to any one of claims 13 to 18, characterized in that it includes a plurality of processing tiles, each processing tile includes a plurality of processing cores, and is connected to other processing tiles by a chip-on network.

20. The hardware accelerator according to any one of claims 13 to 19, characterized in that the computing device is an edge computing device.

Images