Accelerating Artificial Neural Networks Using Hardware-Implemented Lookup Tables

Abstract

The present invention relates in particular to a hardware system (1) designed to implement an artificial neural network (ANN). The hardware system essentially comprises a neural processing device (15), e.g., a crossbar array structure, one or more lookup table circuits (17), and one or more processing units (18). The neural processing device is configured to implement M artificial neurons, where M≧1. The lookup table circuit is configured to implement a lookup table (LUT). The system further comprises M′ processing units, where M≧M′≧1. Each processing unit is connected by at least one neuron such that it can access a first value output by each connected neuron. Furthermore, each processing unit is connected to a LUT circuit for efficiently accessing parameter values ​​of a parameter set from the LUT. Finally, each processing unit is configured to output a second value corresponding to the value of a mathematical function having the first value as an argument. The mathematical function is otherwise determined by a parameter set, and its parameter values ​​are accessed from the LUT by each processing unit during operation. That is, the mathematical function is defined (and thus determined) by a set of parameters, and the parameter values ​​are efficiently looked up from a hardware-implemented LUT. This results in a substantial speedup in the calculation of the function output beyond that which can already be achieved within the neural processing device and processing unit itself. As a result, neuron outputs can be more efficiently processed before being passed to the next layer of neurons. The present invention further relates to a method for operating such a hardware system.

Description

[Technical Field]

[0001] The present invention relates generally to the field of in-memory and near-memory processing techniques (i.e., methods, devices, and systems) and related speed-up techniques for implementing artificial neural networks (ANNs). In particular, the present invention relates to a hardware system including a neural processing device (e.g., having a crossbar array structure) that implements neurons, a processing unit, and a hardware-implemented look-up table (LUT) that stores parameter values ​​that can be quickly accessed by the processing unit to more efficiently apply mathematical functions (such as activation functions) to neuron outputs. [Background technology]

[0002] Deep neural networks (ANNs), such as deep neural networks (DNNs), have revolutionized the field of machine learning by offering unprecedented performance in solving cognitive data analysis tasks. ANN operations often involve matrix-vector multiplication (MVM). MVM operations pose multiple challenges due to their reproducibility, generalizability, computational, and memory requirements. Traditional computer architectures are based on the von Neumann computing concept, according to which processing power and data storage are partitioned into separate physical units. Such architectures suffer from congestion and high power consumption because data must be continuously transferred from memory units to control and computational units through physically constrained and costly interfaces.

[0003] One possibility for accelerating MVM is to use dedicated hardware acceleration devices, such as dedicated circuits with a crossbar array structure. This type of circuit includes input and output lines interconnected at crosspoints that define cells. Each cell contains a memory device (or set of memory devices) designed to store its respective matrix coefficients. Vectors are encoded as signals applied to the input lines of the crossbar array, and MVM is performed using multiply-accumulate (MAC) operations. Such an architecture allows for simple and efficient mapping of MVM. Weights can be updated as needed by reprogramming memory elements to perform successive matrix-vector multiplications. Such an approach breaks the "memory wall" by fusing computation and memory units into a single in-memory computing (IMC) unit, allowing processing to occur much more efficiently in or near memory (i.e., the crossbar array).

[0004] While the primary computational burden of ANNs such as DNNs revolves around MAC operations, the execution of ANNs often involves additional mathematical functions, such as activation functions. Even quantized neural networks require activation functions that are inherently more difficult to compress and often must be performed with floating-point precision.

[0005] We conclude that on hardware platforms designed for efficient execution of DNNs and low power consumption, performing such functions can be cumbersome and expensive in terms of computational resources. One possible solution is to offload the execution of such functions to a digital signal processor (DSP). However, doing so can be very demanding in terms of latency, area, and energy.

[0006] The inventors therefore took up the challenge to achieve a new computational architecture involving non-traditional processing means to speed up the computation of such functions. Summary of the Invention

[0007] According to a first aspect, the present invention is embodied as a hardware system designed to implement an artificial neural network (ANN). The hardware system essentially includes a neural processing device, one or more lookup table circuits, and one or more processing units. The neural processing device is configured to implement M artificial neurons, where M≧1. The one or more lookup table circuits are configured to implement lookup tables (LUTs). The system further includes M′ processing units, where M≧M′≧1. Each of the M′ processing units is connected to at least one of the M artificial neurons so as to access a value (referred to as a “first value”) output by each of the at least one neuron during operation. Additionally, each processing unit is connected to a LUT circuit of one or more LUT circuits so as to access parameter values ​​of the parameter set from the LUT during operation. Finally, each processing unit is configured to output a value (a “second value”) of a mathematical function having the first value as an argument. The mathematical function is otherwise determined by the parameter set. During operation, parameter values ​​of a parameter set are accessed by each of the processing units from the LUT circuit.

[0008] The architecture of the present hardware system differs from conventional computer architectures, which typically use the same digital processor (or set of digital processors) to both calculate neuron output values ​​and apply subsequent mathematical functions (e.g., activation functions). In contrast, in the present hardware system, the processing hardware used to calculate neuron output values ​​is distinct from the processing units used to apply the mathematical functions, although the processing units may be configured as adjacent memory processing devices in the system. Such an architecture is employed for computational efficiency reasons. In particular, the LUT is implemented in hardware by hardware circuits distinct from each of the neural processing unit (used to calculate neuron outputs) and the processing unit (used to apply the mathematical functions). Substantial speedup is achieved by the hardware-implemented LUT: the mathematical functions are defined (and thus determined) by a set of parameters, and the parameter values ​​are efficiently retrieved from the hardware-implemented LUT. This results in a substantial speedup of function output calculation beyond that which can already be achieved within the neural processing unit and processing unit. As a result, neuron outputs can be processed more efficiently before being passed to the next neuron layer.

[0009] Furthermore, given that the LUT stores parameter values ​​instead of mapping input values ​​to output values, very little memory is required. The LUT is not used to look up function outputs directly, as opposed to what is typically done when using a lookup table.

[0010] Finally, the approach is integration friendly: in particular, the LUT circuit, the processing unit and the neural processing unit can be advantageously co-integrated on the same device, e.g., the same chip.

[0011] In an embodiment, each processing unit is configured to output a second value by (i) selecting said parameter set according to a first value, and (ii) performing an operation based on the first value and parameter values ​​of said selected parameter set, with the aim of outputting a second value, which in fact allows reducing the number of parameters required, since a small parameter set is already sufficient for a locally accurate estimation of the function in an interval that includes each potential input value.

[0012] Preferably, each processing unit is further configured to select the parameter set by comparing the first value with bin boundaries to identify a relevant bin, i.e., the bin containing the first value. For this purpose, each processing unit is further configured to access the bin boundaries from the lookup table circuit. A parameter set is then selected during the operation according to the identified bin. Therefore, the bin boundaries can be efficiently accessed, enabling rapid comparison. In this way, the binning problem can be efficiently solved, thereby enabling rapid identification of the relevant parameter set.

[0013] In a preferred embodiment, a dedicated comparator circuit is used to efficiently identify the relevant bin. That is, each processing unit includes at least one comparator circuit. This circuit is designed to compare the first value with the bin boundaries and transmit a selection signal that encodes the selected parameter set. The processing unit can then access the corresponding parameter value based on the transmitted signal.

[0014] A simple binary tree comparison circuit may be relied upon. However, more sophisticated comparison schemes and comparison circuit layouts are contemplated. The comparison circuit may be designed to enable multiple levels of comparison, particularly to speed up binning. In particular, the comparator circuit may advantageously be configured as a multi-level q-ary tree comparison circuit designed to enable multiple levels of comparison, where q is 3 or greater for one or more of the multiple levels.

[0015] In this embodiment, each LUT circuit is a circuit for hard-coding parameter values. Furthermore, each processing unit includes at least one multiplexer, which is connected to each comparator circuit to receive a selection signal, and connected to the LUT circuit to search for the corresponding parameter value according to the selection signal. This design makes parameter search very efficient. The drawback is that the hard-coded data cannot be changed after the LUT circuit is hard-wired.

[0016] Therefore, in a variant, it may be preferable to use a reconfigurable memory. In this way, the mathematical function may be dynamically reconfigured as the calculation progresses or is updated as needed. For example, each LUT circuit may include an addressable memory unit connected to the comparator circuit and receiving a selection signal. In this way, the addressable memory unit can retrieve parameter values ​​of a selected parameter set according to the received selection signal.

[0017] In a preferred embodiment, the mathematical function is a piecewise-defined polynomial function that is polynomial in each of its subdomains. Each subdomain corresponds to a bin. In this case, the selected parameter set corresponds to the polynomial parameters of the piecewise-defined polynomial function. That is, the selected parameter set corresponds to the parameters of the locally relevant polynomial. Such a construct is suitable for fast calculation by an arithmetic unit, since simple arithmetic operations are required to achieve the desired result. Therefore, each processing unit may advantageously include an arithmetic unit connected to the output of the LUT circuit, whereby the operations required to calculate the second value are performed as arithmetic operations by the arithmetic unit.

[0018] Interestingly, such operations can be simply performed using a multiply-accumulate circuit, i.e. a circuit specifically designed to perform multiply-accumulate operations efficiently, and therefore the arithmetic unit preferably comprises a multiply-accumulate circuit, which allows for faster obtaining of the output value of the mathematical function.

[0019] In a preferred embodiment, the neural processing device includes a crossbar array structure including N input lines and M output lines arranged in rows and columns, where N>1 and M>1, thereby enabling the neural processing device to implement a layer of M neurons. The input and output lines are interconnected via memory elements. Each of the M output lines is connected to at least one of the M' processing units. The crossbar array structure fuses the arithmetic and memory units into a single in-memory computing unit, enabling efficient retrieval of neuron outputs.

[0020] Neural processing devices are typically designed to implement several neurons (M>1) at a time. The number of neurons may be, for example, 256 or more or 512 or more (M≧256 or M≧512). Furthermore, the processing units may advantageously be vector processing units, each of the M′ processing units comprising b processing elements so as to be able to operate on a one-dimensional array of dimension b. The number of processing units M′ is preferably equal to 1 or 2.

[0021] Various architectures are possible. For example, several processing units (i.e., M'>1) may be relied upon, but their number may typically be less than or equal to the number of neurons that can be implemented at one time (i.e., M≧M'>1). In such a case, the LUT circuit may include M' separate circuits that are respectively mapped to the M' processing units.

[0022] According to another aspect, the present invention is embodied as a method for operating such a hardware system. The system includes a neural processing device configured to implement M artificial neurons, where M≧1, and M′ processing units, each connected by at least one of the M artificial neurons. The hardware system further includes one or more LUT circuits implementing a LUT. The method includes operating the neural processing device to obtain M first values ​​generated by the M artificial neurons, respectively. The method further relies on the M′ processing units to apply a mathematical function to the neuron outputs. That is, for each first value of the M first values, an output value of the mathematical function is obtained (via the M′ processing units). The mathematical function is, among other things, determined by a parameter set. Thus, the output value of the mathematical function is obtained based on operands including the first value and parameter values ​​of the parameter set, the parameter values ​​being retrieved from one or more LUT circuits.

[0023] Preferably, the output value is obtained, for each said first value, by selecting a parameter set according to the first value and performing an operation based on the first value and the parameter value retrieved according to the selected parameter set.

[0024] In a preferred embodiment, the parameter set is selected by comparing a first value to bin boundaries (retrieved from one or more LUT circuits) to identify an associated bin that contains the first value. The parameter set is then selected according to the identified bin.

[0025] As mentioned above, the applied mathematical function is preferably a piecewise-defined polynomial function. In that case, each parameter set includes two or more polynomial coefficients. The operation performed to calculate the second value may be a simple arithmetic operation. In a preferred embodiment, the mathematical function includes a set of linear polynomials, each corresponding to a respective one of the bins. In this case, the parameter set corresponding to each linear polynomial consists of a scale factor and an offset factor. Again, the arithmetic operation may advantageously be performed by a multiply-accumulate circuit.

[0026] In an embodiment, the method further includes programming one or more LUT circuits implementing the LUT to enable one or more types of mathematical functions, such as activation functions, normalization functions, shrinkage functions, state update functions, classification functions, and / or prediction functions.

[0027] The method may further include an upstream step (i.e., performed at construction time prior to operating the neural network processing device) for determining one or more sets of appropriate bin boundaries, each according to one or more criterion functions (i.e., mathematical functions of potential interest to the ANN execution). In an embodiment, the bin boundaries are determined for each criterion function so as to minimize the number of bins or the maximum error, where the error is measured as the difference between the approximate value of each criterion function calculated based on the parameter values ​​and the theoretical value of that criterion function.

[0028] These and other objects, features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments thereof, which is to be read in connection with the accompanying drawings, which together with the detailed description are intended to clearly indicate that the present invention will be readily understood by those skilled in the art. [Brief explanation of the drawings]

[0029] [Figure 1] 1 is a diagram illustrating a computer network including several hardware systems according to an embodiment of the present invention, which allows users to interact with a server to accelerate machine learning computational tasks that are offloaded to the hardware systems, as in the embodiment. [Figure 2] FIG. 1 is a diagram illustrating schematically selected components of a hardware system including, among other things, a neural processing device with a crossbar array structure, a processing unit, and a hardware-implemented look-up table (LUT), according to an embodiment. [Figure 3] FIG. 1 is a diagram illustrating a possible architecture of a hardware system according to a preferred embodiment, showing how neurons of a neural processing unit connect to a vector processing unit, and how the vector processing unit connects to a LUT circuit. [Figure 4] 1 is a circuit diagram showing a given processing element (e.g., a vector processing unit as shown in FIG. 3) connected to a respective LUT circuit, as in an embodiment. In this example, the processing element includes a comparator and a multiplexer, and the lookup table is implemented by the circuit hard-coding parameter values ​​necessary to apply a mathematical function to the neuron output. [Figure 5] A variation of FIG. 4 in which the LUT circuit is implemented as an addressable memory (no multiplexers are required in this example). [Figure 6] 4 is a flowchart illustrating high-level steps of a method of operating a hardware system such as that shown in FIG. 2 or FIG. 3, according to an embodiment. [Figure 7A] 10 is a graph illustrating how a nonlinear function can be approximated using a piecewise-defined polynomial function thanks to optimized bin boundaries, as in an embodiment. [Figure 7B] 10 is a graph illustrating how a nonlinear function can be approximated with a piecewise-defined polynomial function thanks to optimized bin boundaries, as in an embodiment. [Figure 7C] 10 is a graph illustrating how a nonlinear function can be approximated with a piecewise-defined polynomial function thanks to optimized bin boundaries, as in an embodiment. [Figure 8] 10 is a table showing optimization of the number of comparators included in each level of a multi-level q-ary tree comparison circuit used in an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0030] The accompanying drawings show simplified representations of devices or portions of devices as included in the embodiments, and like or functionally similar elements in the drawings are assigned the same reference numbers unless otherwise indicated.

[0031] By way of non-limiting example, a hardware system and method embodying the present invention will now be described.

[0032] The following description is structured as follows: General embodiments and high-level variants are described in Section 1; Section 2 deals with particularly preferred embodiments and technical implementation details; Section 3 summarizes final remarks; and the method and its variants are collectively referred to as "the method." All reference signs Sn refer to method steps in the flowchart of FIG. 7, and the reference numbers relate to devices, components, and concepts included in embodiments of the invention.

[0033] 1. General Embodiments and High-Level Variations A first aspect of the present invention will now be described in detail with reference to Figures 1 to 5. This aspect relates to a hardware system 1, also referred to herein for simplicity as "the system." System 1 is designed to implement an artificial neural network (ANN) by efficiently evaluating mathematical functions (such as activation functions) applied to neuron outputs.

[0034] An example of such a hardware system 1 is shown in Figure 2. System 1 essentially includes a neural processing unit 15, a hardware-implemented look-up table (LUT) 17, and one or more processing units 18.

[0035] Neural processing unit 15 is configured to implement M artificial neurons, where M≧1. However, in practice, M is typically strictly greater than 1. For example, unit 15 may enable up to 256 or 512 neurons, or possibly more. However, as will be illustrated later, there may be situations where neural processing unit 15 implements a single neuron at a time. Neural processing unit 15 may advantageously have a crossbar array structure 15, as envisioned in FIG. 2.

[0036] The LUT is implemented by one or more LUT circuits 175, as shown in Figure 3. As will be explained in more detail below, several types of LUT circuits 175a, 175b are possible.

[0037] The system further relies on M′ processing units 18 to evaluate the mathematical functions, where M≧M′≧1. As shown in FIG. 3, each processing unit 18 may include several processing elements 185, allowing for several efficient processors.

[0038] As shown in FIGS. 2 and 3, neurons connect to processing units 18, which themselves connect to LUT circuits. Various configurations are possible. FIG. 3 illustrates a preferred architecture. At a minimum, each processing unit 18 is connected to at least one of the M neurons implemented by device 15. In this manner, each processing unit 18 has access to neuron outputs, i.e., values ​​output by at least one, and possibly more, of the neurons. Furthermore, each processing unit 18 is connected to one or more of LUT circuits 175, 175a, 175b to enable fast calculation of the second value. For example, as envisioned in FIG. 3, each processing unit can be connected to a respective LUT circuit 175.

[0039] In the following, a neuron output will be referred to as a "first value," and the corresponding value output by the processing unit 18 will be referred to as a "second value." The "first value" corresponds to one of the M values ​​output by the neuron in each algorithm cycle, while the "second value" corresponds to the value of a mathematical function applied to this first value and evaluated (i.e., calculated) by the processing unit. Note that an algorithm cycle is a cycle of calculation initiated by the neural processing unit 15. Each algorithm cycle begins with a calculation performed by this unit 15 (see step S40 in FIG. 6). Then, each processing unit 18 is configured to access at least one first value (from a connected neuron) and output a second value in each algorithm cycle. Thus, M second values ​​are output by the processing unit during each algorithm cycle. Furthermore, this number of available processing elements may require several calculation sub-cycles to enable the processing unit to output M second values ​​within each algorithm cycle.

[0040] The first value is the argument of the function to be applied. Furthermore, in the present context, any mathematical function applied to a neuron output is further defined (and thus determined) by a set of parameters. The values ​​of the function parameters are efficiently looked up from the LUT, which allows the value of the associated mathematical function to be efficiently calculated. Thus, using a non-traditional hardware architecture, one or more mathematical functions are applied to the neuron output each cycle.

[0041] The above specifications of the system 1 minimize constraints on the architecture of the processing unit 18, the LUT circuit 17 and the neural processing unit 15. Various embodiments are possible. Furthermore, some concepts defined below are relied upon.

[0042] Hardware Architecture Hardware system 1 includes several devices (i.e., one or more processing units 18, one or more LUT circuits 17, and neural processing unit 15) that are connected together to form system 1. However, system 1 itself can be fabricated as a single unit, or even as a single device. In particular, LUT circuit 17, processing unit 18, and neural processing unit 15 may all be co-integrated on the same chip, as envisioned in FIG. 2. Additional components may be included, as described below with reference to FIG. 2.

[0043] In principle, the neural processing unit 15 can be any information processing unit 15 or information processing device capable of implementing the artificial neurons of an ANN. The unit 15 performs the basic function inherent to ANN neurons: they generate signals intended for other neurons, e.g., neurons in the next layer of a feedforward or recurrent neural network configuration. However, such signals typically encode values ​​that require post-processing (e.g., application of activation functions), which is an advantage of having a processing unit 18 connected to the neurons.

[0044] The neural processing unit 15 can optionally be a general-purpose or special-purpose computer. Preferably, however, the processing unit 15 has a crossbar array structure 15 (also referred to herein as a "crossbar array" or simply a "crossbar"). A crossbar array structure is a non-traditional processing unit that, as described in the Background section, is designed to efficiently process analog or digital signals to perform matrix-vector multiplication. Relying on the crossbar array structure 15 allows for substantial speedup of matrix-vector multiplication already when it is included during the training and inference phases of an ANN.

[0045] The crossbar array structure 15 allows M neurons (M>1) at a time and can be used to implement a single neural layer (or part of it) at a time. The neurons are designated v1 to v2 in FIG. MIn principle, M can be any number allowed by the technology used to manufacture device 15. Typically, the number of neurons enabled by the crossbar, M, is 256, 512, or 1024. However, it is possible that the problem to be solved may include an incommensurate ANN layer, i.e., a layer containing a number of neurons different from what is effectively allowed (at one time) by crossbar 15. Therefore, a distinction should be made between the number of neurons M actually enabled by crossbar 15 at one time (which may be referred to as a physical neural layer) and the size of the abstract neural layer involved in the problem to be solved. However, in practice, such a potential discrepancy is not a problem. In fact, ANN layers with fewer than M neurons can be fully processed by device 15, while ANN layers with more than M neurons can be mapped onto crossbar 15 by repeatedly operating crossbar 15. Therefore, in practice, incommensurate ANN layers can be appropriately handled. For completeness, the crossbar array structure 15 included herein may generally be used to map neurons in various ANN architectures, such as, for example, feedforward architectures (including convolutional neural networks), recurrent networks or transformer networks.

[0046] The crossbar array structure 15 can be operated cyclically in a closed loop so that the structure 15 can implement several successively connected neural layers of an ANN. In a variant, several crossbar array structures 15 are cascaded to achieve the implementation of a neural layer. The neural layer implemented by the crossbar array structure 15 can be any layer (or part thereof) of the ANN, including the final layer, which may consist of a single neuron. Thus, in certain cases (e.g., during the final algorithm cycle), the number of neurons enabled by the crossbar array structure can be one.

[0047] The architecture of hardware system 1 differs from conventional computer architectures, which typically use a single digital processor (or a set of single digital processors) to both calculate neuron output values ​​and apply subsequent mathematical functions (e.g., activation functions). In contrast, in the present context, processing hardware 15 used to calculate neuron outputs is distinct from hardware device 18 used to apply the subsequent mathematical functions. Nevertheless, processing unit 18 is highly preferably “close” to neural processing hardware 15. That is, processing unit 18 is preferably configured in system 1 as a proximal memory processing device, as envisioned in FIG. 2 . Note that “proximal memory” here corresponds to considering device 15 as a memory that stores neuron outputs. Neuron outputs are efficiently delivered to processing unit 18, for example, via dedicated readout circuitry 16, which is known per se. In contrast, in conventional computerized systems, neuron outputs typically pass through a conventional computer bus, are stored in the computerized system’s main memory (or cache), and must be retrieved from this memory to apply the mathematical functions. That is, a proximal memory arrangement such as that shown in FIG. 2 differs from the usual cache memory in a CPU chip.

[0048] Furthermore, the processing unit 18 preferably also includes non-traditional computation means, such as a vector processing unit (as envisaged in Figure 3), which allows for even faster calculations.

[0049] Furthermore, in the present context, the LUT is implemented in hardware by a separate hardware circuit 17, i.e., a circuit separate from each of the neural processing unit 15 (used to calculate the neuron outputs) and the processing unit 18 (used to apply the mathematical functions). Thus, not only may the processing hardware 15, 18 differ from conventional hardware, but furthermore, a hardware-implemented LUT is relied upon (implemented by the separate circuit 17) to quickly look up parameter values ​​and thus apply the mathematical functions more efficiently.

[0050] Hardware-Implemented Lookup Table The LUT is implemented in hardware by one or more dedicated circuits 17, which can be considered memory circuits. Each of these circuits may implement the same table of values, or at least partially different tables of values. The circuits may also implement completely separate tables. However, in that case, each of the separate tables may still be considered part of a superset that forms the LUT. Taken as a whole, the table may potentially enable several types of mathematical functions.

[0051] 3, reference numeral 17 generally refers to a set of one or more LUT circuits 175, each implementing a respective table, the values ​​of which may vary from case to case. Each circuit 175 may be, for example, a circuit 175a for hard-coding parameter values ​​or an addressable memory circuit 175b, as envisioned in FIGS. 4 and 5, respectively.

[0052] 5, the LUT is assumed to be implemented by at least one addressable memory circuit 175b that stores parameter values, it being understood that several LUT circuits 175b may be included in place of circuit 175, for example, as in FIG. 3. It should be noted that when the LUT circuits are implemented as addressable memory circuits, system 1 typically includes programming means (not shown) connected to memory circuits 175b so as to rewrite (and thereby update) corresponding parameter values ​​as required.

[0053] In FIG. 4, the LUT is assumed to be implemented at least in part by a hard-coded circuit 175a designed to provide the necessary parameter values, similar to a read-only memory (ROM) circuit. Again, several LUT circuits 175a may be included (instead of circuit 175, as in FIG. 3). However, in practice, hard-wired circuits typically enable only a few functions. For example, circuit 175a shown in FIG. 4 enables a single mathematical function. Therefore, as in FIG. 5, it may be preferable to rely on a rewritable memory circuit, such as a random access memory (RAM), to achieve a reprogrammable LUT.

[0054] 4 and 5, it is assumed that one LUT circuit 175a, 175b is connected to each processing element 185a, 185b. However, system 1 may actually include several processing elements 185a, 185b connected to one or more LUT circuits 175a, 175b. Various architectures are contemplated. Such architectures, at a minimum, allow at least one LUT circuit to be mapped to each processing unit 18 (or its processing element) at a time. That is, each processing unit 18 may be dynamically connected to a separate LUT circuit, possibly switched on-the-fly. Note that at least one LUT circuit should be connected to a processing unit 18 when the unit is active.

[0055] That is, since several LUT circuits can be readily provided in practice (possibly within the same device), a convenient architectural choice is to provide several LUT circuits 17, with at least one LUT circuit 175 permanently connected to each processing unit 18, as envisioned in FIG. 3 . As such, LUT circuit 17 may include at least M′ separate circuits 175 that map to M′ processing units 18, respectively. That is, LUT circuit may include more than M′ circuits in some cases. For example, in the example of FIG. 3 , the number T of LUT circuits 175 exceeds the number M′ of processing units 18 (i.e., T>M′), thereby allowing for redundancy and / or, if necessary, for computation speed, allowing table values ​​to be preloaded (i.e., prefetched) as the computation progresses.

[0056] Parameter Type vs. Parameter Value When retrieved from the LUT, a distinction is made between the type of parameter (e.g., a given polynomial coefficient) and the actual value of each type of parameter (e.g., 2.173). The LUT stores parameter values ​​for one or more types of parameters. The type of parameter may correspond, for example, to a polynomial coefficient if the mathematical function is defined as a polynomial. In practice, if the mathematical function is defined as a piecewise polynomial, as in the embodiment described below, several parameter sets may be associated with each type of function. However, more generally, several types of mathematical functions may be included, and the functions may require different types of parameters.

[0057] Value vs. Signal The values ​​generated by components 15, 18 and the values ​​retrieved from LUT circuit 17 are encoded into respective signals. In practice, signals encoding neuron outputs are passed from neural processing device 15 to processing unit 18 (typically via readout circuit 16, see FIG. 2). The calculations performed by processing unit 18 require further signal transmission from the LUT circuit (to retrieve the required parameter values). A further signal (encoding a second value) is then passed from processing unit 18 to the neural processing device, either the same device 15 or a different processing device, to initiate the execution of another neural layer, and so on. This may require an input / output (I / O) unit 19, as envisioned in FIG. 2. I / O unit 19 may further be used to interface system 1 with other machines.

[0058] Processing units, vector processing units, processing elements and efficient processors Processing unit 18 is a processing circuit, typically in the form of an integrated circuit. As mentioned above, preferably such circuitry is arranged as a close-memory processing device at the output of neural processing unit 15, as shown in Figure 2, to efficiently process neuron outputs.

[0059] In the present context, processing unit 18 may comprise a simple arithmetic logic unit (performing basic arithmetic and logical operations), or a processing element simply requiring only a multiply-accumulate processing element. In alternative embodiments, a more sophisticated type of processing unit is used, which may, among other things, also perform control and I / O operations as required. Such operations may also be performed by other components of system 1, such as I / O unit 19.

[0060] Each processing unit 18 allows for at least one effective processor thanks to at least one processing element 185, 185a, 185b. The processing unit 18 may be a standard microprocessor. However, preferably, this processing unit 18 is a vector processing unit (as envisioned in FIG. 3) that allows for a degree of parallelism when applying mathematical functions. In the example of FIG. 3, each processing unit 18 includes a certain number b of processing elements 185, which allows for efficient operations on one-dimensional arrays of dimension b, which can be advantageously utilized at the output of neurons. Overall, in FIG. 3, the M′ vector processing units 18 include bM′ processing elements 185, where b represents the degree of parallelism enabled by each vector processing unit. However, in principle, the vector processing units 18 may have a different number of processing elements 185.

[0061] In a variant, or in addition to vector processing, conventional parallel processing may also be included. That is, each unit 18 may include several cores. In fact, each processing element 185 may be a multi-core processor. Thus, in general, one or more processing units 18 may include one or more processor cores, each of which may enable one or more effective processors, for example, by threading a physical core into multiple virtual cores. In other words, M' processing units may collectively form M'' effective processors, where M'' may be at least equal to M' and may be strictly greater than M'.

[0062] Number of processing units (or available processors) relative to the number of neurons As mentioned above, in principle, each of M' and M'' can be greater than the number of neurons, M. However, given that up to M functions are typically required in each algorithm cycle executed by the neural processing unit 15, this may not be useful in a configuration such as that shown in FIG. 2. Therefore, a preferred configuration is one in which system 1 includes M' processing units (which may potentially include M'' effective processors), with M≧M''>M'≧1. In a variant, system 1 has additional processing power such that M' (or M'') can be strictly greater than M. Such a configuration may be particularly useful for implementing certain types of activation functions, such as concatenated rectified linear units (CReLU), which preserve both positive- and negative-sequence information while promoting non-saturating nonlinearities. For example, computing CReLU(x)=[ReLU(x),ReLU(−x)], where [,] denotes concatenation, can potentially be done in separate passes. However, thanks to 2M' processing units (or 2M'' processors), performance can be improved by performing this operation in a single pass. In such a case, the number of processing units (or active processors) may advantageously be strictly greater than the number M of neurons activated by the device 15 during each algorithm cycle.

[0063] In summary, the number of neurons M enabled by each device 15 can in some cases be less than M' (or M''). It can also be equal to M', thereby allowing each neuron output to be processed in parallel. Another configuration is to have fewer neurons, i.e., M', if one or more of the M' processing units (or M'' active processors) are shared by at least some of the M neurons. <Mおよび / またはM’’<M(M> 1). In the latter case, the number of processing units (or effective processors) is reduced, and the M artificial neurons rotate through the M' processing units (or M'' effective processors).

[0064] In other words, each neuron connects to one of M' processing units 18, although each processing unit 18 may potentially be connected by multiple neurons. System 1 includes at least one processing unit, which at a minimum includes a single processor (possibly a single core). However, relying on a single processor (core) can substantially impact throughput and thus the benefits of including several processing units or at least several processing cores. Conversely, vector processing is expensive, and therefore a tradeoff is necessary to optimize the number of effective processors relative to the number of neurons.

[0065] For simplicity, in the following description, each processing unit 18 is assumed to include a fixed number b of processing elements 185, as in Figure 3. Furthermore, each processing element 185 is assumed to form a single effective processor (in which case the processing element does not allow virtual processing), as in the examples of Figures 4 and 5. Thus, the total number of processing elements is equal to bM', and the number of effective processors M'' (effectively enabled by vector processing unit 18) is equal to the total number of processing elements (i.e., M''=bM').

[0066] As mentioned above, it may be desirable to optimize the number of processors available relative to the number of neurons. In that regard, the ratio of M" to M is preferably 1 / 8 to 1. A preferred architecture includes one or two vector processing units per neural device 15 (i.e., M' = 1 or 2), where each vector processing unit 18 contains 64 processing elements, so M" is 64 or 128, while the number of neurons, M, is 512 in each device 15. For example, two vector processing units (M' = 2), each containing 64 processing elements (b = 64), can be used, in which case M" is 128 and the ratio of M" to M is 1 / 4.

[0067] As assumed in FIG. 2, all vector processing units may be directly connected to neuron outputs (requiring readout circuits 16). In a variant, some of the vector processing units may be indirectly connected to neurons. More precisely, some or all of the neurons may first be connected to intermediate processing units (not shown), which themselves connect to vector processing units. For example, if two vector processing units are used (M'=2), the first vector processing unit may be directly connected to M neurons at the output of crossbar array 15 (actually the output of readout circuits 16), while the second vector processing unit may be connected to a so-called depthwise processing unit (DWPU), which itself is connected to the output of crossbar array 15. Inserting a DWPU enables depthwise convolution operations.

[0068] Number of LUT circuits vs. number of processing units In principle, it is sufficient for the LUT to be implemented by a single circuit (e.g., a single addressable memory circuit) that services each processing unit 18. However, relying on multiple processing units may require multiple interfaces or data communication channels. Note, however, that if a single LUT circuit is mapped to a single vector processing unit of b processing elements (as in FIG. 3), then only a single port is required on the LUT circuit, and its output signal can be multiplexed to the b processing elements.

[0069] It may be desirable to be able to apply several mathematical functions. For that purpose, it may be desirable to connect one processing unit 18 to several LUT circuits 17. However, this is usually not useful, as a single LUT circuit can already enable a large number of parameter values, thereby allowing multiple functions to be applied to the output of a neuron. Therefore, connecting a single LUT circuit to each processing unit may be sufficient.

[0070] For completeness, the LUT circuitry may be shared by the processing units 18 instead of being shared by the processing elements 185 of each processing unit 18 (as assumed in FIG. 3). That is, the LUT circuitry may consist of J separate circuits, where J<M’であり、これは、M≧M’’≧M’> The configuration is J≧1.

[0071] Processing Units vs. Mathematical Functions The number L of functions available to each processing unit 18 is 1 or greater (L≧1). If several mathematical functions are available, any of the L functions may potentially be selected and formed by corresponding parameter values ​​accessed from the LUT. That is, each of the M′ processing units may potentially apply any available mathematical function.

[0072] One convenient approach, as in the embodiment described below, is to rely on the same general functional construct (e.g., a piecewise-defined polynomial function), which is suitably parameterized so that a variety of functions can ultimately be evaluated using that same construct and applied to each neuron output.

[0073] Again, several configurations are possible. In a simple scenario, the M' processing units 18 apply the same function (i.e., a single function) to the outputs from all neurons of the neural layer implemented by the device 15, i.e., at each algorithmic cycle. Nevertheless, it may be necessary to apply separate functions to the outputs from successive neural layers. Conversely, in a more sophisticated scenario, the M' processing units may implement up to M separate functions (possibly selected from L > M potential functions) at each algorithmic cycle, in which case separate functions are applied to the neuron outputs. In other words, separate functions may be used from one neural layer to another, and separate functions may be applied to neuron outputs from the same layer, if desired.

[0074] Arguments vs. parameters of mathematical functions In principle, arguments are variables passed to mathematical functions to calculate their output values. Function parameters can also be considered variables. However, parameters are variables that determine (i.e., contribute to completely defining) a function, similar to the parameters defined in a function declaration in a programming language. For example, the polynomial function f(x) = αx + β has a single argument x, but includes two parameters a and b, whose values ​​contribute to completely defining the function.

[0075] Similarly, in the present context, any mathematical function of interest takes as an argument a value x, i.e. the value encoded in the signal output by the neuron. Any output value of this mathematical function is therefore calculated by a processing unit 18 (or processing element) based on the value encoded in the signal obtained from the neuron connected to this unit 18 (or processing element). In Figures 4 and 5, the argument of the function is x IN and y OUT denotes the output value of the function. Note that, according to the present method, the parameter values ​​must first be looked up in the LUT in order to be able to calculate the output value.

[0076] Advantages of the proposed solution According to the proposed solution, neuron outputs are calculated by dedicated neural processing hardware 15, and mathematical functions are applied to the neuron outputs using dedicated processing means 18. The hardware implementation of the LUT speeds up the lookup of parameter values ​​needed to calculate the mathematical function.

[0077] Such an architecture allows components 15, 17, and 18 to be individually optimized, as well as to enhance their intercommunication, thereby accelerating computation. For example, processing unit 18 can be configured as a proximal memory processing unit 18 "close" to device 15, accelerating transmission of neuron outputs beyond what can already be achieved by the neural processing device 15 (e.g., a crossbar array) implementing the neurons. In particular, parameter values ​​can be efficiently accessed by processing unit 18 from hardware-implemented lookup tables, resulting in substantially faster computation of function outputs. As a result, neuron outputs can be processed more quickly before being passed to the next layer of neurons.

[0078] It is important to understand that in this context, LUTs are not used to directly look up function outputs (as is typically done when using lookup tables), but rather to more efficiently access parameter values ​​needed to evaluate the function. In this way, even modest-sized LUTs are already capable of implementing a variety of functions (e.g., nonlinear activation functions, regularization functions). Because the LUT stores parameter values ​​instead of mapping input values ​​to output values, very little memory is required. Nevertheless, LUTs may in some cases be designed as reconfigurable tables, whereby mathematical functions may be dynamically reconfigured as computations progress or are updated (either during training or inference) if new types of functions are needed over time.

[0079] Furthermore, as mentioned above, the present approach is compatible with integration. That is, the LUT circuit 17, the processing units 18, and the neural processing unit 15 can be advantageously co-integrated in the same device. In particular, the LUT circuit 17 may be co-integrated in close proximity to the respective processing units 18. Thus, the hardware system 1 may consist of a single device (e.g., a single chip) co-integrating all necessary components. In this manner, the present system 1 may be conveniently used in a special-purpose infrastructure or network to process multiple and simultaneous client requests, as envisioned in FIG. 1.

[0080] All of this will now be explained in more detail with reference to a specific embodiment of the present invention. First, each processing unit 18 may advantageously be configured to obtain a mathematical function value (i.e., a second value) by first selecting all necessary parameters according to the value (first value) output by the neuron, and then searching for the corresponding parameter value accordingly. That is, the calculations performed to obtain the second value are based on the first value, on the one hand, and on the parameter values ​​of the associated parameter set selected according to the first value, and such parameter values ​​are efficiently searched for in the LUT. A set of suitable parameter values ​​can be initially determined at the time of construction. As will be understood, this can actually reduce the number of parameters required for each evaluation, since a small parameter set is already sufficient for locally accurate estimation of the function in the interval including the input value (first value). For example, linear polynomials (each requiring only two coefficients) can accurately fit the curve locally.

[0081] In this regard, with more specific reference to FIGS. 4 and 5, each processing unit 18 may be further configured to select an associated parameter set by comparing the first value (neuron output value) with bin boundaries. This allows the associated bin containing the first value to be identified. Next, the associated parameter set is selected according to the identified bin and then retrieved from the LUT. If necessary, the desired type of function (e.g., ReLU, softmax, binary, etc.) can be selected based on additional parameters. The bin boundaries are stored in the LUT along with the parameter values. Note that the bin boundaries can also be considered as parameters used to calculate a function. However, the function of such parameters is different in detail from the parameter values ​​(e.g., associated polynomial coefficients) used to calculate the function. The bin boundaries can also be efficiently retrieved to enable rapid comparison. In this way, the binning problem can be efficiently solved, thereby allowing the associated parameter set to be quickly identified.

[0082] For this purpose, each processing unit 18 preferably includes at least one comparator circuit 182 (see FIGS. 4 and 5) designed to compare the first value with a bin boundary. That is, a dedicated comparator circuit is used to efficiently identify the relevant bin. As seen in FIGS. 4 and 5, the comparator circuit is further designed to transmit a selection signal encoding the selected parameter set. Finally, the corresponding parameter value is retrieved by the transmitted selection signal. Note that the processing unit 18 may actually include multiple comparator circuits 182. For example, the processing unit 18 may include several processing elements 185 (as in FIG. 3), each of which may be designed according to FIG. 4 or 5, where each element 185a, 185b includes a comparator circuit. In a variant, the comparator circuit may be partially shared within the processing unit.

[0083] The selection signal may in particular be sent to a multiplexer 186 forming part of the processing element 185a, as in Figure 4. In a variant, the selection signal is passed to a LUT circuit, for example to an addressable memory 175b, as in Figure 5.

[0084] 4, the LUT circuits 175a are circuits for hard-coding parameter values. The multiplexers 186 are connected to the respective comparator circuits 182 so as to receive selection signals during operation. The multiplexers 186 are further connected to the LUT circuits 175a, so that the associated parameter values ​​can be selected during operation according to the received selection signals. Such a design makes parameter search very efficient. The drawback is that the hard-coded data cannot be changed after the circuits 175a are hard-wired. Therefore, it may be preferable to use a reconfigurable memory.

[0085] In that regard, the example shown in Figure 5 shows a LUT circuit 175b including an addressable memory unit 175b connected to a comparator circuit 182 to receive a selection signal during operation. In that case, the selection signal is sent directly to the memory 175b (in that case, no multiplexer is required). The LUT circuit 175b is separately configured to retrieve the parameter value of the associated parameter set selected according to the received selection signal. In the rest, the comparator circuit 182 may be equivalent to the circuit used in Figure 4. Figures 4 and 5 are further described in Chapter 2.

[0086] A simple binary tree comparison circuit may be relied upon. However, to speed up binning, more sophisticated comparison schemes and comparison circuit layouts are contemplated that allow for multiple levels of comparison. In particular, comparator circuit 182 may be advantageously configured as a multiple-level q-ary tree comparison circuit, where q is 3 or greater for one or more of the multiple comparison levels enabled by circuit 182, as shown in the table of FIG.

[0087] In detail, in a q-ary comparison tree, the number of comparator levels is log q (K), where K represents the total number of bins used, assuming there are q comparators per level. Now, as shown in Figure 8, the number of levels, the total number of comparators, and the number of comparators at each level can be jointly optimized. The table shows the optimal number of comparators that can be used at each level (row 2), the total number of comparators included (row 3), and the associated computational cost (row 4). The number of levels considered in Figure 8 (row 1) varies from 1 to 6, while the number of comparators per level varies from 1 to 63 (as does the total number of comparators). The number of levels is related to latency, with a higher number of levels resulting in longer latency. The total number of comparators also contributes to cost. Therefore, the total cost can be the number of levels times the total number of comparators, as done in Figure 8.

[0088] A q-ary tree comparison circuit enables q-1 comparators, such that the number of comparators used at each level corresponds to q-1. In principle, the optimal value of q is a cost function (q-1)Log that describes the trade-off between the number of levels and the total number of comparators. K (q) 2 The number q that minimizes * This corresponds to the lower bound of . Minimizing this function gives The resulting image is TIFF2025530542000002.tif7170, which corresponds to the three comparators.

[0089] However, the optimal number of comparators depends on the selected cost function and the number of levels. For example, the inventors performed extensive optimization based on a more sophisticated cost function, resulting in the optimal value shown in Figure 8. According to this optimization, the best is to rely on three comparison levels and a quadtree (i.e., enabling three comparators), with each level having the same number of comparators (i.e., 3).

[0090] As mentioned above, the applied mathematical function is advantageously constructed as a function piecewise defined by polynomials, each polynomial being applied to a different interval within the domain of the function. Thus, such a function (also called a spline) is a polynomial in each of its subdomains and can be mapped to bins. The polynomial coefficients can be adjusted to fit a given basis function (i.e., a theoretical function). Note that the polynomials in this context do not need to be continuous across bin boundaries (they may be, but this depends on the basis function). Thus, for any given first value (argument of the function), a related parameter set can be identified, corresponding to the polynomial parameters of the locally relevant polynomial. The corresponding parameter values ​​are then looked up in the LUT to estimate the output value of the function.

[0091] As will be appreciated, using such a construct lends itself to fast computation by the arithmetic unit, i.e., only arithmetic operations are required to achieve the desired result, where each processing unit 18 (or, indeed, each processing element 185, 185a, 185b) may include an arithmetic unit 188 connected to the LUT circuit 17 to perform the necessary arithmetic operations.

[0092] For example, an addressable memory unit may be used to store at least L×(K×(2+l)−1) parameter values, where L denotes the number of distinct functions performed by the processing unit (L≧1) and l denotes the interpolation order of each interpolating polynomial (l≧1). There are K bins and K−1 bin boundaries. Thus, if processing unit 18 can implement a total of L functions, each with K bins, and implement l-th order interpolation, the minimum number of parameters to be stored in memory is L×(l+1)×K+L×(K−1)=L×(K×(2+l)−1).

[0093] In principle, other constructs (other than ordinary splines) may be relied upon. For example, the applied function may be a B-spline or may include a Bézier curve. However, using splines (especially linear polynomials) allows for very efficient calculations. Moreover, in this case, the calculations can be performed simply using a sum-of-products circuit, as assumed in FIGS. 4 and 5. That is, the calculation unit 188 may simply consist of a sum-of-products circuit. The circuit 188 is specifically designed to perform sum-of-products operations that efficiently achieve the output value of the mathematical function. Note that, where l is the degree of the polynomial, it may be necessary to perform up to l operations. To the extent that similar (or identical) circuit techniques can be used in the neural processing unit 15, relying on the sum-of-products circuit 188 is also advantageous.

[0094] In practice, the neural processing device 15 preferably includes a crossbar array structure 15, i.e., a structure including N input lines 151 and M output lines 152, where N>1 and M>1, as shown in FIG. 2. The input and output lines are arranged in rows and columns interconnected at crosspoints (i.e., junctions) via memory elements 156. Each column corresponds to a neuron, allowing the device 15 to implement a layer of M neurons. Each of the output lines 152 is connected to at least one of the M' processing units 18. The output lines are typically connected to the processing units via readout circuitry 16, as shown in FIG. 2. Note that each output line may optionally be connected to a respective processing unit or processing element. However, preferably, the M neurons partially share the processing unit 18, as shown in FIG. 3.

[0095] The crossbar array structure 15 can be considered to define N×M cells 154, i.e., repeating units corresponding to the intersections of rows and columns. As is known per se, each row and each column may actually require multiple conductors. In a bit-serial implementation, each cell may be connected by a single physical line that serially supplies an input signal carrying an input word. However, in a parallel data acquisition approach, parallel conductors may be used to connect each cell; i.e., bits are injected into each of the cells in parallel via parallel conductors.

[0096] Each cell 154 includes a respective memory system 156 (see FIG. 2) consisting of at least one memory element 156. Thus, N×M cells include N×M memory systems 156, which are individually labeled a 11~ a 44 The memory systems 156 store weights corresponding to matrix elements used to perform matrix-vector multiplication (MVM). Each memory system 156 may include, for example, serially connected memory elements that store each bit of the weight stored in a corresponding cell. In that case, multiply-accumulate (MAC) operations are performed in a bit-serial manner. The memory elements may be, for example, static random access memory (SRAM) devices, although the crossbar structure may in principle comprise various types of electronic memory devices (e.g., SRAM devices, flash cells, memristive devices, etc.). Any type of memristive device is contemplated, such as phase-change memory cells (PCM), resistive random access memory (RRAM), and electrochemical random access memory (ECRAM) devices.

[0097] The memory elements may form part of a multiply-accumulate circuit (not shown in FIG. 2) to efficiently perform MAC operations, with each column including N multiply-accumulate circuits. Vectors are encoded as signals applied to the input lines of the crossbar array structure 15, which performs the MVM via MAC operations. While physically limited to M neurons, the structure 15 can nevertheless be used to map larger matrix-vector multiplications, as described above. If necessary, weights can be pre-read and stored in each cell (e.g., proactively, with multiple memory elements per cell) to speed up the MVM. Generally, MVM can be performed in the digital or analog domain. Implementations in the analog domain can exhibit better performance in terms of area and energy efficiency when compared to fully digital IMC. However, this typically comes at the expense of limited computational precision.

[0098] Another aspect of the invention will now be described with reference to the flow chart of Figure 6. This aspect relates to a method of operating a hardware system 1 as described above. The essential features of this method have already been described, even if implicitly, with reference to the first aspect of the invention. Such features will only be briefly described below.

[0099] According to the method, operating the hardware system 1 requires operating the neural processing unit 15, for example, as performed in steps S20 to S50 of the flow chart of FIG. 6. Generally speaking, operating the neural processing unit 15 results in M ​​first values ​​in each algorithm cycle (see step S40). Such values ​​are generated by M artificial neurons enabled by the unit 15, respectively. If the unit 15 has a crossbar array structure, an array (i.e., a vector) of input values ​​is encoded as input signals that are applied to input lines of the unit 15 to generate output signals at the M line outputs. Such output signals correspond to first signals, according to the terminology introduced above.

[0100] Next, one or more mathematical functions are applied to the first values ​​to obtain second values. That is, for each first value among the M first values ​​generated by the M neurons, an output value of the mathematical function is obtained via M' processing units 18 (steps S60-S110). As explained above, the mathematical function takes the first value as an argument. Furthermore, this function is determined, among other things, by a parameter set whose values ​​are accessed from a hardware-implemented LUT. Thus, each mathematical function is calculated based on operands including the first value and parameter values ​​of the parameter set, which are efficiently retrieved from one or more LUT circuits 17 (S100). Thus, each first value results in a second value, i.e., an output value of the mathematical function. In this way, M second signals encoding M output values ​​corresponding to the evaluation of the mathematical function are obtained (at each algorithm cycle).

[0101] As mentioned above, preferably, the mathematical function is evaluated by first selecting an associated parameter set according to the neuron output (first value) (S70-S80). Next, an operation is performed (S110) based on the first value and the parameter value retrieved according to the selected parameter set (S100). As further seen in FIG. 6, the parameter set is preferably selected by comparing the first value with bin boundaries to identify an associated bin (S70), whereby an associated parameter set can subsequently be selected according to the identified bin (S80). This is performed efficiently, given that the bin boundaries are retrieved from the LUT. Typically, several computational algorithm cycles are performed, as shown in FIG. 6. Each algorithm cycle begins with a calculation (i.e., MVM) performed by the neural processing unit 15.

[0102] A typical flow is as follows. This flow relies on a hardware system 1 as shown in FIG. 2, which performs inference based on an ANN, assumed here for simplicity to have a feedforward configuration. Furthermore, a device 15 is assumed to enable a sufficiently large number of M neurons to map any layer of this ANN. System 1 is provided in step S10. Specifically, it includes a crossbar array structure 15, a LUT circuit 17 (assumed to be a programmable memory circuit), and a proximity memory processing unit 18. The parameter values ​​required to evaluate the mathematical function are first determined in step S5 (at construction). The LUT is initialized accordingly in step S20. This corresponds to programming the LUT circuit to store the appropriate parameter values ​​(S20). If necessary, the LUT circuit may be reprogrammed later to update the function. Apart from the LUT circuit 17, matrix coefficients (i.e., weights) must be initialized in the crossbar array 15 to configure it as the first neural layer of the ANN to be executed (S30).

[0103] Next, the input unit 11 of the system 1 applies the currently selected input vector to the crossbar array 15 to perform MVM (S40). Signals are acquired at the outputs of M columns of the crossbar 15. The acquired signals encode neuron output values ​​(or first values). The corresponding values ​​are read by the dedicated circuitry 16 and passed to the processing unit 18 (S60). A comparator circuit in the unit 18 compares the neuron output values ​​with bin boundaries (S70) to identify the associated bin. The corresponding selection signal is then forwarded by the comparator circuit to the LUT circuit (S80), which retrieves the associated parameter value from the LUT (S100). This allows for efficient calculation of output values ​​for one or more mathematical functions (e.g., activation functions) (S110), which is preferably performed by the sum-of-products circuit 188.

[0104] This process is repeated for each successive layer. If the current neural layer is the last layer (S120 is "yes"), the algorithm may return the last function value to form an inference result (S130). Otherwise (S120 is "no"), the function value obtained in step S110 can be passed to a further processing unit (e.g., a digital processing unit) if necessary (S140). That is, the result of step S110 may have to be passed to a digital processing unit that performs an operation that cannot be performed by the crossbar array 15 or the processing unit 18 (S140). For example, a digital processing unit may be included to perform a max-pooling operation. The value obtained at the output of step S110 (or S140) is then sent to an input unit of the same crossbar array 15 or another cascaded crossbar array 15 (S150). That is, the next input vector is formed, and another algorithm cycle S40-S150 begins. In parallel with steps S60 to S140, new matrix coefficients may be stored in the (next) crossbar array (S50) and configured as the next neural layer.

[0105] The mathematical functions involved are preferably constructed as piecewise-defined polynomial functions for the reasons mentioned above. In that case, each parameter set contains two or more polynomial coefficients, and the only operations required are arithmetic operations (S110). This is done efficiently thanks to the product-sum circuit 188, allowing the same techniques used in the neural processing unit 15 to be reused.

[0106] For example, each mathematical function may be evaluated over the range of interest using a set of linear polynomials, each mapped to a respective bin. If linear polynomials are used, each parameter set may consist of a scale factor and an offset factor, respectively (although the polynomials may be defined in different ways).

[0107] The degree of the polynomial to be used may be selected in step S5. This preliminary step S5 may further include determining a set of suitable bin boundaries, along with the corresponding parameter values, for each criterion function, i.e., a function of potential interest to the ANN. Various methods are possible. In general, determining suitable bin boundaries can be viewed as an optimization problem. Suitable bin boundaries are typically determined (for each criterion function) by minimizing the number of bins (given the maximum error allowed at any point), or by minimizing the maximum error between the approximation of the criterion function (calculated based on the parameter values) and the theoretical value of the criterion function (given a predetermined number of bins to be used) (S5). A joint optimization may also be performed to optimize both the number of bins and the maximum error. A detailed description and examples are provided in Section 2.

[0108] Various types of reference mathematical functions can be considered, regardless of the construct used to estimate them. The LUT circuit 17 can enable a variety of mathematical functions routinely required in ANN computations, such as activation functions, normalization functions, reduction functions, state update functions, as well as analytical classification, prediction, or other inference-like functions.

[0109] Activation functions are an important category of functions because applying such functions to neuron outputs is nearly always required. In particular, nonlinear activation functions such as the so-called binary step, sigmoid, Tanh, ReLU, LeakyReLU, and softmax (i.e., normalized exponential) functions may be used. Specific normalization functions may be required (e.g., batch normalization, layer normalization, etc.). Furthermore, as mentioned above, the applied mathematical function may be any analytical function (classification, prediction) used to perform inference. However, in certain cases, the mathematical function may be bypassed or configured as an identity function.

[0110] The parameter values ​​retrieved from the LUT are often sufficient for processing unit 18 to calculate the complete function output. However, in other cases, additional (external) processing may be required, for example to calculate the sum of exponents required in a softmax function. Furthermore, other types of operations may have to be performed, such as reduction operations on a set of values ​​or arithmetic operations between such values. For completeness, the LUT may also store values ​​that can be used by system 1 to perform other tasks, such as those of a support vector machine algorithm.

[0111] The above embodiments have been briefly described with reference to the accompanying drawings and may be adapted to several variants. Several combinations of the above features are possible. Examples are given in the following paragraphs.

[0112] 2. Specific Embodiments - Technical Implementation Details 2.1 Bin Boundary Determination and Interpolation The LUT preferably takes an input value x, a vector of bin boundaries b b Implements a piecewise polynomial function such that, given a vector of coefficients c, the output o of the function is generally calculated as o=f(x,c[i]), where i is x>b b [i-1], x ≤ b b [i]. A suitable methodology for generating LUT values ​​should ideally allow for a simple definition of the parameters used to calculate the desired function. Furthermore, the methodology should advantageously support various types of approximations to the function.

[0113] Below we describe several methods that can be applied to approximate a function as a piecewise polynomial (spline). In the simplest case, the interpolation is linear and the function is defined by two parameters: a scale factor and an offset factor.

[0114] 7A-7C show a method for binning by linear interpolation of the GELU function. The plain (continuous) line represents the criterion function, and the thick (striped) curve represents the interpolated version approximated using given parameter values. The dark dots represent the optimal bin boundaries.

[0115] The following examples of the method differ mainly in the way the bins are obtained. They all have in common that they attempt to minimize the error between a function and its interpolated version. However, they differ in the way this minimization is performed. Some of the proposed approaches try to minimize the number of bins given the maximum error at any point, while others try to minimize the error given the number of bins used.

[0116] For example, the following method minimizes the number of bins given the maximum error at any point: The underlying algorithm proceeds as follows: (i) The target interval of the mathematical function is defined by the user, and a pointer is assigned to point to this interval. (ii) Subsequently, an interpolation of the points defining the limits of the interval is carried out. (iii) The error between the value of the original function and the value of the interpolated curve at the center of the interval (pointed to by the pointer) is then calculated. (iv) If the error is greater than or equal to the user-defined tolerance, the pointed interval is divided in half and the pointer is moved to the interval on the left after the division. Otherwise, the pointer is moved to the next interval on the right. If there are no more intervals to the right, the algorithm stops; otherwise, it returns to step (ii).

[0117] The comments are in order. For symmetric and antisymmetric functions, the algorithm is applied to the portion of the function that lies to the left or right of the axis of symmetry or antisymmetricity, and the bin values ​​are calculated by mirroring the bins according to the symmetric / antisymmetric pattern. Heuristics may be relied upon to automatically determine the limits of the interval of interest. The interval of interest may also be initially divided into a fixed number of equally spaced bins. In that case, the above algorithm may be applied to each of the initial bin boundaries.

[0118] Assuming linear interpolation is desired, each linear portion requires a slope coefficient (slp) and an offset coefficient (off). Then, a particular output o of a function over a particular subdomain corresponding to a given input x can be estimated as o = slp[i] × x + off[i], where i denotes an integer corresponding to the input x, e.g., i|x>b b [i-1], where x≦b b [i] and b b Here again, represents the bin boundary vector.

[0119] The above binning technique is illustrated using an example where a Gaussian Error Linear Unit (GELU) function is approximated (see FIG. 7A). First, the number of points, n i is determined (or estimated) (Figure 7A), where n i ≧2. Such a point is n i For simplicity, we will assume that only two points are determined initially. Bins are only needed during the nonlinear part of the function. Therefore, the algorithm may first determine the interval where the function is nonlinear. This results in a single interval in the example of Figure 7A. The next step consists of binning only within the nonlinear region. To that end, the algorithm measures the error at the center of the bin and, if the error exceeds the tolerance, can divide this interval into two equal subintervals. This is shown in Figure 7B, where additional points are indicated. The same operation can then be repeated until a suitable number of intervals is achieved, resulting in an acceptable interpolation error. Finally, each interval is assigned an optimal triplet of parameter values. For example, the first interval is bb [0], slp[0], off[0], and the second section corresponds to b b [1], slp[1], off[1], etc. Each set of parameter values ​​is stored in a LUT for later lookup at run time.

[0120] Another method is to calculate the maximum error (E max ), the goal is to minimize the number of bins used (constrained optimization). The algorithm proceeds as follows: (i) The target interval of the function is redefined by the user. (ii) The algorithm starts from one boundary (either left or right) of the interval of interest and considers it to be one of the boundaries of the first bin. In the following, the left boundary is taken as the starting point. In this case, the left boundary of the interval of interest is also the left boundary of the first bin. The initial value (defined via delta) of the other bin boundary (in this case the right bin boundary) is imposed by the user. (iii) Linear interpolation of points within a bin is performed by comparing the interpolating function with the function to be interpolated within a given bin (E bin_max ) is chosen so that the maximum error between it and its left and right boundaries, as well as the center of the bin (of exactly opposite sign). This can be achieved by interpolating the points that define the bin boundaries. The line is then shifted by half the error at the center of the bin, which is E bin_max is equal to. (iv)E bin_max <E max If so, the bin is increased by the bin's initial value (delta) (by moving its right boundary to the right) and step (iii) is repeated to obtain the new E bin_max Then, step (iv) determines E bin_max ≧E max This is repeated until (v) The algorithm is |E max -E bin_maxCheck whether |<epsilon, where epsilon is user-defined. If so, the algorithm proceeds to step (vi). Otherwise, move the right boundary left by delta / 2 and repeat step (iii) to get a new E bin_max Then, each time the boundary is moved, the condition |E max -E bin_max |< Until epsilon is true, E max -E bin_max Step (v) is repeated in the direction determined by the sign of ∇ ∇ (to the right if positive, to the left if negative). The algorithm then proceeds to step (vi). (vi) The algorithm then repeats the same procedure for the next bin, starting from step (ii). (vii) The algorithm continues until all bins are deterministic. Finally, the function can be estimated by interpolation according to the parameter values ​​obtained for each of the bins.

[0121] In steps (iv) and (v), |E max -E bin_max Note that additional methods can be used to find the bin size that tests |< epsilon. Additionally, the limits of the interest interval can be calculated automatically using heuristics.

[0122] Various other algorithms are contemplated. For example, a non-iterative variation of the above algorithm may be devised that determines the bins in a single step. This variation minimizes the error given the number of bins to be used and can include any interpolating polynomial. Given the desired number of bins and the degree d of the interpolating polynomial, the algorithm approximates the d-th derivative of the function. The optimal size of each bin is inversely proportional to the rate of change of the highest-order coefficient and, therefore, the rate of change of the d-th derivative of the function, which is actually the (d+1)-th derivative of the function. The optimal bin is identified based on a cumulative sum of the (d+1)-th derivative of the function calculated at several sampling points (much more than the number of bins).

[0123] Further approaches may be based on neural networks trained to minimize either the number of bins or the maximum error in each bin.

[0124] 2.2 Preferred Hardware Implementation In the following, a hardware implementation assuming linear interpolation is described. In this case, two classes of embodiments are considered. The first class relates to a predetermined (predefined) function implementation, which uses K-1 bin boundaries along with K scaling coefficients and K offset coefficients. Such numbers remain constant during runtime. The typically small number of required coefficients does not require addressable memory and can instead be hard-coded in the LUT circuit 175a (FIG. 4), similar to a ROM circuit. The priority network of the comparator 182 provides a selection signal, which is fed to the multiplexer 186. The multiplexer 186 can thus select the optimal scale and offset parameter values. The multiply-accumulate unit 188 can be implemented, for example, as two separate units (for multiplication and addition) or as a fused multiply-accumulate unit. The value bin.b i (1≦i≦K−1) are the optimal bin boundary values ​​(optimal vector components b b [i] (see previous subsection). i and offs i refers to the scale and offset coefficients that are also hard-coded in circuit 174a.

[0125] Conversely, if an arbitrary function implementation is desired, an addressable memory can be used to store all the necessary bin boundaries, as well as the scaling and offset coefficients, as assumed in FIG. 5. This memory can be reprogrammed. Such an embodiment also includes the comparator priority network 182 and the multiply-accumulate unit 188, as in FIG. 4. However, addressable memory 175b is used to store the bin boundaries, as well as the scaling and offset coefficients for each desired function. In that case, the memory also selects the optimal scale and offset parameter values ​​via its output decoder. Value bin.b i , scl i and offs i refers to the bin boundary values, scale factors, and offset factors, as in Figure 4. However, in that case, an additional value ("bin.b.bits") corresponding to (K-1) times the number of bits used to store the bin boundaries is required. Furthermore, the addresses of the bin boundaries (bin.b.addresses) must be passed to memory so that the memory can retrieve the corresponding values.

[0126] The circuit 175a and memory unit 175b shown in FIGS. 4 and 5 can be mapped to processing elements 185 as shown in FIG. 3. The neural processing unit 15 is preferably embodied as a crossbar array 15 (FIG. 2). Preferably, all components and devices required for system 1 are co-integrated on the same chip, as envisioned in FIG. 2. Thus, system 1 may be assembled into a single device including the crossbar array structure 15, the LUT circuit 17, and the processing unit 18, which is preferably arranged as a nearby memory-processing unit. In addition, device 1 may include an input unit 11 for applying input signals encoding input vector components to the crossbar array 15. Furthermore, device 1 typically includes a readout circuit 16 and an I / O unit 19 for interfacing system 1 to an external computer (not shown in FIG. 2).

[0127] FIG. 1 illustrates a network 5 including several systems 1 (e.g., integrated devices such as those shown in FIG. 2). That is, systems 1 form part of a larger computer system 5 that includes a server 2 interacting with clients 4, which may be natural persons (interacting via personal computers 3), processes, or machines. Each hardware system 1 is configured to read data from and write data to memory units of the server computer 2 in this example. Client requests are managed by units 2, which may be configured, among other things, to map a given computing task to vectors and weights, which are then passed to system 1. The entire computer system 5 may be configured, for example, as a composable distributed infrastructure that may further include other hardware acceleration devices, such as application-specific integrated circuits (ASICs) and / or field-programmable gate arrays (FPGAs).

[0128] Of course, many other architectures are possible. For example, the system 1 may be configured as a stand-alone system or as a computerized system connected to one or more general-purpose computers. The system 1 may be used in particular in distributed computing systems, such as edge computing systems.

[0129] 3. Final findings Computerized devices and systems 1 can be designed to implement embodiments, including methods, of the present invention described herein. In this regard, the methods described herein can be understood to be non-interactive in nature, i.e., automated. The automated portions of such methods can be implemented solely in hardware or as a combination of hardware and software. In exemplary embodiments, the automated portions of the methods described herein are implemented in software as services or executable programs (e.g., applications), with the executable programs executed by a suitable digital processing device. However, all embodiments described herein include computational steps performed by non-traditional hardware, such as hardware-implemented LUTs, neural processing units, such as crossbar array structures, and separate processing units, preferably located as a memory processing unit adjacent to the neural processing unit.

[0130] Furthermore, the methods described herein may further comprise executable programs, scripts, or more generally, executable instructions in any form. For example, the necessary computer-readable program instructions may be downloaded to the processing elements from a computer-readable storage medium, for example over a network such as the Internet.

[0131] Aspects of the present invention are described herein with particular reference to flowcharts and block diagrams. It will be understood that each block or combination of blocks in the flowcharts and block diagrams can be implemented by computer-readable program instructions. The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of System 1, as well as methods of operating possible implementations of System 1, according to various embodiments of the present invention.

[0132] While the present invention has been described with reference to a limited number of embodiments, variations, and accompanying drawings, those skilled in the art will recognize that various modifications can be made and equivalents substituted without departing from the scope of the invention. In particular, features (such as apparatus or methods) listed in a given embodiment, variation, or illustrated in a drawing may be combined with or substituted for other features of other embodiments, variations, or drawings without departing from the scope of the invention. Accordingly, various combinations of the features described in connection with any of the above embodiments or variations are contemplated which remain within the scope of the appended claims. Moreover, many minor modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from the scope of the invention. Therefore, the invention is not limited to the particular embodiments disclosed, but is intended to include all embodiments falling within the scope of the appended claims. Moreover, many other variations other than those expressly mentioned above are contemplated. For example, other types of memory element devices, selection circuits, LUT circuits, and processing units are contemplated.

Claims

1. A hardware system (1) designed to implement an artificial neural network, comprising: a neural processing unit (15) configured to implement M artificial neurons, where M≧1; one or more lookup table circuits (17) configured to implement a lookup table; M' processing units (18), where M≧M′≧1, each processing unit of the M′ processing units: a first value output by each of the at least one of the M artificial neurons; connected to a lookup table circuit (175, 175a, 175b) of said one or more lookup table circuits (17) for accessing parameter values ​​of the parameter set from said lookup tables; configured to output a second value corresponding to a value of a mathematical function having the first value as an argument, the mathematical function being determined by the parameter set, and parameter values ​​of the parameter set being accessed by each of the processing units from the look-up table circuit (175, 175a, 175b) during operation; M' processing units (18); A hardware system (1) comprising:

2. Each of the processing units (18) selecting the parameter set according to the first value; performing an operation based on the first value and the parameter values ​​of the selected parameter set to output the second value; The hardware system (1) of claim 1, configured to output the second value by:

3. Each of the processing units (18) accessing the bin boundaries of the bins from said lookup table circuit (175, 175a, 175b); The parameter set is selected by comparing the first value with the accessed bin boundaries to identify a given bin of bins that includes the first value, whereby the parameter set is selected according to the given bin during operation. The hardware system (1) of claim 2, further configured to:

4. Each of the processing units (18) includes at least one comparator circuit (182), the at least one comparator circuit (182) comprising: comparing the first value to the bin boundaries; transmitting a selection signal encoding a selected set of the parameters and causing the processing unit (18) to access the corresponding parameter values ​​based on the transmitted signal; It is designed to A hardware system (1) according to claim 3.

5. the comparator circuit (182) is configured as a multi-level q-ary tree comparison circuit designed to allow for multiple levels of comparison, where q is 3 or greater for one or more of the multiple levels; A hardware system (1) according to claim 4.

6. the lookup table circuit (175a) is a circuit for hard-coding parameter values; Each of the processing units (18) includes at least one multiplexer (186), the at least one multiplexer (186) a comparator circuit (182) connected to each of the at least one comparator circuits (182) to receive the selection signal; connected to the look-up table circuit (175a) to retrieve the corresponding parameter value according to the selection signal; A hardware system (1) according to claim 4.

7. The look-up table circuit (175b) includes an addressable memory unit, the addressable memory unit comprising: connected to the comparator circuit (182) to receive the selection signal; configured to retrieve the parameter values ​​of the selected parameter set according to the received selection signal; A hardware system (1) according to claim 4.

8. the mathematical function is a piecewise-defined polynomial function that is polynomial in each of its subdomains, each of the subdomains corresponding to a bin; the selected parameter set corresponds to polynomial parameters of the piecewise-defined polynomial function; each said processing unit including an arithmetic unit (188) connected to an output of said look-up table circuit and designed to perform said operation to calculate said second value as an arithmetic operation; A hardware system (1) according to claim 4.

9. The arithmetic unit (188) includes a multiply-accumulate circuit; A hardware system (1) according to claim 8.

10. the one or more lookup table circuits (17) include M' separate circuits (175) respectively mapped to the M' processing units (18), where M≧M'>1; A hardware system (1) according to any one of claims 1 to 9.

11. the neural processing unit (15) comprises a crossbar array structure (15) including N input lines (151) and M output lines (152) arranged in rows and columns such that the neural processing unit (15) implements a layer of M neurons, where N>1 and M>1; the input lines and the output lines are interconnected via a memory element (156); each of the M output lines (152) is connected to at least one of the M' processing units (18); A hardware system (1) according to any one of claims 1 to 9.

12. the one or more look-up table circuits (17), the processing unit (18) and the neural processing unit (15) are co-integrated on the same chip (10); A hardware system (1) according to any one of claims 1 to 9.

13. M>1, preferably M≧256, more preferably M≧512; M′=1 or 2; each of the M' processing units is a vector processing unit (18) including b processing elements for operating on a one-dimensional array of dimension b; A hardware system (1) according to any one of claims 1 to 9.

14. A method of operating a hardware system (1), comprising: a neural processing unit (15) configured to implement M artificial neurons, where M≧1; M' processing units (18), each connected by at least one of the M artificial neurons; one or more lookup table circuits (17) for implementing a lookup table; (S10) providing the hardware system (1) including: Operate the neural processing device (15) (S20 to S50) to obtain M first values ​​generated by the M artificial neurons (S40); via the M' processing units (18), for each first value of the M first values, obtaining (S60-S110) an output value of a mathematical function that takes the first value as an argument and is otherwise determined by a parameter set based on operands including the first value and parameter values ​​of the parameter set, wherein the parameter values ​​are retrieved (S100) from the one or more look-up table circuits (17); A method comprising:

15. For each of the first values, the output value is: Selecting the parameter set according to the first value (S70-S80); performing an operation (S110) based on the first value and the parameter values ​​retrieved (S100) according to the selected parameter set; The method of claim 14, wherein the method is obtained by

16. The parameter set is comparing said first value with bin boundaries of bins to identify a given bin of said bins that contains said first value (S70), said bin boundaries being retrieved (S70) from said one or more look-up table circuits (17); selecting the parameter set according to the identified given bin (S80); The method of claim 15, wherein the compound is selected by:

17. Before operating the neural processing device (15) (S20 to S50), the method determining one or more sets of bin boundaries respectively according to one or more criterion functions (S5); 17. The method of claim 16, further comprising:

18. the bin boundaries of each of the sets are determined for each criterion function of the one or more criterion functions so as to minimize a maximum error between approximations of each criterion function calculated based on the number of bins or parameter values ​​of the criterion function and theoretical values ​​(S5); 18. The method of claim 17.

19. the mathematical function is a piecewise-defined polynomial function, the parameter set includes two or more polynomial coefficients, and the operation performed comprises an arithmetic operation.

17. The method of claim 16.

20. the mathematical function includes a set of linear polynomials, each corresponding to a respective one of the bins, and the set of parameters corresponding to each of the linear polynomials consists of a scale factor and an offset factor.

20. The method of claim 19.

21. The arithmetic operation is performed by a product-sum circuit (188) (S110).

20. The method of claim 19.

22. The method includes programming (S20) the one or more lookup table circuits that implement the lookup table (17) to calculate the mathematical function as: activation function, normalization function, reduction function, state update function, Classification functions, and Prediction Function Enable it as one of 22. The method of any one of claims 14 to 21, further comprising:

Images