Systems and methods using neural networks

Abstract

A neural network-based system comprising at least one neural network processing core, an activation memory, an instruction memory, and at least one control register, the neural network processing core adapted to implement neural network computation, control, and communication primitives, a memory map including regions corresponding to each of the activation memory, the instruction memory, and the at least one control register, and an interface operatively connected to the neural network processor system, the interface adapted to communicate with a host and to expose the memory map.

Description

[Technical Field]

[0001] Embodiments of the present disclosure relate to systems for neural inference, and more particularly to memory-mapped neural network accelerators for deployable inference systems. Summary of the Invention

[0002] According to an embodiment of the present disclosure, a system includes a neural network processor system comprising at least one neural network processing core, an activation memory, an instruction memory, and at least one control register, the neural network processing core adapted to implement neural network computation, control, and communication primitives; a memory map comprising regions corresponding to each of the activation memory, the instruction memory, and the at least one control register; and an interface operatively connected to the neural network processor system, the interface further exposing the memory map for communication with a host. (expose: to make public) A method and a computer program for a system comprising an interface adapted for such a purpose are provided.

[0003] According to embodiments of the present disclosure, a neural network processor system is configured to receive a neural network description via an interface, receive input data via the interface, and provide output data via the interface. In some embodiments, the neural network processor system exposes an API via the interface. (Published) The API includes methods for receiving a neural network description via an interface, receiving input data via the interface, and providing output data via the interface. In some embodiments, the interface includes an AXI, PCIe, USB, Ethernet, or Firewire interface.

[0004] In some embodiments, the system further comprises a redundant neural network processing core, the redundant neural network processing core configured to compute the neural network model in parallel with the neural network processing core. In some embodiments, the neural network processor system is configured to provide redundant computation of the neural network model, or to provide at least one of hardware, software, and model level redundancy, or both. In some embodiments, the neural network processor system comprises programmable firmware, the programmable firmware configurable to process input data and output data. In some embodiments, the processing comprises buffering. In some embodiments, the neural network processor system comprises non-volatile memory. In some embodiments, the neural network processor system is configured to store configuration or operational parameters or program states. In some embodiments, the interface is configured for real-time or faster than real-time operation. In some embodiments, the interface is communicatively coupled to at least one sensor or camera. In some embodiments, the system comprises a plurality of systems as described above interconnected by a network. In some embodiments, a system is provided comprising a plurality of systems as described above and a plurality of computational nodes interconnected by a network. In some embodiments, the system further comprises a plurality of disjoint memory maps, each memory map corresponding to one of a plurality of systems as described above.

[0005] According to another aspect of the present disclosure, there is provided a method, the method including receiving a neural network description in a neural network processor system from a host via an interface, the neural network processor system comprising at least one neural network processing core, an activation memory, an instruction memory, and at least one control register, the neural network processing core adapted to implement neural network computation, control, and communication primitives, the interface being operatively connected to the neural network processor system, the method further including exposing a memory map via the interface, the memory map including regions corresponding to each of the activation memory, the instruction memory, and the at least one control register, the method further including receiving input data in the neural network processor system via the interface, computing output data from the input data based on a neural network model, and providing output data from the neural network processor system via the interface. In some embodiments, the neural network processor system receives the neural network description via the interface, receives the input data via the interface, and provides the output data via the interface. In some embodiments, the neural network processor system exposes an API through an interface, the API including methods for receiving a neural network description through the interface, receiving input data through the interface, and providing output data through the interface, in some embodiments, the interface operates at real-time or faster than real-time rates. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 illustrates an exemplary memory-mapped (MM) system according to an embodiment of the present disclosure. [Figure 2] 1 illustrates an exemplary message passing (MP) system according to an embodiment of the present disclosure. [Figure 3] FIG. 1 illustrates a neural core according to an embodiment of the present disclosure. [Figure 4] FIG. 1 illustrates an exemplary inference processing unit (IPU) according to an embodiment of the present disclosure. [Figure 5] FIG. 1 illustrates an exemplary multi-core inference processing unit (IPU) according to an embodiment of the present disclosure. [Figure 6] FIG. 1 illustrates a neural core and associated network according to an embodiment of the present disclosure. [Figure 7] FIG. 1 illustrates a method of integration between a host system and an IPU, according to an embodiment of the present disclosure. [Figure 8] 1A-1C illustrate an exemplary method of redundancy according to an embodiment of the present disclosure. [Figure 9] FIG. 1 illustrates a system architecture of a memory-mapped neural inference engine according to an embodiment of the present disclosure. [Figure 10] FIG. 1 illustrates an exemplary runtime software stack according to an embodiment of the present disclosure. [Figure 11] FIG. 1 illustrates an exemplary series of executions according to an embodiment of the present disclosure. [Figure 12] FIG. 1 illustrates an exemplary integration of a neural reasoner according to an embodiment of the present disclosure. [Figure 13] FIG. 1 illustrates an exemplary integration of a neural reasoner according to an embodiment of the present disclosure. [Figure 14] FIG. 1 illustrates an exemplary configuration in which a neural inference device is interconnected with a host via a PCIe bridge, according to an embodiment of the present disclosure. [Figure 15] 1 is a flowchart of a method for exposing a memory map in a neural network processor system, according to an embodiment of the present disclosure. [Figure 16] FIG. 2 illustrates a computing node according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0007] Many conventional computing systems communicate between system components via a shared memory / memory-mapped (MM) paradigm. In contrast, many parallel and distributed computing systems, such as neurosynaptic systems, communicate with each other via a message-passing (MP) paradigm. This disclosure provides an efficient interface between these two types of systems.

[0008] An artificial neuron is a mathematical function whose output is a nonlinear function of a linear combination of its inputs. Two neurons are connected if the output of one is an input to the other. A weight is a scalar value that encodes the strength of the connection between the output of one neuron and the input of another neuron.

[0009] A neuron calculates its output, called activation, by applying a nonlinear activation function to a weighted sum of its inputs. A weighted sum is an intermediate result calculated by multiplying each input by its corresponding weight and accumulating the products. A partial sum is a weighted sum of a subset of the inputs. A weighted sum of all inputs can be calculated in stages by accumulating one or more partial sums.

[0010] A neural network is a collection of one or more neurons. Neural networks are often divided into groups of neurons called layers. A layer is a collection of one or more neurons that all receive input from the same layer and all send output to the same layer, usually performing a similar function. An input layer is a layer that receives input from sources outside the neural network. An output layer is a layer that sends output to targets outside the neural network. All other layers are intermediate processing layers. A multilayer neural network is a neural network with more than one layer. A deep neural network is a multilayer neural network with many layers.

[0011] A tensor is a multidimensional array of numbers. A tensor block is a contiguous subarray of elements in a tensor.

[0012] Each neural network layer is associated with a parameter tensor V, a weight tensor W, an input data tensor X, an output data tensor Y, and an intermediate data tensor Z. The parameter tensor contains all of the parameters that control the neuron activation function σ in the layer. The weight tensor contains all of the weights that connect inputs to the layer. The input data tensor contains all of the data that the layer computes as input. The output data tensor contains all of the data that the layer computes as output. The intermediate data tensor contains any data that the layer generates as an intermediate result, such as partial sums.

[0013] The data tensors for a layer (input, output, and intermediate) can be three-dimensional, with the first two dimensions interpreted as encoding spatial location and the third dimension interpreted as encoding a different feature. For example, if the data tensor represents a color image, the first two dimensions encode the vertical and horizontal coordinates within the image, and the third dimension encodes the color at each location. Each element of the input data tensor X can be connected to each neuron by a separate weight, resulting in a total of six dimensions for the weight tensor W, concatenating the three dimensions of the input data tensor (input row a, input column b, input feature c) with the three dimensions of the output data tensor (output row i, output column j, output feature k). The intermediate data tensor Z has the same shape as the output data tensor Y. The parameter tensor V concatenates the three output data tensor dimensions with an additional dimension o, which indexes the parameters of the activation function σ. In some embodiments, the activation function σ does not require any additional parameters, in which case no additional dimensions are required, although in some embodiments the activation function σ requires at least one additional parameter that appears in dimension o.

[0014] The elements of the layer's output data tensor Y can be calculated as in Equation 1, the neuron activation function σ is composed of a vector of activation function parameters V[i,j,k,:], and the weighted sum Z[i,j,k] can be calculated as in Equation 2. Y[i,j,k]=σ(V[i,j,k,:];Z[i,j,k]) formula 1

number

[0015] For simplicity of notation, the weighted sum in Equation 2 may be referred to as the output, and is equivalent to using a linear activation function Y[i,j,k]=σ(Z[i,j,k])=Z[i,j,k], it being understood that, without loss of generality, a similar statement applies when a different activation function is used.

[0016] In various embodiments, the computation of the output data tensor as described above is decomposed into smaller problems, each of which may then be solved in parallel on one or more neural cores, or on one or more cores in a conventional multi-core system.

[0017] Naturally, from the above, neural networks are parallel structures. Neurons in a given layer receive elements x from one or more layers or other inputs. i Each neuron receives an input X with elements w i The neural network computes its state y∈Y based on a weight W with a weight b. In various embodiments, the weighted sum of the inputs is adjusted by a bias b, and then the result is passed to the nonlinearity F(·). For example, a single neuron activation can be calculated as y=F(b+Σx i w i )

[0018] Because all neurons in a given layer receive input from the same layer and compute their outputs independently, neuron activations can be computed in parallel. Due to the overall neural network aspect, performing computations on distributed cores in parallel accelerates the overall computation. Furthermore, within each core, vector operations can be computed in parallel. Even in the case of a recurring input, such as when a layer projects back onto itself, all neurons are still updated simultaneously. In effect, the recurring connection is delayed to align with subsequent inputs to the layer.

[0019] Referring to FIG. 1, an exemplary memory-mapped system 100 is shown. Memory map 101 is segmented, with regions 102-105 allocated to various system components. Computational cores 106-109, e.g., processor cores on one or more chips, are connected to a bus 110. Each core 106-109 is connected to bus 110 and can communicate with each other via shared memories 111-112, which correspond to addressable regions of memory maps 102-103. Each core 106-109 can communicate with a subsystem 113 via addressable region 104 of memory map 101. Similarly, each core 106-109 can communicate with an external system 114 via addressable region 105 of memory map 101.

[0020] Memory Map (MM) addresses are relative to the global memory map and in this example go from 0x00000000 to 0xFFFFFFFF.

[0021] 2, an exemplary message passing (MP) system 200 is shown. Each of multiple cores 201-209 includes a computational core 210, a memory 211, and a communication interface 212. Each of the cores 201-209 is connected by a network 213. The communication interface 212 includes an input buffer 214 and an output buffer 215 for injecting and receiving packets from the network 213. In this manner, the cores 201-209 can communicate with each other by exchanging messages.

[0022] Similarly, subsystem 216 may be connected to network 213 via communication interface 217 having input buffer 218 and output buffer 219. An external system may be connected to network 213 via interface 220. In this manner, cores 201-209 may communicate with subsystems and external systems by exchanging messages.

[0023] Message passing (MP) addresses refer to network addresses that are local to a core. For example, an individual core can be identified by its X,Y location on the chip, while local addresses can be used for buffers or memory that are local to the individual core.

[0024] Referring now to FIG. 3, a neural core according to an embodiment of the present disclosure is shown. Neural core 300 is a tileable computational unit that computes one block of output tensors. Neural core 300 has M inputs and N outputs. In various embodiments, M=N. To compute an output tensor block, the neural core multiplies M×1 input tensor blocks 301 by M×N weight tensor blocks 302 and accumulates the products into a weighted sum, which is stored in a 1×N intermediate tensor block 303. The O×N parameter tensor block contains O parameters that specify each of the N neuron activation functions to be applied to the intermediate tensor blocks 303 to generate a 1×N output tensor block 305.

[0025] The multiple neural cores can be tiled into a neural core array, which in some embodiments is two-dimensional.

[0026] A neural network model is a set of constants that collectively specify the entire computation performed by a neural network, including the graph of connections between neurons and weights and per-neuron activation function parameters. Training is the process of modifying the neural network model to perform a desired function. Inference is the process of applying the neural network to inputs to generate outputs without modifying the neural network model.

[0027] An inference processing unit is a type of processor that performs neural network inference. A neural inference chip is a specific physical instance of an inference processing unit.

[0028] Referring to FIG. 4, an exemplary inference processing unit (IPU) according to an embodiment of the present disclosure is shown. The IPU 400 includes a memory 401 for a neural network model. As described above, the neural network model may include synaptic weights for the neural network to be calculated. The IPU 400 includes an activation memory 402, which may be transient. The activation memory 402 may be divided into an input domain and an output domain and stores neuron activations for processing. The IPU 400 includes a neural computation unit 403 that loads the neural network model from the model memory 401. Input activations are provided from the activation memory 402 before each computation step. Output from the neural computation unit 403 is written back to the activation memory 402 for processing in the same or another neural computation unit.

[0029] In various embodiments, the IPU 400 includes a micro-engine 404. In such embodiments, all operations in the IPU are directed by the micro-engine. As described below, various embodiments may provide a central micro-engine, a distributed micro-engine, or both. A global micro-engine may be referred to as a chip micro-engine, and a local micro-engine may be referred to as a core micro-engine or a local controller. In various embodiments, a micro-engine comprises one or more micro-engines, microcontrollers, state machines, CPUs, or other controllers.

[0030] Referring to FIG. 5, a multi-core inference processing unit (IPU) according to an embodiment of the present disclosure is shown. The IPU 500 includes memory 501 for neural network models and instructions. In some embodiments, the memory 501 is divided into a weight portion 511 and an instruction portion 512. As described above, the neural network model may include synaptic weights for the neural network to be calculated. The IPU 500 includes activation memory 502, which may be transient. The activation memory 502 may be divided into an input domain and an output domain and stores neuron activations for processing.

[0031] The IPU 500 includes an array 506 of neural cores 503. Each core 503 includes a computation unit 533 that is loaded with a neural network model from the model memory 501 and is operable to perform vector computations. Each core also includes a local activation memory 532. Input activations are provided from the local activation memory 532 before each computation step. Outputs from the computation units 533 are written back to the activation memory 532 for processing in the same or other computation units.

[0032] The IPU 500 includes one or more networks-on-chip (NoCs) 505. In some embodiments, a partial sum NoC 551 interconnects the cores 503 and carries partial sums between them. In some embodiments, a separate parameter distribution NoC 552 connects the cores 503 to memory 501 for distributing weights and instructions to the cores 503. Of course, various configurations of NoCs 551 and 552 are suitable for use with the present disclosure. For example, a broadcast network, a row broadcast network, a tree network, and a switched network may be used.

[0033] In various embodiments, a global micro-engine 504 is included in the IPU 500. In various embodiments, a local core controller 534 is included on each core 503. In such embodiments, operational instructions are shared between the global micro-engine (chip micro-engine) and the local core controller (core micro-engine). In particular, at 511, computational instructions are loaded from the model memory 501 by the global micro-engine 504 into the neural computation unit 533 of each core 503. At 512, parameters (e.g., neural network / synaptic weights) are loaded from the model memory 501 by the global micro-engine 504 into the neural computation unit 533 of each core 503. At 513, neural network activation data is loaded from the local activation memory 532 into the neural computation unit 533 of each core 503 by the local core controller 534. As described above, activations are provided to neurons of a particular neural network defined by the model and may originate from the same neural computation unit, other neural computation units, or external to the system. At 514, the neural computation unit 533, when instructed by the local core controller 534, performs a computation to generate an output neuron activation. In particular, this computation involves applying input synaptic weights to the input activation. Of course, various methods are available for performing such a computation, including in silico dendrite and vector multiplication units. At 515, when instructed by the local core controller 534, the results of the computation are stored in the local activation memory 532. As described above, the above steps can be pipelined to achieve efficient use of each core's neural computation units. Of course, inputs and outputs can be transferred from the local activation memory 532 to the global activation memory 502 according to the requirements of a given neural network.

[0034] Thus, the present disclosure provides runtime control of operations in an inference processing unit (IPU). In some embodiments, the micro-engines are centralized (single micro-engine). In some embodiments, the IPU computations are distributed (performed by an array of cores). In some embodiments, the runtime control of operations is hierarchical, involving both central and distributed micro-engines.

[0035] One or more microengines direct the execution of all operations in the IPU. Each microengine instruction corresponds to several sub-operations (e.g., address generation, load, calculation, store, etc.). In the distributed case, core microcode executes on a core microengine (e.g., 534). This core microcode contains instructions to perform a single tensor operation in its entirety, such as a convolution between a weight tensor and a data tensor. In the single-core context, the core microcode contains instructions to perform a single tensor operation on a locally stored subset of the data tensor (and partial sums). Chip microcode executes on a chip microengine (e.g., 504). The microcode contains instructions to perform all of the tensor operations in the neural network.

[0036] Referring now to FIG. 6, an exemplary neural core and associated network according to an embodiment of the present disclosure is shown. Core 601, embodied as described with reference to FIG. 3, is interconnected with additional cores by networks 602-604. In this embodiment, network 602 is responsible for distributing weights and / or instructions, network 603 is responsible for distributing partial sums, and network 604 is responsible for distributing activations. However, it should be understood that various embodiments of the present disclosure may combine these networks or further separate these networks into multiple additional networks.

[0037] Input activations (X) are distributed core 601 from out-of-core via activation network 604 to activation memory 605. Layer instructions are distributed core 601 from out-of-core via weight / instruction network 602 to instruction memory 606. Layer weights (W) and / or parameters are distributed core 601 from out-of-core via weight / instruction network 602 to weight memory 607 and / or parameter memory 608.

[0038] The weight matrix (W) is read from the weight memory 607 by a vector matrix multiplication (VMM) unit 609. The activation vector (V) is read from the activation memory 605 by a vector matrix multiplication (VMM) unit 609. The vector matrix multiplication (VMM) unit 609 then performs a vector-matrix multiplication Z=X T 603. The vector-vector unit 610 computes W and provides the result to the vector-vector unit 610. The vector-vector unit 610 reads additional partial sums from the partial sum memory 611 and receives additional partial sums from off-core via the partial sum network 603. Vector-vector operations are computed by the vector-vector unit 610 from their source partial sums. For example, the various partial sums are added in sequence. The resulting target partial sums are written to the partial sum memory 611, sent off-core via the partial sum network 603, or returned for further processing by the vector-vector unit 610, or some combination thereof.

[0039] This partial sum results from the Vector-Vector Unit 610 and is provided to the Activation Unit 612 for calculation of the output activations after all calculations for a given layer's inputs are complete. The activation vector (Y) is written to the Activation Memory 605. The layer activations (including the results written to the activation memory) are redistributed across cores from the Activation Memory 605 via the Activation Network 604. Upon receipt, the layer activations are written to the local activation memory of the receiving core. Once processing for a given frame is complete, the output activations are read from the Activation Memory 605 and sent out of the core via the Network 604.

[0040] In operation, the core control micro-engine (e.g., 613) orchestrates the data movement and computation of the core accordingly. The micro-engine issues a read activation memory address operation to load an input activation block into the vector-matrix multiplication unit. The micro-engine issues a read weight memory address operation to load a weight block into the vector-matrix multiplication unit. The micro-engine issues a compute operation to the vector-matrix multiplication unit so that the compute array of the vector-matrix multiplication unit computes the partial sum block.

[0041] The microengine issues one or more of a partial sum read / write memory address operation, a vector calculation operation, or a partial sum communication operation to read partial sum data from a partial sum source, calculate using a partial sum calculation unit, or write partial sum data to a partial sum target. Writing partial sum data to a partial sum target may include communicating outside the core via a partial sum network interface or sending the partial sum data to an active calculation unit.

[0042] The micro-engine issues an activation function calculation operation so that the activation function calculation unit calculates the output activation block, and the micro-engine issues a write activation memory address, and the output activation block is written to the activation memory through the activation memory interface.

[0043] Thus, a wide variety of sources, targets, address types, computation types, and control components may be defined for a given core.

[0044] The sources for the vector-vector unit 610 include the vector-matrix multiplication (VMM) unit 609, activation memory 605, constants from parameter memory 608, partial sum memory 611, partial sum results from the previous cycle (TGT partial sums), and partial sum network 603.

[0045] The targets for the vector-vector unit 610 include a partial sum memory 611, a partial sum result for the subsequent cycle (SRC partial sum), an activation unit 612, and a partial sum network 603.

[0046] Thus, a given instruction may read from or write to activation memory 605, read from weight memory 607, or read from or write to partial sum memory 611. Computational operations performed by the core include vector matrix multiplication by VMM unit 609, vector (partial sum) operations by vector unit 610, and activation functions by activation unit 612.

[0047] The control operations include a program counter and loop and / or sequence counters.

[0048] Thereby, memory operations are issued to read weights from addresses in weight memory, read parameters from addresses in parameter memory, read activations from addresses in activation memory, and read / write partial sums to / from addresses in partial sum memory. Computation operations are issued to perform vector-matrix multiplication, vector-vector operations, and activation functions. Communication operations are issued to select vector-vector operands, route messages on the partial sum network, and select partial sum targets. Loops on layer outputs and loops on layer inputs are controlled by control operations that specify the program counter, loop counter, and sequence counter.

[0049] In various embodiments, a memory-mapped architecture is implemented that allows the IPU, as described above, to communicate with the host through memory reads and writes. Referring to Figure 7, an exemplary integration method between a host system and an IPU is shown. At 701, the host prepares data for inference. At 702, the host notifies the IPU that the data is available. At 703, the IPU reads the data. At 704, the IPU performs a computation on the data. At 705, the IPU notifies the host that the computation result is available. At 706, the host reads the result.

[0050] 8(A)-(C), an exemplary redundancy method is shown. Of course, neuromorphic systems such as those described herein above can process data from multiple sensors simultaneously. Multiple networks can exist and run simultaneously. As described herein, in various embodiments, network results are provided using a high-speed I / O interface.

[0051] Referring to Figure 8(A), direct / hardware redundancy is shown. In this example, the same model is run more than once and the outputs are compared. Referring to Figure 8(B), model redundancy is shown. In this example, ensembles of different data and / or different data are run and statistical models (e.g., weighted averaging between models) are applied to arrive at an overall output. Referring to Figure 8(C), apprentice validation is shown. In this example, an apprentice model is validated against a control model (or driver).

[0052] The low power requirements of the architecture described herein allow multiple chips in a system to run redundant networks. Similarly, redundant networks can be run on partitions of chips. Furthermore, fast and partial reconfigurability is provided to switch between drive and test modes to detect / locate / avoid anomalies.

[0053] Of course, an inference processing unit as described herein can be integrated into a wide variety of form factors. For example, a system-on-chip (SoC) can be provided. An SoC allows for scaling to accommodate area budgets. This approach allows for on-die integration with resulting high-speed data transfer capabilities. The SoC form factor can also be easier and cheaper to package than various alternatives. In another example, a system-in-package (SiP) can be provided. The SiP approach combines SoC components with an IPU die and supports the integration of different process technologies. Minimal injection changes to existing components are required.

[0054] Another example is a PCIe (or other expansion card) implementation. This approach allows for independent development cycles for each component. This has the advantage of using a standardized, high-speed interface, allowing for modular integration. This is particularly suitable for early prototypes and data centers. Similarly, an electronic control unit (ECU) may be implemented, which complies with automotive standards, including those for safety and redundancy. ECU modules are suitable for in-vehicle deployment, but generally require additional research and development time.

[0055] 9, a system architecture for a memory-mapped neural inference engine according to an embodiment of the present disclosure is shown. A neural inference engine 901 (such as that detailed above) is connected to a system interconnect 902. A host 903 is also connected to the system interconnect 902.

[0056] In various embodiments, system interconnect 902 conforms to an Advanced Microcontroller Bus Architecture (AMBA), such as the Advanced eXtensible Interface (AXI). In various embodiments, system interconnect 902 is a Peripheral Component Interconnect Express (PCIe) bus or other PCI bus. Of course, a wide variety of other bus architectures known in the art to which this disclosure pertains are suitable for use as described herein. In each case, system interconnect 902 connects host 903 to neural inference engine 901 and provides a flat, memory-mapped view of the neural inference engine in the host's virtual memory.

[0057] The host 903 includes an application 904 and an API / driver 905. In various embodiments, the API includes three functions: configure(), which copies a self-contained neural network program into the neural inference engine 901 via a memory map; push(), which copies input data into the neural inference engine 901 via a memory map and initiates evaluation; and pull(), which retrieves output data from the neural inference engine 901 via a memory map.

[0058] In some embodiments, an interrupt 906 is provided by the neural inference engine 901 to signal the host 903 that the network evaluation is complete.

[0059] Referring to Figure 10, an exemplary runtime software stack is shown according to various embodiments. In this example, a library 1001 is provided for interfacing with a neural inference engine device 1002. API calls are provided for loading networks and for memory management (including standard functions for allocating and freeing memory, copying to memory, and retrieving from memory).

[0060] Referring to Figure 11, an exemplary sequence of execution according to an embodiment of the present disclosure is shown. In this example, offline training results in a network definition file nw.bin1111. During network initialization 1102, the neural inference device is accessed, e.g., by an open API call, and the network definition file 1111 is loaded. During runtime operation phase 1103, data space is allocated on the neural inference device, and input data 1131 (e.g., image data) is copied to a device memory buffer. One or more computational cycles are performed, as detailed above. Upon completion of a computational cycle, output may be received from the device, e.g., by a rcvAPI call.

[0061] The neural inference unit can be memory-mapped for input and output and performs its computations without host instructions and without requiring external memory for either the neural network model or the intermediate activations. This provides a streamlined programming model in which the neural inference unit is simply instructed to compute the neural network, rather than requiring separate instructions for component operations such as matrix multiplication. In particular, there is no translation of convolutions to matrix multiplications, and therefore no need for retranslation. Also, new calls do not need to be issued for each new layer of the network. As noted above with respect to the overall chip design, inter-layer neuron activations never leave the chip. Using this approach, new network model parameters do not need to be loaded during runtime.

[0062] Referring to Figure 12, an exemplary integration of a neural inference unit 1201 is shown. In this example, a FIFO buffer is provided on the data path with internal decoding. This provides a multi-channel DMA configuration without the need for multiple masters. Alternatively, multiple AXI interfaces may be provided with masters, thereby increasing concurrent throughput.

[0063] On the hardware side, a first AXI slave provides a FIFO interface to the neural inference device's activation memory, a second AXI slave provides a FIFO interface from the neural inference device's activation memory, and a third AXI slave provides four FIFO interfaces: one to the instruction memory, one from the instruction memory, one to the parameter / control registers, and one from the parameter / control registers.

[0064] AXI masters initiate data movement to and from the neural inference data path, which is commanded via MC-DMA. The multi-channel DMA controller (MC-DMA) provides a programmable DMA engine that can simultaneously perform data movement for multiple AXI slaves.

[0065] Applications built for this integration scenario use API routines for tasks (e.g., sendTensor, recvTensor), and therefore the runtime library is agnostic to the specific hardware instance, while the driver is built for a given hardware configuration.

[0066] Referring to Figure 13, there is shown an exemplary integration of a neural reasoner 1301. In this example, a fully memory-mapped interface is used.

[0067] On the hardware side, one AXI slave provides a memory-mapped interface to the neural inference unit's activation memory, a second AXI slave provides a memory-mapped interface from the neural inference unit's activation memory, and a third AXI slave provides memory-mapped interfaces: one for instruction memory, one for global memory, and one for parameter / control registers.

[0068] AXI masters initiate data movement to and from the neural inference data path, which is commanded via MC-DMA. The multi-channel DMA controller (MC-DMA) provides a programmable DMA engine that can simultaneously perform data movement for multiple AXI slaves.

[0069] Applications built for this integration scenario use API routines for tasks (e.g., sendTensor, recvTensor), and therefore the runtime library is agnostic to the specific hardware instance, while the driver is built for a given hardware configuration.

[0070] Referring to FIG. 14, an exemplary configuration is shown in which a neural reasoning device 1401 is interconnected to a host via a PCIe bridge.

[0071] In some embodiments, the runtime is provided at the application layer. In such embodiments, the application exposes a primary interface (e.g., Configure, Put Tensor, Get Tensor) to other applications. The base software layer communicates with the neural reasoner via a PCIe driver, creating an abstraction layer. The neural reasoner is then connected to the system as a peripheral device via a high-speed interface.

[0072] In some embodiments, a runtime driver is provided that exposes the primary interface (e.g., Configure, Put Tensor, Get Tensor) to other AUTOSAR applications. The neural inference device is then connected to the system as a peripheral device via a high-speed interface.

[0073] The techniques and layouts described above enable a wide variety of multiple neural inference device models. In some embodiments, multiple neural inference modules communicate with a host via a select high-speed interface. In some embodiments, multiple neural inference chips communicate with each other and with a host via a high-speed interface, potentially using glue logic. In some embodiments, multiple neural inference dies communicate with either a host or other neural inference dies via a dedicated interface, potentially using glue logic (on-chip or on an interposer). In some embodiments, multiple neural inference systems-in-package communicate with each other and / or with an on-die host via a high-speed interface. Exemplary interfaces include PCIe gen4 / 5, AXI4, SerDes, and specialized interfaces.

[0074] Referring to FIG. 15, a method 1500 is shown for receiving 1501 a neural network description from a host via an interface in a neural network processor system, the neural network processor system comprising at least one neural network processing core, an activation memory, an instruction memory, and at least one control register, the neural network processing core adapted to implement neural network computation, control, and communication primitives, and an interface operatively connected to the neural network processor system. The method further includes exposing 1502 a memory map via the interface, the memory map comprising regions corresponding to each of the activation memory, the instruction memory, and the at least one control register. The method further includes receiving 1503 input data in the neural network processor system via the interface. The method further includes calculating 1504 output data from the input data based on the neural network model. The method further includes providing 1505 output data from the neural network processor system via the interface. In some embodiments, the method includes 1506 receiving a neural network description via the interface, receiving input data via the interface, and providing output data via the interface.

[0075] As described above, various embodiments provide a memory-mapped neural inference engine that includes one or more neural inference chips with peripheral communication interfaces for communication to a host, sensors, other inference engines, or a combination thereof. In some embodiments, each neural inference chip is memory-mapped and uses a reduced set of communication API primitives, such as configure_network(), push_data(), and pull_data(). In some embodiments, interchangeable interfaces, such as AXI, PCIe, USB, Ethernet, Firewire, or wireless, are used to communicate with the neural inference engine. In some embodiments, multiple levels of hardware, software, and model-level redundancy are used to increase system yield and ensure correct system operation. In some embodiments, firmware is used to manipulate and buffer incoming / outgoing data for improved performance. In some embodiments, a runtime programming model is used to control the neural accelerator chip. In some embodiments, a hardware-firmware-software stack is used to implement multiple applications on the neural inference engine.

[0076] In some embodiments, the system operates in a stand-alone mode by incorporating on-board non-volatile memory (such as a flash card or SD card) for storing system configuration and operating parameters or for resuming from a previous state. In some embodiments, the performance of the system and communications infrastructure described above supports real-time operation and communication with the neural accelerator chip. In some embodiments, the performance of the system and communications infrastructure described above supports faster than real-time operation and communication with the neural accelerator chip.

[0077] In some embodiments, the neural inference chip, firmware, software, and communication protocols allow multiple such systems to be arrayed into larger systems (e.g., multi-chip systems, multi-board systems, racks, data centers, etc.). In some embodiments, the neural inference chip and microprocessor chip comprise an energy-efficient, real-time processing hybrid cloud computing system. In some embodiments, the neural inference chip is used in a cloud system for sensor-based, neural-based, video-based, and / or audio-based applications, as well as modeling applications. In some embodiments, the interface controller is used for communication with other cloud segments / hosts, which may use a variety of communication interfaces.

[0078] In some embodiments, a firmware stack and a software stack (including drivers) perform the inference engine / microprocessor, inference engine / host, and microprocessor / host interactions. In some embodiments, a runtime API is provided that performs low-level interactions with the neural inference chip. In some embodiments, a software stack is provided that includes an operating system that automatically maps workloads and user applications to the system's devices and executes them in order.

[0079] 16, a schematic of an example computational node is shown. Computational node 10 is merely one example of a suitable computational node and is not intended to suggest any limitation as to the scope of use or functionality of the inventive embodiments described herein, although computational node 10 is capable of implementing and / or performing any of the functions described above.

[0080] There is computer system / server 12 operable with numerous other general purpose or special purpose computing system environments or configurations in compute node 10. Examples of well-known computing systems, environments, or configurations, or combinations thereof, that may be suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0081] The computer system / server 12 may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system / server 12 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0082] 16, computer system / server 12 in compute node 10 is shown in the form of a general-purpose computing device. Components of computer system / server 12 include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that couples various system components, including system memory 28, to processor 16.

[0083] Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a wide variety of bus architectures, including, by way of example and without limitation, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0084] Computer system / server 12 typically includes a wide variety of computer system-readable media, which may be any available media that can be accessed by computer system / server 12 and includes both volatile and nonvolatile media, removable and non-removable media.

[0085] System memory 28 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer system / server 12 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 may be provided for reading from and writing to non-removable, non-volatile magnetic media (not shown, commonly referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to removable, non-volatile magnetic disks (e.g., "floppy disks"), and an optical disk drive may be provided for reading from and writing to removable, non-volatile optical disks, such as CD-ROMs, DVD-ROMs, or other optical media. In such cases, each may be connected to bus 18 by one or more data media interfaces. As illustrated and further described below, memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present invention.

[0086] By way of example, and not limitation, a program / utility 40 having a set (at least one) of program modules 42, as well as an operating system, one or more application programs, other program modules, and program data, may be stored in memory 28. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may include an implementation of a networking environment. The program modules 42 generally perform the functions and / or methodologies of embodiments of the present invention as described herein.

[0087] The computer system / server 12 may further communicate with one or more external devices 14, such as a keyboard, pointing device, display 24, one or more devices that allow a user to interact with the computer system / server 12, or any device (e.g., a network card, modem, etc.) that allows the computer system / server 12 to communicate with one or more other computing devices, or a combination thereof. Such communication may occur via an input / output (I / O) interface 22. Furthermore, the computer system / server 12 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or a combination thereof, via a network adapter 20. As indicated above, the network adapter 20 communicates with other components of the computer system / server 12 via a bus 18. It should be understood that other hardware and / or software components, not shown, may be used in conjunction with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data mass storage systems.

[0088] The present invention may be a system, method, and / or computer program product, which may include computer-readable storage medium(s) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0089] A computer-readable storage medium may be any tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disks (DVDs), memory sticks, floppy disks, mechanical encryption devices such as punch cards or ridge structures with instructions recorded on them, and any suitable combination of the above. As used herein, a computer-readable storage medium should not be construed as being, per se, a transitory signal such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., an optical pulse passing through a fiber optic cable), or an electrical signal transmitted by an electrical wire.

[0090] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, or a wireless network, or any combination thereof. This network may comprise copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or any combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0091] Computer-readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and traditional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer, and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer readable program instructions to personalize the electronic circuitry by utilizing state information of the computer readable program instructions to perform aspects of the present invention.

[0092] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0093] The computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to manufacture a machine, such that the instructions, when executed by the processor of the computer or other programmable data processing apparatus, create means for performing the functions / acts specified in the blocks of the flowcharts and / or block diagrams. These computer-readable program instructions may further be stored on a computer-readable storage medium that causes a computer, programmable data processing apparatus, or other apparatus to function in a particular way or combination thereof, such that the computer-readable storage medium having the instructions stored thereon comprises an article of manufacture containing instructions that implement aspects of the functions / acts specified in the blocks of the flowcharts and / or block diagrams.

[0094] The computer-readable program instructions may be further loaded into a computer, other programmable data processing device, or other device to cause a series of operational steps to be executed on the computer or other programmable device or other device to create a computer-implemented process, such that the instructions, which execute on the computer, other programmable device, or other device, perform the functions / operations specified in the flowchart and / or block diagram blocks.

[0095] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specialized logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending on the functionality involved. It will also be recognized that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs a specialized function or operation, or executes a combination of dedicated hardware and computer instructions.

[0096] The description of various embodiments of the present invention has been provided for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein have been chosen to best explain the principles of the embodiments, practical applications or technical improvements of existing technologies, or to enable others of ordinary skill in the art to which the disclosure pertains to understand the embodiments disclosed herein.

Claims

1. 1. A system comprising: a neural network processor system comprising at least one neural network processing core, an activation memory, an instruction memory, and at least one control register, said neural network processing core adapted to implement neural network computation, control, and communication primitives; a memory map corresponding to a shared memory connected to a communication bus, the shared memory having areas corresponding to each of the activation memory, instruction memory, and at least one control register, wherein each of the neural network processing cores is connected to the communication bus, and each of the neural network processing cores intercommunicates with neural network processing cores, including non-adjacent neural network processing cores, via shared memory corresponding to addressable areas of the memory map; an interface operatively connecting the neural network processor system to the communication bus, the interface adapted to communicate with a host over the communication bus and expose the memory map; A system comprising:

2. 10. The system of claim 1, wherein the neural network processor system is configured to receive a neural network description via the interface, receive input data via the interface, and provide output data via the interface.

3. 3. The system of claim 2, wherein the neural network processor system exposes an API through the interface, the API including receiving the neural network description through the interface, receiving input data through the interface, and providing output data through the interface.

4. 10. The system of claim 1, wherein the interface comprises an AXI, PCIe, USB, Ethernet, or Firewire interface.

5. 10. The system of claim 1, further comprising a redundant neural network processing core, the redundant neural network processing core configured to compute neural network models in parallel with the neural network processing core.

6. 10. The system of claim 1, wherein the neural network processor system is configured to provide at least one of hardware redundancy, which runs the same model multiple times and compares the outputs, model redundancy, which runs different ensembles of data or different data, and software redundancy, which uses apprentice validation for control models.

7. 3. The system of claim 2, wherein said neural network processor system comprises programmable firmware, said programmable firmware being configurable to process said input data and output data.

8. The system of claim 7 , wherein the processing includes buffering.

9. 10. The system of claim 1, wherein the neural network processor system comprises non-volatile memory.

10. 10. The system of claim 9, wherein the neural network processor system is configured to store configuration or operating parameters or program states of the neural network processor system in the non-volatile memory.

11. 11. The system of claim 10, wherein the neural network processor system is configured to operate in a stand-alone mode by storing in the non-volatile memory.

12. The system of claim 1 , wherein the interface is communicatively coupled to at least one sensor or camera.

13. A system comprising a plurality of the systems of claim 1 interconnected by a network.

14. A system comprising a plurality of the systems according to claim 1 and a plurality of computing nodes, the systems being interconnected by a network.

15. 15. The system of claim 14, further comprising a plurality of non-overlapping memory maps, each memory map corresponding to one of the plurality of systems of claim 1.

16. 1. A method, comprising: receiving a neural network description in the neural network processor system from a host via an interface; the neural network processor system comprising at least one neural network processing core, an activation memory, an instruction memory, and at least one control register, the neural network processing core adapted to implement neural network computation, control, and communication primitives; the interface is operably connected to the neural network processor system via a communication bus; The method further includes exposing a memory map via the interface, the memory map corresponding to a shared memory connected to the communication bus, the memory map comprising areas corresponding to each of the activation memory, instruction memory, and at least one control register, each of the neural network processing cores being connected to the communication bus, and each of the neural network processing cores intercommunicating with neural network processing cores, including non-adjacent neural network processing cores, via shared memory corresponding to addressable areas of the memory map; The method further includes receiving input data at the neural network processor system via the interface; calculating output data from the input data based on a neural network model; providing said output data from said neural network processor system via said interface; A method comprising:

17. 17. The method of claim 16, wherein the neural network processor system receives a neural network description through the interface, receives input data through the interface, and provides output data through the interface.

18. 18. The method of claim 17, wherein the neural network processor system exposes an API through the interface, the API including receiving the neural network description through the interface, receiving input data through the interface, and providing output data through the interface.

19. 17. The method of claim 16, wherein the neural network processor system is configured to operate in a stand-alone mode by storing configuration or operating parameters or program states of the neural network processor system in non-volatile memory.

Images