Central scheduler and instruction dispatcher for neural inference processors

Abstract

A neural inference processor is provided. In various embodiments, the processor includes a plurality of cores. Each core includes a neural computing unit, an activation memory, and a local controller. The neural computing unit is adapted to apply a plurality of synaptic weights to a plurality of input activations to produce a plurality of output activations. The activation memory is adapted to store input activations and output activations. The local controller is adapted to load input activations from the activation memory to the neural computing unit and to store a plurality of output activations from the neural computing unit to the activation memory. The processor includes a neural network model memory adapted to store network parameters including a plurality of synaptic weights. The processor includes a global scheduler operatively coupled to the plurality of cores, adapted to provide synaptic weights from the neural network model memory to each core.

Description

Background Art

[0001] Embodiments of the present disclosure relate to neural network processing components, and more particularly, to a central scheduler and instruction distributor for a neural inference processor. Summary of the invention

[0002] According to an embodiment of the present disclosure, a neural reasoning processor is provided. A processor includes multiple cores. Each core includes a neural computing unit, an activation memory, and a local controller. The neural computing unit is suitable for applying multiple synaptic weights to multiple input activations to generate multiple output activations. The activation memory is suitable for storing input activations and output activations. The local controller is suitable for loading input activations from the activation memory to the neural computing unit and for storing the multiple output activations from the neural computing unit to the activation memory. The processor includes a neural network model memory, which is suitable for storing multiple synaptic weights. The processor includes a global scheduler, which is operably coupled to the multiple cores and is suitable for providing synaptic weights from the neural network model memory to each core.

[0003] According to an embodiment of the present disclosure, a neural inference processor is provided. A neural computing unit is provided. The neural computing unit is adapted to apply multiple synaptic weights to multiple input activations to generate multiple output activations. A neural network model memory is adapted to store multiple synaptic weights. An activation memory is adapted to store input activations and output activations. A scheduler is operatively coupled to the neural computing unit, the neural network model memory, and the activation memory. The scheduler is adapted to: load the synaptic weights from the neural network model memory to the neural computing unit; load the input activations from the activation memory to the neural computing unit; and store multiple output activations from the neural computing unit to the activation memory.

[0004] According to an embodiment of the present disclosure, a method and a computer program product for operating a neural inference processor are provided. A global scheduler provides synaptic weights from a neural network model memory to each of a plurality of cores. A local controller of each core loads input activations from an activation memory of the core to a neural computing unit of the core. The neural computing unit of each core applies the plurality of synaptic weights to the plurality of input activations to generate a plurality of output activations. The local controller of each core stores a plurality of output activations from the neural computing unit of the core to the activation memory of the core. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0006] Figure 1 An exemplary inference processing unit (IPU) according to an embodiment of the present disclosure is shown.

[0007] Figure 2 yes Figure 1 IPU pipeline timing diagram.

[0008] Figure 3 A multi-core inference processing unit (IPU) according to an embodiment of the present disclosure is shown.

[0009] Figure 4 A multi-core inference processing unit (IPU) according to an embodiment of the present disclosure is shown.

[0010] Figure 5 An exemplary IPU Network-on-Chip (NoC) according to an embodiment of the present disclosure is shown.

[0011] Figure 6 An exemplary IPU Network-on-Chip (NoC) according to an embodiment of the present disclosure is shown.

[0012] Figure 7 An exemplary IPU Network-on-Chip (NoC) according to an embodiment of the present disclosure is shown.

[0013] Figure 8 The embodiment according to the present invention is shown Figure 7 An exemplary memory configuration of the IPU.

[0014] Fig. 9 A method of operating a neural inference processor according to an embodiment of the present disclosure is shown.

[0015] Fig.10 A computing node according to an embodiment of the invention is described. DETAILED DESCRIPTION

[0016] An artificial neuron is a mathematical function whose output is a nonlinear function of a linear combination of its inputs. Two neurons are connected if the output of one neuron is the input of another neuron. A weight is a scalar value that encodes the strength of the connection between the output of one neuron and the input of another neuron.

[0017] A neuron computes its output, called an activation, by applying a nonlinear activation function to a weighted sum of its inputs. A weighted sum is an intermediate result computed by multiplying each input by its corresponding weight and accumulating the products. A partial sum is a weighted sum of a subset of the inputs. The weighted sum of all inputs can be computed in stages by accumulating one or more partial sums.

[0018] A neural network is a collection of one or more neurons. Neural networks are often divided into groups of neurons called layers. A layer is a collection of one or more neurons that all receive input from the same layer and all send output to the same layer, and generally perform similar functions. An input layer is a layer that receives input from a source external to the neural network. An output layer is a layer that sends output to a target external to the neural network. All other layers are intermediate processing layers. A multi-layer neural network is a neural network with more than one layer. A deep neural network is a multi-layer neural network with multiple layers.

[0019] A tensor is a multidimensional array of values. A tensor block is a contiguous subarray of the elements in a tensor.

[0020] Each neural network layer is associated with a weight tensor, a parameter tensor, an input tensor, an output tensor, and intermediate tensors. The weight tensor contains all the weights that connect the inputs to the layer. The parameter tensor contains all the parameters that control the activation functions of the neurons in the layer. The input tensor contains all the data consumed by the layer as input. The output tensor contains all the data calculated by the layer as output. Intermediate tensors contain any data produced by the layer as intermediate calculations, such as partial sums.

[0021] A neural core is a tileable computational unit that computes a block of output tensors. A neural core has M inputs and N outputs. In various embodiments, M=N. To compute an output tensor block, the neural core multiplies M×1 input tensor blocks with M×N weight tensor blocks and accumulates the products as a weighted sum, which is stored in 1×N intermediate tensor blocks. U×N parameter tensor blocks contain U parameters that specify each of the N neuron activation functions, which are applied to the intermediate tensor blocks to produce 1×N output tensor blocks.

[0022] Multiple neural cores can be tiled in a neural core array. In some embodiments, the array is 2-dimensional.

[0023] A neural network model is a set of constants that together specify the entire computation performed by the neural network, including the connection graph between neurons and the weights and activation function parameters of each neuron. Training is the process of modifying the neural network model to perform the desired function. Inference is the process of applying the neural network to inputs to produce outputs, without modifying the neural network model.

[0024] An IPU is a type of processor that performs neural network inference. A neural inference chip is a specific physical instance of an IPU.

[0025] refer to Figure 1, an exemplary inference processing unit (IPU) is shown according to an embodiment of the present disclosure. IPU 100 includes a memory 101 for a neural network model. As described above, the neural network model may include synaptic weights of the neural network to be calculated. IPU 100 includes an activation memory 102 which may be a transient state. Activation memory 102 may be divided into input and output areas, and stores neuron activations for processing. IPU 100 includes a neural computing unit 103, which is loaded with a neural network model from model memory 101. Input activations are provided from activation memory 102 before each computation step. The output from neural computing unit 103 is written back to activation memory 102 for processing on the same or another neural computing unit.

[0026] In various embodiments, scheduler 104 is included in IPU 100. In such embodiments, all operations in the IPU are directed by the scheduler. As described below, a central and / or distributed scheduler may be provided in various embodiments. The global scheduler may be referred to as a chip microengine, and the local scheduler may be referred to as a core microengine or a local controller. In various embodiments, the scheduler includes one or more microengines, microcontrollers, state machines, CPUs, or other controllers.

[0027] refer to Figure 2 ,for Figure 1 The IPU of provides a pipeline timing diagram, at 111, the calculation instructions are loaded from the model memory 101 to the neural computing unit 103. At 112, the parameters (e.g., neural network / synaptic weights) are loaded from the model memory 101 to the neural computing unit 103. At 113, the neural network activation data is loaded from the activation memory 102 to the neural computing unit 103. As described above, the activation is provided to the axons of the specific neural network defined by the model, and can originate from the same or another neural computing unit, or from outside the system. At 114, the neural computing unit 103 performs calculations to generate output neuron activations. Specifically, the calculation includes applying the input synaptic weights to the input activations. It should be understood that various methods can be used to perform such calculations, including in silicon dendrites and vector multiplication units. At 115, the results from the calculations are stored in the activation memory 102. As shown, these stages can be pipelined to provide efficient use of the neural computing unit.

[0028] refer to Figure 3, a multi-core inference processing unit (IPU) is shown according to an embodiment of the present disclosure. IPU300 includes a model memory 301 for a neural network model. As described above, the neural network model may include the synaptic weights of the neural network to be calculated. IPU300 includes an activation memory 302 that may be transient. Activation memory 302 may be divided into input and output areas and stores neuron activations for processing. IPU300 includes multiple cores 303. Each core 303 includes a neural computing unit 333, which is loaded with a neural network model from model memory 301. Each core also includes a local activation memory 332. Input activation is provided from the local activation memory 332 before each calculation step. The output from the neural computing unit 333 is written back to the activation memory 332 for processing on the same or another neural computing unit.

[0029] In various embodiments, a global scheduler 304 is included in the IPU 300. In various embodiments, a local core controller 334 is included on each core 303. In such embodiments, the direction of operation is shared between the global scheduler (chip microengine) and the local core controller (core microengine). Specifically, at 311, the global scheduler 304 loads computational instructions from the model memory 301 to the neural computing unit 333 on each core 303. At 312, the global scheduler 304 loads parameters (e.g., neural network / synaptic weights) from the model memory 301 to the neural computing unit 333 on each core 303. At 313, neural network activation data is loaded from the activation local activation memory 332 to the neural computing unit 333 on each core 303 by the local core controller 334. As described above, activation is provided to the axons of a specific neural network defined by the model, and may originate from the same or another neural computing unit, or from outside the system. At 314, the neural compute unit 333 performs the computation to generate output neuron activations as directed by the local core controller 334. Specifically, the computation includes applying the input synaptic weights to the input activations. It should be appreciated that a variety of methods may be used to perform such computations, including in silicon dendrites and vector multiplication units. At 315, the results of the computations are stored in the local activation memory 332 as directed by the local core controller 334. As described above, these stages may be pipelined to provide efficient use of the neural compute unit on each core. It should also be appreciated that inputs and outputs may be passed from the local activation memory 332 to the global activation memory 302, depending on the requirements of a given neural network.

[0030] Thus, the present disclosure provides runtime scheduling of operations in an inference processing unit (IPU). In some embodiments, the operation scheduler is centralized (single scheduler). In some embodiments, IPU computations are distributed (performed by an array of cores). In some embodiments, runtime scheduling of operations is hierarchical—both central and distributed schedulers participate.

[0031] The scheduler or schedulers direct the execution of all operations in the IPU. Each scheduler instruction corresponds to several sub-operations (e.g., address generation, loading, calculation, storage, etc.). In the distributed case, the core microcode is run on the core microengine (e.g., 334). The core microcode includes instructions to perform complete single tensor operations. For example, convolution between weight tensors and data tensors. In the context of a single core, the core microcode includes instructions to perform single tensor operations on a subset of locally stored data tensors (and partial sums). The chip microcode runs on a chip microengine (e.g., 304). The microcode includes instructions to perform all tensor operations in a neural network.

[0032] refer to Figure 4 , a multi-core inference processing unit (IPU) is shown according to an embodiment of the present disclosure. IPU400 includes a model memory 401 for a neural network model. As described above, the neural network model may include synaptic weights of the neural network to be calculated. In some embodiments, the model memory 401 includes one or more physical memories, which can be allocated to synaptic weights and instructions, respectively. IPU400 includes an activation memory 402 that can be transient. Activation memory 402 can be divided into input and output areas, and stores neuron activations for processing. IPU400 includes multiple cores 403. Each core 403 can be as described above with respect to Figure 3 Configure as described.

[0033] IPU 400 includes one or more Networks on Chip (NoC) 404. As described above, network weights and instructions are stored in memory 401. To configure each core, scheduler (chip microengine) 404 generates a read address and creates a read transaction from (weight / instruction) parameter memory 401. The read weights and instructions are sent to the destination core (e.g., 403) through NoC 404. Using a distributed set of weights, distributed instructions are executed on each core on the local data of that core.

[0034] Thus, in some embodiments, the scheduler distributes weight blocks from parameter memory via a network on chip (NoC). In some embodiments, the scheduler distributes instructions from instruction memory via a network on chip (NoC). In some embodiments, the instructions include memory operations, computational operations, and control operations. In some embodiments, a NoC is provided. In some embodiments, multiple NoCs are provided. In some such embodiments, weights are distributed on a NoC different from instructions. In some embodiments, the instruction memory is the same physical memory as the parameter memory, while in some embodiments, it is separate.

[0035] refer to Figure 5 , an exemplary IPU on-chip network (NoC) is shown according to an embodiment of the present disclosure. The IPU 400 includes a tree-shaped on-chip network 504. It can be understood that such a tree-shaped network can be used to distribute weights and instructions to each core included in the IPU.

[0036] In particular, in some embodiments, the IPU can operate in single instruction, multiple data (SIMD) mode. In such an embodiment, the same weights and instructions are distributed to all cores. Therefore, although each core has different activation data, all weights and instructions are the same for each core.

[0037] In some embodiments, the IPU can operate in multiple instruction, multiple data (MIMD) mode. In such embodiments, different weights and instructions are distributed to different cores. As described above, each core has different activation data. In some such MIMD embodiments, the scheduler sends different weights and instructions to each core sequentially. If the weights and instructions must be distributed frequently, this method has a relatively low throughput. If the weights and instructions are not distributed frequently, this method can have a reasonable throughput.

[0038] refer to Figure 6 , an exemplary IPU on-chip network (NoC) is shown according to an embodiment of the present disclosure. IPU 600 includes a row broadcast network on chip 604. Network 604 includes multiple row networks 641...644, each connecting a row of cores within IPU 600. Such embodiments are suitable for use with a single scheduler, multiple parallel schedulers, or a hierarchical configuration in which a single scheduler drives multiple schedulers in parallel.

[0039] In some embodiments, the IPU can operate in multiple instruction, multiple data (MIMD) mode. In such an embodiment, each core has different activation data. The weights and instructions are the same within each row of the core. However, between rows, each row of cores has different weights and / or instructions. In some embodiments, each parallel row network 641...644 has a corresponding memory bank 611...614, which stores the weights and instructions for the corresponding core row. Therefore, by reading the same data from each memory bank 611...614 (as shown by the highlighted parallel addresses at 615), the same weights and instructions can be sent to all cores. By reading different data from each memory bank 611, 611...614, 614 (as shown by the highlighted addresses at 616), different weights and instructions can be sent to the cores in each row.

[0040] In some embodiments, a single scheduler is used to implement MIMD mode in IPU 600. In such embodiments, the same source address may be used across all memory banks (as shown at 615). Alternatively, a different source address may be used for each memory bank (as shown at 616). In some such embodiments, the scheduler code includes an address for each memory bank. In other such embodiments, the scheduler code uses a single address to index into a lookup table (LUT) storing the address of each memory bank.

[0041] In some embodiments, multiple schedulers are used to implement more MIMD in IPU 600. In some such embodiments, one scheduler is provided for each bank 611 ... 614. Each bank scheduler runs independent microcode and generates source addresses for the corresponding memory bank.

[0042] In some embodiments, a hierarchical scheduler is used to implement MIMD mode in IPU600. In some embodiments, a scheduler is provided for each memory bank, and a supervisory scheduler is provided. In such an embodiment, each memory bank scheduler runs independent microcode and generates source addresses for the corresponding memory bank. The supervisory scheduler synchronizes operations between the memory bank schedulers.

[0043] refer to Figure 7, an exemplary IPU on-chip network (NoC) is shown according to an embodiment of the present disclosure. The IPU 700 includes a row broadcast network 704 on the chip. The network 704 includes a plurality of row networks 741 ... 744, each connecting a row of cores within the IPU 700. The row networks 741 ... 744 are coupled to a memory bypass bus 705 and a global bus 706. The row distribution networks 741 ... 744 can provide weights and instructions from the memory bypass bus 705, thereby obtaining inputs from the memory storage bodies 711 ... 714 associated with each corresponding row. In particular, the memory bypass bus can switch between the memory input and the global input for each row. The row distribution networks 741 ... 744 can provide weights and instructions from the global bus, and the global bus can extract inputs from any one of the memory storage bodies 711 ... 714. In particular, the global bus can switch between the various storage bodies. In this way, each row can receive a row-specific input or a selected global input. It should be understood that a variety of interconnects are suitable for such switching, including various switches and gates known in the art.

[0044] Such embodiments are suitable for operating in SIMD and MIMD modes. In various embodiments, SIMD or MIMD operation is programmable. For a given microcode instruction, the system can be SIMD or MIMD (but not both). Likewise, it can switch between SIMD and MIMD operating modes (and vice versa) after each microcode instruction.

[0045] Reference now Figure 8 According to the embodiment of the present disclosure, Figure 7 811 is divided into a plurality of sub-banks 815. From each sub-bank, data is selected from non-adjacent memory locations 816. In this way, a synaptic weight block can be assembled from multiple sub-vectors extracted from non-contiguous locations in the memory. As shown in the figure, different sub-banks provide each sub-vector w b The chip microengine addresses each sub-memory bank, for example directly, algorithmically, or using a LUT.

[0046] The weights of a neural network, which may be referred to as a weight block, may be represented as a matrix W. In some embodiments, W is of B×B dimensions, where B is referred to as the block size. The W matrix may be decomposed into B sub-vectors w b , each of which is 1B× dimensional. Instead of reading the matrix W from adjacent locations in memory, we can read the subvectors w b Construct a matrix W, the subvector w b Read from non-contiguous locations in memory. If the subvector w b If the weight blocks W are common among several weight blocks, then only w needs to be stored in the weight memory. bThis makes it possible to reduce the amount of physical weight memory required and increase the effective number of weight blocks that can be stored in the same amount of physical weight memory.

[0047] For example, a weight matrix W of size 4 may be defined as a combination of vectors as shown in Equation 1.

[0048]

[0049] w0=[3,5,1,6]

[0050] w1 = [2, 4, 2, -1]

[0051] w2 = [-2, -4, 7, 0]

[0052] w3 = [1, 2, -2, -1]

[0053] Equation 1

[0054] Reference now Fig. 9 , a method of operating a neural inference processor is shown according to an embodiment of the present disclosure. At 901, a global scheduler provides synaptic weights from a neural network model memory to each of a plurality of cores. At 902, a local controller of each core loads input activations from an activation memory of the core to a neural computing unit of the core. At 903, the neural computing unit of each core applies the plurality of synaptic weights to the plurality of input activations to generate a plurality of output activations. At 904, the local controller of each core stores a plurality of output activations from the neural computing unit of the core to the activation memory of the core.

[0055] Reference now Fig.10 , a schematic diagram of an example of a computing node is shown. Computing node 10 is only one example of a suitable computing node and is not intended to suggest any limitation on the scope of use or functionality of the embodiments of the present invention described herein. Regardless, computing node 10 is capable of implementing and / or performing any of the functions set forth above.

[0056] In computing node 10, there is a computer system / server 12, which can operate with many other general or special computing system environments or configurations. Examples of well-known computing systems, environments and / or configurations suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, fat clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems or devices, etc.

[0057] Computer system / server 12 may be described in the general context of computer system executable instructions, such as program modules, executed by the computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform specific tasks or implement specific abstract data types. Computer system / server 12 may be practiced in a distributed cloud computing environment, where tasks are performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in local and remote computer system storage media including memory storage devices.

[0058] like Fig.10 As shown, the computer system / server 12 in the computing node 10 is shown in the form of a general-purpose computing device. The components of the computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including the system memory 28 to the processor 16.

[0059] Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, these architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0060] Computer system / server 12 typically includes a variety of computer system readable media. Such media can be any available media that can be accessed by computer system / server 12, and it includes volatile and nonvolatile media, removable and non-removable media.

[0061] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer system / server 12 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, and commonly referred to as a "hard drive"). Although not shown, a disk drive for reading from and writing to a removable, non-volatile disk (e.g., a "floppy disk") may be provided, as well as an optical drive for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media. In this case, each may be connected to the bus 18 via one or more data media interfaces. As will be further depicted and described below, the memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of an embodiment of the present invention.

[0062] A program / utility 40 having a set (at least one) of program modules 42, as well as an operating system, one or more application programs, other program modules, and program data, may be stored in memory 28 as an example and not by way of limitation. Each of the operating system, one or more application programs, other program modules, and program data, or some combination thereof, may include an implementation of a networking environment. Program modules 42 generally perform the functions and / or methods of embodiments of the present invention described herein.

[0063] The computer system / server 12 may also communicate with one or more external devices 14 such as a keyboard, pointing device, display 24, etc.; one or more devices that enable a user to interact with the computer system / server 12; and / or any device that enables the computer system / server 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may occur via an input / output (I / O) interface 22, however, the computer system / server 12 may communicate with one or more networks such as a local area network (LAN), a general purpose wide area network (WAN), and / or a public network (e.g., the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system / server 12 via a bus 18, and it should be understood that, although not shown, other hardware and / or software components may be used in conjunction with the computer system / server 12, examples of which include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0064] The present invention may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium (or multiple media) having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to perform various aspects of the present invention.

[0065] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be interpreted as a temporary signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0066] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0067] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​(such as Smalltalk, C++, etc.) and conventional procedural programming languages ​​(such as "C" programming language or similar programming languages). The computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider through the Internet). In certain embodiments, in order to perform various aspects of the present invention, the electronic circuit including, for example, a programmable logic circuit, a field programmable gate array (FPGA) or a programmable logic array (PLA) can execute the computer-readable program instructions to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions.

[0068] Various aspects of the present invention are described herein with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to embodiments of the present invention. It will be understood that each frame of the flowchart and / or block diagram and the combination of frames in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0069] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more boxes of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can guide the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0070] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operational steps are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0071] Flowcharts and block diagrams in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each frame in the flow chart or block diagram can represent a module, segment or part of an instruction, which includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions mentioned in the frame may not occur in the order mentioned in the figure. For example, two frames shown in succession can actually be performed substantially simultaneously, or these frames can sometimes be performed in reverse order, depending on the function involved. It will also be noted that each frame of the block diagram and / or flow chart illustration and the combination of the frames in the block diagram and / or flow chart illustration can be realized by a dedicated hardware-based system that performs a specified function or action or performs a combination of special hardware and computer instructions.

[0072] The description of various embodiments of the present invention has been given for the purpose of illustration, but it is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are selected to best explain the principles of the embodiments, practical applications, or technical improvements existing in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A system comprising: Multiple cores, each core includes a neural computation unit adapted to apply a plurality of synaptic weights to a plurality of input activations to produce a plurality of output activations, an activation memory adapted to store said input activations and said output activations, a local controller adapted to load the input activations from the activation memory to the neural computation unit and to store the plurality of output activations from the neural computation unit to the activation memory; A neural network model memory, adapted to store the plurality of synaptic weights; A global scheduler operatively coupled to the plurality of cores, adapted to load the synaptic weights and instructions from the neural network model memory to each core without using the local controllers, wherein the global scheduler communicates with each local controller to share scheduling operations of the system, wherein the global scheduler provides the instructions directly to the neural computational units.

2. The system according to claim 1, wherein: The global scheduler is adapted to provide local controller instructions to each core.

3. The system of claim 1, wherein the global scheduler provides the same synaptic weight to each of the cores. 4 . The system of claim 2 , wherein the global scheduler provides the same local controller instructions to each of the cores.

5. The system of claim 1, wherein the global scheduler provides different synaptic weights to at least some of the cores.

6. The system of claim 2, wherein the global scheduler provides different local controller instructions to at least some of the cores.

7. The system of claim 1, further comprising a network coupled to the plurality of cores.

8. The system of claim 7, wherein the global scheduler provides the synaptic weights via the network.

9. The system of claim 7, wherein the global scheduler provides local controller instructions to each core via the network.

10. The system of claim 7, wherein the network comprises a tree network.

11. The system of claim 7, wherein the network comprises a line broadcast network.

12. The system of claim 11, wherein the network comprises a plurality of rows, each row being connected to a subset of the plurality of cores.

13. The system of claim 12, wherein each row includes an interconnect operable to select between a broadcast input and a row-specific input.

14. The system of claim 13, wherein the interconnect is operable by the global scheduler.

15. The system of claim 1, wherein the global scheduler is adapted to provide the synaptic weights from a plurality of non-contiguous regions of the neural network model memory.

16. A system comprising: a neural computation unit adapted to apply a plurality of synaptic weights to a plurality of input activations to produce a plurality of output activations; A neural network model memory, adapted to store the plurality of synaptic weights; an activation memory adapted to store said input activations and said output activations; A global scheduler and a local controller are operatively coupled to the neural computing unit, the neural network model memory, and the activation memory, the global scheduler being adapted to: loading the synaptic weights and instructions from the neural network model memory to the neural computing unit without using the local controller; and The local controller is suitable for: loading the input activations from the activation memory into the neural computation unit; The plurality of output activations from the neural compute units are stored in the activation memory, wherein the global scheduler communicates with the local controllers to share scheduling operations of the system, wherein the global scheduler provides the instructions directly to the neural compute units.

17. A method comprising: A global scheduler loads synaptic weights and instructions from a neural network model memory to each of the multiple cores without using a local controller; loading, by the local controller of each core, input activations from an activation memory of the core to a neural compute unit of the core; applying, by a neural computation unit of each core, the plurality of synaptic weights to the plurality of input activations to produce a plurality of output activations; The multiple output activations from the neural compute unit of each core are stored in the activation memory of the core by the local controller of each core, wherein the global scheduler communicates with the local controller of each core to share scheduling core operations, and wherein the global scheduler directly provides the instructions to the neural compute unit of each core.

18. The method according to claim 17, further comprising: Local controller instructions are provided by the global scheduler to each of the plurality of cores.

19. The method of claim 17, wherein the global scheduler provides the same synaptic weight to each of the plurality of cores.

20. The method of claim 18, wherein the global scheduler provides the same local controller instructions to each core of the plurality of cores.

21. The method of claim 17, wherein the global scheduler provides different synaptic weights to at least some of the cores.

22. The method of claim 18, wherein the global scheduler provides different local controller instructions to at least some of the cores.

23. The method of claim 17, wherein the global scheduler provides the synaptic weights via a network.

24. The method of claim 17, wherein the global scheduler provides local controller instructions to each core via a network.

25. The method of claim 17, wherein the global scheduler provides the synaptic weights from multiple non-contiguous regions of the neural network model memory.

Images