Neural network processor comprising multiple levels of hierarchical aggregation of compute and memory elements

Abstract

A novel and useful neural network (NN) processing core suitable for implementing artificial neural networks (ANN). The NN processor is composed of independent compute units organized in a hierarchical architecture. Homogeneity enables simpler management and control of similar compute units aggregated in multiple levels. Compute units are designed with minimal overhead possible, with additional features and capabilities aggregated at higher levels in the hierarchy. On-chip memory provides storage for what is inherently needed for basic operations at a particular level of the hierarchy, coupled with compute resources at an optimal ratio. Lean control provides just enough signaling to manage operations needed only at particular levels. Dynamic resource assignment agility is provided, which can be adjusted as needed according to resource availability and capacity of the device.

Description

TECHNICAL FIELD

[0001] The subject matter disclosed herein relates to the field of neural networks, and more particularly, to a neural network (NN) processing engine suitable for implementing an artificial neural network (ANN). BACKGROUND

[0002] Artificial neural networks (ANNs) are computing systems inspired by biological neural networks that constitute animal brains. These systems learn (i.e., improve performance) by considering examples to complete tasks, often without needing to be specifically programmed with tasks-specific programming by extracting key features in those tasks and generalizing from a large number of examples. For example, in image recognition, they can learn to identify images containing cats by analyzing exemplary images that have been manually labeled as "cat" or "not cat" and using the results of the analysis to identify cats in other images. They have found uses in most of the applications in application programs that have been difficult to express using rule-based programming in traditional computer algorithms.

[0003] ANNs are based on a set of connected units called artificial neurons (analogous to neurons in a biological brain). Each connection or synapse between neurons can pass a signal to another neuron. A receiving or postsynaptic neuron is connected to another or several neurons and can process the signal and then signal downstream neurons connected to it through synapses (also called axons). A neuron can have a state, generally represented by a real number, typically between 0 and 1. Neurons and synapses can also have weights that change with learning, which can increase or decrease the strength of the signal it sends to downstream. In addition, they can have a threshold, such that a downstream signal is only sent if the aggregate signal is below or above the level.

[0004] Generally, neurons are organized in layers. Different layers can perform different types of transformations on their inputs. A signal can travel from a first (i.e., input) layer to a last (i.e., output) layer after multiple passes through the layers.

[0005] The initial goal of neural network methods was to solve problems in the same way that the human brain does. Over time, the focus has centered on matching specific mental abilities, which has led to a departure from biology, such as backpropagation, or passing information backward and adjusting the network to reflect that information.

[0006] The components of an artificial neural network include: (1) neurons with an activation threshold; (2) connections and weights for passing the output of a neuron; (3) a propagation function for computing the input to a neuron from the output of the previous generation of neurons; and (4) a learning rule, which is an algorithm that modifies the neural network parameters so that a given input produces a desired result, which is typically equivalent to modifying the weights and thresholds.

[0007] Given a specific task to be solved and a class of functions F, learning requires using a set of observations to find a function that solves the task in some best sense. A cost function C is defined such that: for the best solution, no other solution has a cost less than the cost of the best solution.

[0008] The cost function C is a measure of the distance of a particular solution from the best solution to the problem to be solved. Learning algorithms search the solution space to find a function with the smallest possible cost.

[0009] Neural networks can be trained using backpropagation, which is a method of computing the gradient of a loss function with respect to the weights in an ANN. The weight updates of backpropagation can be done by the well-known stochastic gradient descent technique. It should be noted that the choice of cost function depends on factors such as the type of learning (e.g., supervised, unsupervised, reinforcement) and the activation function.

[0010] There are three main learning paradigms, and each corresponds to a specific learning task: supervised learning, unsupervised learning, and reinforcement learning. Supervised learning uses a set of example pairs, and the goal is to find a function in the allowed class of functions that matches the examples. A commonly used cost is the mean squared error, which attempts to minimize the average squared error between the network output and the target value over all example pairs. Minimizing this cost using gradient descent with a class of neural networks called multilayer perceptrons (MLPs) results in the backpropagation algorithm for training neural networks. Examples of supervised learning include image recognition, i.e., classification and regression, i.e., function approximation.

[0011] In unsupervised learning, some data is given and a cost function is minimized, which can be any function of the data and the network output. The cost function depends on the task (i.e., the model domain) and any prior assumptions (i.e., the implicit properties of the model, its parameters, and the observed variables). Tasks that fall into the unsupervised learning paradigm are often estimation problems; applications include clustering, estimation of statistical distributions, compression, and filtering.

[0012] In reinforcement learning, data is typically not provided, but is generated by the agent's interaction with the environment. At each time point, the agent performs an action, and the environment generates an observation and an instantaneous cost according to some usually unknown dynamics. The goal is to discover a policy for selecting actions that minimizes some measure of long-term cost (e.g., expected cumulative cost). The dynamics of the environment and the long-term cost of each policy are usually unknown but estimable.

[0013] One common application of neural networks today is the analysis of video streams, i.e., machine vision. Examples include: industrial factories, where machine vision is used on assembly lines that manufacture goods; autonomous vehicles, where machine vision is used to detect objects in the vehicle's path and around the vehicle, etc.

[0014] Artificial neural networks (ANN) have an inherent structure that greatly depends on a set of parameters attributed to so-called 'network models'. These parameters are often referred to as the 'weights' of the network, as they tend to operate as scaling factors for other intermediate values as they propagate along the network. The process for determining the weight values is called training, as described above. When training is complete, the network enters a steady state and can now be used with new (i.e. unknown) data to extract information. This phase is called the 'inference' phase.

[0015] During inference, the resulting set of parameters (i.e. weights) can be observed and manipulated to produce better performance (i.e. representation). Methods for pruning and quantizing weights are known. However, these methods are applied to the trained model only prior to entering the inference phase. This approach does produce better performance in execution. However, it does not fully explore and exploit the possibility of modifying the weights. Additionally, existing solutions apply quantization of weights only after training, when the weights of the ANN have already converged to a satisfactory level. SUMMARY

[0016] The present invention is a neural network (NN) processing engine suitable for implementing artificial neural networks (ANN). The grain nature of the NN processing engine or processor (also known as a neural computer or neural chip) enables easy identification of the basis of neural networks and implementation of a wide range of neural network models in a very efficient manner. The NN processor offers a certain flexibility in choosing the balance between (1) overgeneralizing about the architecture of the computation and (2) aggregating specialized compute units. The present invention provides an improved balance specific to neural networks and attempts to meet the required capacity with the appropriate capacity. As a result, the resulting architecture is more efficient and provides significantly higher compute unit density and much lower power consumption per unit.

[0017] Several key features of the architecture of the NN processor of the present invention include the following: (1) the compute units are independent and configured to exploit to the fullest their target tasks; (2) the hierarchical architecture provides homogeneity and self-similarity, enabling simpler management and control of similar compute units aggregated in multiple levels in the hierarchy; (3) the compute units are designed with as minimal overhead as possible, with additional features and capabilities placed at higher levels in the hierarchy (i.e. aggregated); (4) on-chip memory provides storage for what is inherently needed for the basic operations at a specific hierarchy, coupled with the compute resources at the optimal ratio; (5) lean control provides just enough control to manage operations needed only at specific levels; and (6) dynamic resource assignment flexibility can be adjusted as needed according to availability and capacity.

[0018] Such, additional and / or other aspects and / or advantages of embodiments of the present application are set forth in the detailed description which follows; possibly inferable from the detailed description; and / or learnable by practice of an embodiment of the present application.

[0019] According to the present application there is therefore provided a neural network (NN) processor circuit for performing neural network computations for an artificial neural network (ANN) having one or more network layers, comprising: a plurality of compute circuits, each compute circuit comprising a compute element, an associated dedicated memory element and related control logic, and operating to process an input data stream associated with the ANN; wherein the plurality of compute circuits are aggregated in a plurality of levels to form a hierarchy; and wherein higher levels in the hierarchy are generally more complex and comprise a smaller number of instantiations than lower levels.

[0020] According to the present application there is therefore provided a neural network (NN) processor circuit for performing neural network computations for an artificial neural network (ANN) having one or more network layers, comprising: a plurality of compute circuits, each compute circuit comprising a compute element, an associated dedicated memory element and related control logic, and operating to process an input data stream associated with the ANN; wherein the plurality of compute circuits are aggregated in a plurality of levels to form a hierarchy; and wherein higher levels in the hierarchy are generally more complex and comprise a smaller number of instantiations than lower levels.

[0021] According to the present application there is therefore provided a neural network (NN) processor circuit for performing neural network computations for an artificial neural network (ANN) having one or more network layers, comprising: a plurality of compute circuits, each compute circuit comprising a compute element, an associated dedicated memory element and related control logic, and operating to process an input data stream associated with the ANN; wherein the plurality of compute circuits are aggregated in a plurality of levels to form a hierarchy; and wherein higher levels in the hierarchy are generally more complex and comprise a smaller number of instantiations than lower levels. BRIEF DESCRIPTION OF DRAWINGS

[0022] The application is explained in further detail, in the following exemplary embodiments and with reference to the drawings, wherein identical or similar elements can be denoted by identical or similar reference numerals, and wherein features of various exemplary embodiments can be combinable. The application is described by way of example only in the accompanying drawings, which are depicted by way of illustration only, and in which:

[0023] Figure 1 is a block diagram showing an exemplary computer processing system suitable for implementing one or more portions of the present application;

[0024] Figure 2 is a diagram showing a first example artificial neural network;

[0025] Figure 3 is a diagram showing an example multi-level abstraction of a neural network processing system;

[0026] Figure 4 is a high-level block diagram showing an example SoC-based NN processing system including one or more NN processing cores;

[0027] Figure 5 is a high-level block diagram showing an example NN processing core in more detail;

[0028] Figure 6 is a block diagram showing a first example low-level processing element (PE) in more detail;

[0029] Figure 7A is a block diagram showing a second example low-level processing element (PE) in more detail;

[0030] Figure 7B is a block diagram showing a quad multiplier of a PE in more detail;

[0031] Figure 8 is a high-level block diagram showing a first example sub-cluster in more detail;

[0032] Figure 9 is a high-level block diagram showing a second example sub-cluster in more detail;

[0033] Figure 10 is a high-level block diagram showing a first example cluster in more detail;

[0034] Figure 11 is a high-level block diagram showing a second example cluster in more detail;

[0035] Figure 12 is a high-level block diagram showing an inter-cluster crossbar in more detail;

[0036] Figure 13 is a diagram showing a first example memory windowing scheme;

[0037] Figure 14 is a diagram showing a second example memory windowing scheme;

[0038] Figure 15 is a diagram showing a first example memory accessibility between a compute element and a memory element, including window size and compute access configurability;

[0039] Figure 16is a diagram showing a second exemplary memory accessibility between compute elements and memory elements;

[0040] Figure 17 is a diagram showing an exemplary scatter / gather based resource windowing technique;

[0041] Figure 18 is a block diagram showing an exemplary memory contention resolution scheme;

[0042] Figure 19 is a high level block diagram showing a first exemplary tier controller in more detail;

[0043] Figure 20 is a high level block diagram showing the tier controller interface to L3 memory and sub-clusters in more detail;

[0044] Figure 21 is a high level block diagram showing a second exemplary tier controller in more detail;

[0045] Figure 22 is a high level block diagram showing an exemplary NN processor compiler / SDK;

[0046] Figure 23 is a diagram showing the flexible processing granularity of a NN processor and the tradeoff of related memory versus latency;

[0047] Figure 24 is a diagram showing a first exemplary multi-NN processor SoC system of the present invention;

[0048] Figure 25 is a diagram showing a second exemplary multi-NN processor SoC system of the present invention;

[0049] Figure 26 is a diagram showing a first exemplary multi-NN processor SoC system of the present invention;

[0050] Figure 27 is a diagram showing a first exemplary multi-NN processor SoC system of the present invention;

[0051] Figure 28 is a diagram showing an exemplary mapping strategy for a first exemplary artificial neural network of Figure 2

[0052] Figure 29 is a diagram showing a second exemplary artificial neural network;

[0053] Figure 30 is a diagram showing an exemplary multi-NN processor SoC system for an ANN of Figure 29

[0054] Figure 31 ​​This is a diagram illustrating a third exemplary artificial neural network;

[0055] Figure 32 It is a display Figure 31 A diagram of the first exemplary multi-NN processor SoC system of ANN; and

[0056] Figure 33 It is a display Figure 31 A diagram of a second exemplary multi-NN processor SoC system of ANN. Detailed Implementation

[0057] In the following detailed description, numerous specific details are set forth in order to provide a full understanding of the invention. However, those skilled in the art will understand that the invention can be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the invention.

[0058] Other objects and advantages of the invention will become apparent from the following description taken in conjunction with the accompanying drawings, among the benefits and improvements already disclosed. Detailed embodiments of the invention are disclosed herein; however, it should be understood that the disclosed embodiments are merely illustrative of the invention as it may be manifested in various forms. Furthermore, each of the examples given in connection with the various embodiments of the invention is illustrative and not restrictive.

[0059] The subject matter considered to be the invention is specifically pointed out and clearly claimed in the concluding section of this specification. However, the invention can be best understood with reference to the following detailed description while reading the accompanying drawings, regarding its organization and methods of operation, along with its objectives, features, and advantages.

[0060] The accompanying drawings form part of this specification and include illustrative embodiments of the invention, illustrating its various objectives and features. Furthermore, the drawings are not necessarily drawn to scale, and some features may be enlarged to show details of specific components. Additionally, any measurements, specifications, etc., shown in the drawings are intended to be illustrative rather than restrictive. Therefore, the specific structural and functional details disclosed herein should not be construed as restrictive, but merely as a representative basis for teaching those skilled in the art to employ the invention in different ways. Furthermore, reference numerals may be repeated in these drawings where deemed appropriate to indicate corresponding or similar elements.

[0061] Because the illustrated embodiments of the present invention can be implemented in most cases using electronic components and circuits known to those skilled in the art, the details will not be explained to any greater extent than is deemed necessary in order to understand and comprehend the basic concepts of the invention and to avoid obscuring or diverting the teachings of the invention.

[0062] Any reference in the specification to a method should be applied mutatis mutandis to a system capable of executing the method. Any reference in the specification to a system should be applied mutatis mutandis to a method that can be executed by the system.

[0063] Throughout the specification and claims, the following terms take the meanings explicitly associated herein, unless the context clearly dictates otherwise. The phrase "in one embodiment," "in one exemplary embodiment," and "in some embodiments," as used herein does not necessarily refer to one and the same embodiment, though it may. Also, the phrase "in another embodiment," "in an alternative embodiment," and "in some other embodiments," as used herein does not necessarily refer to a different embodiment, although it can. Thus, various embodiments of the application can be readily combined into one or more embodiments without departing from the scope or spirit of the application.

[0064] Additionally, the term "or" as used herein, unless otherwise indicated, is the inclusive "or" operator and is equivalent to the term "and / or". The term "based on" is not exclusive and allows for being based on additional factors, unless otherwise indicated. Also, throughout the specification, "a," "an," and "the" are meant to include plural references unless the context clearly indicates otherwise. "In" includes "in" and "on."

[0065] As will be appreciated by one of skill in the art, the present application can be embodied as a system, method, computer program product, or any combination thereof. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that can all generally be referred to herein as a "circuit," "module" or "system." Furthermore, embodiments of the present application can take the form of a computer program product embodied in any tangible medium of expression having computer usable program code embodied in the medium.

[0066] The present application can be described in the general context of computer- executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The present application can be practiced in a distributed computing environment where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote computer storage media including memory storage devices.

[0067] Any combination of one or more computer usable or computer readable medium(s) can be utilized. The computer-usable or computer-readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. More specific examples (a non- exhaustive list) of the computer-readable medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CDROM), an optical storage device, a transmission media such as those supporting the Internet or an intranet, or a magnetic storage device. Note that the computer-usable or computer-readable medium could even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, and then stored in a computer memory. In the context of this document, a computer-usable or computer-readable medium can be any medium that can contain or store the program for use by or in connection with an instruction execution system, apparatus, or device.

[0068] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++, C#, or the like, conventional procedural programming languages, such as the "C" programming language, or the like, and functional programming languages, such as Prolog and Lisp, machine code, assembler, or any other suitable programming language. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer, for example, through the Internet using an Internet Service Provider.

[0069] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0070] These computer program instructions can also be stored in a computer readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0071] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0072] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0073] In addition, the present application can operate in systems including video and still cameras, sensors, etc., such as exist in automated factories, autonomous vehicles, in mobile devices such as tablets and smartphones, smart meters installed in power grids, and control systems for robotic networks. In general, any computing device that can host an agent can be used to implement the present application.

[0074] In Figure 1A block diagram showing an exemplary computer processing system that exhibits suitable for implementing one or more portions of the present application is shown in FIG. 1. An exemplary computer processing system (generally designated 10) for implementing the present application includes a general purpose computing device 11. The computing device 11 includes a central processing unit (CPU) 12, a host / PIC / cache bridge 20, and a main memory 24.

[0075] The CPU 12 includes one or more general purpose CPU cores 14 and optionally one or more special purpose cores 16 (e.g., DSP cores, floating point, GPU, and neural network optimized cores). The one or more general purpose cores execute general purpose opcodes, while the special purpose cores perform functions specific to their purpose. The CPU 12 is coupled to the host / PCI / cache bridge 20 through a CPU local bus 18. A second level (i.e., L2) cache memory (not shown) can be coupled to a cache controller in the chipset. For some processors, an external cache can include an LI or first level cache. The host / PCI / cache bridge 20 is coupled to the main memory 24 through a memory bus. The main memory includes dynamic random access memory (DRAM) or extended data out (EDO) memory, or other types of memory such as ROM, static RAM, flash and non-volatile static random access memory (NVSRAM), bubble memory, etc.

[0076] The computing device 11 also includes various system components coupled to the CPU through a system bus 26 (e.g., PCI). The host / PCI / cache bridge 20 interfaces to the system bus 26, such as a peripheral component interconnect (PCI) bus. The system bus 26 can include any of several types of bus structures using any of a variety of bus architectures. Exemplary architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI), also known as Mezzanine bus.

[0077] The various components connected to the system bus include, among other things, non-volatile memory 28 (e.g., a hard disk-based data storage device), video adapter 30 connected to display 32, user input interface (I / F) 31 connected to one or more input devices such as mouse 34, tablet 35, microphone 36, keyboard 38, and modem 40, network interface 42, peripheral interface 52 connected to one or more external peripheral devices such as printer 54 and speakers 56. Network interface 42 is coupled to one or more devices such as data storage device 46, remote computer 48 running one or more remote applications 50, through network 44, which can include the Internet cloud, a local area network (LAN), a wide area network (WAN), a storage area network (SAN), and the like. A small computer system interface (SCSI) adapter (not shown) can also be coupled to the system bus. The SCSI adapter can be coupled to various SCSI devices, such as a CD-ROM drive, a tape drive and the like.

[0078] Non-volatile memory 28 can include various non-removable / removable, volatile / non-volatile computer storage media, such as a hard disk drive that reads from or writes to non-removable, non-volatile magnetic media, a magnetic disk drive that reads from or writes to a removable, non-volatile magnetic disk, and an optical disk drive that reads from or writes to a removable, non-volatile optical disk such as a CD ROM or other optical media. Other non-removable / removable, volatile / non-volatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape, flash memory cards, solid state RAM, and solid state ROM, and the like.

[0079] A user can enter commands and information into the computer through input devices connected to user input interface 31. Examples of input devices include a keyboard and pointing device, mouse, trackball or touchpad. Other input devices can include a microphone, joystick, game pad, satellite dish, scanner, or the like.

[0080] Computing device 11 can operate in a networked environment using logical connections to one or more remote computers, such as remote computer 48. The remote computer can include a personal computer (PC), a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above. Such a networked environment is common in an office, enterprise-wide computer network, intranet and the Internet.

[0081] When used in a LAN networking environment, the computing device 11 is connected to the LAN through a network interface 42. When used in a WAN networking environment, the computing device 11 typically includes a modem 40 or other means for establishing communications over the WAN, such as the Internet. The modem 40, which can be internal or external, is connected to the system bus 26 via the user input interface 31, or other appropriate mechanism.

[0082] The computing system environment is one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the application. Neither should the computing environment be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment.

[0083] In one embodiment, software adapted to implement the systems and methods of the present application can also reside within the cloud. Cloud computing provides computation, software, data access, and storage services that do not require end-user knowledge of the physical location and configuration of the system that delivers the services. Cloud computing encompasses any subscription-based or pay-per- use service in which a user does not directly own or control the physical resources that deliver the service. Cloud computing typically involves providing dynamically scalable and often virtualized resources via the Internet.

[0084] In another embodiment, software adapted to implement the systems and methods of the present application is adapted to reside on a computer-readable medium. A computer-readable medium can be any available medium or means that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by a computer. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The software adapted to implement the systems and methods of the present application can also reside within static or dynamic memory, or on a firmware of a processor within a computer system (i.e., within the internal memory of a microcontroller, microprocessor, or microcomputer).

[0085] Other digital computer system configurations can be used to implement the systems and methods of the present application and, as such, are intended to be within the spirit and scope of the systems and methods of the present application Figure 1 of a representative digital computer system and are intended to be within the spirit and scope of the present application.

[0086] Such digital computer systems, when programmed to perform particular functions pursuant to instructions from program software implementing the systems and methods of the present application, become in effect a special purpose computer dedicated to the method of the present application. The techniques necessary to do this are well known to those skilled in the art of computer systems.

[0087] It should be noted that the computer programs implementing the systems and methods of the present application will typically be distributed to users on a distribution medium such as floppy disk, CD ROM, DVD, flash memory, portable hard drive, etc. From there, they will typically be copied to a hard disk or similar intermediate storage medium. When the programs are to be run, they will be loaded from their distribution medium or their intermediate storage medium into the execution memory of the computer, thereby configuring the computer to act according to the method of the present application. All of these operations are well-known to those skilled in the art of computer systems.

[0088] The flow and block diagrams in the drawings show architectural, functional, and operational representations of possible implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow and block diagrams can represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing one or more specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block and / or flow diagrams and combinations thereof can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0089] Neural network (NN) processing core

[0090] At a very high level, an ANN is essentially a function with a large number of parameters that maps between an input space to an output space. Thus, an ANN can be viewed as a series of computations. However, an ANN has a particular internal structure and a set of properties. In view of this unique structure, a neural network (NN) processor includes a plurality of basic computing units that perform the same or similar mathematical operations that, when combined together, form a neural network.

[0091] The following set of symbols is used herein to uniquely describe the network:

[0092] ANN ∝ {X <s>< / s> <s>Y <t>< / t> ,M <w>< / w>} (1)

[0093] wherein:

[0094] X <s>< / s> <s>denotes an input dataset, characterized by a specific structure S;

[0095] Y <t>< / t> denotes an output dataset, having a format indicated by T;

[0096] Y <w>< / w> denotes an ANN model, in view of a set of parameters or weights (W), which is a function mapping inputs to outputs;

[0097] In Figure 2 a diagram illustrating an exemplary artificial neural network is shown. The exemplary ANN (generally designated 350) includes four network layers 352, including network layer 1 through network layer 4. Each network layer includes a plurality of neurons 354. Inputs X1 through X 14 are inputs to network layer 1. Weights 358 are applied to the inputs of each neuron in a network layer. The outputs of one network layer are generated, which form the inputs to the next network layer, until the final outputs 359 (output 1 through output 3) are generated.

[0098] In one embodiment, the architecture of the present invention includes a multi-layer architecture (i.e., not referring to ANN layers), which satisfies the computational needs of an artificial neural network in its entirety. The term multi-layer refers to an approach similar to the well-known ISO OSI layer model for networking, which describes the overall solution at different levels of abstraction.

[0099] In Figure 3 a diagram illustrating an exemplary multi-layer abstraction of a neural network processing system is shown. An equivalent model of neural network processing (generally designated 410) includes six layers, including: layer 1 (physical layer 412), which includes the physical primitives that make up the various units; layer 2 (unit layer 414), which includes the basic computational units that form the basis of a neural network; layer 3 (interconnect layer 416), which includes the interconnect fabric that provides network connectivity; layer 4 (management layer 418), which provides network level flow control, monitoring, and diagnostics; layer 5 (interface layer 420), which provides application layer interfaces and maps to the architecture primitives; and layer 6 (application layer 422), which includes the neural network-based applications.

[0100] In Figure 4 A high-level block diagram showing an exemplary system-on-a-chip (SoC) NN processing system including one or more NN processing cores is shown in FIG. 1. The SoC NN processing system (generally referenced 100) includes at least one NN processor integrated circuit (core) 102, optionally coupled to one or more additional internal or external NN processors 104 through one or more suitable chip-to-chip interfaces, a bus structure 106 adapted to couple the NN processors to various system-on-a-chip elements (SoC) 108, a microcontroller unit (MCU) subsystem 118, and one or more interfaces 126.

[0101] In one embodiment, the SoC 108 includes a bootstrap circuit 110, a debug circuit 112, a power supply circuit 114, and a clock circuit 116. The MCU subsystem 118 includes a controller 120, an instruction memory 122, and a data memory 124. The interfaces 126 include a pin multiplexer 139 and one or more well-known interfaces including a camera serial interface (CSI) 128, a display serial interface (DSI), an Ethernet 132, a universal serial bus (USB) 134, an internal integrated circuit (I 2 C) 136, a serial peripheral interface (SPI) 137, and a controller area network (CAN) 138. It is noted that these interfaces are shown as examples, as any combination of different interfaces can be implemented.

[0102] In Figure 5 A high-level block diagram showing an exemplary NN processing core in more detail is shown in FIG. 2. The NN processor (core) 60 includes several tiers of computing units. The lowest tier is a processing element (PE) 76 with its own dedicated internal tier 1 or Ll memory 78, where an individual neuron is implemented. N multiple PEs 76, along with a dedicated tier 2 or L2 memory 74, make up the next tier, called a sub-cluster 70. M multiple sub-clusters 70, along with a dedicated tier 3 or L3 memory 72, multiple activation functions 80, and multiple layer controllers (LCs) 82, make up a cluster 66. L multiple clusters, along with a dedicated tier 4 or L4 memory 64, are in the NN processor (core) 60, which also includes a NN manager 62 and a memory interface 68 to an off-chip tier 5 or L5 memory 98. Multiple bus interfaces 86 (i.e., chip-to-chip interfaces) couple the NN processor to other off-chip NN processor chips for additional network capacity. Additional bus interfaces 84 (i.e., chip-to-chip interfaces) couple the NN processor to a regular rule-based machine (RBM) co-processor 88, including a CPU 90, an instruction memory 92, and a data memory 94. In alternative embodiments, the RBM co-processor is optionally coupled to the NN processor (core) 60 through a suitable interface (e.g., GPU, IC, etc.). 2

[0103] ​It should be noted that in the exemplary NN processor implementation, the PEs include P = 16 neurons, the sub-clusters include N = 64 PEs, the clusters include M = 64 sub-clusters, and the NN core includes L = 8 clusters. It should be appreciated that the NN processor can be implemented with any desired number of levels and any number of compute units within each level, and is not limited to the examples provided herein for illustrative purposes only. Furthermore, any number of activation functions 80 and layer controllers 82 can be implemented at the cluster level or any other level, depending on the design goals and specific implementation of the NN processor.

[0104] In one embodiment, the NN manager 62 is a dedicated processor that controls, along with the functions of the driving network fabric, two data pipelines: one parallel pipeline and one serial pipeline. This processor performs dedicated operations that are dedicated to the control plane of the neural network itself. Exemplary operations include, but are not limited to, inference, training, loading weights, and updating weights. Load balancing and resource allocation are handled by an external software toolchain, which includes a set of tools that address these tasks, including a compiler, a mapper, and an allocator.

[0105] In one embodiment, the NN processor includes a shared memory for storing weights, and a dedicated memory element for storing context, enabling a relatively high data processing bandwidth. Furthermore, the NN processor includes a data plane and a control plane that are strictly separated from each other and provide out-of-band control to the compute elements. Furthermore, the NN processor includes a configurable interconnecter between the aggregation levels for producing dynamic and programmable data pipelines.

[0106] In another embodiment, the NN processor is capable of implementing multiple ANNs in parallel, where each ANN has one or more network layers. The NN processor is adapted to process one or more input data streams associated with the ANNs simultaneously. Since the architecture of the NN device is similar to the structure of the ANNs, the multiple ANNs can be viewed as a single wide ANN. It should be noted that when deploying multiple ANNs, the mapper in the external toolchain operates to map the available resources, given sufficient resources, while the NN manager manages the event triggers. In this case, each group of resources grouped within the "layers" of the ANNs are independent of each other due to the massive parallelism of the device.

[0107] Furthermore, the compute elements of the NN processor operate to function at any desired granularity of a subset of the input data streams, trading off memory elements usage versus latency, as described in greater detail below.

[0108] The NN processor of the present invention uses several design principles in its implementation, including: (1) timely use of system resources; (2) dynamic allocation of system resources as needed; (3) utilization of time and spatial domains to optimize utilization and efficiency; and (4) balancing the load on available system resources.

[0109] It should be noted that the present invention is well suited to implementing ANNs. Generally, ANNs are implemented in three phases: modeling, training, and inference, all of which are addressed to some extent by the NN processor of the present invention.

[0110] With respect to modeling, the NN processor is capable of changing model representations statically and dynamically, reflecting its flexibility. The use of the term "processor" rather than "accelerator" is used because the latter is typically a priori adapted to perform a predefined set of operations. With respect to training, the NN processor supports immediate and complementary training operations that allow for the implementation of training procedures. This includes: (1) running back and forth through the network (i.e., backpropagation); (2) dynamic application of dropout; and (3) immediate evaluation of layer performance and misbehavior detection. During inference mode, the ANN is executed in an optimal and efficient manner and applied to new inputs.

[0111] The NN processor of the present invention combines several features that are combined together to provide extremely high computational rates, small chip footprint, low power consumption, scalability, programmability, and flexibility to process many types of neural networks.

[0112] The first feature includes the computational structure (or computational capability) provided by the compute units, which are organized into various levels of aggregation or hierarchy, such as PEs, sub-clusters, clusters, NN cores, as described in the exemplary systems disclosed herein. The computational structure includes basic computational elements that are configured to meet the special nature of the computational needs of ANNs. Several features of the computational structure include: (1) a lean circuit architecture, allowing for a relatively large number of physical entities to be implemented; (2) a large number of product-sum operations performed at a time, with the addition performed as accumulation; (3) flexibility in the representation of numbers, including integers and floating points, and different bit widths; (4) four-way multiplier support, allowing for higher resolution computations; and (5) N-way ALU support for providing the ability to optimize memory bandwidth, i.e., rather than performing a single operation (such as y <- y + w*x) per cycle, more complex operations (such as y <- y + w1*x1 + w2*x2) can be implemented, reflecting a tradeoff between increased silicon complexity and reduced memory accesses needed.

[0113] The second feature is the strict separation of the control plane and control fabric from the data fabric, which enables the aggregation of control and very "lean” or "minimal” control of the entire data fabric (i.e., the data plane). The control plane is separated from the data plane, and it is thus aggregated in the sense that it can control a large number of compute units using relatively few control lines (e.g., in some cases by a single control line). For example, consider the multiplication circuit in a PE, a single control signal initiates multiplication operations in thousands of PEs simultaneously. In addition, the programmability of the control plane is separated from the programmability of the data plane. The large-scale parallelism of the data fabric of the NN core matches the lean fabric of the control plane.

[0114] This is in contrast to typical prior art approaches to in-band control, in which control signals are applied close to the data, which requires the replication of control signals through multiple compute elements. In addition, out-of-band control is in contrast to traditional microcontroller-based techniques, as it is not a von Neumann machine-based technique.

[0115] Another advantage of the separation of control and data fabric is that control is still programmable. The non-rigid implementation of the control fabric and the general nature of the compute units (i.e., PEs, sub-clusters, clusters, etc.) allow the NN core to handle multiple types of ANNs (such as convolutional NNs (CNNs), recurrent NNs (RNNs), deep NNs (DNNs), MLPs, etc.), as well as more complex implementations of the above in convolutional mode and fine combinations and characteristics of each (e.g., strides, padding, etc.).

[0116] The third feature is the construction of the memory fabric, including memory windowing. In addition to the positioning and hierarchical structure of the memory, what is also provided simultaneously with the large number of compute units is high-bandwidth access to the memory. This is achieved by zooming in the access of a particular compute unit to only a small portion of the memory. Thus, full random access to the entire memory is not provided. Rather, access is provided to only a relatively small window of the memory. This allows simultaneous access across thousands of compute units, thus representing a tradeoff between bandwidth and random accessibility. Since the single compute unit memory access pattern is constructed by the ANN and explicitly defined, and full random access to the entire memory is not required, the access can be “windowed” to only those few blocks of memory needed by the particular compute unit. Thus, extremely high memory bandwidth is achieved, whereby thousands of compute units can access the memory simultaneously, with the tradeoff being that only the memory that is “local” to the compute unit is accessed.

[0117] In one embodiment, the architecture of the NN processor includes a control plane and a data plane (or control fabric and data fabric). The control plane is responsible for configuring and controlling all the data computation units in the NN processor. It includes a dataflow machine or processor that in one embodiment includes microcode tailored for neural network operations. In the exemplary NN processor described herein, the control plane manages clusters 66 that serve as aggregators for the next aggregation layer (i.e., subclusters 70). The subclusters in turn include the most basic unit, a processing element (PE) 76, which is composed of a multiply-accumulate (MAC) circuit and local memory. The PE hierarchy contains a set of neuron entities that exist in a typical neural network.

[0118] One important aspect of implementing an ANN in a NN processor is the control and interconnection of all the computation elements. The present invention utilizes a very large number of computation elements in the ANN. One feature of the device control fabric is that it is relatively very lean, as it is shared among a large number of computation resources. In one embodiment, the NN processor features (1) a strict separation between data and control, where control signaling is performed out-of-band and does not include any data-driven memory access; (2) dynamic mapping between control and additional computation resources; and (3) flexibility and programmability (i.e., at compile time) of the control fabric. In addition, the NN processor includes a layer controller that includes a microcode machine that allows full access control signaling of the computation elements, memory, etc.

[0119] Note that data-driven memory access indicates an access that involves observing the data flowing through the data pipes. The NN processor does not need this. Note that data-driven memory access is common in rule-based machines, as the nature of the rules depends on the data, and thus the control has to be interleaved with the data. For example, consider the statement: if (x > some_value) then do A. This implies that every input ‘x’ needs to be observed. In contrast, consider a machine that compares many inputs to a threshold. In this case, the microcode only needs to trigger the operation that applies a large number of comparators. However, this approach cannot be employed in RBMs, as it implies a large number of operations that must be hardwired, which negates the possibility of programming the machine.

[0120] In contrast, the NN processor uses a very limited set of operations to operate on the data. The nature of the processing flow does not involve the values of the data. Thus, the control can be aggregated and a relatively small number of control signals can be used to drive the vast number of computation elements. For example, in a NN device, a control bus of 64 control signals is needed to control thousands of computation units.

[0121] In one embodiment, the NN processor is implemented such that functionality is provided at several aggregation points where it is needed, as described in more detail below. In addition, the NN processor is configured to be largely balanced in terms of compute and memory resources to ensure that the system achieves maximum utilization.

[0122] In the event that the capacity of the NN processor is insufficient for a particular neural network, the bus interface 86 provides for interconnecting additional NN processors 96 to extend beyond the limits of a single processor.

[0123] In one embodiment, the RBM coprocessor 88 is configured to support one or more primitives not supported by the NN processor. In addition, the coprocessor functions as a switch for tasks extracted from the ANN and assigned to the RBM.

[0124] The NN processor operates essentially as a dataflow machine, which means that computations are performed based only on the availability of data. Dataflow is divided between layers and layers (similar to layers in an ANN). The computation units inside a layer act in synchrony (start when data is ready at the input of the layer and end when they need new data and / or need to pass results to the next layer), which is when the state machine of the layer synchronizes with the state machines of the previous and / or next layer.

[0125] As one example, an MLP network with two dense layers can be mapped as (1) one layer that receives input from outside the core, (2) two layers that represent the neural network layers, and (3) one layer that sends results to outside the core.

[0126] In one embodiment, the input layer waits until it receives all inputs (e.g., 784 inputs for the well-known MNIST dataset), and then signals layer 1 that its inputs are ready. Layer 1 then performs all the required multiply-accumulate (MAC) operations, activation functions, and finally signals layer 2, which in turn repeats the same steps. When layer 2 is complete, it signals the output layer to send results outside the NN core.

[0127] In another embodiment, considering the same network, the NN core starts MACs on a smaller portion of the input data in layer 1, reducing the required buffering between the input layer and layer 1, at the expense of the complexity of the state machine in layer 1 and possible loss of computational efficiency during signaling.

[0128] Data is passed within a cluster 66 in the NN core over a shared L3 memory 72, while signaling is performed over dedicated interconnectors, described in more detail below. In one embodiment, the AXI4-Stream protocol is used between clusters, which handles both the data plane and the control plane. To prevent idling, the interconnectors between layers provide a double buffering mechanism, so that one layer writes its output to one buffer, while a second layer reads the previous output from a second buffer as its input.

[0129] In one embodiment, the use of a dataflow architecture in the neural network, together with a relatively limited set of basic operations, enables a significant reduction in the requirements for control distribution.

[0130] First, most of the information about the computations being performed is statically known at the time the network model is defined, and can therefore be loaded a priori over a narrowband interface, reducing the number of control lines required during computation. As a result, the code implementing the 'kernel' of a layer is split between a quasi-static configuration that is constant for each network model, and dynamic instructions that change throughout the computation.

[0131] Second, each dynamic 'instruction' actually comprises multiple instructions that instruct all the computational elements in a layer what to do in each cycle. Since each computational element has relatively simple functionality, the basic instructions themselves are relatively simple. Repetition (i.e. loop) and jump instructions are provided out-of-band to avoid wasting cycles.

[0132] Third, the static order of computations, combined with the appropriate arrangement of parameters in memory, enables sequential access to memory. Therefore, only address increment instructions are required for accessing memory, rather than full addressing.

[0133] Fourth, since the microcode is very compact, it can reside in on-chip SRAM without the need for prefetching, branch prediction, etc.

[0134] Fifth, while a layer comprises many processing elements (PEs), only one central state machine and a small number of slave state machines storing only sub-states are required to control the computation steps of the entire layer, where each of the state machines controls multiple PEs. In one embodiment, a global enable bit starts the execution of all state machines, and a global synchronous reset signal returns them to the initial state. Note that the reset has no effect on the configuration memory and data memory, since the control plane ensures that no invalid data is used.

[0135] Note that the term'model' is used to describe the quasi-static configuration that defines the dynamic behavior of all the computational units in the NN core. The model is typically similar to an ANN model, but other types of models can exist, such as models loaded for debugging purposes or models used to load weights into memory.

[0136] The configuration space is exposed in a memory-like interface, where modules are addressed using a hierarchical address space. Weight loading is typically performed prior to the configuration model and is achieved through configuration control signaling that copies the weights into the relevant memory blocks and sets the enable bits. The inference model is then loaded when the cluster is disabled, the controls are reset, and finally the cluster is enabled.

[0137] Memory hierarchy

[0138] In one embodiment, the memory structure of the NN processor is designed to meet the inherent properties of ANNs. Thus, the memory is structured in a hierarchical manner to meet the needs of various memory consumers. These consumers include: (1) inter-layer data (i.e., cross-layer input / output); (2) intra-layer information (i.e., context or intermediate results); and (3) weights. The various memory levels (e.g., five in the exemplary embodiment disclosed herein) vary from smaller, efficient, more localized memory to larger, less efficient global memory.

[0139] In one embodiment, the memory structure is organized and structured with the following: (1) localization of memory, where a computing element needs to access local data that allows for the accessibility of any given computing element to a predefined and limited memory entity; (2) structured organization, whereby the memory content is a priori organized in a given consistent matter; (3) limited recollection properties (i.e., read once), where most of the data is intrinsically volatile and once processed is completely consumed to the extent that it needs or does not need further access to it; (4) pipelined operation, where the output data of one computing element is used as input data of another computing element.

[0140] As mentioned above, each level contains its own local memory. PEs include LI memory, sub-clusters include L2 memory, clusters include L3 memory, NN cores include L4 memory, and L5 memory is located outside the SoC. One exemplary memory hierarchy is presented below in Table 1.

[0141] Table 1: Memory hierarchy

[0142]

[0143] where N represents the number of processing elements in a sub-cluster, M is the number of sub-clusters in a cluster, and L is the number of clusters in the NN processor device. It should be noted that the sizes indicated for each memory level LI to L5 are for demonstration purposes only. It should be understood that any desired memory size for the various memory levels can be implemented without departing from the scope of the present application.

[0144] It should be noted that the size of the lower memory layer (e.g., L1 in PE) is set to be small, but the memory layer has a large bandwidth. The size of the upper memory layer (e.g., L4 in NN core) is set to be much larger, but the memory has much less communication.

[0145] According to the present invention, as much memory as possible is kept as close as possible to where it is needed, while leveraging the locality of memory usage in an ANN to avoid providing full mesh access between the entire memory and computing elements. To overcome the limitations imposed by the above strategy, memory allocation to consumers is done in a "progressive" manner, such that each memory level with a specific role is supplemented by a higher level when it needs more resources, wherein the higher-level memory is used for "resource load balancing" among multiple layers with different requirements in the ANN.

[0146] It should be noted that in one implementation, this 'overflow' is a quasi-static feature because resource requirements are known at the time of model selection and therefore complex arbitration is not required. This feature allows for a significantly smaller amount of memory resources to be statically allocated in each tier, as they are allocated based on nominal conditions rather than worst-case scenarios.

[0147] Furthermore, the "gradual" allocation of memory is also characterized by the sliding window mechanism briefly described above, which is used in L3 memory and will be described in more detail below.

[0148] Processing element (PE)

[0149] In one implementation, the basic computing unit is a processing element (PE). Figure 6 The diagram shows a more detailed block diagram of an exemplary low-level processing element (PE). The PE (generally labeled 140) includes one or more multipliers 142 controlled by multiplication flip-flop 177, an adder 144 controlled by adder flip-flop 171, an L1 memory 150 including multiple registers 152, a destination multiplexer 146 controlled by destination control 175, a source multiplexer 148 controlled by source control 173, a write multiplexer 154 controlled by output control 178, and a read multiplexer 156 controlled by input control 179.

[0150] Input (X) 161 from input (X) memory 158 and weight (W) 163 from weight (W) memory 160 are provided to one or more multipliers 142 according to input control and weight control, respectively.

[0151] The most basic mathematical operations of neurons in a neural network are defined by the following:

[0152]

[0153] where:

[0154] x(i) indicates an input dataset, which is organized into a ID vector;

[0155] w(i,j) indicates a weight representing the ith input contribution to output j;

[0156] σ represents an activation function, which is typically a non-linear scalar function;

[0157] The basic computational unit is the PE and includes a multiply / accumulate entity that reflects the intrinsic operation of a neuron. Intermediate results or artifacts are stored in the LI memory 150, which is local to the PE. In the examples described herein, the LI memory has a particular depth and width, e.g., P = 16 neurons, each of which is 16 bits wide. It should be understood that LI memories having any desired depth and width can be used. The depth P of the LI memory reflects the number of simultaneous ‘neurons’ or ‘contexts’ that the PE can process. Note that more than P neurons (i.e., contexts) can be processed by storing intermediate results of additional neurons in L2 / L3 memory. Latency is affected because additional time is required to process additional neurons. Providing P neurons both exploits the spatial domain by constraining the computation fabric to an absolute minimum, while also exploiting the temporal domain by storing multiple contexts.

[0158] The ability to process internal contexts provides a number of capabilities, such as: (1) the ability to assign multiple logical neurons to a single physical neuron (each context stores the output of one neuron); (2) the storage of multiple intermediate results for the same input, resulting in simultaneous operations and hypothesis testing of different versions of weights (e.g., backpropagation results, gradient-based correction values, etc.); (3) multi-threaded inference of the same input for the purpose of applying the collective methods and majority vote extraction of a network committee; (4) running multiple networks when resources are available; and (5) load balancing based on overall network capacity managed by the NN manager.

[0159] In operation, the above Equation 2, which reflects the functionality of a neuron, is propagated over multiple time instances and implemented as provided in Listing 1 below. Note that this is merely an exemplary implementation, as other sequences can be used by loading different microcode into the layer controller (LC) 642 Figure 20 ) of the PE 640.

[0160] Listing 1: Neuron Functionality

[0161]

[0162]

[0163]

[0164] Reference Figure 6 , the PE includes count elements controlled separately for weights (w) and inputs (x) and separate control of the representation format for adders and multipliers. It also includes separately controlled entry / exit LI entry indices that allow manipulation of the order of computation. Intermediate results of accumulation functions are stored locally in registers 152 of the LI memory. In addition, pre-processing during initialization enables the LI memory to be pre-loaded with default values (e.g., previous intermediate results, bias values, etc.). The PE also includes intermediate memory aggregation control, i.e., allocation step size. In addition, activation functions are aggregated to minimize area overhead and are not implemented at the PE or sub-cluster level, but at the cluster level. The PE also supports activation bypass to allow cascading.

[0165] Pre-synthesis configurability allows: (1) N-way product accumulation (i.e., Y = Y + A1*B1 +... + A N *B N ); (2) representation format span (e.g., each entry supports k0...k N bits with m-bit mantissa and e-bit exponent, where k = m + e); and (3) selection of local storage depth P.

[0166] In operation, data flow within the PE is quite flexible. The output 151 of the adder 144 can be diverted through the destination multiplexer 146 using the destination control 175 to either (1) through path 162 to the activation function; (2) through path 164 to the L2 or L3 memory; or (3) through path 166 to the source multiplexer 148. The source multiplexer 148 selects either (1) the output from the adder; or (2) the intermediate result from the L2 or L3 memory 168 through the source control 173. The write multiplexer selects one of the registers 152 through the output control 178 to write the output of the source multiplexer through one of the P paths 172. The data written to the LI memory typically includes intermediate results generated due to product accumulation operations performed during many cycles.

[0167] Data is read from the LI memory by connecting the neuron register to the read multiplexer 156 and selecting one of the P paths 174 through the input control 179. The output 176 of the read multiplexer forms one of the two inputs to the adder 144. The other input to the adder is the output of the multiplier 142. Note that in the case of multiple multipliers 142, a pre-adder (not shown) is used to add the outputs of the multipliers to generate a single sum that is then input to the adder 144.

[0168] In Figure 7A A block diagram showing a second exemplary low-level processing element (PE) in more detail is shown. As noted above, the PE is the most basic computing element of the NN processor. The neurons of the ANN are implemented in the PE, essentially in the LI memory. The processing element (generally labeled 450) includes an input data representation circuit 452, a multiplier 454, a representation conversion / rounding 456, an accumulator (i.e., adder) 458, an LI memory 460, a negate circuit 472, and a multiplexer 474.

[0169] In operation, the input (X) 468 and the weight (W) 470 are inputs from the L3 memory to the input data representation circuit 452. This circuit operates to convert the representation of the input data and / or the weight from integer to floating point (FP) format and vice versa, depending on the INT / FP signal 462 (which is also input to the multiplier). The resulting X 504 and W 506 are input to the multiplier 454. Note that the multiplier 454 is capable of multiplying both integer and floating point numbers. The product output of the multiplier 486 is input to the representation conversion / rounding 456. The FP accumulator and FP input 508, 510, respectively, control the representation conversion / rounding 456 as to whether the product is an integer or FP format. In addition, the representation conversion / rounding 456 is used to perform rounding of the product prior to input to the accumulator. Figure 6 and Figure 7A Either of the two PE embodiments shown can be used in the NN apparatus of the present application.

[0170] In one embodiment, the multiplier includes several multipliers operating in parallel. The multiplier is capable of multiplying both integer and floating point numbers. The number of significant bits of the input data and the weight can also be different, as set by the control input data significant bits 464, weight significant bits 466, respectively. The product output of the multiplier 486 is input to the representation conversion / rounding 456. The FP accumulator and FP input 508, 510, respectively, control the representation conversion / rounding 456 as to whether the product is an integer or FP format. In addition, the representation conversion / rounding 456 is used to perform rounding of the product prior to input to the accumulator.

[0171] The output 488 of the representation conversion / rounding 456 is input to the accumulator (adder) 458. The second input 496 to the accumulator includes either the context (i.e., intermediate result) 490 from the L2 or L3 memory or the output of the local LI memory 460. The multiplexer 474 selects between the two according to the SEL 476. The output 494 is input to the negate circuit 472, where the second output 496 is negated prior to input to the accumulator according to the negate 478.

[0172] Additional configuration controls for the accumulators include accumulator shift (accumulator_shift) 498, accumulator enable (accum_en) 500, and FP accumulator 502. The output of the accumulators 484 is written to the LI memory. The LI memory also includes LI output select 480 and zero skip 482. The intermediate results (i.e., context) from the LI memory output are input to the accumulators through path 493 or written to the L2 or L3 memory through path 492. In one embodiment, the accumulation (i.e., intermediate) results are written to and read from the LI memory sequentially, i.e., there is no random access to the neuron registers in the LI memory. Note that any suitable predefined pattern (e.g., sequential (one-by-one), skip one, skip two, etc.) can be used to access the LI memory in addition to random. This greatly simplifies the addressing required to access the neuron registers. Furthermore, if there is not enough local LI memory available for a particular ANN, then a pass to the L2 and L3 memory layers is provided. In this case, the intermediate results are stored in the higher memory layers to accommodate the particular ANN. However, the tradeoff is an increase in latency when accessing the higher memory layers.

[0173] In an alternative embodiment, higher precision multiplication (e.g., 16 bits) can be performed by combining four low precision (e.g., 8 bits) multipliers to produce a high (or double) precision (e.g., 16 bits) product. Figure 7B A block diagram showing the quad multiplier of the PE in more detail is shown. The quad multiplier (generally labeled 870) includes four lower precision (e.g., 8 bits) multipliers 872, namely Q0, Q1, Q2, and Q3. The inputs to the quad multiplier are: a double precision input X, which is composed of two low precision (e.g., 8 bits) values, namely X L 873 and X H 871; and a double precision weight W, which also includes two low precision (e.g., 8 bits) values, namely W L 880 and X H 882.

[0174] In operation, each basic cell Q i receives low precision (e.g., 8 bits) W and X values and based thereon, the quad multiplier circuit generates the result Y <- Y + W*X. Considering the double precision X and W values, we denote the upper and lower parts of the weight, input data, and output as W H 882, X H 871, Y H 876 and W L 880, X L 873, Y L 875. Three carries C0 874, C1 878, and C2 879 are also generated.

[0175] Y <- Y + W * X is extended to

[0176] (Y H << 16 + Y L ) <- (W H << 8 + W L ) * (X H << 8 + X L ) (3)

[0177] Yields the following

[0178] Y L <- W L * X L + [(W L * X H + W H * X L ) << 8] L + C0 << 9 (4)

[0179] And

[0180] Y H <- W H * X H + [(W L * X H + W H * X L ) << 8] H + C1 << 9 + C2 << 9 (5)

[0181] Note that each output Y L and Y H represents a 16-bit number to produce a 32-bit multiplication product Y. It will be appreciated that results with greater precision can be obtained using additional multipliers and suitable combinations of input, weight, and carry components.

[0182] Subcluster

[0183] A high-level block diagram showing a first exemplary subcluster in more detail is shown in Figure 8 The subcluster (generally labeled 180) includes N multiple processing elements (PEs) 182 (each individual PE 182 includes a local LI memory 184), an interconnect fabric 186, a dedicated local L2 memory 188 divided into a plurality of allocated memory blocks 190, a configuration / decode 192, and control / data signals 181. The configuration / decode 192 receives instructions from an external control bus 194. Each subcluster 180 also communicates with an input / output alignment 196 and an activation function 198, which in the exemplary embodiments presented herein are located in the cluster hierarchy as described in more detail below.

[0184] In one embodiment, the function of a sub-cluster is to aggregate N multiple PEs, e.g., N = 64. All PEs in a sub-cluster belong to the same layer of the neural network, which greatly simplifies the required control logic. For example, no cycle-by-cycle operation control is needed except for a priori static configuration.

[0185] In addition, a sub-cluster encapsulates the next level of memory hierarchy, i.e., the L2 memory layer that stores the inter-layer and intermediate results. In one embodiment, it also includes the activation function circuit (i.e., denoted by σ in Equation 2 above). However, for efficiency, the exemplary NN core moves the activation function to the cluster level. The activation function receives the output of the neuron regardless of its location, and is triggered once every N product-sum operations. Note that the number and location of the activation function circuits are chosen to reflect the best utilization of the hardware.

[0186] Several features of a sub-cluster include: (1) a distributed control scheme for managing memory access; (2) dynamic allocation of weights and intermediate results for L2 memory; (3) inherent intermediate result shuffling support for seamless scaling of L1 memory; (4) layer-centric information and diagnostic storage; (5) layer-centric pre-processing; (6) layer-centric post-processing; and (7) intra-layer partitioning support (e.g., for quantization subdivision).

[0187] A high-level block diagram showing a second exemplary sub-cluster in more detail is shown in Figure 9 While Figure 8 reflects a mostly logical view of a sub-cluster, the view in Figure 8 reflects a more physical view. The sub-cluster (generally labeled 200) includes a dedicated local L2 memory 210, N multiple PEs 212 each with its own L1 memory 214 and receiving an enable EN 211, PE control 213, and PE configuration 215, an input interconnect 206, an output interconnect 208, a sub-cluster configuration 202 receiving instructions from a sub-cluster control bus 230 and outputting an L2_cbus 236, and a sub-cluster decoder 204 receiving layer control 232 and group control 234 and outputting an address ADDR 238, an enable EN 240, and a select SEL 242.

[0188] In operation, inputs 216 and weights 218 are provided from the cluster-level L3 memory to the input interconnect 206 according to control signals 201. The input interconnect feeds inputs 244 and weights 246 to the PEs 212. A zero_skip 217 informs the PEs that the input data or weights have zero values and, therefore, no multiplication and addition operations are needed. Note that weights 220 can also come from the dedicated local L2 memory 210, which receives an address ADDR 205, an enable EN 207, and a control L2_cbus 209.

[0189] Once the neurons in the PE have accumulated the computations required for a particular layer, the contents of the neurons now representing the intermediate results 248 are read out and signaled to the output interconnect 208 through the control 203. The intermediate results can then be written to the local L2 memory through path 226, or to the L3 memory through path 221, multiplexer 222, and path 228. Additionally, the intermediate results 224 can be read from the L2 memory and transmitted through the multiplexer 222 to the L3 memory or to the output interconnect, which then forwards it to the PE through path 249.

[0190] Thus, each sub-cluster includes flexible and programmable paths for feeding input data and weights to the neurons in the PE, as well as steering intermediate results from and to the L2 or L3 memory.

[0191] In one embodiment, a sub-cluster is dedicated to performing a single ANN layer or a portion thereof. Its function is to receive external inputs from the L3 memory, perform multiplication and addition with weights from either the local L2 memory or the external L3 memory, store the intermediate results (also called "contexts") in the PE LI memory (or in the local L2 memory if the LI memory is insufficient), and finally send the results to an external activation function for normalization and activation.

[0192] The sub-cluster decoder 204 is used to combine the static inputs from the sub-cluster configuration 202 with the dynamic inputs (common layer control and timing group control). The state it stores includes counters that hold the following addresses: (1) weight read / write address; (2) context read address; (3) context write address; (4) activation source address (PE output to for reading).

[0193] The input interconnect operates to (1) select between external weights (i.e., L3 memory) or local weights (i.e., from L2 memory); (2) select the width of the weight memory, i.e., the number of weights selected and the depth of the memory, where the maximum width allows all PEs to receive different weights from either the L2 memory or from the L3 external memory; (3) select the weights from the selected weight source to pass to the PEs (using the MSB of the address); select the width of the input bus; and (4) select the inputs from the selected input source to pass to the PEs (using the MSB of the address).

[0194] It should be noted that the dedicated local L2 memory 210 is used to store both weights and contexts in the same block. The weight addresses start at zero and count upwards, while the context addresses start at the end of the memory. It is the responsibility of the control plane to prevent overflow.

[0195] Cluster

[0196] In Figure 10 A high level block diagram showing a first exemplary cluster in more detail is shown in FIG. 3. The cluster (generally labeled 250) includes M multiple sub-clusters (each sub-cluster 266 has its own L2 memory 268), a dedicated local L3 memory 262 split into multiple allocated memory blocks 264, a memory management unit (MMU) 260 adapted to interface the L3 memory with the sub-clusters, management and control 252 including a control synchronizer 254 and multiple layer controllers 256, multiple input aligners 274, and multiple activation functions 276. Input / output (I / O) ports 270 interface each cluster with an inter-cluster cross-connect 272.

[0197] In one embodiment, a cluster is an aggregation of typically more than one neural network layer. It contains sub-clusters with PE basic computing entities contained within and interconnect fabric between the sub-clusters. This provides flexibility to the NN core to represent different neural network models by controlling the connectivity between the sub-clusters. A dedicated local L3 memory 262 is used to store the inter-layer results in one or more allocated memory blocks 264.

[0198] Several features of the cluster include: (1) a distributed control scheme for managing memory access; (2) a flexible configurable routing matrix for supporting representing the total M sub-clusters into multiple layers; (3) dynamic allocation of L3 memory for weights and intermediate results (relatively infrequent); and (4) inter-layer control for allowing data flow throttling and load balancing.

[0199] Additional features include: (1) weight / input data balancing; (2) pre / post processing blocks; (3) dynamic bus width and memory bit cells; (4) input data and weight interchangeability in the MMU; (5) ability to provide event driven behavior and pipelining; (6) control plane separation from data plane; (7) optional zero pipelining capability; and (8) balance capability for runtime configuration modifications.

[0200] In Figure 11 A high level block diagram showing a second exemplary cluster in more detail is shown in FIG. 4. The cluster (generally labeled 280) includes a cluster interconnect 282, input buffers 284, output buffers 292, M multiple sub-clusters 306, a sub-cluster interconnect 304, multiple activation functions / pooling 300, multiple input aligners 302, and an L3 memory 296 including multiple allocated memory blocks 298.

[0201] Input data and weights 286 are stored in an input buffer 284. Data and weights 288 are input from the input buffer to a cluster interconnect 282. Data 305 and weights 307 can also be written to and read from an L3 memory 296. Input from the cluster interconnect 281 is input to an input aligner 302 before input to a sub-cluster interconnect 304. Input 285 is fed from the sub-cluster interconnect to a sub-cluster 306, while output 283 from the sub-cluster is sent to the sub-cluster interconnect. Output 309 is input to an activation function / pooling 300, where the result output 308 is input to the cluster interconnect 282. Data 290 is written to an output buffer 292. Data 294 is then sent to other clusters or off-chip.

[0202] In one embodiment, the NN core supports multiple neural networks in parallel. Each cluster operates to expose control interfaces (e.g., clock, reset, enable, etc.), configuration interfaces (like memory), and data interfaces (e.g., Advanced eXtensible Interface (AXI)). Each cluster is adapted to implement one or more ANN layers, which can be from more than one ANN. The AXI interconnect exposes control interfaces and is used to connect clusters, DMA engines of the ARM controller in the NN core, and external ports. The ARM exposes the AXI interface through the DMA engines, control and configuration interfaces of the clusters and interconnect, and external standard interfaces.

[0203] In one embodiment, a cluster includes: (1) a configuration circuit; (2) a memory management unit (MMU); (3) a control interconnect; (4) a trigger interconnect; (5) a plurality of sub-clusters; (6) a plurality of layer controllers (LCs); (7) a plurality of specialized units; (8) a plurality of input units; (9) a plurality of output units; and (10) a plurality of memory blocks (i.e., L3 memory).

[0204] In one embodiment, a cluster supports multiple ANN layers, which can be from multiple ANNs, in parallel. It should be noted that a network layer can be implemented as a layer controller (LC) with one or more sub-clusters connected through the control interconnect or a specific unit (specialized, input, or output) within it containing the control. The layer uses signaling for flow control through the trigger interconnect, transfers data through allocated memory blocks 298 in the L3 memory 296, all defined by the configuration. The allocated memory blocks are also used as weight memory for the sub-clusters. All control signals from the layers to the L3 memory are translated from virtual to physical addresses by the MMU 260 using the configuration.

[0205] The MMU uses a sliding overlap window mechanism between two sets of communication ports, such as a read port of the L3 memory and an input port of a sub-cluster. Each sub-cluster can select its input from a set of memory ports around its relative position in the list of sub-clusters. The window mechanism is described in more detail below.

[0206] To be able to efficiently utilize the pipes in the NN cores, the allocation of sub-clusters per ANN layer is preferably proportional to the number of computations needed in the ANN layer per feed. The allocation is determined by a control interconnect that maps the sub-clusters to the LCs. The mapping is performed in two levels: (1) each sub-cluster is assigned to an LC by a sliding overlapping window mechanism (i.e., similar to the one used in MMU); (2) the sub-clusters are assigned to timing groups within the ANN layer. Timing groups propagate actions that require common resources (such as write ports of L3 used after activation) over time. An ANN layer can include one or more timing groups, each containing one or more sub-clusters. Controls common to all timing groups are not passed through the second level of selection, thus reducing the multiplexing complexity of the circuit.

[0207] In one embodiment, the signaling mechanism between ANN layers is based on two bidirectional lines that negotiate about the state of the double buffers between them. Thus, two bidirectional lines are needed to connect two consecutive layers, i.e., four bidirectional lines are used per layer, two for the previous layer and two for the next layer. Two reverse signals indicate whether the buffers are ready to receive new data for each of the two buffers between the layers, and two forward signals indicate whether the data in the buffers is valid for both buffers. To simplify the interface, the controller can use a dedicated instruction to flip the meaning of the two buffers in both directions (i.e., active and passive).

[0208] In Figure 12 A high-level block diagram showing the inter-cluster cross-connect in more detail is shown in FIG. 43. The sub-cluster inter-cluster interconnect structure / cross-connect (generally labeled 430) includes a plurality of multiplexers 432 and de-multiplexers 440 that enable communication between clusters 436. In one embodiment, each cluster J includes a plurality of ports including input ports 396 and output ports 398. Four input ports and four output ports are shown in the example, but any number can be implemented.

[0209] The multiplexers 432 on the input side are controlled by SEL lines 438. The input 434 of each multiplexer includes output lines from adjacent clusters (e.g., clusters J-2, J-1, J, J+1). The output 444 from each multiplexer is input to a separate input port 396 in the cluster. Similarly, the de-multiplexers 440 on the output side generate outputs 442 that are fed to input lines of adjacent clusters (e.g., clusters J-1, J, J+1, J+2). The output 446 from each output port 398 of the cluster is input to a separate de-multiplexer 440. The NN manager 392 is used to control the configuration of the sub-cluster inter-cluster interconnect structure / crossbar 430. In one embodiment, the possible connections from one cluster to another are intentionally limited to reduce addressing and control routing and improve bandwidth. For example, the connections by input 434 to cluster J are limited to clusters J-2, J-1, J, and J+1, i.e., to adjacent clusters (and themselves). Similarly, the connections from cluster J to outputs 442 are limited to clusters J-2, J-1, J, and J+1. Note that while direct connections to other clusters are limited, any cluster is still able to communicate with any other cluster indirectly by traversing one or more intermediate clusters.

[0210] Note that crossbaring occurs at all levels, starting at the cluster level, through the highest level of NN processor cores, and between devices to devices. The L clusters in a NN processor are connected using a cyclic interconnect structure that enables an output port from one cluster to be mapped to an adjacent cluster. The crossbar also enables an output of a cluster to be routed to itself (i.e., self-routing). Note that the range of access in the crossbar is configurable and allows a tradeoff between design complexity and accessibility. Also note that a "scatter / gather” mechanism allows an output (i.e., by a de-multiplexer) to be split into multiple copies so that the same output is fed to multiple inputs in parallel. The NN manager 392 provides control of the crossbar through control lines 431.

[0211] Sliding overlapping memory windowing

[0212] In Figure 13 A diagram illustrating a first exemplary memory windowing scheme is shown in FIG. 1. To maintain flexibility, each memory consumer in the processor has access to different memory segments for data exchange. The term memory windowing refers to a scheme whereby a compute element or entity is given access to only a certain subset of the available memory resources rather than a wider range of memory resources. Limiting the access of compute elements to memory using the memory windowing scheme significantly improves the available bandwidth while greatly reducing the required address and control routing. It should be noted that the memory structure can dynamically rearrange the memory windowing scheme whereby the memory resources accessible to a compute element are programmable and configurable (e.g., at compile time, run time, etc.). The windowing scheme is based on the scatter / gather technique described in more detail below.

[0213] In the example shown (generally labeled 580), two compute elements 582 access memory blocks 584, 586, 588. No compute element has access to the entire memory, but only a limited window. This is because a compute element never needs to access the entire memory structure at once. It should be noted that the windowing can be different for control, ingress data, egress data, and weights. Further, the windows typically overlap to enable sharing and pipelining. Further, the memory resources themselves are multi-purpose, where it can be used to store more than one type of information.

[0214] In the illustrative example, compute element 1 control spans memory blocks 584, 586, and 588, indicated by control 1 arrow 590. Compute element 1 includes an ingress data window to memory block 586, indicated by ingress data arrow 592. Similarly, compute element 1 includes an egress data window to memory block 588, indicated by egress data arrow 594. Weights are stored in memory block 584 as well as memory block 588, which is also used to store egress data. In a similar manner, the other compute elements also include control, ingress, egress, and weight windows. For example, compute element 2 includes a control window 596 that spans memory block 588 as well as one or more other memory blocks (not shown).

[0215] In Figure 14 A diagram illustrating a second exemplary memory windowing scheme is shown in FIG. 2. In one embodiment, data flowing through compute elements in a NN processor is pipelined, where PEs in a subset cluster receive data as input and generate output that is then used as input to some other subset cluster for a subsequent computation. Memory in each layer is localized as much as possible and utilized to maximize the accessibility and efficiency of the compute elements served by each layer. Since compute elements only need to access a limited subset of memory, routing (i.e., address lines, control, etc.) can be limited to the cross-connect memory blocks and the compute elements that actually need it. Figure 15 、 Figure 16 and Figure 17 The configurability of the memory access window is demonstrated, through which the allocation of each resource is managed and configured, and is equipped with resources that address specific needs.

[0216] The windowed memory scheme (generally labeled 340) includes multiple sub-clusters 348 (each sub-cluster 348 including multiple PEs 349), L3 memory (unshared) 342, and L3 memory (shared) 344. In operation, the sub-clusters receive weights 345 from the unshared portion of the L3 memory. Inputs 341 to the sub-clusters are received from allocated memory blocks 346 from the shared portion of the L3 memory. The PEs within the sub-cluster process the weights and input data and generate outputs 343. However, the outputs are written to different (e.g., adjacent) allocated memory blocks (i.e., not the memory blocks from which the inputs were read). These outputs are then read as inputs to another sub-cluster (e.g., a neuron in a subsequent layer of the ANN). In this way, ANN input data 347 is input to the shared L3 memory, read from allocated memory blocks, processed by PEs in one or more sub-clusters, output to adjacent memory blocks, and ultimately output from the shared L3 memory as ANN output data 349 after traversing layers in the ANN.

[0217] It should be noted, however, that the sub-clusters do not have direct random access capability to the L3 memory, but only to the immediate or nearby neighborhood of the allocated memory blocks. For example, sub-cluster H can access sub-clusters H-2, H-1, H (itself), and H+1. This greatly reduces the addressing and control routing requirements for memory access. Thus, each sub-cluster can only'see' a relatively small window of memory sufficient for its PEs to perform their functions.

[0218] In Figure 15 A diagram illustrating a first exemplary memory accessibility between window sizes of compute elements and memory elements and computer access configurability is shown in FIG. 5. This diagram illustrates a memory windowing scheme whereby compute elements as well as memory elements can have limited access to each other. For example, consider memory elements 1 through D and compute elements 1 through E. The hatched regions 520 represent the resources accessible to each element. Thus, compute elements 1 through 3 can only access memory elements 1 through 12. Similarly, memory elements 1 through 12 can only connect to compute elements 1 through 3. As shown, the memory elements accessible to the compute elements form sliding access windows that overlap each other. The access windows have sizes (i.e., spans) and specific connectivity that can be dynamically configured rather than hardwired or fixed. One key feature is that any single compute element cannot randomly access the entire memory. Rather, each compute element can only access a portion of the memory elements, e.g., adjacent memory elements or nearby memory elements. The portion of the memory that is not accessible to the compute elements is represented by the white regions 522.

[0219] It is also noted that the number of compute elements accessible to a memory is programmable and configurable, as represented by the vertical arrow 523. Similarly, the number of memory elements accessible to a compute element is programmable and configurable, as represented by the horizontal arrow 521.

[0220] A diagram illustrating a second exemplary memory accessibility between compute elements and memory elements is shown in Figure 16 A diagram illustrating an exemplary scatter / gather based resource windowing technique is shown in

[0221] A diagram illustrating an exemplary scatter / gather based resource windowing technique is shown in Figure 17 A diagram illustrating an exemplary scatter / gather based resource windowing technique is shown in A diagram illustrating an exemplary scatter / gather based resource windowing technique is shown in

[0222] Control of the splitters and multiplexers is provided by a layer controller (LC) 548. Control lines 549 of the LC output are input to a series of multiplexers 546 in a control fabric 544, which select one control from the LC according to SEL lines 547 originating from the LCU and which can be further decoded within the LC. Control of the multiplexers 546 is programmable and configurable, such as at compile or runtime, enabling flexible mapping between A resources and B resources.

[0223] According to the present application, the memory access fabric of the NN processor is characterized by being able to operate with substantially higher parallelism. This is a benefit of the inherent decoupling between the compute resources and the memory attached to them. For example, weights are only explicitly connected to the relevant sub-cluster. One exception, however, is the case of a shared allocated memory block and a collision occurring. While such events are typically rare, the NN processor provides the ability to resolve contention resulting from the collision. In one embodiment, memory contention is resolved at the control layer, where two compute entities sharing a common memory block handle collision avoidance at the signaling level, as described below. Note that back pressure is typically temporary and brief, and the overall aggregate bandwidth is guaranteed by the design of the NN processor.

[0224] In Figure 18 a block diagram illustrating an exemplary memory contention resolution solution is shown. The memory contention resolution circuit (generally labeled 600) includes L3 memory 602 (which includes a plurality of memory blocks 632), MMU 626, LCU A 604, LCU B 606, one or more sub-clusters 618 forming ANN layer G 614, and one or more sub-clusters 620 forming ANN layer G+1 616.

[0225] In this illustrative example, both layer G and layer G+1 of the ANN read and write data to and from memory blocks 634 in the L3 memory. The output of layer G is used as the input to layer G+1. Occasionally, however, both layers can attempt to access the same memory block at the same time. This is indicated by the memory block labeled "X" 636. When contention for the same memory block occurs, the MMU 626 detects the event and generates a contention alert 608 to the LCUs (A and B in this example) in their respective LCs. In response to the contention alert, one of the LCUs generates a stop 610, 612 command that is input to the sub-clusters. The sub-cluster that receives the stop command inhibits access to the memory block in the L3 memory until the read or write operation is completed.

[0226] Note that memory contention always occurs between ANN layers and not within a layer, because within a layer, the sub-clusters that make up the layer are configured such that contention for memory never occurs. Typically, contention occurs when one layer is writing while the other layer is reading. In response to the contention alert, either the write or read operation can be inhibited. In one embodiment, the write operation is inhibited because the nature of the ANN is that write operations are a much rarer event. Moreover, inhibiting the read operation would stall a significant portion of the data processing pipeline. Thus, the write operation is inhibited rather than the read operation. A stop 610 command is issued to the layer to be inhibited to layer G or 612 to layer G+1). Note also that the decision of whether to inhibit the write or read operation is programmable and configurable a priori at compile time.

[0227] layer controller

[0228] In Figure 19 A high-level block diagram showing an exemplary layer controller in more detail is shown in FIG. 7. The layer controller (LC) (generally designated 310) includes a layer control unit (LCU) 314 that is responsible for decoding and executing microcode instructions 311 read from instruction memory 312. From the instructions, one or more command signals 313 are output to various control and decode blocks, including input aligner control 316, activation control 318, input address decoder 320, weight address decoder 322, output address decoder 324, and PE control 326. Control and address signals from these six blocks are output to input aligner 328, activation function circuit 330, input memory 332, weight memory 334, output window 335, and control window 336, respectively. PE control signals 315 are output from control window 336 to PE circuits in subset cluster 338.

[0229] In Figure 20 A high-level block diagram showing a layer controller interface to L3 memory and subset cluster in more detail is shown in FIG. 8. The exemplary cluster (generally designated 640) includes L3 memory 644, LC 642, a plurality of subset clusters 662, post-memory 666, and windowing for control, write data, read data, and weights, as described above in connection with FIG. 6. Figure 17 LC 642 includes LCU 656, one or more pre-processors 652, instruction memory 654, one or more decoder circuits 658, and MMU 660.

[0230] In particular, control windowing includes window circuit 674 and control line 684; weight windowing includes window circuits 646, 648 and signal line 650; entry data windowing includes window circuits 676, 678, 672 and signal lines 690, 692; exit data windowing includes window circuits 680, 682, 668 and signal lines 686, 688. Note that the entry and exit windows to L3 memory overlap, as indicated by the dashed lines. Memory window control (MWC) signals 670, generated by the LCU and decoders and input to window circuits 674, 646, 648, 676, 678, 672, 680, 682, and 668, provide control to the windowing (i.e., selection for the splitters and multiplexers).

[0231] In operation, entry data is read from the L3 memory and input to the pre-processor 652. These circuits are used to optionally reshape the data, perform operations on the input data, such as shifting, etc. The pre-processed data is output to the sub-clusters, where the PEs 664 multiply the input data with weights also read from the L3 memory. The intermediate results (i.e., the contexts) are output from the sub-clusters to the post-memory 666 through memory windowing. The post-processing circuits are part of the data processing pipeline and operate to apply the activation function and optionally apply alignment.

[0232] It should be noted that each LC is assigned one or more sub-clusters, which make up a layer in the ANN. Each cluster includes multiple LCs (e.g., eight). Thus, the illustrated sub-clusters 662 are only a subset of the M sub-clusters within each cluster, where each LC controls a different set of sub-clusters that can be selected using the same windowing concept described above. Furthermore, the N PEs within a sub-cluster are not separated, meaning that all PEs in a sub-cluster are controlled as a single unit. This simplifies the control of the compute elements and allows for relatively lean control signaling, as only a few control lines control a large number of PEs and ultimately the neurons. Similarly, each of the decoder circuits 658 is configured to control a different set of memory blocks. The encoded control signals 698, which are generated by the LCUs in one embodiment, are input to the decoder circuits 658. The LCUs themselves are controlled by the contents of the instruction memory 654. The execution of each instruction results in the generation of an encoded control signal, which is then decoded by the decoder and output to the compute elements through the control window circuit 674. It should be noted that in addition to the control signals that control the compute elements in the sub-clusters, the LCUs also generate control signals that are used to control the control windows (along with the weight, entry, and exit data windows) as well (i.e., MWC selection control). When configured (at compile time), the static routing control signals, weights, entry, and exit data. The MMU 660 generates control signals for the L3 memory windowing and is used to perform the virtual to physical mapping. It is also used to generate a contention alert 694 in response to a memory contention event between two layers in the ANN. As described above, the LCUs resolve contention events by issuing a stop command to one of the layers.

[0233] In Figure 21 A high-level block diagram showing a second exemplary layer controller in more detail is shown in FIG. 55. The exemplary LC (generally designated 550) includes an instruction memory 552 (which includes a plurality of instructions 554), an LCU 556, a decoder 566, a flip-flop window cross-connect 558, and a flip-flop processor 560. The LCU 556 includes a state machine 562 and an instruction register 564.

[0234] In operation, instructions 551 are read from instruction memory into instruction registers 564 in the LCUs that decide and execute them. One or more portions of the instructions 568 that are configured to directly control hardware are sent to one or more decoders 566 for decoding. The output of the decoders includes direct control signaling that is sent to the subclusters to control internal PE operations as shown and described above in Figure 20

[0235] In one embodiment, one or more triggers 555 are generated by the state machine and input to a trigger window crossbar 558. Trigger functions are similar to "interrupts" in that activity can be stopped and delayed until some event occurs. Trigger signals are used to trigger activity. Triggers can be issued to activate other triggers. They represent an asynchronous mechanism for synchronizing activity in the NN processor. For example, a trigger can be issued to stop processing until a write buffer, or until a layer finishes processing (or otherwise used as an indication that some event has occurred and further processing can begin).

[0236] In addition, triggers can be issued to trigger activity in LCUs in different LCs. This process is called "switching". The switching mechanism can trigger activity from one LC to another LC, for example, a trigger can be used when one ANN layer finishes and sends results to another layer in the ANN. The trigger window crossbar is used to divert the output trigger signals 559 to the trigger handlers in the appropriate LC where they are used to control activity in the LCUs through signals 557.

[0237] With respect to the separation between the data plane and the control plane, in one embodiment, the microcode that manages the control plane is executed in the LCs and has no access to the data. An additional capability of the microcode machine in the LCs is that there are no conditional statements or conditional branches. This is advantageous for data pipelining because the need to manage branch prediction or other pipelining overhead is avoided. Thus, the execution is completely predictable. This is in contrast to typical prior art microcode that can branch to cause execution to depend on inputs. In the NN processor, when the microcode executes, the evolution of the data flow is completely predictable, i.e., the generation of each control signal can be predicted at each time instance.

[0238] ​In one embodiment, each microcode instruction operation performed in the microcode-based controller generates control signaling for both compute resources and memory resources. In other words, the microcode has no "overhead" as there is no any operation responsible for internal processing that also does not impose actual control signaling to the output. Thus, no microcode instruction operation is wasted on internal housekeeping of the microcode machine (the only exception is the 'NOP' operation).

[0239] Another capability of the microcode machine in the LC is the trigger operation. While branching is not supported, the execution flow can be triggered by an external signal that indicates the start / stop of the execution to implement data pipeline handshakes, e.g., switching from one LCU to another LCU.

[0240] Yet another capability of the microcode machine in the LC is the repeat operation support, supporting inline repetition of operations (i.e., loops that run inline), such that the repeat operation can be indicated within the opcode itself, avoiding the unnecessary cycles of setting up and managing loops and the associated fetches. It should be noted that this feature is useful for loops with few operations compared to the overhead of loop management. The latter is very common in neural network operations (e.g., many multiply-accumulate (MAC) operations following an activation). This is very important in a data pipeline machine when the ratio between control and data is such that very few controls define the behavior of a relatively large data pipeline.

[0241] For example, consider a conventional processor configured to perform 1000 multiply-accumulate (MAC) operations. An exemplary pseudo code is provided in Listing 2 below.

[0242] Listing 2: Exemplary conventional processor pseudo code loop

[0243]

[0244] In the above pseudo code, there are four opcodes in the loop (i.e., four cycles) in the above pseudo code, two of which are operational, with a usage of 50%. Assume that this loop controls 1024 MAC circuits, which means that only 512 are effectively operating at full capacity.

[0245] In contrast, inline repetition is supported in the NN processor. Moreover, the overhead of internal control is zero, which eliminates the need for "spare" opcodes (i.e., opcodes that are only used for machine internal management or housekeeping). The pseudo code of Listing 2 is converted to the following pseudo code presented in Listing 3 below.

[0246] Listing 3: Exemplary NN processor pseudo code loop

[0247]

[0248] As shown above, all loop information is embedded in the functional opcode, and the MAC utilization increases to 100%.

[0249] It should be noted that having a deep separation between the control plane and the data plane also serves to make it somewhat inherently immune to control plane security breaches. This is because a common technique for an illegitimate intruder is to feed it with data that interferes with the control plane. Since the two planes are strictly separated, interfering with one plane does not affect the other plane.

[0250] Compiler

[0251] A high-level block diagram showing an exemplary NN processor compiler / SDK is shown in Figure 22 The compiler (generally labeled 770) is used to configure the NN processor based on an input ANN model. It is executed in a process that is performed off-chip as part of an external software toolchain used and initiated by a user. Typically, the compiler has knowledge of the NN processor, NN processor SoC, or multiple NN processor SoCs that will be the target of the source ANN model.

[0252] In particular, the translator 772 is used to receive a user model and generate an intermediate format of the model. The optimizer 774 is used to perform model-level optimizations, post-translation model performance tuning, and numerical retargeting to different bit widths. The resource allocator 778 allocates and assigns physical resources (e.g., compute elements and memory elements, etc.) according to the intermediate model. The profiler 786 performs performance evaluations, including, for example, expected power consumption, throughput, latency, etc. The emulator 782 is used to perform bit-accurate numerical emulation of the NN processor using the intermediate model output of the translator 772.

[0253] In one embodiment, the user is provided with several target options to implement the external toolchain. The three target options include: (1) the NN device 780, (2) the emulator 782, and (3) the simulator 784, which includes a software model of the hardware that simulates the functionality of the NN device. Thus, the user can choose to use the NN device itself, a hardware emulation of the NN device, or a software emulation of the NN device to execute the toolchain.

[0254] Multiple operational granularities of the NN processor and related memory / delay tradeoffs

[0255] The ability and advantage of the present invention is that the pipeline in the NN processor can operate at any desired granularity of any subset of the input, where memory trades off for latency and vice versa. More specifically, when the input data has some internal structure (e.g., a video frame, and each frame is composed of multiple rows (or buffers, packets, etc.), the NN processor architecture can trigger activity of the next layer from any aggregation of a single such row, buffer, packet, etc., or a multiple thereof.

[0256] In the case of lower aggregation, additional intermediate result (i.e., context) storage is required to store the intermediate results. However, the latency is minimal, as the subsequent processing elements are freed up for further processing earlier in the pipeline, which allows the incoming traffic to be consumed but not blocked. Thus, the higher memory storage requirement trades off for lower context latency.

[0257] On the other hand, in the case of higher aggregation, i.e., less context memory is desired or an ANN model is to be implemented that requires a large amount of context, a tradeoff can be made using less context memory to exchange buffer memory, thereby implementing additional buffering of the input, resulting in a reduction in the number of contexts needed at any one time, but an increase in latency. In one embodiment, this tradeoff is implemented by microcode in the LC, and is thus configurable and programmable.

[0258] In Figure 23 A diagram showing the flexible processing granularity of the NN processor and the related tradeoff of memory to latency is shown in FIG. 9. A data pipeline instance (generally labeled 930) highlights the option of utilizing the data pipeline to support minimal latency and operate at a low input domain granularity. Consider an exemplary input tensor 932 comprising input data 938, which can be at the beginning of the network or at any point in the network. Then, one of the network layers applies a NN operation 934 to the input data (e.g., a 3x3 convolution in this example), followed by an output domain 936 comprising storage blocks 931 and 939.

[0259] In this example, the input data stream is completely consumed and all required computations are applied, while minimizing latency and not requiring retrieval of the input data, as all computations are committed to intermediate results stored in memory. In alternative embodiments, this functionality can be performed by (1) waiting for the entire frame and applying a batch operation, whereby all data is committed to the output immediately to avoid intermediate results; (2) waiting for a minimum group of rows to avoid intermediate results (three in this example); (3) using intermediate results stored in external memory with an increase in memory access latency; or (4) invoking the input as needed (i.e., reading the same data multiple times) to avoid having to store intermediate results.

[0260] NN processor SoC, intra- and inter-chip connectivity

[0261] As described in detail above, the NN processor can be used to implement an ANN. However, if the ANN to be implemented exceeds the capacity of the NN processor, the present invention provides the ability to implement an ANN model using several NN processors. As described above, the NN processor includes a plurality of bus interfaces (e.g., chip-to-chip interfaces) for communicating between the NN processor cores. In the examples disclosed herein, two chip-to-chip interfaces are provided, but any number can be implemented. Thus, large ANN models can be accommodated by combining the processing power of multiple NN processor cores.

[0262] It should be noted that deploying a network of interconnected NN processors over a chip-to-chip interface is substantially seamless. With device-to-device communication, the network behaves identically to an ANN contained on a single NN device. In one embodiment, the chip-to-chip interface is consistent with techniques that narrow bandwidth on layer boundaries. The physical layer of the interface can include any suitable protocol that is synchronous and guarantees the required bandwidth. The next layer is a packet layer, which has a frame format that can be removed by the receiving chip. The structure of the frame format attempts to minimize the overhead in transitions between devices and is similar to the structure of Ethernet, which includes a number of fields, including, for example, a flow ID, a destination layer, a data format, etc. For example, consider a layer with a WxHxF output tensor. The protocol identifies the structure, flow ID, and network ID in the next device before any processing occurs. The required bandwidth is then (WxHxF + overhead) x frames / second.

[0263] In Figure 24 A diagram showing a first exemplary multi-NN processor SoC system of the present invention is shown in FIG. 1. In one embodiment, as described above and in FIG. 2, the NN processor cores (or engines) are replicated and implemented as a system on a chip (SoC). The intellectual property (IP) of the NN processor cores can be used to implement a single-chip integrated circuit (IC). Alternatively, the physical NN processor core dies can be integrated and implemented on a SoC. Figure 4 and Figure 5 The NN processor cores (or engines) shown in FIGS. 1 and 2 can be replicated and implemented as a system on a chip (SoC). The intellectual property (IP) of the NN processor cores can be used to implement a single-chip integrated circuit (IC). Alternatively, the physical NN processor core dies can be integrated and implemented on a SoC.

[0264] The NN processor SoC (generally designated 700), implemented as a single-chip semiconductor or SoC, includes a plurality of NN processors 706, one or more external interfaces 702, one or more 'external' L5 memory circuits 708, a bootstrap and pre-processor 704, and a post-processor 712, interconnected by an internal bus 710. It should be noted that the number of NN processor cores, L5 memory circuits, etc. is not limited to the number shown, as those skilled in the semiconductor arts can implement an IC or SoC with any number of NN processor cores and other components.

[0265] In operation, ANN input 714 is written to the NN processor SoC 700 through external interface 702. The bootstrap and preprocessor 704 operates to perform one or more functions depending on the implementation, including, for example, buffering, clocking, power management, data throttling, etc. The data is then fed to the NN processor 706 for processing. The NN processor cores communicate with each other through internal bus 710. It should be noted that the connectivity between the NN processor cores can include any desired routing type, including, for example, full mesh, token ring, link, etc., depending on the implementation and not critical to the present invention. It should be noted that other circuit components also communicate through the bus, including the bootstrap and preprocessor 704, external interface 702, L5 memory 708, and postprocessor 712.

[0266] In Figure 25 A diagram showing a second exemplary multi-NN processor SoC system embodying the present invention is shown in FIG. 8. In this exemplary system (generally labeled 790), multiple NN processors / SoCs 794 are connected in series. ANN input 792 is input to the left-most NN processor, and ANN output 799 exits the right-most NN processor. The multiple NN processors together implement the ANN model layer by layer.

[0267] In Figure 26 A diagram showing a first exemplary multi-NN processor SoC system embodying the present invention is shown in FIG. 7. In this exemplary system (generally labeled 800), three NN processors / SoCs 804, 806, 808 are combined in a 2→1 aggregation scheme and together implement an ANN model. ANN input 802 is input to NN processors / SoCs 804, 806 through input ports. In this example, two NN processors / SoCs 804, 806 in parallel are required to implement the ANN model, for example, (1) the model contains a very large number of neurons in one or more layers or (2) the number of neurons exceeds any resource constraints (e.g., control, memory, or compute) of a single device. The output of each NN processor / SoC 804, 806 is input to NN processor / SoC 808 through chip-to-chip input ports, which is used to generate ANN output 809.

[0268] In Figure 27 A diagram showing a first exemplary multi-NN processor SoC system embodying the present application is shown in FIG. 1. In this exemplary system (generally labeled 810), three NN processors / SoCs 814, 816, 818 are combined in a 1→2 fan-out scheme and together implement an ANN model. ANN input 812 is input to NN processor / SoC 814 through an input port. The output of NN processor / SoC 814 is input to both NN processors / SoCs 816, 818. In this instance, two NN processors / SoCs 816, 818 in parallel are required to implement the ANN model, for example, (1) the model contains a very large number of neurons in one or more layers or (2) the number of neurons exceeds any resource constraints (e.g., control, memory, or compute) of a single device. The outputs generated by each NN processor / SoC 816, 818 are combined to form ANN output 819.

[0269] Exemplary ANN mapping strategies

[0270] As noted above, if the ANN's requirements exceed the compute and / or memory resources of a single NN processor core, the ANN model can be split into several devices. The compiler / SDK seamlessly takes advantage of the typical cellular nature of ANNs, which allows for splitting and merging between and above the network layers. Within the compiler, the split is done while taking into account the bandwidth requirements at the input and output of the sub-networks mapped to each device, in addition to relying on the fact that the inter-layer bandwidth is inherently much lower than the intra-layer bandwidth. Several exemplary mapping possibilities and strategies are presented.

[0271] In general, the device-to-device mapping performed by the compiler is driven by the number of input and output ports present in the device (e.g., two in this instance). In the exemplary case of two input and output ports on the device, the flexibility of mapping 1→2 (i.e., fan-out), 2→1 (i.e., fan-in), and 1→1 (i.e., feed-forward) allows for the construction of the system arrangements shown.

[0272] In Figure 28 A diagram showing a second exemplary multi-NN processor SoC system embodying the present application is shown in FIG. 2. In this exemplary system (generally labeled 820), three NN processors / SoCs 824, 826, 828 are combined in a 1→2 fan-out scheme and together implement an ANN model. ANN input 822 is input to NN processor / SoC 824 through an input port. The output of NN processor / SoC 824 is input to both NN processors / SoCs 826, 828. In this instance, two NN processors / SoCs 826, 828 in parallel are required to implement the ANN model, for example, (1) the model contains a very large number of neurons in one or more layers or (2) the number of neurons exceeds any resource constraints (e.g., control, memory, or compute) of a single device. The outputs generated by each NN processor / SoC 826, 828 are combined to form ANN output 829. Figure 2 A diagram illustrating an exemplary mapping strategy for the first exemplary ANN. As described above, the compiler / SDK is used to map the logical ANN model to a physical NN processor device. As a result of its analysis, in this example, the compiler determines that the entire ANN can be implemented in a single cluster 362 within a single NN processor device. Each network layer 365 in the ANN is mapped to one or more sub-clusters 364 and is also assigned a network controller (LC) 361. Thus, for example, network layer 1 is mapped to three sub-clusters, namely sub-clusters 1, 2, and 3, which also receive ANN inputs 363. These three sub-clusters are configured and controlled by LC 1. Similarly, neurons in network layer 2 are mapped by the compiler to sub-clusters 4, 5, and 6 and assigned to LC 2. Neurons in network layer 3 are mapped to sub-clusters 7 and 8 and assigned to LC 3. Finally, network layer 4 is mapped to sub-cluster 9 and is configured and controlled by LC 4. The ANN output 369 is generated by sub-cluster 9.

[0273] exist Figure 29 The diagram illustrates a second exemplary artificial neural network. This exemplary ANN (generally labeled 720), which may be a convolutional neural network, includes multiple layers 726, including layers 1 through 6. Layer 1 receives ANN input 722, and layer 6 generates ANN output 724.

[0274] exist Figure 30 The middle shows the display Figure 29 A diagram of an exemplary multi-NN processor SoC system of an ANN. The NN system (generally labeled 730) represents the mapping of the ANN 720 to the NN processor system of the present invention. Each NN processor 736 includes a separate IC, or alternatively includes a separate die in the SoC.

[0275] During the complication process, the compiler and SDK function to map the logical ANN model to the physical NN processor configuration. In this example, layer 1 is mapped to the entire NN processor 1 because its capacity in terms of computational elements, memory structure, etc., is only sufficient to implement layer 1. NN processor 1 also receives ANN input 732. Layers 2 and 3 make it impossible to implement them in a single device, thus requiring two devices (i.e., parallel NN processors 2 and 3) and processing is separated between them. Layer 4 is large, but the compiler determines that it can be implemented in a single device. Therefore, the entire NN processor 4 is mapped to layer 4. Layers 5 and 6 are analyzed and mapped by the compiler to a single NN processor 5 device. NN processor 5 generates ANN output 734. It should be noted that the NN processors communicate with each other in a feedforward manner through chip-to-chip interfaces in each device.

[0276] exist Figure 31 The diagram illustrates a third exemplary artificial neural network. The exemplary ANN (generally labeled 740) is intended to represent any desired ANN. It comprises multiple neurons 744 organized into different network layers. Input data X 746 is input to the first layer, and output data Y 748 is generated by the last layer.

[0277] exist Figure 32 The middle shows the display Figure 31 A diagram of a first exemplary multi-NN processor SoC system for an ANN. In this example, the first mapping (generally labeled 750) is generated by the compiler / SDK and includes several NN processor devices. Specifically, neurons 756 in the first two network layers are mapped to NN processor 1, the third network layer is mapped to NN processor 2, and the last three network layers are mapped to NN processor 3. ANN input data 752 is input to the first layer in NN processor 1. NN processor 3 generates ANN output data 754.

[0278] exist Figure 33 The middle shows the display Figure 31 A diagram of a second exemplary multi-NN processor SoC system for an ANN is shown. In this example, different mappings (generally labeled 760) are generated by the compiler / SDK and include several NN processor devices. Specifically, neurons 766 in the first four network layers are separated between two devices (i.e., NN processors 1 and 2) because they exceed the capacity of a single device. The last two network layers are mapped to NN processor 3. ANN input data 762 is input into the first layer in NN processors 1 and 2. NN processor 3 generates ANN output data 764.

[0279] Those skilled in the art will recognize that the boundaries between logic and circuit modules are merely illustrative and that alternative implementations may combine logic modules or circuit elements or impose alternative functional decompositions on various logic modules or circuit elements. Therefore, it should be understood that the architectures described herein are merely exemplary, and in fact, many other architectures that achieve the same functionality can be implemented.

[0280] Any arrangement of components used to achieve the same functionality is effectively "associated" to achieve the desired functionality. Thus, any two components combined herein to achieve a particular functionality can be considered "associated" with each other to achieve the desired functionality, regardless of architecture or intermediate components. Similarly, any two such associated components can also be considered "operably connected" or "operably coupled" to each other to achieve the desired functionality.

[0281] Furthermore, those skilled in the art will recognize that the boundaries between operations as described above are merely illustrative. Multiple operations may be combined into a single operation, which may be distributed among other operations and may be performed during periods of at least partial temporal overlap. Additionally, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be varied in a variety of other embodiments.

[0282] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, unless the context clearly indicates otherwise, the singular forms "a," "an," and "described" are also intended to include the plural forms. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0283] In the claims, any reference numerals placed between parentheses shall not be construed as limiting the claims. The use of introductory phrases such as "at least one" and "one or more" in the claims shall not be construed as implying that another claim element introduced by the indefinite article "a(a)" or "an" limits any particular claim containing such introduced claim element to an invention containing only one such element, even when the same claim includes the introductory phrase "one or more" or "at least one" and an indefinite article such as "a(a)" or "an". The same applies to the use of definite articles. Unless otherwise indicated, terms such as "first," "second," etc., are used to make arbitrary distinctions between elements described by such terms. Therefore, these terms are not necessarily intended to indicate a temporal or other priority order of such elements. The undisputed fact that certain measurements are recited in mutually different claims does not indicate that a combination of these measurements cannot be used to obtain an advantage.

[0284] Corresponding structure, material, acts and equivalents of all means or step plus function elements are intended to include any structure, material, or act for performing the function in combination with other claimed The description of the application has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the application to the precise forms disclosed. Various modifications and variations are possible in light of the above teachings without departing from the scope or spirit of the application. It is therefore intended that the application will not be limited to the limited number of embodiments described, but will include all variations, modifications and equivalents falling within the spirit and scope of the application. The embodiments were chosen and described in order to best explain the principles of the application and the practical application, and to enable others skilled in the art to understand the application for various embodiments with various modifications as are suited to the particular use contemplated.< / s> < / s>

Claims

1. A neural network processor circuit for performing neural network computations for artificial neural networks having one or more network layers, wherein the neural network processor circuit comprises: a hierarchical storage fabric usable for a plurality of uses including storing inter-layer data, intra-layer information, and weights, the hierarchical storage fabric having a plurality of storage levels, wherein each storage level contains its own local memory usable for the uses of the plurality corresponding to the corresponding storage level; a hierarchical compute fabric including a plurality of compute units aggregated into a plurality of levels to form a hierarchy, the plurality of compute units configured for performing a large number of multiply-accumulate operations at a time and processing input data streams associated with artificial neural networks, wherein the weights are stored in a shared memory; a programmable control fabric coupled to the plurality of compute units, wherein control is achieved by using a relatively small number of control lines to control the plurality of compute units, wherein control signaling is performed outside of data processing bandwidth without the need for data-driven memory accesses; and wherein a compute level in the hierarchy of compute units that is higher than a low compute level in the hierarchy of compute units is more complex and includes a smaller number of instantiations than the low compute level.

2. The circuit of claim 1, wherein the artificial neural network to be implemented is mapped into the hierarchy of compute units.

3. The circuit of claim 1, further comprising circuitry operative to receive weights and input data for input to one or more of the compute units.

4. The circuit of claim 1, wherein the plurality of compute units include circuitry to enable neural network computations to be performed in parallel, thereby maximizing utilization of the plurality of compute units.

5. The circuit of claim 1, wherein compute unit resources and memory resources in the neural network processor circuit are balanced in order to maximize utilization.

6. The circuit of claim 1, further comprising a rule-based machine subsystem in communication with the neural network processor circuit and operative to perform one or more tasks.

7. The circuit of claim 1, wherein each level in the hierarchy includes minimal overhead circuitry.

8. The circuit of claim 1, wherein required capabilities that exceed the capabilities provided by the plurality of compute units at a particular hierarchy level are performed at a higher aggregation level in the hierarchy.

9. The circuit of claim 1, wherein the plurality of compute units require minimal control, including just enough control to perform their operations.

10. The circuit of claim 1, wherein the shared memory is operable to pass data between aggregation levels in the hierarchical storage fabric.

11. The circuit of claim 1, wherein functionality is provided in a plurality of aggregation points that require the artificial neural network to be implemented. ​ 12. The circuit of claim 1, wherein the local memory is operable to store context, enabling relatively high data processing bandwidth.

13. The circuit of claim 1, wherein the hierarchy storage fabric and the hierarchy compute fabric and the programmable control fabric are strictly separated from each other, wherein control signaling to the compute units is performed outside the data processing bandwidth and does not include data-driven memory access.

14. The circuit of claim 1, further comprising configurable interconnects between aggregation levels to produce programmable dynamic data pipelines.

15. The circuit of claim 1, wherein multiple of the compute units operate to: implement multiple artificial neural networks in parallel, each artificial neural network having one or more network layers; and process one or more input data streams concurrently.

16. The circuit of claim 1, wherein multiple of the compute units operate to function at a desired granularity of a subset of the input data streams, trading off memory usage versus latency.

17. A method implemented by an integrated circuit comprising a neural network processor circuit for performing neural network computations for artificial neural networks having one or more network layers, the method comprising: providing a hierarchy storage fabric, the hierarchy storage fabric being usable for multiple purposes, the purposes including storing inter-layer data, intra-layer information, and weights, the hierarchy storage fabric having multiple storage levels, wherein each storage level contains its own local memory usable for a purpose of the multiple purposes corresponding to the corresponding storage level; aggregating multiple compute units into multiple levels to form a hierarchy compute fabric, multiple of the compute units being configured for performing a large number of multiply-accumulate operations at a time; processing input data streams associated with artificial neural networks via multiple of the compute units, wherein the weights are stored in shared memory; providing a programmable control fabric, the programmable control fabric being coupled to multiple of the compute units, control being aggregated by controlling multiple of the compute units using relatively few control lines; performing control signaling outside of data processing bandwidth, without data-driven memory access; and wherein a compute level in the hierarchy of compute units that is higher than a low compute level in the hierarchy of compute units is more complex and includes a smaller number of instantiations than the low compute level.

18. The method of claim 17, further comprising mapping the artificial neural networks to be implemented to the hierarchy compute fabric.

19. The method of claim 17, further comprising performing neural network computations in parallel, thereby maximizing utilization of multiple of the compute units.

20. The method of claim 17, further comprising balancing compute unit resources and memory resources in the neural network processor circuit to maximize resource utilization.

21. The method of claim 17, further comprising performing one or more tasks on a rule-based machine subsystem in communication with the neural network processor circuit.

22. The method of claim 17, further comprising an aggregation level in the hierarchical computing structure that provides a capability exceeding the capability of the plurality of said computing units to perform at said particular hierarchical level.

23. The method of claim 17, further comprising providing the plurality of said computing units with just enough minimal control to perform their operations.

24. The method of claim 17, further comprising transferring data between aggregation levels in the hierarchical computing structure via the shared memory.

25. The method of claim 17, further comprising: Multiple artificial neural networks are implemented in parallel by multiple computing units, each artificial neural network having one or more network layers; as well as It can process one or more input data streams at the same time.

26. The method of claim 17, further comprising operating the plurality of said computing units at a desired granularity of a subset of said input data stream, thereby compromising memory usage against latency.

Images