Method and apparatus for distributed and collaborative computing in artificial neural networks

Abstract

Methods and apparatus for distributed and collaborative computing in artificial neural networks are described. For example, one embodiment of the apparatus includes: an input / output (I / O) interface; a plurality of processing units communicatively coupled to the I / O interface to receive data for input neurons and synaptic weights associated with each input neuron, each of the plurality of processing units processing at least a portion of the synaptic weights and data of the input neurons to generate a partial result; and an interconnect communicatively coupling the plurality of processing units, each of the processing units sharing the partial results with one or more other processing units via the interconnect, the other processing units using the partial results to generate additional partial results or a final result. The processing units may share data and weights including input neurons on a shared input bus.

Description

Technical Field

[0001] The present invention generally relates to the field of computer processors. More specifically, the present invention relates to a method and apparatus for neural network acceleration. Background Art

[0002] ANNs are generally presented as a system of interconnected "neurons" that can compute values ​​from inputs. ANNs represent one of the most relevant and widespread techniques for learning and recognizing patterns. As a result, ANNs have emerged as an effective solution for intuitive human / device interaction that improves user experience, a new computing paradigm called "cognitive computing." ANNs can be used for imaging processing, speech and object recognition, or natural language processing, among other uses. Convolutional Neural Networks (CNNs) or Deep Belief Networks (DBNs) are just a few examples of computing paradigms that employ ANN algorithms. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] A better understanding of the present invention may be obtained from the following detailed description taken in conjunction with the following drawings, in which:

[0004] Figure 1A is a block diagram illustrating both an exemplary in-order fetch, decode, retire pipeline and an exemplary register renaming, out-of-order issue / execution pipeline according to an embodiment of the present invention;

[0005] Figure 1B is a block diagram illustrating both an exemplary embodiment of an in-order fetch, decode, retire core and an exemplary register renaming, out-of-order issue / execution architecture core included in a processor according to an embodiment of the present invention;

[0006] Figure 2 is a block diagram of a multi-core processor and a single-core processor with integrated memory controller and graphics according to an embodiment of the present invention;

[0007] Figure 3 A block diagram of a system according to one embodiment of the present invention is illustrated;

[0008] Figure 4 illustrates a block diagram of a second system according to an embodiment of the present invention;

[0009] Figure 5 illustrates a block diagram of a third system according to an embodiment of the present invention;

[0010] Figure 6 illustrates a block diagram of a system on chip (SoC) according to an embodiment of the present invention;

[0011] Figure 7A block diagram illustrating the use of a software instruction converter to convert binary instructions in a source instruction set into binary instructions in a target instruction set according to an embodiment of the present invention is illustrated;

[0012] Figure 8 A scheme of a fully connected artificial neural network (ANN) with multiple layers is illustrated;

[0013] Fig. 9 An exemplary neuromorphic accelerator architecture is illustrated;

[0014] Fig.10 illustrates bandwidth requirements for both a monolithic configuration and an embodiment of the present invention utilizing a distributed configuration based on the number of neurons per layer;

[0015] Fig.11 Illustration of the comparison between monolithic architecture and distributed neuromorphic architecture based on the number of neurons per layer;

[0016] Fig. 12A illustrates an embodiment in which a neuromorphic accelerator is communicatively coupled to a processor;

[0017] Fig. 12B illustrates an embodiment in which a neuromorphic accelerator is integrated within a processor;

[0018] Fig.13 One embodiment of a neuromorphic accelerator comprising a collection of processing units (PUs) is illustrated;

[0019] Figure 14A-14B illustrates how a fully connected 1-to-1 operation may be mapped according to one embodiment of the present invention;

[0020] Figure 15A-15B illustrates how a fully connected 1-to-N operation may be mapped according to another embodiment of the present invention;

[0021] Fig.16 Figure 1 shows the energy required to perform convolutions with varying filter sizes and multiple memory sizes.

[0022] Figure 17A-17B illustrates different storage arrangements for executing clusters;

[0023] Fig.18A One embodiment of a memory organization is illustrated in which memory banks and interconnects are shared between different types of data;

[0024] Fig.18B illustrates an exemplary image to which a filter may be applied according to one embodiment;

[0025] Fig.19 A scratchpad memory including a plurality of blocks according to an embodiment of the present invention is illustrated;

[0026] Figure 20A-20B illustrates different exemplary utilizations for scratchpad memory;

[0027] Fig.21 illustrates the percentage reduction in external memory accesses as a function of multiple filters;

[0028] Fig. 22 A convolutional neural network accelerator according to an embodiment of the present invention is illustrated;

[0029] Fig.23 An example set of input data used in a convolver is shown;

[0030] Fig.24 Shown from Fig.23 How elements of can be mapped in physical memory blocks;

[0031] Fig.25 illustrates a high-level block diagram of a general-purpose accelerator for image processing;

[0032] Fig.26 An exemplary image is shown in which a 4×4 pixel kernel is applied from the upper left to the lower right corner in steps of 1 pixel;

[0033] Fig. 27 illustrates multiple read ports as a function of convolution configuration;

[0034] Fig.28 An embodiment of an accelerator having 4 clusters is illustrated;

[0035] Fig.29 A data container according to an embodiment of the present invention is illustrated;

[0036] Fig.30 illustrates a table showing an alignment pattern implemented in one embodiment of the present invention;

[0037] Fig.31 illustrates a table showing the logic employed in one embodiment of the present invention;

[0038] Fig.32 illustrates how containers, alignment logic, and selection logic are interconnected to produce output packets according to one embodiment of the present invention;

[0039] Fig.33An exemplary activation function (AF) for computing the output of a neuron based on the neuron's weighted inputs is illustrated;

[0040] Figure 34A-Figure 34B illustrates the calculation of each point in the function approximated by a fixed value using a lookup table (LUT);

[0041] Figure 35A-Figure 35B It illustrates that the calculation of each point can be approximated by a line segment with a given slope using a LUT;

[0042] Fig.36 An embodiment of the present invention including a piecewise interpolation approximation unit optimized for the characteristics of AF is illustrated;

[0043] Figure 37A-Figure 37B illustrates a data set showing how an embodiment of a polymorphic decoder maps each input to a range in abscissa space;

[0044] Fig.38 The maximum error is 2 -5 and an example of a sigmoid function with 5 line segments;

[0045] Fig.39 illustrates one embodiment of an architecture including a LUT and a polystate decoder; and

[0046] Fig.40 One embodiment of a polymorphic decoder is illustrated. DETAILED DESCRIPTION

[0047] In the following description, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the embodiments of the invention described below. However, it will be clear to those skilled in the art that embodiments of the invention may be practiced without some of these specific details. In other cases, well-known structures and devices are shown in block diagram form to avoid obscuring the basic principles of embodiments of the invention.

[0048] Exemplary Processor Architectures and Data Types

[0049] Figure 1A is a block diagram illustrating both an exemplary in-order fetch, decode, retire pipeline and an exemplary register renaming, out-of-order issue / execution pipeline according to embodiments of the present invention. Figure 1B is a block diagram illustrating an in-order fetch, decode, retire core and an exemplary register renaming, out-of-order issue / execution architecture core included in a processor according to an embodiment of the present invention. Figure 1A-1BThe solid boxes in illustrate the in-order portions of the pipeline and core, while the optional addition of dashed boxes illustrates the register renaming, out-of-order issue / execution pipeline and core.

[0050] exist Figure 1A , the processor pipeline 100 includes a fetch stage 102, a length decode stage 104, a decode stage 106, an allocation stage 108, a rename stage 110, a schedule (also called dispatch or issue) stage 112, a register read / memory read stage 114, an execute stage 116, a write back / memory write stage 118, an exception handling stage 122, and a commit stage 124.

[0051] Figure 1B A processor core 190 is shown, which includes a front end unit 130 coupled to an execution engine unit 150, and both are coupled to a memory unit 170. The core 190 can be a reduced instruction set computing (RISC) core, a complex instruction set computing (CISC) core, a very long instruction word (VLIW) core, or a hybrid or alternative core type. As a further option, the core 190 can be a special-purpose core, such as, for example, a network or communication core, a compression engine, a coprocessor core, a general-purpose computing graphics processing unit (GPGPU) core, a graphics core, etc.

[0052] The front end unit 130 includes a branch prediction unit 132 coupled to an instruction cache unit 134, which is coupled to an instruction translation lookaside buffer (TLB) 136, which is coupled to an instruction fetch unit 138, which is coupled to a decode unit 140. The decode unit 140 (or decoder) can decode the instruction and generate one or more micro-operations, microcode entry points, microinstructions, other instructions or other control signals as output, which are decoded from, or otherwise reflect, or derived from the original instruction. The decode unit 140 can be implemented using a variety of different mechanisms. Examples of suitable mechanisms include, but are not limited to, lookup tables, hardware implementations, programmable logic arrays (PLA), microcode read-only memories (ROMs), etc. In one embodiment, the core 190 includes a microcode ROM or other medium that stores microcode for certain macroinstructions (e.g., in the decode unit 140 or otherwise within the front end unit 130). Decode unit 140 is coupled to rename / allocator unit 152 in execution engine unit 150 .

[0053] The execution engine unit 150 includes a rename / allocator unit 152 coupled to a retirement unit 154 and a set of one or more scheduler units 156. The (one or more) scheduler units 156 represent any number of different schedulers, including reservation stations, central instruction windows, etc. The (one or more) scheduler units 156 are coupled to (one or more) physical register file units 158. Each of the (one or more) physical register file units 158 represents one or more physical register files, where different physical register files store one or more different data types, such as scalar integers, scalar floating points, packed integers, packed floating points, vector integers, vector floating points, status (e.g., an instruction pointer that is the address of the next instruction to be executed), etc. In one embodiment, the (one or more) physical register file units 158 include vector register units, write mask register units, and scalar register units. These register units can provide architectural vector registers, vector mask registers, and general purpose registers. The physical register file(s) units 158 are overlapped by the retirement unit 154 to illustrate various ways in which register renaming and out-of-order execution may be implemented (e.g., using reorder buffer(s) and retirement register file(s); using future file(s), history buffer(s) and retirement register file(s); using register map and register pool, etc.). The retirement unit 154 and the physical register file(s) units 158 are coupled to the execution cluster(s) 160. The execution cluster(s) 160 includes a set of one or more execution units 162 and a set of one or more memory access units 164. The execution units 162 may and perform various operations (e.g., shift, addition, subtraction, multiplication) on various types of data (e.g., scalar floating point, packed integer, packed floating point, vector integer, vector floating point). While some embodiments may include several execution units dedicated to a particular function or set of functions, other embodiments may include only one execution unit or multiple execution units that all perform all functions.Scheduler unit(s) 156, physical register file unit(s) 158, and execution cluster(s) 160 are shown as possibly plural because some embodiments create separate pipelines for certain types of data / operations (e.g., scalar integer pipelines, scalar floating point / packed integer / packed floating point / vector integer / vector floating point pipelines, and / or memory access pipelines, each of which have their respective scheduler unit(s), physical register file unit(s), and / or execution clusters—and in the case of separate memory access pipelines, some embodiments are implemented where only the execution cluster for that pipeline has memory access unit(s) 164). It should also be understood that where separate pipelines are used, one or more of these pipelines may be out-of-order issue / execution, and the rest may be in-order.

[0054] A set of memory access units 164 are coupled to a memory unit 170, which includes a data TLB unit 172, which is coupled to a data cache unit 174, which is coupled to a level 2 (L2) cache unit 176. In an exemplary embodiment, the memory access unit 164 may include a load unit, a store address unit, and a store data unit, each of which is coupled to the data TLB unit 172 in the memory unit 170. The instruction cache unit 134 is further coupled to a level 2 (L2) cache unit 176 in the memory unit 170. The L2 cache unit 176 is coupled to one or more other levels of cache and ultimately to main memory.

[0055] By way of example, the exemplary register renaming, out-of-order issue / execution core architecture may implement pipeline 100 as follows: 1) instruction fetch unit 138 performs fetch stage 102 and length decode stage 104; 2) decode unit 140 performs decode stage 106; 3) rename / allocator unit 152 performs allocate stage 108 and rename stage 110; 4) (one or more) scheduler units 156 perform schedule stage 112; 5) (one or more) physical register file units 158 and memory units 170 perform register read / memory read stage 114; execution cluster 160 performs execute stage 116; 6) memory unit 170 and (one or more) physical register file units 158 perform write back / memory write stage 118; 7) various units may be involved in exception handling stage 122; and 8) retirement unit 154 and (one or more) physical register file units 158 perform commit stage 124.

[0056] Core 190 may support one or more instruction sets (e.g., the x86 instruction set (with some extensions that have been added to newer versions); the MIPS instruction set from MIPS Technologies of Sunnyvale, California; the ARM instruction set from ARM Holdings of Sunnyvale, California (with optional additional extensions such as NEON)), including the instruction(s) described herein. In one embodiment, core 190 includes logic to support packed data instruction set extensions (e.g., AVX1, AVX2, and / or some form of universal vector friendly instruction format (U=0 and / or U=1)), as described below), thereby allowing operations used by many multimedia applications to be performed using packed data.

[0057] It should be understood that a core may support multithreading (executing two or more parallel sets of operations or threads), and may do so in a variety of ways, including time-sliced ​​multithreading, simultaneous multithreading (where a single physical core provides a logical core for each of the threads that the physical core is simultaneously multithreaded), or a combination thereof (e.g., such as in Hyper-Threading technology in the subsequent time-slicing acquisition and decoding and simultaneous multi-threading).

[0058] Although register renaming is described in the context of out-of-order execution, it should be understood that register renaming can be used in an in-order architecture. Although the illustrated embodiment of the processor also includes separate instruction and data cache units 134 / 174 and a shared L2 cache unit 176, alternative embodiments may have a single internal cache such as, for example, a level 1 (L1) internal cache for both instructions and data, or multiple levels of internal caches. In some embodiments, the system may include a combination of internal caches and external caches external to the core and / or processor. Alternatively, all caches may be external to the core and / or processor.

[0059] Figure 2 is a block diagram of a processor 200 that may have more than one core, may have an integrated memory controller, and may have integrated graphics according to an embodiment of the present invention. Figure 2 The solid line box in the figure shows a processor 200 having a single core 202A, a system agent unit 210, a set of one or more bus controller units 216, while the optional addition of the dashed line box illustrates an alternative processor 200 having multiple cores 202A-N, a set of one or more integrated memory controller units 214 in the system agent unit 210, and dedicated logic 208.

[0060] Thus, different implementations of processor 200 may include: 1) a CPU having dedicated logic 208 (which may include one or more cores) that is integrated graphics and / or scientific (throughput) logic, and cores 202A-N that are one or more general purpose cores (e.g., general purpose in-order cores, general purpose out-of-order cores, a combination of both); 2) a coprocessor having cores 202A-N that are a large number of dedicated cores primarily for graphics and / or scientific (throughput); and 3) a coprocessor having cores 202A-N that are a large number of general purpose in-order cores. Thus, processor 200 may be a general purpose processor, a coprocessor, or a dedicated processor, such as, for example, a network or communication processor, a compression engine, a graphics processor, a GPGPU (general purpose graphics processing unit), a high throughput multi-integrated core (MIC) coprocessor (including 30 or more cores), an embedded processor, etc. The processor may be implemented on one or more chips. Processor 200 may be part of one or more substrates and / or implemented on one or more substrates using any of a variety of process technologies such as, for example, BiCMOS, CMOS, or NMOS.

[0061] The memory hierarchy includes one or more levels of cache within the core, a set or one or more shared cache units 206 and external memory (not shown) coupled to a set of integrated memory controller units 214. The set of shared cache units 206 may include one or more mid-level caches, such as level 2 (L2), level 3 (L3), level 4 (L4) or other levels of cache, a last level cache (LLC), and / or combinations thereof. While in one embodiment, a ring-based interconnect unit 212 interconnects the dedicated logic 208, the set of shared cache units 206, and the system agent unit 210 / (one or more) integrated memory controller units 214, alternative embodiments may use any number of known techniques for interconnecting such units. In one embodiment, coherency is maintained between the one or more cache units 206 and the cores 202-AN.

[0062] In some embodiments, one or more cores 202A-N may be multithreaded. System agent unit 210 includes those components that coordinate and operate cores 202A-N. System agent unit 210 may include, for example, a power control unit (PCU) and a display unit. The PCU may be or include logic and components required to adjust the power state of cores 202A-N and dedicated logic 208. The display unit is used to drive one or more externally connected displays.

[0063] In terms of architectural instruction sets, cores 202A-N may be homogeneous or heterogeneous; that is, two or more of cores 202A-N may be capable of executing the same instruction set, while other cores may be capable of executing only a subset of the instruction set or a different instruction set. In one embodiment, cores 202A-N are heterogeneous and include both "small" cores and "big" cores described below.

[0064] Figure 3-Figure 6 is a block diagram of an exemplary computer architecture. Other system designs and configurations known in the art for laptop computers, desktop computers, handheld PCs, personal digital assistants, engineering workstations, servers, network devices, network hubs, switches, embedded processors, digital signal processors (DSPs), graphics devices, video game devices, set-top boxes, microcontrollers, cellular phones, portable media players, handheld devices, and various other electronic devices are also suitable. In general, a wide variety of systems or electronic devices that can incorporate processors and / or other execution logic as disclosed herein are generally suitable.

[0065] Reference now Figure 3 , a block diagram of a system 300 according to one embodiment of the present invention is shown. The system 300 may include one or more processors 310, 315 coupled to a controller hub 320. In one embodiment, the controller hub 320 includes a graphics memory controller hub (GMCH) 390 and an input / output hub (IOH) 350 (which may be on separate chips); the GMCH 390 includes memory and a graphics controller, a memory 340 and a coprocessor 345 are coupled to the GMCH 390; and the IOH 350 couples an input / output (I / O) device 360 ​​to the GMCH 390. Alternatively, one or both of the memory and the graphics controller are integrated within the processor (as described herein), the memory 340 and the coprocessor 345 are coupled directly to the processor 310, and are coupled to the controller hub 320 in a single chip using the IOH 350.

[0066] Figure 3 The optional nature of the additional processor 315 is indicated by dashed lines in FIG.

[0067] The memory 340 may be, for example, dynamic random access memory (DRAM), phase change memory (PCM), or a combination of both. For at least one embodiment, the controller hub 320 communicates with the processor(s) 310, 315 via a multi-drop bus such as a front-side bus (FSB), a point-to-point interface such as a QuickPath Interconnect (QPI), or similar connection 395.

[0068] In one embodiment, coprocessor 345 is a special purpose processor such as, for example, a high throughput MIC processor, a network or communication processor, a compression engine, a graphics processor, a GPGPU, an embedded processor, etc. In one embodiment, controller hub 320 may include an integrated graphics accelerator.

[0069] There may be various differences between the processors 310 , 315 in a range of quality metrics including architectural, microarchitectural, thermal, power consumption characteristics, and the like.

[0070] In one embodiment, processor 310 executes instructions that control general types of data processing operations. Embedded within the instructions may be coprocessor instructions. Processor 310 recognizes these coprocessor instructions as being of a type that should be executed by an attached coprocessor 345. Accordingly, processor 310 issues these coprocessor instructions (or control signals representing coprocessor instructions) on a coprocessor bus or other interconnect to coprocessor 345. Coprocessor(s) 345 accept and execute the received coprocessor instructions.

[0071] Reference now Figure 4 , which is a block diagram of a first more specific exemplary multiprocessor system 400 according to an embodiment of the present invention. Figure 4 As shown in FIG. 4 , the multiprocessor system 400 is a point-to-point interconnect system and includes a first processor 470 and a second processor 480 coupled via a point-to-point (PP) interconnect interface 450. Each of the processors 470 and 480 may be a version of the processor 200. In one embodiment of the present invention, the processors 470 and 480 are processors 310 and 315, respectively, and the coprocessor 438 is the coprocessor 345. In another embodiment, the processors 470 and 480 are processors 310 and coprocessor 345, respectively.

[0072] Processors 470 and 480 are shown to include integrated memory controller (IMC) units 472 and 482, respectively. Processor 470 also includes point-to-point (PP) interfaces 476 and 478 as part of its bus controller unit; similarly, second processor 480 includes PP interfaces 486 and 488. Processors 470, 480 can exchange information via point-to-point (PP) interconnect interface 450 using PP interfaces 478, 488. Figure 4 As shown in FIG. 4 , IMCs 472 and 482 couple the processors to respective memories, namely, memory 432 and memory 434 , which may be portions of main memory locally attached to the respective processors.

[0073] Each of the processors 470, 480 may exchange information with the chipset 490 via separate PP interfaces 452, 454 using point-to-point interfaces 476, 494, 486, 498. The chipset 490 may optionally exchange information with the coprocessor 438 via a high performance interface 439. In one embodiment, the coprocessor 438 is a special purpose processor such as, for example, a high throughput MIC processor, a network or communication processor, a compression engine, a graphics processor, a GPGPU, an embedded processor, etc.

[0074] A shared cache (not shown) may be included in either processor or external to both processors but still connected to the processors via the PP interconnect so that local cache information of one or both processors may be stored in the shared cache if the processors are placed in a low power mode.

[0075] Chipset 490 may be coupled to first bus 416 via interface 496. In one embodiment, first bus 416 may be a peripheral component interconnect (PCI) bus or a bus such as a PCI Express bus or another third generation I / O interconnect bus, but the scope of the invention is not limited in this regard.

[0076] like Figure 4 As shown in , various I / O devices 414 may be coupled to first bus 416 along with a bus bridge 418 that couples first bus 416 to a second bus 420. In one embodiment, one or more additional processors 415, such as a coprocessor, a high throughput MIC processor, a GPGPU, an accelerator (such as, for example, a graphics accelerator or a digital signal processing (DSP) unit), a field programmable gate array, or any other processor, are coupled to first bus 416. In one embodiment, second bus 420 may be a low pin count (LPC) bus. In one embodiment, various devices may be coupled to second bus 420 including, for example, a keyboard and / or mouse 422, communication devices 427, and a storage unit 428 such as a disk drive or other mass storage device that may include instructions / code and data 430. Additionally, audio I / O 424 may be coupled to second bus 420. Note that other architectures are possible. For example, instead of Figure 4 Instead of a point-to-point architecture, the system may implement a multi-point bus or other such architecture.

[0077] Reference now Figure 5 , shown is a block diagram of a second more specific exemplary system 500 according to an embodiment of the present invention. Figure 4 and Figure 5 Similar elements in the drawings have similar reference numerals and have been changed from Figure 5 Omitted Figure 4 to avoid blurring certain aspects of Figure 5 other aspects.

[0078] Figure 5 It is illustrated that processors 470, 480 may include integrated memory and I / O control logic ("CL") 572 and 582, respectively. Thus, CL 572, 582 includes an integrated memory controller unit and includes I / O control logic. Figure 5 Not only are memories 432, 434 illustrated coupled to CL 572, 582, but I / O devices 514 are also illustrated coupled to control logic 572, 582. Legacy I / O devices 515 are coupled to chipset 490.

[0079] Reference now Figure 6 , which is a block diagram of a SoC 600 according to an embodiment of the present invention. Figure 2 Similar elements in the SoC have similar reference numerals. In addition, dashed boxes are optional features on more advanced SoCs. Figure 6 , the interconnect unit(s) 602 are coupled to: an application processor 610, which includes a set of one or more cores 502A-N and a shared cache unit(s) 506; a system agent unit 510; a bus controller unit(s) 516; an integrated memory controller unit(s) 614; a set or one or more coprocessors 620, which may include integrated graphics logic, image processors, audio processors, and video processors; a static random access memory (SRAM) unit 630; a direct memory access (DMA) unit 632; and a display unit 640 for coupling to one or more external displays. In one embodiment, the coprocessor(s) 620 include a special purpose processor, such as, for example, a network or communication processor, a compression engine, a GPGPU, a high throughput MIC processor, an embedded processor, etc.

[0080] The embodiments of the mechanisms disclosed herein may be implemented in hardware, software, firmware, or a combination of such implementation methods. Embodiments of the present invention may be implemented as a computer program or program code executed on a programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0081] Such as Figure 4 Program code such as the code 430 illustrated in FIG. 4 is applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0082] The program code can be implemented in a high-level program or object-oriented programming language to communicate with the processing system. If necessary, the program code can also be implemented in assembly or machine language. In fact, the mechanism described herein is not limited to any specific programming language in scope. In any case, the language may be a compiled or interpreted language.

[0083] One or more aspects of at least one embodiment may be implemented through representative instructions stored on a machine-readable medium representing various logic within a processor, which when read by a machine causes the machine to fabricate logic for performing the techniques described herein. Such representations, known as "IP cores," may be stored on a tangible, machine-readable medium and supplied to various customers or manufacturing plants to load into manufacturing machines that actually fabricate the logic or processor.

[0084] Such machine-readable storage media may include, but are not limited to, a non-transitory, tangible arrangement of an article of manufacture manufactured or formed by a machine or apparatus, including storage media such as hard disks, any other type of disk including floppy disks, optical disks, compact disk-read only memory (CD-ROM), compact disk-rewritable (CD-RW), and magneto-optical disks, semiconductor devices, such as read-only memory (ROM), random access memory (RAM) such as dynamic random access memory (DRAM), static random access memory (SRAM), erasable programmable read-only memory (EPROM), flash memory, electrically erasable programmable read-only memory (EEPROM), phase change memory (PCM), magnetic or optical cards, or any other type of medium suitable for storing electronic instructions.

[0085] Therefore, embodiments of the present invention also include non-transitory, tangible machine-readable media containing instructions or containing design data, such as hardware description languages ​​(HDL) that define the structures, circuits, devices, processors and / or system features described herein. Such embodiments may also be referred to as program products.

[0086] In some cases, an instruction converter may be used to convert instructions from a source instruction set to a target instruction set. For example, the instruction converter may morph, emulate, or otherwise convert an instruction translation (e.g., using a static binary translation, including a dynamically compiled dynamic binary translation) to one or more other instructions to be processed by the core. The instruction converter may be implemented in software, hardware, firmware, or a combination thereof. The instruction converter may be on the processor, outside the processor, or partially on the processor and partially outside the processor.

[0087] Figure 71 is a block diagram of a software instruction converter that converts binary instructions in a source instruction set into binary instructions in a target instruction set according to an embodiment of the present invention. In the illustrated embodiment, the instruction converter is a software instruction converter, but alternatively, the instruction converter can be implemented in software, firmware, hardware, or various combinations thereof. Figure 7 It is shown that a program in a high-level language 702 can be compiled using an x86 compiler 704 to generate x86 binary code 706 that can be natively executed by a processor 716 having at least one x86 instruction set core. The processor 716 having at least one x86 instruction set core represents any processor that can perform substantially the same functions as an Intel processor having at least one x86 instruction set core by compatibly executing or otherwise processing (1) a large portion of the instruction set of the Intel x86 instruction set core or (2) an object code version of an application or other software that is targeted to run on an Intel processor having at least one x86 instruction set core in order to achieve substantially the same results as the Intel processor having at least one x86 instruction set core. The x86 compiler 704 represents a compiler that can operate to generate x86 binary code 706 (e.g., object code) that can be executed on a processor 716 having at least one x86 instruction set core with or without additional linkage processing. Similarly, Figure 7 It is shown that a program in a high-level language 702 can be compiled using an alternative instruction set compiler 708 to generate an alternative instruction set binary code 710 that can be executed locally by a processor 714 that does not have at least one x86 instruction set core (e.g., a processor having a core that executes the MIPS instruction set of MIPS Technologies of Sunnyvale, California and / or executes the ARM instruction set of ARM Holdings of Sunnyvale, California). The instruction converter 712 is used to convert the x86 binary code 706 into code that can be executed locally by the processor 714 that does not have an x86 instruction set core. This converted code is unlikely to be the same as the alternative instruction set binary code 710 because an instruction converter that can achieve this is difficult to manufacture; however, the converted code will complete the general operation and be composed of instructions from the alternative instruction set. Therefore, the instruction converter 712 represents software, firmware, hardware, or a combination thereof that allows a processor or other electronic device that does not have an x86 instruction set processor or core to execute the x86 binary code 706 through simulation, emulation, or any other process.

[0088] Method and apparatus for distributed and collaborative computing in artificial neural networks

[0089] Artificial neural networks (ANNs) can be designed as a collection of fully connected layers containing a large number of "neurons". Each neuron is connected to all neurons belonging to adjacent layers through "synapses". The strength or magnitude of the connection between two neurons across a synapse is called a "synaptic weight" (or just "weight"), which can be represented as a numerical value. Therefore, the output of a neuron can be calculated by adding all the input neurons from the previous layer weighed by their synaptic weights, an operation called a dot product. Figure 8 illustrates a scheme of a fully connected ANN with multiple layers (layer 1, layer i-1, layer i, layer N) and shows how multiple weights w can be combined using a dot product operation oj -w (m-1)j .

[0090] ANNs such as brain-inspired algorithms may require very high computational requirements, which may be prohibitive in traditional low-power devices due to their limited power budget and processing capabilities. To overcome this problem, both industry and academia have recently shown great interest in introducing specialized neuromorphic architectures that provide energy efficiency orders of magnitude better than conventional methods.

[0091] Unfortunately, the proposed neuromorphic architecture requires huge memory bandwidth, which is not available in low-power system-on-chip (SoC) architectures. For example, it has been shown that the bandwidth required by the fully connected layers in a simple CNN system is 120GB / s, while the available bandwidth in an SoC is much smaller. Even with the use of tiling techniques and internal memory, the bandwidth requirements remain prohibitive.

[0092] Fig. 9 A neuromorphic accelerator architecture 900 is shown, where a single processing unit ("PU") 901 is responsible for computing dot product operations for each logical neuron, and accumulating partial results until all input neurons have been traversed and the result is final. Inputs and weights are brought to the PU 901 from an input / output (IO) interface 902 using a point-to-point bus that connects each element from the unit with internal memory or the outside world.

[0093] Such as Fig. 9The main problem with the current design shown in is that inputs and weights are not reused inside the PU 901. Since each logical neuron has its own weight, new data must be fetched every cycle, adding significant traffic between the PU 901 and the IO interface 902. In addition, since only a single logical neuron is mapped in hardware, the input must be accessed again for each neuron contained in a given fully connected layer, thereby losing the opportunity to reuse the input. In addition, since both inputs and weights must be delivered at each calculation, the bus from the IO interface 902 to the PU is too wide, making the design inefficient. Better reuse of input data and reducing the amount of data transmitted for each calculation will greatly improve the energy efficiency of the architecture.

[0094] Embodiments of the invention described below include an optimized architecture for computing fully connected neural networks very efficiently. One embodiment of the architecture consists of a collection of distributed processing units (PUs) that work collaboratively to minimize bandwidth requirements and reduce the number of external reads and writes. A collection of specific mechanisms are implemented to (i) reuse as much data as possible within the PU, (ii) share data between PUs to avoid broadcasting data, and (iii) request data in advance to avoid idle PU cycles. The scalar and modular approach allows the system to be extremely efficient in many market segments ranging from high-performance devices to low-power devices.

[0095] ANNs can be executed in conventional architectures (e.g., high-end processors) or in specialized neuromorphic accelerators. It has been shown that the latter offer orders of magnitude improved energy efficiency compared to the former and are therefore more suitable for those divisions where power is limited, such as SoCs or wearable devices. However, current neuromorphic architectures present a monolithic structure with huge bandwidth requirements, which is not available in the above-mentioned fields. This obstacle hinders the implementation of those accelerators in market divisions where the cognitive capabilities of ANNs would be particularly useful (e.g., mobile devices).

[0096] The embodiments of the invention described below enable the integration of neuromorphic accelerators in low-power devices, reduce data traffic by maximizing data reuse on fully connected operations, and reduce the width of the bus connecting the PU to the IO interface. This design allows the system to save area (because fewer connections are required) and power (because fewer external memories are accessed). In addition, the external bandwidth is significantly reduced. These properties allow embodiments of the invention to meet the physical constraints in current SoC technology.

[0097] Fig.10Bandwidth requirements for both a monolithic configuration and an embodiment of the present invention utilizing a distributed configuration are shown. The Y-axis shows bandwidth (GB / s), while the X-axis shows the number of output neurons in a given ANN layer (the number of input neurons is a fixed number). The monolithic configuration introduces a single PU that can operate at 1 GHz (full speed mode) or 0.25 GHz (bandwidth saving mode) and includes 64 8-bit multiply-accumulate (MAC) units. An embodiment of a distributed configuration operates at 0.25 GHz, supports two configurations (1 to 1 and 1 to N) and includes sixteen PUs each with 4 8-bit multipliers (a total of 64 8-bit MAC units, the same resources as a monolithic configuration).

[0098] Fig.10 It is shown that a single-chip configuration requires up to 128GB / s (1GHz) in a large ANN, and 32GB / s is required when operating at a quarter of their frequency. This is a fixed number regardless of the number of output neurons of the ANN layer. If the chip connection is shared between different components, low-power partitioning cannot support this bandwidth. By supporting both 1-to-1 and 1-to-N configurations, the distributed architecture requires less than 14GB / s (the maximum bandwidth requirement that is independent of the number of output neurons), which is reduced to 1 / 2.3 compared to the bandwidth required for a single-chip architecture operating at the same frequency and reduced to about 1 / 9.8 compared to the bandwidth required for a single-chip architecture operating at full speed. In one embodiment, this is achieved because the calculation is distributed between different PUs to better allow data reuse and less access to external memory. Note that the 1-to-1 configuration is better suited for ANNs with several output neurons, while the 1-to-N configuration works better when calculating large ANNs with a large number of output neurons. The distributed architecture presented in this article supports two execution modes.

[0099] Furthermore, a monolithic architecture requires more connections to external components (such as internal or external memory) than a distributed architecture. Fig.10 In the configuration example shown in , the monolithic configuration requires a 128-byte wide bus for delivering 64B data and 64B weights to a single PU per cycle, while the distributed solution presented in this paper only requires a 20-byte wide bus shared between PUs to deliver 4B data and 16B weights. This results in a 75% area savings in the circuit and a similar reduction in interconnect power consumption.

[0100] Fig.11A comparison between monolithic and distributed neuromorphic architectures is illustrated, where the Y-axis shows energy consumption (in mJ) and the X-axis shows the number of output neurons. Looking at the figure, it can be seen that the benefit on energy consumption is obtained when implementing a distributed architecture, which translates into more than a 2x improvement in energy consumption. This is because accessing main memory is a power-hungry operation, which can be mitigated when multiple small PUs work in collaboration.

[0101] Figure 12A-12B Different system architectures on which embodiments of the present invention may be implemented are illustrated. Both exemplary architectures include a processor 1200 including a plurality of cores 1201-1203, N and an "uncore" component 1210 including a communication bus structure or construction for interconnecting the cores 1201-1203, N. The uncore component 1210 also includes a memory controller 1211 that communicatively couples each of the cores 1201-1203, N to a system memory 1220 (e.g., a dynamic random access memory (DRAM) or other type of storage device).

[0102] exist Fig. 12A In the embodiment shown in , the neuromorphic accelerator 1230 used to implement embodiments of the present invention described herein is configured as an external accelerator that is coupled to the processor via a bus or communication fabric. Fig. 12B In an alternative embodiment shown in , the neuromorphic accelerator 1231 is integrated within the processor 1200. In either case, access to the system memory 1220 is provided to the neuromorphic accelerator 1230 via the integrated memory controller 1211. Although for illustration purposes only, the system memory 1220 is provided to the neuromorphic accelerator 1230 via the integrated memory controller 1211. Figure 12A-12B These two specific architectures are shown in , but it should be noted that the underlying principles of the invention are not limited to any specific processor architecture.

[0103] like Fig.13 As shown in , one embodiment of a neuromorphic accelerator 1230 includes a collection of PUs 1300-1303 connected to an IO interface 1310 via a shared data and partial input bus (labeled 1 in the figure) and a shared result output bus (labeled 2). The first bus (1) is used to bring data, kernels, and weights to the PUs 1300-1303, while the second (2) is used to deliver partial or final results. The PUs 1300-1303 are interconnected using unidirectional point-to-point (P2P) links (labeled 3), which are used to send partial computations from one PU to another. A global control unit 1305 is responsible for driving data from the IO interface 1310 to the PUs 1300-1303, as well as configuring each PU to perform the appropriate fully connected configuration at any point in time.

[0104] In one embodiment, to optimize power / performance, the PUs 1300-1303 may operate at a lower frequency than the IO interface 1310. The underlying principles of the invention are not limited to any number of PUs 1300-1303 sharing a bus and / or any specific resources contained in each PU. In one embodiment, each PU 1300-1303 contains sixteen 8-bit multipliers, and the architecture includes four distributed PUs 1300-1303 working in collaboration. Various bus widths may be employed, but in one particular embodiment, the width of the shared data input bus (labeled 1 in the figure) is 20B.

[0105] Fig.13 The collaborative approach illustrated in Fig. 9 Many benefits of the monolithic neuromorphic architecture shown in . First, the monolithic architecture contains all processing elements (MAC units and specific circuits) in a single PU, while embodiments of the present invention distribute calculations among multiple PUs 1300-1303, requiring internal latches and specific communication channels between PUs to exchange partial results. Second, the monolithic architecture requires a wider bus between the PU and the IO interface to send / receive as much data as possible per cycle, while embodiments of the present invention share the bus between PUs. Third, embodiments of the present invention introduce some internal latches that are used to reuse data inputs and store other data required for future calculations. Finally, in one embodiment, PUs 1300-1303 operate at a lower frequency when executing fully connected layers to meet bandwidth constraints (although other layers can operate at full speed). Assuming four PUs 1300-1303 are suspended from the same bus, each PU can operate at one-quarter of the bus frequency (1PU cycle = 4 bus cycles). Note that in one embodiment, PUs 1300-1303 always keep running, although less data is sent per second.

[0106] As mentioned, one embodiment of the present invention supports two execution modes: (1) fully connected 1-to-1 and (2) fully connected 1-to-N operation. As explained above, depending on the target ANN configuration, it may be more appropriate to execute in one mode or the other. The main difference between the execution modes is the way inputs and weights are processed. While the first mode requires inputs and weights to arrive at a given PU 1300-1303 at the same time, the second mode first loads the inputs of the neurons and then traverses the output neurons by only taking the weights at each cycle. Figure 14A-14B illustrates how a fully connected 1-to-1 operation may be mapped according to one embodiment of the present invention and Figure 15A-15B It illustrates how a fully connected 1-to-N operation may be mapped according to another embodiment of the present invention.

[0107] First go to Figure 14A-14B , a fully connected 1-to-1 operation maps one logical neuron to one physical neuron computed in a single PU 1300-1303. To perform a dot product operation, both the input neuron and the synapse (weight) 1400-1403 are sent. Since each neuron requires its own weight, both the input neuron and the weight must be obtained for each operation. Each entity hanging from the bus receives their weights packaged together at a different time (encoded using 16B in one embodiment). However, the input neurons are shared between PUs 1300-1303, and therefore they can be broadcast to avoid multiple reads from the IO interface 1310, requiring only 4B per PU cycle. Therefore, the total bandwidth requirement for the illustrated embodiment is 20B at a time, as shown in FIG. Fig. 14B Therefore, in Fig. 14B In the embodiment shown, after four PU cycles (1-4), all PUs 1300-1303 have both the inputs and weights required for the calculation (a total of 128B in the illustrated embodiment), so they can perform the dot product operation and accumulate the results with previous calculations if the neuron has many inputs. In one embodiment, PUs 1300-1303 send the output results back to IO interface 1310 only when the last input of the logic neuron is processed.

[0108] Now go to Figure 15A-15B , when operating with many input neurons, PUs 1300-1303 can be configured to map one logical neuron among multiple physical neurons and thus be more efficient. In this approach, each PU 1300-1303 computes only a subset of the logical neurons and passes partial results to the contiguous PU, which accumulates its own computations with previous results. Therefore, the final output of the neuron is only available in the last PU.

[0109] Note that PUs 1300-1303 can compute different logic neurons every cycle. That is, they need new weights, but they do not fetch values ​​from input neurons again, these values ​​are kept in local latches to increase data reuse. Fig. 15B shows how the inputs and weights are distributed when operating with a 1-to-N configuration. As in the 1-to-1 case, PUs 1300-1303 can also operate at one-quarter of their maximum frequency. Nevertheless, inputs are not broadcast every PU cycle; they are only sent when input neurons are changed (e.g. when a new layer needs to be calculated). To avoid idle cycles, the IO interface 1310 can pre-send inputs for the logical neurons to be calculated next, and these values ​​are internally latched in the PU until they are actually used. Fig. 15BAs shown in , those inputs are sent in separate PU cycles and, along with the weights, are consumed by only a single PU. Thus, in the illustrated embodiment, a 20B wide bus is sufficient to sustain the entire computation.

[0110] Although several specific architecture examples are provided above for the purpose of explanation, the basic principles of the present invention can be implemented in a variety of different architectures including mobile devices (e.g., smart phones), tablet computers equipped with cameras or microphones, or wearable devices. In fact, these embodiments can be implemented on any form of device to reduce the bandwidth requirements of machine learning algorithms and improve energy efficiency on new computer paradigms like artificial neural networks (e.g., convolutional neural networks or deep belief neural networks).

[0111] Storage device and method for performing convolution operation

[0112] Image processing is becoming increasingly important in the design of system-on-chip (SoC) architectures. Image signal processors (ISPs) in mobile devices are responsible for processing increasingly large images and are expected to maintain or reduce their power budget. Convolutional neural networks (CNNs), biologically inspired algorithms that have gained interest due to their applications in novel human-machine interfaces, also make extensive use of 2D convolutions. Currently, the best object recognition algorithms utilize convolutional neural networks and have achieved recognition rates that are significantly better than previous top-performing algorithms.

[0113] Because convolution operations rely on a high degree of data reuse, they benefit greatly from having a dedicated memory device. Fig.16 The energy required to perform convolutions for varying filter sizes and multiple memory sizes is shown. It can be seen that the energy required can be reduced by orders of magnitude, especially for large filter sizes where the required data does not fit in the processing unit. Therefore, if there is sufficient area budget, it is desirable to use dedicated memory devices in the accelerator in order to increase the overall energy efficiency.

[0114] Several accelerators for these kinds of applications have been proposed in industry and academia and most rely on Fig.17A 1701-1706 for different types of data and processing units as illustrated in FIG. However, as discussed below, for a fixed amount of storage, the optimal memory mapping varies depending on the problem type and size. Therefore, having static memory partitioning results in suboptimal memory usage.

[0115] A straightforward solution to this problem is to have a unified data / partial store 1710 with multiple read / write ports dedicated to different types of data, such as Fig. 17BHowever, the area and energy consumed by the memory block increases significantly with the number of ports. By adding only one additional port, the array area can be doubled and both dynamic and leakage energy can be increased by about 35%. In addition, if a minimum quality of service is required for both types of data, a dedicated interconnect or virtual channel should be allocated to each type of data.

[0116] Therefore, the memory of existing accelerators is limited by two conflicting goals. The dedicated memory 1701-1706 is simple and energy efficient, but does not provide efficient, adaptable use of storage. In addition, the data / partial storage 1710 may be suitable for the problem requirements but requires more complex and less efficient memory arrays and interconnects.

[0117] The embodiments of the invention described below include scratchpad memory designs for hardware convolvers and neural network accelerators. These embodiments can use simple one-port memory arrays and shared interconnects, and can also adapt memory allocation based on the problem / application.

[0118] Fig.18A One embodiment of a memory organization is illustrated in which memory blocks 1801-1806 and interconnects 1811-1816 are shared between different types of data (e.g., input images and partial results) executed within an execution cluster 1800 (e.g., including multiple processing units (PUs)). Since the access patterns are well known for a particular problem, no tags are required in the memory array, saving significant area and energy. The percentage of storage dedicated to each type of data can be varied depending on the problem requirements, thus optimizing its use.

[0119] By exploiting the regular access pattern of the convolution operation, independence between the two types of data can be achieved. Fig.18B An image 1820 is shown on which the filter will be applied. In a given cycle of the convolution, the logic unit must have data for a convolution window 1821, which is based on the number of instances of the filter to be processed simultaneously and the filter size. This convolution window 1821 is based on a stride 1823 and moves to the right on each convolution operation within a convolution row 1822 until it reaches the edge of the image 1820. When the edge of the image is reached, the convolution window 1821 moves down and processes the next convolution row. This process is repeated until the end of the image is reached. At steady-state operation of the convolution unit, most of the convolution window 1821 is reused and only new portions need to be loaded from memory. The data that needs to be loaded each cycle depends on the problem and is determined by the height of the convolution window 1821 and by the stride 1823.

[0120] In order to achieve the desired quality of service for both types of data, one embodiment of the present invention uses a mapping that ensures complete independence between the two data types (typically input data and partial results). Fig.19 The operating principle of a scratchpad memory 1900 including a plurality of chunks 1-6 according to one embodiment of the present invention is shown. One idea of ​​this mapping is to distribute the data in such a way that a different chunk 1-6 is accessed in each cycle. Four cycles from T=1 to T=4 are shown. In the illustrated example, chunk 2 is accessed at T=0; chunks 2 and 3 are accessed at T=1; chunks 2-4 are accessed at T=2; and chunks 3-5 are accessed at T=3. One technique maps each row of the image / matrix so that it starts in a different chunk for each adjacent operation. In this way, if a column of N elements needs to be read in steady state, all can be read in one cycle because they are in different chunks. This mapping also ensures that we will access N chunks per cycle and that these chunks will change for each access.

[0121] Since chunks are accessed in a regular and uniform manner, any partial results that need to be stored will be done before a fixed and known maximum time. Therefore, in one embodiment, partial results are buffered in a small input buffer until they can access their destination chunks. The fact that partial results may stall for some cycles is not a problem because they are not in the critical path.

[0122] Therefore, one embodiment of the present invention includes a unified scratchpad memory 1900 for both types of data in the convolution accelerator, input data and partial results. In this scratchpad memory, all chunks are partitioned into two areas (input data and partial results), and the amount dedicated to each data type can be varied depending on the problem / application. Sharing the available storage capacity allows for optimal use for all problem sizes, which results in lower bandwidth requirements and lower energy per operation.

[0123] Embodiments of the present invention also include a mapping technique that ensures a minimum quality of service for both types of data even when using memory blocks with only one read / write port and shared interconnects. Allowing the use of memory blocks with only one port reduces the area and energy required for scratchpad memory 1900.

[0124] One advantage of this unified design is that it achieves optimal utilization of the available capacity of the scratchpad memory 1900, and most importantly, does not require multi-port memory blocks or additional array buses which typically require more area and consume more power. In addition, better scratchpad memory utilization results in significant external memory bandwidth reduction, and therefore lower power and energy consumption.

[0125] exist Fig. 20A , the upper shaded area shows the portion of a typical scratchpad memory that remains unused in relation to the number of convolution filters. More clearly, existing designs with dedicated scratchpad memories of equal size (i.e., 50:50) for input data and partial results do not efficiently utilize the total available memory area. In fact, as Fig. 20A As shown in , for the configuration parameters used in convolutional layers 1 and 2 of a face detection CNN, static partitioning only utilizes 65% and 58% of the available scratchpad capacity, respectively. On average, a 50:50 split scratchpad configuration leaves 30% of the scratchpad memory unused.

[0126] In contrast, the best way to Fig. 20B The unified scratchpad 1900, the results of which are shown in FIG, is partitioned dynamically so that the entire capacity of the scratchpad memory is actually utilized for any given convolution problem configuration. Note that a static scratchpad organization does not fully utilize the available capacity because the ratio between input data and partial results varies significantly depending on the number and size of filters and the number and dimensionality of inputs. By comparing the bottom region (input data) and the middle region (partial) for different numbers of filters (x-axis), one can easily see the difference between the two. Fig. 20A This was observed in .

[0127] Efficient utilization is important, and as a rule of thumb, higher scratchpad utilization results in fewer accesses to external memory. The reason for this is that typically, the available internal memory is not enough to store the complete problem in local memory (e.g., 1 channel of a full HD image is ~2MB). Therefore, the input must be broken into partitions of appropriate size in order to fit both the input data and the partial results in the scratchpad memory. However, partitioning results in fetching a portion of the input data from external memory more than once. Therefore, the more partitions, the greater the external memory bandwidth overhead. Fig.21 The figure shows the reduction in external memory accesses achieved using the proposed unified scratchpad design. The benefit for a given total scratchpad memory size may vary depending on the number of channels and filters. For example, for Figure 20A-20B and Fig.21 For the example of a face detection CNN used in , the reduction in external memory accesses ranges from 3.1% to 11.3% for the configurations of convolutional layers 1 and 2, respectively. It is important to note that as we move to higher definition standards, the impact of partitioning becomes even higher. For example, for 4K UHD input images, the reduction in external memory accesses increases to 5.6% and 17.7% for layers 1 and 2, respectively.

[0128] More clearly, the flexibility of the unified design is the key to reducing the energy of external memory accesses. In fact, external memory accesses dominate the entire energy cost of CNN computations, accounting for more than 95% of the total energy for a wide set of CNN configurations. These findings further emphasize the importance of techniques to reduce redundant external memory accesses.

[0129] Most importantly, embodiments of the present invention provide this reduction of external memory bandwidth without the need for multi-port memory blocks or additional temporary register array buses that usually require more area and consume more power. Instead, the mechanism described in detail below enables the use of memory blocks with only one read / write port, similar to the high-density memory blocks of the intermediate level cache (MLC) used in general-purpose processor cores. In addition, the same shared array bus can be used to serve both the read and write requests for multiple memory blocks, which avoids the excessive increase in the area of ​​the dedicated bus. However, using a simpler memory array requires processing the read / write conflict and special data mapping in the shared bus to ensure the required quality of service for both input data and partial results. However, as shown below, two problems can be solved in a direct manner and there is no significant power and area cost.

[0130] The basic principle of the present invention can be implemented in many ways, but in a specific embodiment, Fig. 22 As shown in the figure, it has been used for Neura TM Accelerator, an ultra-low power convolutional neural network accelerator for wearable and mobile devices was developed. In this case, 4×32 memory blocks 2201 with 1 read / write port connected via 32 buses SAB0-SAB31 are used. Having 4 blocks per bus (A0-D0, A1-D1, etc.) allows writes and reads to operate at 1 / 4 of the maximum accelerator frequency. Fig. 22 The four additional structures shown in are input multiplexer (MUXI) 2202, output multiplexer (MUXO) 2203, partial write back buffer (PWBB) 2204 and control unit 2205.

[0131] In one embodiment, MUXI 2202 is a set of multiplexers that aligns data coming out of the accelerator and is used to fill the scratchpad memory block 2201 when the data is first accessed. MUXO 2203 is another set of multiplexers that aligns data coming out of the internal bus and sent to the processing units (PUs) of the execution cluster 1800 via the PU interface 2206.

[0132] In one embodiment, PWBB 2204 is a set of buffers responsible for holding partial results provided by PUs of execution cluster 1800 when the destination chunk is busy. Since the worst case pattern is known, the dimensions of these buffers can be set to store the maximum number of partials that can be generated in the worst case scenario. PWBB 2204 also handles partial writes and requests control unit 2205 to write to them when different buses are not being used to read data.

[0133] Finally, the control unit 2205 is responsible for controlling the memory group block 2201. In particular, one embodiment of the control unit 2205 indicates to the multiplexers 2202-2203 which buses SAB0-SAB31 are active in each cycle and indicates to the memory group block 2201 when to start a read or write operation and on which lines. The main operations handled by the control unit 2205 are read data, write data, read parts, and bypass data (for example, used when data comes from outside the accelerator). It also grants permission to the PWBB 2204 to write parts in idle buses and groups.

[0134] In one embodiment, the data mapping performed for each block 2201 uses the first N rows for input data (e.g., input image) and the rest for parts. The partial results can be stored in any homogeneous manner, usually in order of arrival. The input data is stored so that the accessed block changes in each cycle. Because there are 4 blocks per bus (Ax, Bx, Cx, and Dx) in this embodiment, the image can be divided so that the Ax block stores even row and column elements, the Bx block stores odd row and even column elements, the Cx block stores even row and odd column elements, and the Dx block stores odd row and column elements.

[0135] Fig.23 shows an example of input data (image) used in the convolver, and Fig.24 shows how these elements can be mapped in physical memory chunks. In the figure you can see the mapping of row 2 (i.e. (2,0), (2,2), (2,4)). Since it is an even row, it is stored in chunk A and chunk B as shown. It can be seen that the row starts in the second A chunk (i.e. (2,0) shown in A1) and that subsequent rows start on different chunks to the right.

[0136] The number of rows accessed varies depending on the problem to be solved (e.g., based on the filter size, the number of filters calculated simultaneously, etc.). Therefore, depending on the problem, a different number of blocks need to be accessed simultaneously. For example, if a convolution row consists of 6 image rows, the system will access (A0, A1, A2-B0, B1, B2-C0, C1, C2-D0, D1, D2-A1, A2, A3...). The memory organization described herein supports different convolution row sizes, which only requires one element from each block. Therefore, this particular embodiment supports accessing all new data required for convolution in two memory cycles. The number of cycles required can be selected by adding more dedicated buses depending on the problem requirements and power / area constraints.

[0137] One beneficial aspect of the above embodiment is that data does not need to identify individual tags. Since the access pattern is highly regular, the control unit 2205 is able to track the index of different chunks without the need for a tag array. Not using tags has significant advantages in terms of area, power and latency, and provides significantly improved energy efficiency over traditional storage structures.

[0138] Fine-grained storage interface and method for low-power accelerators

[0139] Image processing algorithms have gained interest due to their multiple applications in novel human-machine interfaces, which enable better user experiences. One of the most important operations in image processing is convolution. Among other applications, convolution is widely used to apply filters to images in image signal processors (ISPs) and for image recognition in convolutional neural networks (CNNs).

[0140] The convolution operation multiplies two arrays of numbers together, which are usually of different sizes but of the same dimensions, to produce a third output array. In image processing, one of the input arrays is the input image. The second array is called the kernel and is usually much smaller than the input image. The convolution operation is performed by sliding the kernel over the image, usually starting from the top left corner. Each kernel application generates an output pixel calculated by multiplying the kernel's value with the base sub-image value, and all the results are added together. Mathematically, convolution can be described as:

[0141]

[0142] Where l is the input image, K is the kernel and O(x,y) represents the pixel at coordinates x,y of the output image. The constants m and n are the kernel width and height respectively.

[0143] Image processing algorithms have very high computational requirements, which can be prohibitive for traditional low-power partitions due to their limited power budget and processing capabilities. To overcome this problem, many groups have recently been working on developing specialized architectures called "accelerators" that provide orders of magnitude better energy efficiency than conventional approaches. These architectures typically have several processing units (PUs) that perform massively parallel multiplications and additions. Fig.25 A high-level block diagram of a general-purpose accelerator for image processing is shown. PUs 0-15 form an execution cluster 2500. The outputs from the PUs are collected and processed in a synchronized manner to produce the expected results. Data is transferred from memory 2501 to the execution cluster 2500 via I / O interface 2503 or from external interface 2502 to the outside world via I / O interface 2503 (e.g., using a point-to-point bus). PUs are typically grouped into clusters that communicate with memory 2501 and external interface 2502 through an interconnect network.

[0144] However, these architectures require huge memory bandwidth to feed all processing units when performing convolutions, which reduces the overall energy efficiency of the system and requires the implementation of complex memory interconnects - prohibitive requirements for low-power system-on-chip (SOC) designs.

[0145] Current designs propose traditional data cache organizations to reduce bandwidth requirements. In these designs, the cache structure is placed between the I / O interface 2503 and the execution cluster 2500. However, these solutions do not fully exploit the characteristics of convolution operations, which leads to non-optimal results. In these approaches, each processing unit or a subset of processing units requests data individually, which requires a high number of cache read ports. In fact, in a similar Fig.25 When performing 16x16 stride 1 convolutions in the accelerator presented in , up to 18 read ports are required to provide enough data. Given that the area and energy consumed by the memory block increase significantly with the number of ports (adding just one additional port, the array area can double and both dynamic and leakage energy can increase by about 36%), the number of ports is expensive to implement the accelerator. Furthermore, this problem worsens when the number of processing units is increased to perform convolutions faster. Fortunately, the access patterns of convolutions are very predictable and they exhibit high spatial and temporal data locality, which can be exploited to reduce the number of memory accesses and the number of read ports required.

[0146] Embodiments of the present invention include a fine-grained memory interface that allows convolutions to be performed very efficiently in neural network accelerators and image processors with constrained bandwidth, area, and power requirements. The presented embodiments utilize a novel storage organization and data shuffler mechanism that work collaboratively to provide image data to the execution cluster, thereby minimizing the number of reads to higher cache levels. The scheme exploits the regular access pattern of convolution operations and enables processing of images as a series of pixel columns, pixel rows, or a combination of both, which is very convenient for efficiently providing data to the processing unit. In addition, embodiments of the present invention are modular and scalable, covering a wide variety of convolution problems ranging from applications with small filters (2x2 pixels) to large filters (64×64 pixels).

[0147] The proposed embodiments enable the integration of neural network accelerators and image processing targeting convolution operations in low power and bandwidth limited devices. The purpose of these embodiments is to process convolution operations very efficiently by providing enough image data to feed all processing units while keeping memory traffic within reasonable limits. This is achieved in one embodiment by processing the image in the most convenient way to perform the convolution operation.

[0148] Convolution has very high spatial and temporal data locality. Successive applications of the kernel share a large amount of input image data. Fig.26 An image 2600 is shown with a kernel of 4×4 pixels 2601 applied from the upper left to the lower right corner in steps of 1 pixel. When the convolution operation begins, the kernel is applied to a square image area of ​​4×4 pixels 2601. Once the first output pixel is calculated, the kernel is shifted to the right as indicated by the dashed area 2602 in the image. Since other pixels have been provided to the previous kernel operation, only four additional pixels representing columns of four rows need to be requested to perform the new operation. In addition, when the kernel is shifted downward as indicated by the alternating dashed area 2603 in the image, only a single row of four pixels is required since other pixels have been previously provided.

[0149] Conventional approaches store image data as continuous rows or columns of pixels, requiring multiple memory accesses to generate columns when the data is stored as rows, or multiple memory accesses to generate rows when the data is stored as columns. These additional accesses significantly increase the number of memory read ports, and therefore increase the overall power consumption of the accelerator and bandwidth. This is exacerbated when multiple instances of the same kernel are executed simultaneously in the accelerator. Embodiments of the present invention internally organize images to exploit data locality and provide fine-grained access to provide each kernel application with exactly the pixels it needs, thereby significantly reducing the number of ports required.

[0150] Fig. 27The diagram shows Fig.25 The number of read ports required to perform different types of convolutions in an accelerator similar to the accelerator presented in . The convolution configuration (kernel size and stride) is shown on the x-axis, while the number of ports is shown on the y-axis. An accelerator implementing a traditional cache structure is compared with and without using an embodiment of the present invention (cache + data shuffler vs. traditional cache). As can be observed, the number of read ports required in the traditional design varies from 4 in the case of convolutions of 4×4 pixels with stride 2 to 18 in the case of convolutions of 16×16 pixels with stride 1. In contrast, using an embodiment of the present invention, the number of read ports is always equal to 1, regardless of the convolution configuration. This drastic reduction in the number of cache read ports is achieved by exploiting the regular patterns observed in convolutions and making high use of data locality described in detail below.

[0151] As mentioned, embodiments of the present invention enable very efficient execution of convolutions in image processors and neural network accelerators with constrained bandwidth, area, and power requirements by significantly reducing the number of memory read ports. These properties are achieved by:

[0152] 1. Data organization and indexing based on patterns used for convolution. Data is organized in a way that exploits spatial and temporal locality. Images can be processed by traversing pixel rows, pixel columns, or a combination of both. The number of accesses to external memory and / or higher cache levels is lower than in traditional approaches, which significantly reduces the power consumption and bandwidth of the accelerator.

[0153] 2. Tag-less memory organization, which keeps the accelerator area within reasonable limits.

[0154] 3. Distributed data organization, which simplifies the connection between higher cache levels and processing units.

[0155] The above mentioned properties allow embodiments of the invention described herein to meet the physical constraints of current SoC technology.

[0156] A high-level block diagram of one embodiment of an accelerator with 4 clusters is shown in Fig.28. This embodiment of the present invention includes four components: (1) input and distribution logic 2850, (2) containers 2800-2803, (3) alignment logic 2810-2813, and (4) selection logic 2820-2823. The container, alignment logic, and selection logic components can be divided into groups, each connected to a different execution cluster 2830 of a processing unit. In contrast, in one embodiment, the input and distribution logic 2850 is shared by all clusters. As shown, the four components are connected sequentially. The input and distribution logic 2850 is directly connected to the containers 2800-2803, the containers are connected to the alignment logic 2810-2813, and the alignment logic is connected to the selection logic 2820-2823. In one embodiment, all components work in a synchronous manner, and they cooperate to provide image data to all processing unit execution clusters 2830 at the same time.

[0157] Input and distribution logic

[0158] In one embodiment, convolution is performed by applying a kernel over the entire original image. Processing unit execution clusters 2830 collaborate to traverse the image, apply convolution operations to pixels (multiplication and addition), and group the results to generate the final output image. In this process, each execution cluster 2830 of the processing unit works with a subset of pixels from the input image. The input and distribution logic 2850 obtains the image data from the higher cache levels and selectively broadcasts it to the containers 2800-2803 according to the computational requirements of each cluster of the processing unit. In fact, the pixels are stored internally in the containers of each cluster for their later use. In fact, this modular and scalable approach allows the design to cover multiple convolution configurations ranging from small kernels (2x2 pixels) to very large kernels (64x64 pixels).

[0159] Containers, alignment logic, and selection logic

[0160] In one embodiment, when performing convolution, the accelerator processes two types of information: input image data (pixels) and kernel weights. Both types of information are stored in containers 2800-2803, which can be implemented as unlabeled storage units that allow multiple consecutive pixels to be read and written in parallel. Fig.29An embodiment of a container is illustrated in FIG, which includes a set of fixed width buffers 2901 and simple logic for handling the writing and reading of elements. In particular, the input to the buffer 2901 is controlled by a write enable signal (wr_en) applied to a 7b bit mask shifter 2902. In one embodiment, the output of a counter 2904 controlled by a read enable signal (rd_en) and a write enable signal (wr_en) is ORed with the write enable signal as illustrated to control the shifter 2902. In one embodiment, the shifter 2902 shifts through the address to select each buffer entry 2901 in response to a shift signal (Init / Shift). The output of the bit mask shifter 2902 selects the buffer entry where the input data is written. Although pixels are always written sequentially (as in a FIFO buffer), they can be read randomly. In particular, in one embodiment, the multiplexer 2903 selects one of the buffer entries to output (D_out) in response to a read address signal (rd_idx).

[0161] In one embodiment of the present invention, two different types of containers are employed: row containers for image data and weight containers for kernel weights. In one specific embodiment, there are 48 row containers and 4 weight containers, representing a total of 2048 bytes.

[0162] Image pixels are stored in row containers before being sent to the execution cluster. In practice, each individual row container stores multiple pixels of a single image row. The union of multiple row containers covers a portion of the original input image. The containers make it possible to obtain access to multiple pixels from different rows simultaneously without requiring multiple accesses to higher cache levels. In one embodiment, the full set of row containers is capable of providing up to 4 consecutive pixels from N different rows, where N is the number of row containers. In order to perform convolution, the data is stored in a natural way, and the pixels can be accessed by identifying the row to which the operation will be applied. Therefore, the containers of one embodiment do not require labels.

[0163] In one embodiment, the outputs from the row containers are connected to alignment logic 2810-2813, which gives access to the individual pixels of the row entry. The alignment logic 2810-2813 makes it possible to apply simple data transformations such as the data transpose required to generate the image columns. Fig.30 The table in FIG. 1 shows a specific alignment mode implemented in one embodiment of the present invention. The first column identifies the type of alignment, while the other columns indicate the alignment of a given row from a generic row container ( x ) content in the case of alignment output (AO x ). This logic can be easily extended to support other alignments in different embodiments of the present invention.

[0164] In one embodiment, the weights are stored in memory in the same format as expected by the processing unit. Therefore, the weight container outputs do not need to be aligned and are directly connected to the outputs of the selection logic 2820-2823.

[0165] In one embodiment, the weights are stored in memory in the same format that the processing unit expects the weights.Thus, the weight container outputs do not need to be aligned and are directly connected to the outputs of the selection logic 2820-2823.

[0166] In one embodiment, selection logic 2820-2823 is in charge of creating output data in a format that is ready to be consumed by the processing unit. Typically, selection logic 2820-2823 knows what data needs to be taken from which containers and where it needs to be placed to generate output packets that will feed the processing unit. To this end, one embodiment of selection logic 2820-2823 reads the aligned output from one or more containers 2800-2803 and places this information into output packets for execution cluster 2830. The logic used in one embodiment of the present invention is Fig.31 The first column refers to the image access mode being considered: row, column, or square. The four following columns refer to the output groupings. Note that C x refers to container number X, and AO x Refers to the aligned output of the container in position X. As shown, the output is generated by reading the four aligned outputs of a single container in 4B row mode. In 4B column mode, the output is generated by selecting the first aligned position of four different containers. Finally, in 4B square mode, the output is generated by selecting the first two aligned positions of two different containers (e.g., a 2×2 pixel square area). These modes allow traversal of image data in pixel rows, pixel columns, or a combination of both. This can be very convenient to provide convolution modes to the execution cluster 2830 to highly exploit the data locality observed in convolution.

[0167] Fig.32 Illustrated is how one embodiment of the containers 2800-2803, alignment logic 2810-2813, and selection logic 2820-2923 are interconnected to produce output packets 3210 for execution cluster 2830. Each set of multiplexers 3201-3204 selects a portion of the output packets 3210 from one of the outputs from the alignment logic 2810-2813. Note that components across different clusters work in a collaborative manner and produce output packets 3210 simultaneously. After receiving data from the selection logic 2820-2823, the processing units execute in lockstep.

[0168] The containers 2800-2803, alignment logic 2810, and selection logic 2820-2823 together make it possible to provide data to all processing units in the most convenient way for performing convolutions. Although each execution cluster 2830 of processing units may have different data requirements, this logic provides data without multiple accesses to higher cache levels. The data is temporarily stored in the containers 2800-2803, and the alignment logic 2810-2813 and selection logic 2820-2823 provide it in a flexible manner to meet the needs of each execution cluster 2830. For example, the illustrated embodiment can provide multiple image columns from non-contiguous rows simultaneously, requiring only one memory access in total, while in traditional cache schemes, this operation requires a single access for each accessed row.

[0169] Reconfigurable functional unit and method for artificial neural network

[0170] As mentioned above, an ANN is configured as a collection of interconnected "neuron" layers with neurons connected by "synapses." A synaptic weight (or just weight) refers to the strength or magnitude of a connection between two neurons, which can be represented as a numerical value. Thus, the output of a neuron can be calculated by adding together all of the input neurons from the previous layer weighed by their synaptic weights. The output of this calculation is then passed through an activation function (AF), which is a mathematical function that calculates the output of a neuron based on its weighted inputs, such as Fig.33 as shown in .

[0171] In order for a neural network to approximate or predict nonlinear behavior, a nonlinear activation function must be used. Although sigmoid, bounded adjustable (rectified) linear and hyperbolic tangent are the most common functions, several nonlinear functions can be implemented. One problem in convolutional neural networks (CNNs) is what AF to choose. Studies have shown that different AFs can provide different accuracy and may also require larger or shorter training times. Ideally, a flexible hardware design should be able to implement a variety of AFs, each targeted at a specific problem.

[0172] Current solutions offered on general-purpose CPUs provide flexibility, but brain-inspired algorithms can have very high computational requirements, which can be prohibitive in traditional low-power partitions due to limited power budgets and processing capabilities. To overcome this problem, both industry and academia have shown great interest in specialized neuromorphic architectures that offer orders of magnitude better energy efficiency than conventional approaches.

[0173] However, current neuromorphic architectures provide non-optimal AF implementations. For example, many existing implementations support only one type of AF, typically the AF that provides the best results on a huge set of benchmarks. For example, the selected AF may be excellent for a face detection problem, but suboptimal for speech recognition. An architecture that has multiple AF implementations or can be reconfigured to provide multiple AF implementations could address this problem.

[0174] On the other hand, although there are some proposals to implement neuromorphic kernels with reconfigurable AFs, the methods they use are not efficient. First, those existing implementations use uniform segmentation, which is not optimal because the size of the segments is limited by the worst case (e.g., the smallest size). In addition, the calculation of each point in the function is approximated by a fixed value, which is not very useful. Fig.34A As shown in FIG, it can be implemented using a simple lookup table (LUT) 3400. Fig.34B An exemplary sigmoid approximation with the point is shown. Alternatively, Fig.35A As shown in , the calculation of each point can be approximated by a line segment with a given slope using LUT 3500. Fig.35B An exemplary sigmoid approximation with a linear function is shown.

[0175] One embodiment of the present invention includes an optimized and flexible micro-architecture for computing AF in neural networks. Fig.36 As illustrated in , one embodiment of the architecture is based on a piecewise interpolation approximation unit 3603 optimized for the characteristics of AF. It also includes a polymorphic decoder 3601 that supports non-uniform function segmentation and a LUT 3602 with parameters for different segments. As discussed below, it may also include additional logic to perform calculations. Considering the characteristics of AF used in CNN, the proposed solution minimizes the number of approximation segments for minimum accuracy requirements while also reducing energy. The design can be easily scaled based on accuracy requirements and allows the system to be extremely efficient in many market segments ranging from high performance to low power.

[0176] ANNs can be executed in conventional architectures (e.g. high-end processors) or in specialized neuromorphic accelerators. Several studies have shown that the latter offer energy efficiency orders of magnitude better than the former. They are therefore more suitable for power-constrained partitions such as SoCs or wearables. However, current AF implementations in neuromorphic accelerators are not optimal, have significant constraints in terms of flexibility (e.g., they support only one AF), or when reconfigurable, are not efficient for acceptable accuracy levels because they use simple point approximations with uniform quantization that require large LUT blocks (see Fig.34A); or a piecewise approximation with uniform quantization that requires smaller LUT blocks but larger arithmetic units (multipliers and adders) (see Fig.35A ).

[0177] The embodiments of the invention described herein overcome these limitations and are able to provide both high accuracy and efficiency with small LUTs and reasonably sized arithmetic units. To illustrate these issues, Table 1 shows the maximum error 2 for a fixed point [s2.7] representation and a fixed -7 The number of LUT entries required in the current solutions (A and B) and in the embodiments of the present invention. This was determined to be the minimum acceptable error in the face detection CNN implementation. The first three rows in the table show the number of LUT entries required to support each of the three types of AF. The last row shows the number of entries required for a reconfigurable solution that supports all three AFs. For 2 -7 To fix the error, C only needs 50% of the entries compared to B and 3.1% of the entries compared to A.

[0178] AF A-Uniform(point) B-Uniform (segmented) C-Examples of the Invention Hyperbolic tangent 512 32 16 Sigmoid 256 16 16 Bounded ReLU 512 4 4 all 512 32 16

[0179] Table 1.

[0180] Table 2 shows how the error varies when the same number of LUT entries (16) is considered for all implementation types (ie, the case where constraints are imposed on area and power consumption).

[0181] AF A-Uniform(point) B-Uniform (segmented) C-Proposed technical solution Hyperbolic tangent <![CDATA[2 -3 ]]> <![CDATA[2 -6 ]]> <![CDATA[2 -7 ]]> Sigmoid <![CDATA[2 -4 ]]> <![CDATA[2 -7 ]]> <![CDATA[2 -7 ]]> Bounded ReLU <![CDATA[2 -2 ]]> <![CDATA[2 -7 ]]> <![CDATA[2 -7 ]]>

[0182] Table 2.

[0183] Furthermore, for arithmetic calculations, embodiments of the present invention require only shifters instead of multipliers typically used in piecewise approximation solutions, thus also reducing power and area requirements.

[0184] Embodiments of the present invention include a microarchitecture that is optimized for efficiently approximating activation functions commonly used in ANNs, namely, hyperbolic tangent, sigmoid, and bounded ReLU. However, the microarchitecture is flexible enough to support other functions as long as they share common characteristics commonly found in AFs used in ANNs, namely, functions that are limited to input changes near the origin and functions that are more sensitive to input changes near the origin. The proposed design is based on the principle of piecewise approximation using linear functions. Furthermore, in one embodiment, each line segment is of the type shown in equation (1):

[0185] y(x)=α+βx (1)

[0186] Where β = {-2 t ,0,2 t}; t∈N

[0187] Overview of Activation Function Unit

[0188] One embodiment of an activation function (AF) unit includes Fig.36 There are three main components shown in: (a) polymorphic decoder 3601, (b) LUT 3602 and (c) piecewise interpolation approximation unit 3603.

[0189] In one embodiment, polymorphic decoder 3601 maps each input X to a range in the abscissa space and exploits the fact that the activation function under consideration requires more precision (i.e., smaller quantization interval) near the origin and less precision for larger |X| (see Fig.37A ). This property is common to activation functions used to model the nonlinearities present in the responses of neurons. Thus, if we consider a fixed initial set of input quantization intervals as Fig.37A , then the polymorphic decoder 3601 can be configured to select a smaller non-uniform subset of the intervals, such as Fig.37B 2602 ). The method supports efficient and flexible transformations between input X and the indices of the segmented line segments stored in LUT 2602 according to the characteristics of the approximate activation function. Therefore, the intervals selected for a given function are related to its characteristics.

[0190] In one embodiment, LUT unit 3602 is a component that stores the parameters of the linear interpolation segment. As shown, it uses the result of polymorphic decoder 3601 as input. For a given line segment, the parameters stored in LUT 3602 are used by the segment interpolation approximation unit 3603 to calculate the final result according to the above equation (1).

[0191] Method Overview

[0192] According to one embodiment of the present invention, the following method is implemented. First, the activation function under consideration is divided into line segments using any piecewise approximation algorithm that guarantees: (1) the maximum error ∈ and (2) all slopes obey the rule β = {-2 t ,0,2}

[0193] For illustration purposes, Fig.38 The maximum error is 2 -5and 5 line segments. The parameters obtained offline for different line segments are stored in LUT cells 3602, and the polymorphic decoder 3601 is configured according to the corresponding X range. Since the maximum error ∈ can be controlled, and since the number of line segments required depends on the error itself (see Tables 1 and 2), it is possible to use this method for approximate computing solutions, where, depending on the target error, the number of entries in LUT 3602 is re-adjusted to save additional power (e.g., using clock gating).

[0194] Additionally, the AF unit may be incorporated into a processing unit of the neuromorphic accelerator 1230, such as Figure 12A-12B In addition, to take advantage of the configurability of the unit, the following mechanisms may be exposed to software:

[0195] (1) A mechanism for loading approximation parameters into LUT 3602 according to the target function. This mechanism may be exposed through instructions such as "Load M(Rx)->AF".

[0196] (2) If multiple AFs are supported (e.g., by using shadow registers or other types of embodiments that support duplicate register states), additional instructions are implemented to switch between different AFs, such as "SwitchAFAFID#".

[0197] An exemplary embodiment

[0198] Fig.39 An exemplary embodiment is illustrated in FIG. 3900 , which also includes a polymorphic decoder 3902 and a LUT 3903. The module X block (|X|) 3901 exploits the fact that the hyperbolic and sigmoid functions have some symmetry to further halve the number of LUT entries. As mentioned above, the LUT 3902 maintains the different parameters required for AF linear segmentation. In this case, there are 6 parameters (t,α′,α",z′,z",e)

[0199] One embodiment of a piecewise approximate arithmetic unit includes the following components that perform the specified functions:

[0200] (i) Sgn(x) block 3910 - The sign of X is used to select the offset parameter (α), which is zero when β = 0(z), (see equation (1) above), and to adjust the final sign of Y. Mathematically we have:

[0201]

[0202] (ii) Shifter block 3911 - Shifter block is used to calculate β = 2 tWhen shifting, some loss of precision may occur. One embodiment of the present invention reduces this loss in two ways: (1) when right shifting, the LSB is stored as a guard bit and used as a carry-in in addition; and (2) depending on "α" and "t", the programmer can configure the arithmetic unit to calculate equation (1) in two ways:

[0203]

[0204] The first case, where the second can be used to avoid right shifting before the addition occurs.

[0205] (iii) Adder Block 3912 - Finally, the adder block is used to perform addition.

[0206] Fig.40 A possible embodiment of a polymorphic decoder 3902 is shown in FIG. It consists of a group of small LUTs 4001, 4002, 4003, such as Fig.37A As shown in , the group provides variable precision intervals for different segments of the range of function X. The LUT entries can then be configured to map the X intervals to a certain segment segment.

[0207] With fixed precision [s2.7] input X and [s0.7] output Y and for a maximum error of 2 -7 An embodiment of an AF unit that supports sigmoid, bounded adjustable linear, and hyperbolic tangent functions would require a LUT with 16 entries and a polystate decoder with 2 LUTs, one with 25×4 bits and the second with 24×4 bits.

[0208] In the foregoing description, the present invention has been described with reference to specific exemplary embodiments of the present invention. However, it is clear that various modifications and changes may be made thereto without departing from the broader spirit and scope of the present invention as set forth in the appended claims. Therefore, the description and drawings are to be regarded as illustrative rather than restrictive.

[0209] Embodiments of the present invention may include the various steps described above. These steps may be embodied in machine executable instructions, which may be used to cause a general or special purpose processor to perform the steps. Alternatively, these steps may be performed by specific hardware components containing hardwired logic for performing the steps or by any combination of programmed computer components and custom hardware components.

[0210] As described herein, instructions may refer to a specific configuration of hardware configured to perform certain operations or having predetermined functions or software instructions stored in a memory embodied in a non-transitory computer-readable medium, such as an application-specific integrated circuit (ASIC). Thus, the techniques shown in the figures may be implemented using code and data stored and executed on one or more electronic devices (e.g., an end station, a network element, etc.). Such electronic devices use computer machine-readable media such as non-transitory computer machine-readable storage media (e.g., magnetic disks; optical disks; random access memory; read-only memory; flash memory devices; phase change memory) and transitory computer machine-readable communication media (e.g., electrical, optical, acoustic or other forms of propagated signals - such as carrier waves, infrared signals, digital signals, etc.) to store and transmit code and data (internally and / or over a network with other electronic devices).

[0211] In addition, such electronic devices typically include a set of one or more processors coupled to one or more other components, such as one or more storage devices (non-temporary machine-readable storage media), user input / output devices (e.g., keyboards, touch screens, and / or displays), and network connections. The coupling of the processor set and other components is typically through one or more buses and bridges (also referred to as bus controllers). The storage devices and signals carrying network traffic represent one or more machine-readable storage media and machine-readable communication media, respectively. Therefore, the storage device of a given electronic device typically stores code and / or data for execution on a set of one or more processors of the electronic device. Of course, different combinations of software, firmware, and / or hardware can be used to implement one or more parts of an embodiment of the present invention. Throughout this specific embodiment, for the purpose of explanation, many specific details are set forth in order to provide a thorough understanding of the present invention. However, it is clear to those skilled in the art that the present invention can be practiced without some of these specific details. In some cases, in order to avoid obscuring the subject matter of the present invention, known structures and functions are not described in detail. Therefore, the scope and spirit of the present invention should be judged according to the following claims.

Claims

1. An apparatus for performing artificial neural network processing, comprising: Input / output interface; a plurality of processing units communicatively coupled to the input / output interface to receive data for input neurons and synaptic weights associated with each of the input neurons, each of the plurality of processing units processing at least a portion of the data and synaptic weights for the input neurons to generate a partial result; an interconnect communicatively coupling the plurality of processing units, each of the processing units sharing the partial results with one or more other processing units via the interconnect, the other processing units using the partial results to generate additional partial results or a final result; and A plurality of latches for storing received data and synaptic weights for input neurons such that input data and weights for a given computation are shared among the plurality of processing units by providing the latched input data and weights to the plurality of processing units over a plurality of processing cycles to avoid reading the received data and weights from the input / output interface multiple times during a given computation.

2. The apparatus according to claim 1, further comprising: a shared input bus for delivering the data and synaptic weights for the input neurons to each of the processing units; and A shared output bus is used to deliver the partial results or the final results to external components.

3. The device according to claim 2, wherein: The processing units are used to share data including input neurons and weights on the shared input bus.

4. The device according to claim 1, wherein: The interconnect comprises a unidirectional point-to-point link between the processing units.

5. The apparatus according to claim 1, further comprising: A global control unit is configured to drive data for the input neurons and synaptic weights associated with each of the input neurons from the input / output interface to each of the processing units and configure each processing unit to process a corresponding portion of the data to generate the partial results.

6. The device according to claim 1, wherein: The input / output interface operates at a first frequency, and the processing unit operates at a second frequency lower than the first frequency.

7. The device according to claim 1, wherein: Each processing unit includes a plurality of multipliers and adders for performing dot product operations to generate the partial results or the final result.

8. The device according to claim 1, wherein: Data for input neurons are provided to each of the processing units on each cycle for a plurality of cycles, and synaptic weights are provided to a different processing unit on each cycle.

9. The device according to claim 1, wherein: For multiple cycles, data for the input neurons are provided to the same processing unit on each cycle and synaptic weights are provided to a different processing unit on each cycle.

10. A computing device comprising: Dynamic random access memory input / output interface; A processor, coupled to the input / output interface, the processor having multiple cores; a plurality of processing units coupled to the input / output interface, the plurality of processing units performing neural network processing, the plurality of processing units comprising a first processing unit and a second processing unit, The first processing unit receives data for neurons and weights associated with the neurons via the input / output interface, and the first processing unit processes the data for neurons and weights associated with the neurons to generate a partial result; as well as The second processing unit processes the partial results to generate additional partial results or a final result; as well as An interconnect is coupled between the first and second processing units to provide a partial result from the first processing unit to the second processing unit.

11. The device according to claim 10, wherein: The second processing unit is communicatively coupled with the first processing unit via the interconnect through a unidirectional point-to-point link.

12. The apparatus of claim 10, further comprising circuitry to store the partial result before providing the partial result to the second processing unit.

13. The apparatus of claim 10, further comprising a plurality of latches to store partial results before providing the partial results to the second processing unit.

14. The device according to claim 10, wherein: The first processing unit includes a plurality of latches to store data for neurons.

15. The device according to claim 10, wherein: The partial result will be provided to the second processing unit without being sent to the dynamic random access memory through the input / output interface.

16. The device according to claim 10, wherein: The first processing unit generates a partial result representing only a portion of a layer of the neural network, and wherein the second processing unit uses the partial result to further process the layer.

17. The apparatus according to claim 10, further comprising a control unit that controls providing data to the first processing unit.

18. The device according to claim 10, wherein: The first processing unit includes a multiplier and an adder that perform a dot product operation.

Images