Pipelining for analog memory-based neural networks with global local storage

Abstract

A pipeline of an analog memory based neural network with global partial storage is provided. A first synapse array in a hidden layer receives an input array from a previous layer during a feedforward operation. The input array is stored by the first synapse array during the feedforward operation. An input array is received by a second synapse array in the hidden layer during the feedforward operation. The second synapse array computes an output from the input array during the feedforward operation based on weights of the second synapse array. The stored input array is provided to the second synapse array from the first synapse array during a backpropagation operation. A correction value is received by the second synapse array during the backpropagation operation. The weights of the second synapse array are updated based on the correction value and the stored input array.

Description

BACKGROUND

[0001] Embodiments of the present disclosure relate to neural network circuits, and more specifically to pipelining for analog memory based neural networks with global partial storage. SUMMARY

[0002] According to embodiments of the present disclosure, an artificial neural network is provided. In various embodiments, the artificial neural network includes a plurality of synaptic arrays. Each synaptic array of the plurality of synaptic arrays includes a plurality of ordered input lines, a plurality of ordered output lines, and a plurality of synapses. Each synapse is operatively coupled to one of the plurality of input lines and one of the plurality of output lines. Each synapse of the plurality of synapses includes a resistive element configured to store a weight. The plurality of synaptic arrays is configured in a plurality of layers including at least one input layer, one hidden layer, and one output layer. A first synaptic array of the at least one synaptic array in the at least one hidden layer is configured to receive and store an input array from a previous layer during a forward pass operation. A second synaptic array of the at least one synaptic array in the at least one hidden layer is configured to receive the input array from the previous layer and compute an output from the at least one hidden layer during the forward pass operation based on the weights of the second synaptic array. The first synaptic array of the at least one synaptic array is configured to provide the stored input array to the second synaptic array of the at least one synaptic array during a backpropagation operation. The second synaptic array of the at least one synaptic array is configured to receive a correction value during the backpropagation operation and update its weights based on the correction value and the stored input array.

[0003] According to embodiments of the present disclosure, an apparatus including first and second synaptic arrays is provided. Each synaptic array of the first and second synaptic arrays includes a plurality of ordered input lines, a plurality of ordered output lines, and a plurality of synapses. Each synapse of the plurality of synapses is operatively coupled to one of the plurality of input lines and one of the plurality of output lines. Each synapse of the plurality of synapses includes a resistive element configured to store a weight. The first synaptic array is configured to receive and store an input array from a previous layer of an artificial neural network during a forward pass operation. The second synaptic array is configured to receive the input array from the previous layer and compute an output during the forward pass operation based on the weights of the second synaptic array. The first synaptic array is configured to provide the stored input array to the second synaptic array during a backpropagation operation. The second synaptic array is configured to receive a correction value during the backpropagation operation and update its weights based on the correction value and the stored input array.

[0004] According to embodiments of the present disclosure, methods and computer program products for operating a neural network circuit are provided. During a forward pass operation, an input array is received from a previous layer by a first synapse array in a hidden layer. The input array is stored by the first synapse array during the forward pass operation. The input array is received by a second synapse array in the hidden layer during the forward pass operation. The second synapse array computes an output from the input array based on weights of the second synapse array during the forward pass operation. The stored input array is provided from the first synapse array to the second synapse array during a backpropagation operation. A correction value is received by the second synapse array during the backpropagation operation. The weights of the second synapse array are updated based on the correction value and the stored input array. BRIEF DESCRIPTION OF DRAWINGS

[0005] Figure 1 An exemplary non-volatile memory based crossbar array or crossbar memory according to embodiments of the present disclosure is shown.

[0006] Figure 2 An exemplary synapse within a neural network according to embodiments of the present disclosure is shown.

[0007] Figure 3 An exemplary neural kernel array according to embodiments of the present disclosure is shown.

[0008] Figure 4 An exemplary neural network according to embodiments of the present disclosure is shown.

[0009] Figure 5A A step of forward propagation according to embodiments of the present disclosure is shown.

[0010] Figure 6A A step of backpropagation according to embodiments of the present disclosure is shown.

[0011] Figure 7A A simultaneous step for forward and backpropagation according to embodiments of the present disclosure is shown.

[0012] Figure 8 A method of operating a neural network according to embodiments of the present disclosure is shown.

[0013] Figure 9 A compute node according to embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0014] Artificial neural networks (ANN) are distributed computing systems that include a plurality of neurons interconnected by junctions called synapses. Each synapse encodes the strength of a connection between the output of one neuron and the input of another. The output of each neuron is determined by the aggregate input received from other neurons connected to that neuron. As such, the output of a given neuron is based on the outputs of connected neurons from a previous layer and the strength of the connection as determined by the synapse weight. ANNs are trained to solve a particular problem (e.g., pattern recognition) by adjusting the weights of the synapses such that a particular class of input produces a desired output.

[0015] ANNs can be implemented on different types of hardware, including crossbar arrays, also known as cross-point arrays or cross-line arrays. A basic crossbar array configuration includes a set of conductive row lines and a set of conductive column lines formed to intersect the set of conductive row lines. The intersection points between the two sets of lines are separated by cross-point devices. The cross-point devices function as the weighted connections between neurons of the ANN.

[0016] In different embodiments, non-volatile memory based crossbar arrays or crossbar memory are provided. A plurality of junctions are formed by the intersection of row lines and column lines. A resistive memory element (e.g., non-volatile memory) is coupled in series with a selector at each of the junctions between one of the row lines and one of the column lines. The selector can be a volatile switch or transistor, various types of which are known in the art. It will be appreciated that various resistive memory elements are suitable for use as described herein, including memristors, phase change memory, conductive bridging RAM, and spin transfer torque RAM.

[0017] A fixed number of synapses can be provided on a core, and then multiple cores are connected to provide a complete neural network. In this embodiment, interconnectivity between the cores is provided to pass the output of a neuron on one core to another core, for example, via a packet switched or circuit switched network. In a packet switched network, greater flexibility of interconnectivity can be achieved at a power and speed cost due to the need to transmit, read, and act on address bits. In a circuit switched network, no address bits are needed, so flexibility and reconfigurability must be achieved by other means.

[0018] In different exemplary networks, multiple cores are arranged in an array on a chip. In this embodiment, the relative location of the cores can be referred to by cardinal directions (north, south, east, west). Data carried by neural signals can be encoded in the pulse duration carried by each line using a digital voltage level or other form of digital signal suitable for buffering.

[0019] One approach to routing is to provide an analog-to-digital converter at the output edge of each core, paired with a chip-on-die digital network for fast routing of packets to any other core, and a digital-to-analog converter at the input edge of each core.

[0020] Training of a deep neural network (DNN) involves three distinct steps: 1) forward inference of a training example through the entire network to an output; 2) backpropagation of deltas or corrections based on the difference between the guessed output for that training example and the known ground-truth output; and 3) weight update of each weight in the network by combining the raw forward excitation (x) associated with the neuron just upstream of the synapse weight with the backpropagated delta associated with the neuron just downstream of the synapse weight.

[0021] Due to the fact that the two pieces of data required for the weight update are produced at vastly different times, pipelining of this training process is complicated. The incoming excitation values (x vectors) are produced during the forward pass, while the incoming delta values (delta vectors) are not produced until the entire forward pass is complete and the backward pass has returned to the same neural network layer. For layers located early in the neural network, this means that the x vector data that will be needed later must be stored simultaneously, and the amount of this vector that must be stored and later fetched can be very large.

[0022] In particular, to perform a weight update at layer q, the excitation corresponding to the input m (e.g., an image) produced at some time step t is required. In addition, the delta for layer q is required, which is not available until time step t + 2l, where l is the number of layers between q and the output of the network.

[0023] Meanwhile, only the forward inference pipelining approach, which does not require long-term storage of x vectors, can efficiently pass these vectors from one array core implementing a neural network layer to the next with extremely local routing, such that all layers can process data simultaneously. For example, the array core(s) associated with the Nth DNN layer can work on the Nth data example while the array core(s) of the N-lth layer work on the N-lth data example. This approach, in which multiple data blocks are processed through the hardware system in stages, is referred to as pipelining. This is particularly efficient because each component remains busy continuously, even if adjacent components can be working on different parts of the same problem or data example, or even on completely different data examples.

[0024] A method for pipelined training that digitizes all x-vectors and delta vectors and stores them elsewhere on the chip is described. This method requires digitization, long distance routing of digital data, and a large amount of memory, and as the number of neural network layers gets large, any of these elements can become a bottleneck.

[0025] Thus, there is a need for a technique that allows pipelining of deep neural network training that provides the same scalability for large networks by eliminating all long distance data traffic.

[0026] The present disclosure provides a 5-step sequence in which two or more logic array cores are assigned to each neural network layer. These array cores can be uniquely provisioned or can otherwise be identical. One array core is responsible for extremely local short-term storage of x-vectors produced during forward pass; the other array core operates in a generally crossbar or RPU (resistive processing unit) mode of forward propagation (producing next x-vectors), backpropagation (producing delta vectors), and weight update.

[0027] In some embodiments, short-term storage can be distributed across multiple array cores, and RPU / crossbar functionality can also be distributed across multiple array cores. At the other end of the distribution spectrum, both short-term storage and crossbar functionality can be implemented on one physical array core or tile.

[0028] Referring to Figure 1 , an exemplary non-volatile memory-based crossbar array or crossbar memory is shown. A plurality of junctions 101 are formed by the intersection of row lines 102 with column lines 103. At each of the junctions 101 coupled between one of the row lines 102 and one of the column lines 103, a resistive memory element 104 (e.g., a non-volatile memory) is in series with a selector 105. The selector can be a volatile switch or transistor, various types of which are known in the art.

[0029] It will be understood that a variety of resistive memory elements are suitable for use as described herein, including memristors, phase change memory, conductive-bridge RAM, spin-transfer torque RAM.

[0030] Referring to Figure 2 , an exemplary synapse within a neural network is shown. A plurality of inputs x1...x n are multiplied by corresponding weights w ij At node 202, the weights and the sum x i w ij are provided to a function f(·) to obtain a value It will be understood that a neural network will include a plurality of such connections between layers, and this is merely exemplary.

[0031] Referring now to Figure 3 , an exemplary array of cores is shown in accordance with embodiments of the present disclosure. The array 300 includes a plurality of cores 301. The cores in the array 300 are interconnected by lines 302, as further described below. In this example, the array is two-dimensional. However, it will be appreciated that the present disclosure is applicable to one-dimensional or three-dimensional arrays of cores. The core 301 includes a non-volatile memory array 311, which implements synapses as described above. The core 301 includes a west side and a south side, each of which can serve as an input, while the other side serves as an output. It will be appreciated that the west / south nomenclature is employed merely for ease of reference to relative positioning, and is not meant to limit the direction of inputs and outputs.

[0032] In different exemplary embodiments, the west side includes support circuits 312 (dedicated to the entire side of the core 301), shared circuits 313 (dedicated to a subset of rows), and per-row circuits 314 (dedicated to individual rows). In different embodiments, the south side likewise includes support circuits 315 dedicated to the entire side of the core 301, shared circuits 316 dedicated to a subset of columns, and per-column circuits 317 dedicated to individual columns.

[0033] Referring now to Figure 4 , an exemplary neural network is shown. In this example, a plurality of input nodes 401 are interconnected with a plurality of intermediate nodes 402. In turn, the intermediate nodes 402 are interconnected with output nodes 403. It will be appreciated that this simple feed-forward network is presented for illustrative purposes only, and that the present disclosure is applicable independent of a particular neural network arrangement.

[0034] Referring now to Figure 5A -E, steps of forward propagation are shown in accordance with embodiments of the present disclosure. Figure 5A -E each illustrates the operation of a pair of arrays at a time slice.

[0035] In a first step, in Figure 5A , a parallel data vector containing the x-vector of layer q of image m is propagated across the array cores 501, 502 to reach the RPU array core 502 responsible for the layer q computation. The x-vector is also saved in the east side periphery of the array core 501 responsible for layer q storage. A multiply-accumulate operation takes place, establishing the next x-vector.

[0036] The boxes 503...505 at the west edge of each cross indicate the row-wise and shared peripheral circuits associated with the rows of the cross array, for driving the forward excitation, for analog measurement of the integrated current during backpropagation, and for applying the derived forward excitation during the weight update phase.

[0037] Similarly, boxes 506…508 on the south edge indicate column-based and shared peripheral circuitry associated with the column for simulating measurements of the integral current during forward excitation, for driving reverse excitation onto the column, and for applying these reverse excitations during the weighted update phase.

[0038] Arrow 509 indicates the propagation of the data vector along the parallel routing lines through each array, while boxes 510 and 511 mark the capacitors that are updated (e.g., filled or emptied) during this first step. Arrow 512 indicates the current integration (multiplicative accumulation) on the array. During this step, excitations are collected at the eastern edge of the left-hand array core as they pass, and these excitations drive the rows in the right-hand array core. This results in current integration along the columns where the massively parallel multiplicative accumulation operation is performed. At the end of this step, the integrated charge, representing the simulation result of these operations, is located in the capacitor at the southern edge of the right-hand array core, as indicated by box 511.

[0039] In the second step, such as Figure 5B As shown, the x-vector data will be held on the eastern periphery of the storage array core. The data is written column by column into the data column 513 associated with the image m. In some embodiments, this is accomplished using a high-endurance NVM or 3T1C (three transistors and one capacitor) or similar synaptic circuit elements that provide near-infinite endurance and a storage lifetime of several milliseconds.

[0040] Boxes 514 and 515 mark capacitors that retain values ​​from previous time steps—in this case, at the eastern edge of the left-hand array core and at the southern edge of the right-hand array core. Arrow 516 indicates parallel row-by-row writes to a 3T1C (three transistors + one capacitor) device or any other device capable of writing analog states quickly and accurately with very high endurance.

[0041] In the third step, such as Figure 5C As shown, the next x-vector data located south of the computation array core is placed on the routing network and sent to layer q+1. This process may inherently include squashing function operations, or the squashing function may be applied at points along the routing path before reaching the final destination.

[0042] exist Figures 5D-5E In steps three and four shown, no action is required. These time slices will be used for other training tasks before the next image can be processed.

[0043] While this list details operations on the array core associated with layer q, it means that layer q+1 performs exactly these same operations with a phase shift of 2 steps. This means that the arrow 517 in the third step (corresponding to data leaving layer q) is equivalent to the arrow 509 seen in the first step for layer q+1 (corresponding to data arriving at layer q+1). By extension, layer q+2 performs these same operations again, shifted in phase by 4 steps from the original layer. In other words, during forward propagation, all array cores are busy on 3 out of 5 stages.

[0044] Referring now to Figure 6A -E, the steps of backward propagation are illustrated according to embodiments of the disclosure. Figure 6A -E, each illustrates the operation of a pair of arrays at a time slice.

[0045] During the first step, as Figure 6A illustrated, a copy of the x vector of image n is taken that was previously stored so that it is available at the west side periphery of the layer q storage array core. Note that this could have been stored at some time in the past when image n was processed for forward propagation.

[0046] During the second step, as Figure 6B illustrated, the parallel delta vector of layer q of image n is propagated through the routing network to arrive at the south side of the same RPU array core, resulting in a transpose multiply accumulate operation (column driven, integrate along rows) resulting in a stored charge representing the next delta vector in the west side capacitor of the layer q compute array core. A copy of the delta vector is saved in the south side peripheral circuitry (indicated by block 601).

[0047] During the third step, as Figure 6C illustrated, the previously taken x vector is transferred from the storage array core to the compute array core so that it is now available at the west side periphery of the layer q compute array core.

[0048] During the fourth step, as Figure 6D illustrated, the x vector information at the west side periphery and the delta vector information at the south side periphery are combined to perform a cross compatible weight update (RPU array neural network weight update).

[0049] During the fifth step, as Figure 6E illustrated, any derivative information available at the west side periphery is applied to the next delta vector produced in the second step. This information is then placed on the overhead routing network, passed on the left hand array core to arrive at the next earlier layer q-1.

[0050] The phase difference between each column of array cores is consistent with the phase difference observed during the forward propagation step. Thus, each layer of the network does useful work during each time step of operation, allowing full pipelining of the training.

[0051] Referring now to Figure 7A -E, a simultaneous step for both forward and backward propagation is shown, according to embodiments of the disclosure. As these composite images show, Figure 5A -E, and Figure 6A The steps provided in -E are fully consistent and can be performed simultaneously in five time steps. This means that all storage is local and that the mechanism can be scaled to arbitrarily large neural networks, as long as routing paths can be executed without contention. Because during the time period between the initial pass of a data example during forward propagation and the final arrival of the deltas for that data example during backward propagation, one column of intermediate storage is used on each set of five steps, the maximum depth of the network that will be supported is limited by the number of columns available to store x-vectors. After the column that holds the delta values is taken and used for weight updates in the fourth step, it can be discarded and the column reused to store the forward activation data for the next incoming data example. Thus, at each layer of the network two pointers are maintained and updated, one for the incoming example m that is now being forward propagated and one for the incoming example n that is now being backward propagated.

[0052] As outlined above, a second RPU array is used for each layer to locally hold on activations and provide a throughput of one data example on fully connected layers for every five clock cycles. In this way, the throughput is maximized while the remote transfer of data is eliminated. The technique is independent of the number of layers in the network and is applicable to a variety of networks, including LSTMs and CNNs with exsitu weight updates.

[0053] Referring to Figure 8 , a method of operating a neural network is shown, according to embodiments of the disclosure. At 801, an input array is received by a first array of synapses in a hidden layer from a previous layer during a feedforward operation. At 802, the input array is stored by the first array of synapses during the feedforward operation. At 803, the input array is received by a second array of synapses in the hidden layer during the feedforward operation. At 804, an output is computed by the second array of synapses from the input array based on weights of the second array of synapses during the feedforward operation. At 805, the stored input array is provided from the first array of synapses to the second array of synapses during a backpropagation operation. At 806, a correction value is received by the second array of synapses during the backpropagation operation. At 807, the weights of the second array of synapses are updated based on the correction value and the stored input array.

[0054] Accordingly, in various embodiments, a series of tasks are used to process the training data, which tasks implement forward propagation, back propagation, and weight updates.

[0055] In the first task, a parallel data vector containing the x vector for layer q of image m is propagated across the array core to reach the RPU array core responsible for layer q computation, while this parallel data vector is also saved in the east side periphery of the array core responsible for layer q storage. Multiply-accumulate operations occur to establish the next x vector.

[0056] In the second task, the x vector data held in the east side periphery of the storage array core is written column by column to the data column associated with image m. In some embodiments, this will be done using high endurance NVM or 3T1C synaptic circuit elements providing near infinite endurance and several milliseconds of storage lifetime.

[0057] In the third task, the next x vector data at the south side of the compute array core is placed on the routing network and sent to the q+1 layer. This process can inherently include a compression function operation, or a compression function can be applied at a point along the routing path prior to the final destination.

[0058] In the first task of a subsequent iteration through the training data, at the point in time when the delta vector for layer q of image m is ready to be sent, a previously stored copy of the x vector for that same image m is fetched so that it is available in the west side periphery of the layer q storage array core.

[0059] In the second task of the subsequent iteration, the parallel delta vector for layer q of image m is propagated through the routing network to reach the south side of the same RPU array core, resulting in a transpose multiply-accumulate operation (column driven, integrate along rows), resulting in stored charge representing the next delta vector in the west side capacitor of the layer q compute array core. A copy of the delta vector that arrived is saved in the south side peripheral circuitry.

[0060] In the third task of the subsequent iteration, the previously fetched x vector is sent from the storage array core to the compute array core so that it is now available to the west side periphery of the layer q compute array core.

[0061] In the fourth task of the subsequent iteration, the x vector information at the west side periphery and the delta vector information at the south side periphery are combined to perform the conventional cross-compatible weight update normally used for RPU array neural network weight updates.

[0062] In the fifth task of the subsequent iteration, any derivative information available in the west side periphery is applied to the next delta vector produced in the second task.

[0063] Referring now to Figure 9FIG. 1 shows a schematic diagram illustrating an example of a computing node. The computing node 10 is but one example of a suitable computing node and is not intended to suggest any limitation as to the scope of use or functionality of embodiments described herein. Regardless, the computing node 10 is capable of being implemented and / or performing any of the functionality set forth above.

[0064] In computing node 10, there is a computer system / server 12, which is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well- known computing systems, environments, and / or configurations that can be suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and the like.

[0065] Computer system / server 12 can be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules can include routines, programs, objects, components, logic, data structures, and so on that perform particular tasks or implement particular abstract data types. Computer system / server 12 can be practiced in distributed cloud computing environments with remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules can be located in both local and remote computer system storage media including memory storage devices.

[0066] As Figure 9 shown in FIG. 1, computer system / server 12 in computing node 10 is shown in the form of a general-purpose computing device. The components of computer system / server 12 can include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including system memory 28 to processor 16.

[0067] Bus 18 represents one or more of any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example, and not limitation, such architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, Peripheral Component Interconnect (PCI) bus, PCI Express (PCIe), and Advanced Microcontroller Bus Architecture (AMBA) O

[0068] Computer system / server 12 typically includes a variety of computer system readable media. Such media can be any available media that is accessible by computer system / server 12 and includes both volatile and non-volatile media, removable and non-removable media.

[0069] The system memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer system / server 12 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, a storage system 34 can be provided for reading from and writing to non-removable, non-volatile magnetic media (not shown and typically called a "hard drive"). Although not specifically shown, a magnetic disk drive can also be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive can be provided for reading from or writing to a removable, non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical media). In this regard, each can be connected to bus 18 by one or more data media interfaces. As will be further depicted and described below, memory 28 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the disclosure.

[0070] Program / utility 40 having a set (at least one) of program modules 42 can be stored in memory 28 by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, can include an implementation of a network environment. Program modules 42 generally carry out the functions and / or methodologies of embodiments as described herein.

[0071] Computer system / server 12 can also communicate with one or more external devices 14 (such as a keyboard, pointing device, display 24, etc.); and / or any device that enables computer system / server 12 to communicate with one or more other computing devices (e.g., network interface card, modem, etc.). This communication can be performed via input / output (I / O) interface 22. Furthermore, computer system / server 12 can communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet), via network adapter 20. As shown, network adapter 20 communicates with other components of computer system / server 12 via bus 18. It should be understood that, although not shown, other hardware and / or software components can be used in conjunction with computer system / server 12. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archiving storage systems.

[0072] This disclosure can be implemented as a system, method, and / or computer program product. The computer program product may comprise a computer-readable storage medium (or medium) having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0073] Computer-readable storage media can be tangible devices capable of retaining and storing instructions used by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punched cards, or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0074] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions to a computer readable storage medium within the respective computing / processing device for storage and / or execution.

[0075] Computer readable program instructions for carrying out operations of the present disclosure can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or any combination of source code or object code written in one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computing device, partly on the user's computing device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, for example, through the Internet using an Internet Service Provider. In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0076] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0077] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include a non-transitory computer readable storage medium that can be a computer- readable storage medium having no data storage of any type on it. These instructions can be executed by a computer, data processing apparatus, or other device to cause a series of operations to be performed by the computer, data processing apparatus, or other device to produce a computer-implemented process such that the instructions which execute via the computer or other programmable apparatus create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0078] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0079] The flow and block diagrams in the figures illustrate the architecture, functionality, and operations of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow and block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0080] The description of the different embodiments of the disclosure has been presented for purposes of illustration but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An artificial neural network comprising a plurality of synaptic arrays, wherein: each synaptic array of the plurality of synaptic arrays comprises a plurality of ordered input lines, a plurality of ordered output lines, and a plurality of synapses; each synapse of the plurality of synapses is operatively coupled to one of the plurality of input lines and one of the plurality of output lines; each synapse of the plurality of synapses comprises a resistive element configured to store a weight; the plurality of synaptic arrays are configured in a plurality of layers comprising at least one input layer, one hidden layer, and one output layer; a first synaptic array of at least one synaptic array of the at least one hidden layer is configured to receive and store an input array from a previous layer during a forward pass operation; a second synaptic array of the at least one synaptic array of the at least one hidden layer is configured to receive the input array from the previous layer and compute an output from the at least one hidden layer during the forward pass operation based on weights of the second synaptic array; the first synaptic array of the at least one synaptic array is configured to provide the stored input array to the second synaptic array of the at least one synaptic array during a backpropagation operation; and the second synaptic array of the at least one synaptic array is configured to receive correction values during the backpropagation operation and update its weights based on the correction values and the stored input array. the forward pass operation is pipelined.

2. The artificial neural network of claim 1, wherein, the backpropagation operation is pipelined.

3. The artificial neural network of claim 1, wherein, the forward pass operation and the backpropagation operation are concurrently performed.

4. The artificial neural network of claim 1, wherein, the first synaptic array of the at least one synaptic array is configured to store one input array per column.

5. The artificial neural network of claim 1, wherein, each synapse of the plurality of synapses comprises a memory element.

6. The artificial neural network of claim 1, wherein, each synapse of the plurality of synapses comprises NVM or 3T1C.

7. The artificial neural network of claim 1, wherein, 8. An apparatus comprising: a first synaptic array and a second synaptic array, each synaptic array of the first synaptic array and the second synaptic array comprising a plurality of ordered input lines, a plurality of ordered output lines, and a plurality of synapses, wherein: each synapse of the plurality of synapses is operatively coupled to one of the plurality of input lines and one of the plurality of output lines; each synapse of the plurality of synapses comprises a resistive element configured to store a weight; the first synaptic array is configured to receive and store an input array from a previous layer of an artificial neural network during a forward pass operation; the second synaptic array is configured to receive the input array from the previous layer and compute an output during the forward pass operation based on weights of the second synaptic array; the first synaptic array is configured to provide the stored input array to the second synaptic array during a backpropagation operation; and the second synaptic array is configured to receive correction values during the backpropagation operation and update its weights based on the correction values and the stored input array. the forward pass operation is pipelined.

9. The apparatus of claim 8, wherein, the backpropagation operation is pipelined.

10. The apparatus of claim 8, wherein, the forward pass operation and the backpropagation operation are concurrently performed.

11. The apparatus of claim 8, wherein, ​ 12. The apparatus of claim 8, wherein, The first array of synapses is configured to store one input array per column.

13. The apparatus of claim 8, wherein, Each synapse of the plurality of synapses comprises a memory element.

14. The apparatus of claim 8, wherein, Each synapse of the plurality of synapses comprises an NVM or a 3T1C.

15. A method for an artificial neural network, comprising: receiving, during a forward pass operation, an input array from a previous layer through a first array of synapses in a hidden layer; storing, during the forward pass operation, the input array by the first array of synapses; receiving, during the forward pass operation, the input array through a second array of synapses in the hidden layer; computing, during the forward pass operation, an output from the input array based on weights of the second array of synapses by the second array of synapses; providing, during a backpropagation operation, the stored input array from the first array of synapses to the second array of synapses; receiving, during the backpropagation operation, a correction value by the second array of synapses; and updating, based on the correction value and the stored input array, the weights of the second array of synapses. The forward pass operation is pipelined.

16. The method of claim 15, wherein, The backpropagation operation is pipelined.

17. The method of claim 15, wherein, The forward pass operation and the backpropagation operation are performed in parallel.

18. The method of claim 15, wherein, The first array of synapses is configured to store one input array per column.

19. The method of claim 15, wherein, Each synapse of the plurality of synapses comprises a memory element.

20. The method of claim 15, wherein, 21. A computer program comprising program code adapted to perform the method steps of any of claims 15 to 20 when the program is run on a computer. ​

Images