Pipelined Processing of Analog Memory-Based Neural Networks with All-Local Storage

Abstract

This paper implements pipelined processing for analog memory-based neural networks with all local storage. An array of inputs is received by the first synaptic array in the hidden layer from the previous layer during the feed-forward operation. The array of inputs is stored by the first synaptic array during the feed-forward operation. The array of inputs is received by the second synaptic array in the hidden layer during the feed-forward operation. The second synaptic array calculates outputs from the array of inputs based on the weights of the second synaptic array during the feed-forward operation. The stored array of inputs is provided from the first synaptic array to the second synaptic array during the back-propagation operation. Correction values ​​are received by the second synaptic array during the back-propagation operation. The weights of the second synaptic array are updated based on the correction values ​​and the stored array of inputs.

Description

[Technical field]

[0001] TECHNICAL FIELD Embodiments of the present disclosure relate to neural network circuits, and more particularly, to pipelined processing of analog memory-based neural networks with all-local storage. Summary of the Invention

[0002] According to an embodiment of the present disclosure, an artificial neural network is provided. In various embodiments, the artificial neural network comprises a plurality of synaptic arrays. Each of the plurality of synaptic arrays comprises a plurality of ordered input wires, a plurality of ordered output wires, and a plurality of synapses. Each of the synapses is operatively coupled to one of the plurality of input wires and one of the plurality of output wires. Each of the plurality of synapses comprises a resistive element configured to store a weight. The plurality of synaptic arrays are arranged in a plurality of layers, the plurality of layers comprising at least one input layer, one hidden layer, and one output layer. At least one first synaptic array of the at least one hidden layer synaptic arrays is configured to receive and store an array of inputs from a previous layer during a feed-forward operation. At least one second synaptic array of the at least one hidden layer synaptic arrays is configured to receive an array of inputs from a previous layer during a feed-forward operation and calculate an output from the at least one hidden layer based on the weights of the second synaptic array. A first synaptic array of at least one of the synaptic arrays is configured to provide an array of stored inputs to a second synaptic array of at least one of the synaptic arrays during the back-propagation operation. The second synaptic array of at least one of the synaptic arrays is configured to receive a correction value and update weights of the second synaptic array based on the correction value and the array of stored inputs during the back-propagation operation.

[0003] According to an embodiment of the present disclosure, a device is provided that includes a first synaptic array and a second synaptic array. Each of the first synaptic array and the second synaptic array comprises a plurality of ordered input wires, a plurality of ordered output wires, and a plurality of synapses. Each of the plurality of synapses is operably coupled to one of the plurality of input wires and one of the plurality of output wires. Each of the plurality of synapses comprises a resistive element configured to store a weight. The first synaptic array is configured to receive and store an array of inputs from a previous layer of an artificial neural network during a feed-forward operation. The second synaptic array is configured to receive an array of inputs from the previous layer during a feed-forward operation and calculate an output based on the weights of the second synaptic array. The first synaptic array is configured to provide the stored array of inputs to the second synaptic array during a back-propagation operation. The second synaptic array is configured to receive the correction value during the backpropagation operation and to update weights of the second synaptic array based on the correction value and the array of stored inputs.

[0004] According to an embodiment of the present disclosure, a method and computer program product are provided for operating a neural network circuit. An array of inputs is received by a first synaptic array of a hidden layer from a front layer during a feed forward operation. The array of inputs is stored by the first synaptic array during the feed forward operation. The array of inputs is received by a second synaptic array of the hidden layer during the feed forward operation. The second synaptic array calculates an output from the array of inputs based on weights of the second synaptic array during the feed forward operation. The stored array of inputs is provided from the first synaptic array to the second synaptic array during a back propagation operation. A correction value is received by the second synaptic array during the back propagation operation. The weights of the second synaptic array are updated based on the correction value and the stored array of inputs. [Brief description of the drawings]

[0005] [Figure 1] 1 illustrates an exemplary non-volatile memory based crossbar array, or crossbar memory, in accordance with an embodiment of the present disclosure. [Diagram 2] FIG. 1 illustrates an exemplary synapse in a neural network, according to an embodiment of the present disclosure. [Diagram 3] FIG. 2 illustrates an exemplary array of neural cores, according to an embodiment of the present disclosure. [Figure 4] FIG. 1 illustrates an exemplary neural network, according to an embodiment of the present disclosure. [Figure 5A] FIG. 2 illustrates a first step of forward propagation according to an embodiment of the present disclosure. [Figure 5B] FIG. 13 illustrates a second step of forward propagation according to an embodiment of the present disclosure. [Figure 5C] FIG. 13 illustrates a third step of forward propagation according to an embodiment of the present disclosure. [Figure 5D] FIG. 13 illustrates a fourth step of forward propagation according to an embodiment of the present disclosure. [Figure 5E] FIG. 13 illustrates a fifth step of forward propagation according to an embodiment of the present disclosure. [Figure 6A] FIG. 2 illustrates a first step of backpropagation according to an embodiment of the present disclosure. [Figure 6B] FIG. 13 illustrates a second step of backpropagation according to an embodiment of the present disclosure. [Figure 6C] FIG. 13 illustrates a third step of backpropagation according to an embodiment of the present disclosure. [Figure 6D] FIG. 13 illustrates a fourth step of backpropagation according to an embodiment of the present disclosure. [Figure 6E] FIG. 13 illustrates a fifth step of backpropagation according to an embodiment of the present disclosure. [Figure 7A] FIG. 2 illustrates a first step in which both forward and back propagation are performed simultaneously, according to an embodiment of the present disclosure. [Figure 7B] FIG. 13 illustrates a second step in which both forward and back propagation are performed simultaneously, according to an embodiment of the present disclosure. [Figure 7C] FIG. 13 illustrates a third step in which both forward propagation and back propagation are performed simultaneously, according to an embodiment of the present disclosure. [Figure 7D] FIG. 13 illustrates a fourth step in which both forward propagation and back propagation are performed simultaneously, according to an embodiment of the present disclosure. [Figure 7E] FIG. 13 illustrates a fifth step in which both forward propagation and back propagation are performed simultaneously, according to an embodiment of the present disclosure. [Figure 8] FIG. 1 illustrates a method of operating a neural network according to an embodiment of the present disclosure. [Figure 9] FIG. 2 illustrates a computing node according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] An artificial neural network (ANN) is a distributed computing system consisting of several neurons interconnected through connection points called synapses. Each synapse encodes the strength of the connection between the output of one neuron and the input of another neuron. The output of each neuron is determined by the sum of the inputs it receives from the other neurons connected to it. Thus, the output of a given neuron is determined based on the output of its connected neurons from the previous layer and the strength of the connection, determined by the synaptic weights. ANNs are trained to solve a particular problem (e.g., pattern recognition) by adjusting the synaptic weights so that a particular magnitude of input produces a desired output.

[0007] ANNs can be implemented on various types of hardware, including crossbar arrays, also known as crosspoint arrays or crosswire arrays. The basic crossbar array configuration includes a set of conductive row wires and a set of conductive column wires that are formed to cross the set of conductive row wires. The intersections of the two sets of wires are separated by crosspoint devices. The crosspoint devices act as weighted connections between the neurons of the ANN.

[0008] In various embodiments, a non-volatile memory based crossbar array, or crossbar memory, is provided. A plurality of crosspoints are formed by row lines crossing column lines. A resistive memory element, such as a non-volatile memory, is coupled between one of the row lines and one of the column lines in series with a selector at each of the crosspoints. The selector may be a volatile switch or transistor, many types of which are known in the art. It will be appreciated that a variety of resistive memory elements are suitable for use as described herein, including memristors, phase change memories, conductive bridge RAMs, and spin-transfer torque RAMs.

[0009] A fixed number of synapses may be provided on a core, and then multiple cores may be connected to provide a complete neural network. In such an embodiment, interconnectivity between the cores is provided to communicate the output of neurons on one core to another core, for example, via a packet-switched or circuit-switched network. In a packet-switched network, flexible interconnection may be achieved, but at the expense of power and speed due to the need to send, read, and operate on address bits. In a circuit-switched network, no address bits are required, so flexibility and reconfigurability must be achieved by other means.

[0010] In various exemplary networks, multiple cores are arranged in an array on a chip. In such embodiments, the relative positions of the cores may be represented by a direction (north, south, east, west). The data carried by the neural signals may be encoded into the pulse duration carried by each wire using digital voltage levels suitable for buffering or other forms of digital signal restoration.

[0011] One approach to routing is to have an analog-to-digital converter at the input of each core, combined with a digital network-on-chip to quickly route packets to any other core.

[0012] Training a deep neural network (DNN) involves three distinct steps: 1) forward inference of training examples through the network to the output; 2) back-propagating a delta, or correction, based on the difference between the inferred output for that training example and the known ground truth output; and 3) weight update for each weight in the network by combining an initial forward excitation (χ) due to the neuron immediately upstream from the synaptic weight with the back-propagated delta due to the neuron immediately downstream of the synaptic weight.

[0013] This pipelining of the training process is complicated by the fact that these two pieces of data needed to update the weights are generated at widely different times: the input excitation values ​​(χ-vectors) are generated during the forward pass, but the input delta values ​​(delta vectors) are not generated until the entire forward pass is complete and the reverse pass returns to the same neural network layer. For layers that are early in the neural network, this means that the χ-vector data that will be needed later must be stored for a while, and the number of such vectors that must be stored and later retrieved can be very large.

[0014] In particular, to perform a weight update at a layer q, we need the excitation corresponding to the input m (e.g., an image) generated at a time step t. Furthermore, we need the delta for layer q, which is not available until time step t+2l, where l is the number of layers between layer q and the output of the network.

[0015] On the other hand, a forward-only pipelined approach that does not require long-term storage of χ-vectors can efficiently pass these vectors from one array core implementing a neural network layer to the next with highly localized routing, so that all layers can work on the data simultaneously. For example, the array core associated with the Nth DNN layer works on the Nth data example while the array core of the N-1th layer works on the N-1th data example. This approach of multiple chunks of data progressing incrementally through a hardware system is known as pipelining. It is particularly efficient because each component remains busy at all times, even if adjacent components are working on different parts of the same problem or data examples, or on completely different data examples.

[0016] A pipelined training approach is described that digitizes all the chi and delta vectors and stores them elsewhere on the chip. Such an approach requires digitization, long-distance routing of digital data, and large amounts of memory, any of which can become a bottleneck as the number of neural network layers grows.

[0017] Therefore, there is a need for a technique that allows for pipelining of deep neural network training, while providing the same scalability for large networks by eliminating all long-distance data traffic.

[0018] This disclosure provides a five-step sequence with two or more logic array cores assigned to each neural network layer. These array cores can be either unique or identical. One array core is responsible for the highly localized short-term memory of the χ-vectors generated in the forward pass, while the other array core performs forward propagation (generating the next χ-vector), reverse propagation (generating the delta vector), and weight updates in the usual crossbar array or resistive processing unit (RPU) mode.

[0019] In some embodiments, short-term memory may be distributed across multiple array cores and the RPU / crossbar functions may also be distributed across multiple array cores. At the other end of the distribution spectrum, the dual roles of short-term memory and crossbar functions may be implemented in one physical array core or tile.

[0020] 1, an exemplary non-volatile memory based crossbar array, or crossbar memory, is illustrated. A number of crosspoints 101 are formed by row lines 102 crossing column lines 103. At each of the crosspoints 101, a resistive memory element 104, such as a non-volatile memory, is coupled between one of the row lines 102 and one of the column lines 103 in series with a selector 105. The selector can be a volatile switch or transistor, many types of which are known in the art.

[0021] It will be appreciated that a variety of resistive memory elements are suitable for use as described herein, including memristors, phase change memories, conductive bridge RAM, and spin-transfer torque RAM.

[0022] Referring to FIG. 2, an exemplary synapse in a neural network is illustrated. Node 201 has multiple inputs χ 1 χ n is the corresponding weight w ijThe sum of the weights is Σχ i w ij is given to the function f(·) of node 202, and the value

number

[0023] Referring now to FIG. 3, an exemplary array of neural cores is illustrated, according to an embodiment of the present disclosure. Array 300 includes a plurality of cores 301. The cores in array 300 are interconnected by wiring 302, as further described below. In this example, the array is two-dimensional. However, it will be appreciated that the present disclosure may apply to one-dimensional or three-dimensional arrays of cores. Core 301 includes a non-volatile memory array 311 that implements synapses as described above. Core 301 includes a west side and a south side, each of which may function as an input and the other as an output. It will be appreciated that the west / south designation is adopted merely for ease of reference to relative positions and does not limit the direction of input / output.

[0024] In various exemplary embodiments, the west side includes support circuitry 312 dedicated to an entire side of core 301, shared circuitry 313 dedicated to a subset of the rows, and row-wise circuitry 314 dedicated to individual rows. In various embodiments, the south side similarly includes support circuitry 315 dedicated to an entire side of core 301, shared circuitry 316 dedicated to a subset of the columns, and per-column circuitry 317 dedicated to individual columns.

[0025] 4, an exemplary neural network is illustrated. In this example, a number of input nodes 401 are interconnected with a number of hidden nodes 402. The hidden nodes 402 are in turn interconnected with an output node 403. It will be understood that this simple feed-forward network is presented for illustrative purposes only, and the present disclosure is applicable regardless of this particular neural network arrangement.

[0026] 5A-5E, the steps of forward propagation are illustrated according to an embodiment of the present disclosure, each of which illustrates the operation of a pair of arrays during one time slice.

[0027] In the first step shown in Figure 5A, a parallel data vector containing the χ vector for layer q of image m propagates across array cores 501, 502 and arrives at RPU array core 502 responsible for the computation of layer q. The χ vector is stored in the eastern periphery of array core 501 responsible for the storage of layer q. A multiply-and-accumulate operation is performed to set the next χ vector.

[0028] Boxes 503...505 at the western end of each crossbar indicate the peripheral circuitry within the row and the shared peripheral circuitry associated with the row of the crossbar array, which produces the forward excitation, makes analog measurements of the integrated current during reverse propagation, and applies the obtained forward excitation during the weight update stage.

[0029] Similarly, the southernmost boxes 506...508 show peripheral circuitry within the columns and shared peripheral circuitry associated with the columns, which makes analog measurements of integrated currents during forward excitation, produces reverse excitations on the columns, and applies those reverse excitations during the weight update stage.

[0030] Arrow 509 indicates the propagation of the data vector on the parallel routing wires passing through each array, and boxes 510, 511 point to the capacitors that are updated (e.g., charged or discharged) during this first step. Arrow 512 indicates the integration (sum of products) of the currents on the array. During this step, excitations are captured at the east end of the left array core as it passes through it, and these excitations drive the rows of the right array core. This results in the integration of the currents along the columns performing massively parallel sum of products operations. At the end of this step, an integrated charge representing the analog result of these operations is present in the capacitors at the south end of the right array core, as shown in box 511.

[0031] In the second step, shown in Figure 5B, the χ vector data (

number

[0032] Boxes 514, 515 indicate capacitors holding values ​​from the previous time step - in this case the east end of the left array core and the south end of the right array core - and arrows 516 indicate parallel row-wise writes to 3T1C (3 transistors + 1 capacitor) devices, or any other device capable of writing analog states quickly and accurately with very high persistence.

[0033] In the third step shown in Figure 5C, the next χ vector data south of the computational array core is placed on the routing network and sent to the q+1th layer. This process may inherently involve a squashing function operation, or a squashing function may be applied at a point along the routing path before the final destination.

[0034] In the third and fourth steps, shown in Figures 5D-E, nothing needs to be done: these time slices will be used for other training tasks before the next image can be processed.

[0035] This list details the operations on the array cores associated with the qth layer, which means that the q+1th layer performs these exact same operations, but with a phase shift of two steps. This means that the arrow 517 in the third step (corresponding to data leaving layer q) is equivalent to the arrow 509 seen in the first step of the q+1th layer (corresponding to data arriving at layer q+1). Continuing in the same way, the q+2th layer performs these same operations again, but with a phase shift of four steps from the original layer q. In other words, during forward propagation, all the array cores are busy in three out of five phases.

[0036] 6A-6E, the steps of backpropagation are illustrated according to an embodiment of the present disclosure, each of which shows the operation of a pair of arrays during one time slice.

[0037] During the first step shown in Figure 6A, a previously stored copy of the χ vector for image n is retrieved and available at the western periphery of the storage array core of layer q. Note that this was likely stored some time in the past when image n was processed for forward propagation.

[0038] During the second step shown in Figure 6B, the parallel delta vector for layer q of image n propagates through the routing network to the south side of the same RPU array core, where a transpose multiply-accumulate operation (columns driven row-wise integration) is performed, resulting in the accumulation of charge representing the next delta vector on the west capacitors of layer q's computational array core. A copy of the arriving delta vector is stored in the south peripheral circuitry (shown in box 601).

[0039] During the third step shown in Figure 6C, the previously retrieved χ vector is transferred from the storage array cores to the computational array cores, so that it is now available at the western periphery of the computational array cores of layer q.

[0040] During the fourth step shown in Figure 6D, the chi vector information of the western periphery and the delta vector information of the southern periphery are combined to perform a crossbar compatible weight update (RPU array neural network weight update).

[0041] During the fifth step, shown in Figure 6E, all derivative information available at the west periphery is applied to the next delta vector generated in the second step, which then travels through the overhead routing network and passes over the left array core to the previous layer, q-1.

[0042] The phase mismatch between each column of the array core is self-consistent with that observed during the forward propagation step, thus allowing for a fully pipelined training since each layer of the network does useful work during each time step of operation.

[0043] 7A-7E, steps are illustrated in which both forward and back propagation occur simultaneously, according to an embodiment of the present disclosure. As shown in these composite images, the steps given in FIGS. 5A-5E and 6A-6E can be performed simultaneously for five time steps without any self-consistency. This means that all storage is local, and the scheme can be scaled to arbitrarily large neural networks, as long as the routing paths can be performed contention-free. Since one column of intermediate storage is used for each set of five steps during the time period between the first pass of a data example during forward propagation and the final arrival of that data example's delta during reverse propagation, the maximum depth of the network that may be supported is limited by the number of columns available for storing the χ vector. Once the column of delta values ​​has been retrieved and used for the fourth step weight update, the column can be discarded and reused to store forward excitation data for the next input data example. Thus, two pointers - one to the input example m currently being forward-propagated, and one to the input example n currently being reverse-propagated - are maintained and updated at each layer of the network.

[0044] As outlined above, a second RPU array is used for each layer to hold the excitation locally, providing a throughput of one data example every five clock cycles with all layers connected. In this way, throughput is maximized while eliminating long-distance transmission of data. This technique is independent of the number of layers in the network and can be applied to a variety of networks, including long short term memory (LSTM) and convolutional neural networks (CNN) with external weight updates.

[0045] Referring to FIG. 8, a method of operating a neural network according to an embodiment of the present disclosure is shown. At 801, during a feed forward operation, an array of inputs is received by a first synaptic array of a hidden layer from a front layer. At 802, during a feed forward operation, the array of inputs is stored by the first synaptic array. At 803, during a feed forward operation, the array of inputs is received by a second synaptic array of a hidden layer. At 804, during a feed forward operation, the second synaptic array calculates an output from the array of inputs based on the weights of the second synaptic array. At 805, during a back propagation operation, the stored array of inputs is provided from the first synaptic array to a second synaptic array. At 806, during a back propagation operation, a correction value is received by the second synaptic array. At 807, the weights of the second synaptic array are updated based on the correction values ​​and the array of stored inputs.

[0046] Thus, in various embodiments, the training data is processed using a series of tasks that perform forward propagation, back propagation, and weight updates.

[0047] In the first task, a parallel data vector containing the χ vector for layer q of image m propagates across the array cores to arrive at the RPU array core responsible for the computation of layer q, and is also stored at the eastern periphery of the array core responsible for the storage of layer q. A multiply-and-accumulate operation is performed to set the next χ vector.

[0048] In the second task, the chi vector data held in the eastern periphery of the storage array core is written column-wise into the data columns associated with image m. In some embodiments, this will be done using high-persistence NVM or 3T1C synapse circuit elements that exhibit near-infinite persistence and memory lifetimes of a few milliseconds.

[0049] In the third task, the next x vector data south of the computational array core is placed on the routing network and sent to the q+1th layer. This process may include an inherent squashing function operation, or a squashing function may be applied at a point along the routing path before the final destination.

[0050] At the first task of a subsequent iteration of the training data, corresponding to the time when the delta vector of layer q for image m is ready to be transmitted, a copy of the previously stored χ vector for this same image m is retrieved and made available at the western periphery of the storage array core of layer q.

[0051] In the second task of the subsequent iteration, the parallel delta vector of layer q of image m propagates through the routing network to the south side of the same RPU array core, where a transpose multiply-accumulate operation (columns driven row-wise integration) is performed, resulting in the accumulation of charge representing the next delta vector on the west capacitors of layer q's computational array core. A copy of the arriving delta vector is stored in the south peripheral circuitry.

[0052] In the third task of the subsequent iteration, the previously retrieved χ vector is transferred from the storage array core to the computational array core, so that it is now available at the western periphery of the computational array cores of layer q.

[0053] In the fourth task of the subsequent iteration, the chi-vector information from the western periphery and the delta-vector information from the southern periphery are combined to perform a regular crossbar-based weight update typical of neural network weight updates in an RPU array.

[0054] In the fifth task of the subsequent iteration, all derived information available in the western periphery is applied to the next delta vector generated in the second task.

[0055] 9, a schematic diagram of an example computing node is shown. Computing node 10 is merely one example of a suitable computing node and is not intended to suggest any limitation as to the scope of use or functionality of the embodiments described herein. In any case, computing node 10 is capable of implementing and / or performing any of the functions described herein.

[0056] Computing node 10 includes computer system / server 12 that is operable in numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable appliances, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0057] The computer system / server 12 may be described in the general context of computer system executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system / server 12 may be operated in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0058] 9, computer system / server 12 in computing node 10 is shown in the form of a general purpose computing device. Components of computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that couples various system components including system memory 28 to processor 16.

[0059] Bus 18 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures, including, by way of example and not limitation, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, Peripheral Component Interconnect (PCI) bus, Peripheral Component Interconnect Express (PCIe), and Advanced Microcontroller Bus Architecture (AMBA).

[0060] Computer system / server 12 typically includes a variety of computer system readable media. Such media may be any available media that is accessible by computer system / server 12 and includes both volatile and nonvolatile media, removable and non-removable media.

[0061] The system memory 28 may include computer system readable media in the form of volatile memory such as random access memory (RAM) 30 and / or cache memory 32. The computer system / server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may provide for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, typically referred to as a "hard drive"). Also provided, not shown, may be a magnetic disk drive for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading from and writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical medium. In such cases, each may be connected to the bus 18 by one or more data media interfaces. As further shown and described below, memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform functions of embodiments of the present disclosure.

[0062] A program / utility 40 having a set (at least one) of program modules 42 may be stored in memory 28, as well as, by way of example and not limitation, an operating system, one or more application programs, other program modules, and program data. Each or any combination of the operating system, one or more application programs, other program modules, and program data may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0063] The computer system / server 12 may also communicate with one or more external devices 14, such as a keyboard, pointing device, display 24; one or more devices that allow a user to interact with the computer system / server 12; and / or any device (e.g., network card, modem, etc.) that allows the computer system / server 12 to communicate with one or more other computer devices. Such communication may occur through an input / output (I / O) interface 22. Moreover, the computer system / server 12 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet), or combinations thereof, through a network adapter 20. As shown, the network adapter 20 communicates with other components of the computer system / server 12 through a bus 18. It is understood that other hardware and / or software components, not shown, may be used in combination with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processors, external disk drive arrays, RAID (Redundant Arrays of Inexpensive Disk) systems, tape drives, data archive storage systems, etc.

[0064] The present disclosure may be embodied as a system, method, or computer program product, or a combination thereof. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the present disclosure.

[0065] A computer-readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes®, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROMs or flash memories®), static random access memories (SRAMs), portable compact disks read-only memories (CD-ROMs), digital versatile disks (DVDs®), memory sticks®, floppy® disks®, mechanically encoded devices such as punch cards or ridge-in-the-groove structures having instructions recorded thereon, and any suitable combination thereof. As used herein, a computer-readable storage medium should not itself be construed as a transitory signal, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or an electrical signal transmitted over an electrical wire.

[0066] The computer readable program instructions described herein may be downloaded from the computer readable storage medium to the respective computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may comprise copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions to the computer readable storage medium in the respective computing / processing device for storage.

[0067] The computer readable program instructions for carrying out the operations of the present disclosure may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine language instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or traditional procedural programming languages, such as object oriented programming languages ​​such as Smalltalk®, C++, and the “C” programming language, or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions by utilizing state information of the computer readable program instructions to customize the electronic circuitry to perform aspects of the present disclosure.

[0068] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0069] Such computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus for manufacturing a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for performing the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams. Such computer-readable program instructions may also be stored on a computer-readable storage medium, such that the computer-readable storage medium having instructions stored therein includes an article of manufacture including instructions that implement aspects of the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams, and may direct a computer, programmable data processing apparatus, or other device, or combination thereof, to function in a particular manner.

[0070] The computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to generate a computer-implemented process and cause the computer, other programmable apparatus, or other device to perform a series of operational steps, such that the instructions, which execute on the computer, other programmable apparatus, or other device, perform the functions / operations specified in one or more blocks of the flowcharts and / or block diagrams.

[0071] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously or in the reverse order depending on interacting functions. It should be noted that each block in the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, may be implemented by a special purpose hardware-based system that executes a specified function or operation or executes a combination of special purpose hardware and computer instructions.

[0072] The description of various embodiments of the present disclosure has been presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terms used in this specification are selected to best explain the principles of the embodiments, practical applications or technical improvements to the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. An artificial neural network system comprising a plurality of synaptic arrays, comprising: each of the plurality of synapse arrays comprises a plurality of ordered input wires, a plurality of ordered output wires, and a plurality of synapses; each of the plurality of synapses is operatively coupled to one of the plurality of input wires and one of the plurality of output wires; each of the plurality of synapses comprising a resistive element configured to store a weight; the plurality of synapse arrays are arranged into a plurality of layers including at least one input layer, at least one hidden layer, and at least one output layer; a first synaptic array of the plurality of synaptic arrays in the at least one hidden layer configured to receive and store an array of inputs from a previous layer during a feed forward operation; a second synaptic array of the plurality of synaptic arrays in the at least one hidden layer is configured to receive the array of inputs from the previous layer during the feed forward operation and to calculate an output from the at least one hidden layer based on the weights of the second synaptic array; the first synapse array is configured to provide the array of stored inputs to the second synapse array during a back-propagation operation; the second synapse array is configured to receive a correction value during the back-propagation operation and to update weights of the second synapse array based on the correction value and the stored array of inputs. Artificial Neural Network Systems.

2. 2. The artificial neural network system of claim 1, wherein said feed forward operations are pipelined.

3. 2. The artificial neural network system of claim 1, wherein said backpropagation operations are pipelined.

4. 2. The artificial neural network system of claim 1, wherein said feed forward operation and said back propagation operation are performed simultaneously.

5. 2. The artificial neural network system of claim 1, wherein said first one of said synapse arrays is configured to store an array of inputs, one per column.

6. 2. The artificial neural network system of claim 1, wherein each of said plurality of synapses comprises a memory element.

7. 2. The artificial neural network system of claim 1, wherein each of said plurality of synapses comprises an NVM or a 3T1C.

8. 1. A device comprising a first synapse array and a second synapse array, each of the first synapse array and the second synapse array comprises a plurality of ordered input wires, a plurality of ordered output wires, and a plurality of synapses; each of the plurality of synapses is operatively coupled to one of the plurality of input wires and one of the plurality of output wires; each of the plurality of synapses comprising a resistive element configured to store a weight; the first synaptic array is configured to receive and store an array of inputs from a previous layer of an artificial neural network during a feed-forward operation; the second synaptic array is configured to receive the array of inputs from the previous layer during the feed forward operation and to calculate outputs based on the weights of the second synaptic array; the first synaptic array is configured to provide the array of stored inputs to the second synaptic array during a back-propagation operation; the second synapse array is configured to receive a correction value during the back-propagation operation and to update weights of the second synapse array based on the correction value and the array of stored inputs.

9. The device of claim 8 , wherein the feed forward operations are pipelined.

10. The device of claim 8 , wherein the backpropagation operations are pipelined.

11. The device of claim 8 , wherein the feed forward operation and the back propagation operation are performed simultaneously.

12. 9. The device of claim 8, wherein the first synapse array is configured to store an array of inputs, one per column.

13. The device of claim 8 , wherein each of the plurality of synapses comprises a memory element.

14. The device of claim 8 , wherein each of the plurality of synapses comprises an NVM or a 3T1C.

15. receiving an array of inputs by a first synaptic array of the hidden layer from a previous layer during a feed forward operation; storing, by the first synaptic array, the array of inputs during the feed forward operation; receiving the array of inputs by a second synaptic array of the hidden layer during the feed forward operation; computing, during the feed forward operation, by the second synaptic array, outputs from the array of inputs based on weights of the second synaptic array; providing the array of stored inputs from the first synapse array to the second synapse array during a back-propagation operation; receiving a correction value by the second synapse array during the back-propagation operation; updating the weights of the second synapse array based on the correction value and the array of stored inputs; A computer-implemented method comprising:

16. The method of claim 15 , wherein the feed forward operation is pipelined.

17. 16. The method of claim 15, wherein the backpropagation operations are pipelined.

18. The method of claim 15 , wherein the feed forward operation and the back propagation operation are performed simultaneously.

19. 16. The method of claim 15, wherein the first synapse array is configured to store an array of inputs, one per column.

20. A computer program product causing a computer to carry out the steps of the method according to any one of claims 15 to 19.

Images