Efficient tile mapping of row-by-row convolutional neural network maps for simulating AI network inference

Abstract

Implementing a convolutional neural network (CNN) includes configuring a cross-point array to implement a convolution layer in the CNN. The convolution kernels of the layer are stored in the cross-point devices of the array. The calculations of the CNN are performed by iterating a set of operations a predetermined number of times. The operations include sending voltage pulses corresponding to sub-portions of the vector of input data to the cross-point array. The voltage pulses generate a current representing a multiplication operation performed at the cross-point device based on the weight values stored at the cross-point device. A set of integrators accumulates charge based on the output currents from the corresponding cross-point devices. The cross-point array outputs the accumulated charge after iterating a predetermined number of times. The accumulated charge represents the multiplication and addition result of the vector of input data and one or more convolution kernels.

Description

Background Art

[0001] The present invention relates generally to computing technology and, more particularly, to artificial neural networks (ANNs). In particular, embodiments of the present invention relate to mapping convolutional neural networks (CNNs) to cross-point devices in a cross-point array, such as in analog memory-based hardware for providing outputs from a trained CNN during a forward inference phase.

[0002] Technical problems such as computer character and image recognition are known to be well addressed using machine learning techniques. "Machine learning" is a term used broadly to describe the primary function of electronic systems that learn from data. In machine learning and cognitive science, neural networks are a family of statistical learning models inspired by the biological neural networks of animals, particularly the brain. Neural networks can be used to estimate or approximate systems and functions that depend on a large number of inputs and are generally unknown. Neural networks use a class of algorithms based on the concept of interconnected "neurons." In a typical neural network, neurons have a given activation function that operates on the input. By determining appropriate connection weights (a process also known as "training"), neural networks achieve efficient recognition of desired patterns such as images and characters. Typically, these neurons are grouped into "layers" to make the connections between groups more explicit and organize the computational process. Using these appropriate connection weights, other patterns of interest that the network has never seen during training can also be correctly recognized, a process known as "forward reasoning." Summary of the Invention

[0003] According to one or more embodiments of the present invention, a computer-implemented method for implementing a convolutional neural network (CNN) using a cross-point array is described. The method includes configuring a cross-point array that implements a convolution layer in the CNN. The configuration is performed by storing one or more convolution kernels of the convolution layer in one or more cross-point devices of the cross-point array. The method also includes performing calculations of the CNN via the cross-point array by iterating a set of operations a predetermined number of times. The set of operations includes transmitting voltage pulses corresponding to sub-portions of a vector of input data of the convolution layer to the cross-point array. The set of operations also includes outputting a current representing a multiplication operation performed at one or more cross-point devices in the cross-point array, the current being based on weight values stored by the cross-point devices and the voltage pulses from the input data. The set of operations also includes accumulating charge by a set of integrators based on the output currents from the cross-point devices. The method also includes outputting an accumulated charge by the set of integrators after iterating a predetermined number of times, the accumulated charge representing a multiplication-add result of the vector of input data and the one or more convolution kernels.

[0004] In one or more embodiments of the invention, outputting the accumulated charge in the set of integrators includes pooling the accumulated charge.In one or more embodiments of the invention, a sub-portion of each vector of input data is associated with the set of integrators.

[0005] In one or more embodiments of the present invention, the cross-point array includes a plurality of cross-point arrays, wherein a first sub-portion of the vector of input data is sent to a first cross-point array, and a second sub-portion of the vector of input data is sent to a second cross-point array. In one or more embodiments of the present invention, accumulating charge by a group of integrators includes accumulating charge accumulated by a group of integrators of the second cross-point array with a group of integrators of the first cross-point array.

[0006] In one or more embodiments of the present invention, the cross-point device is arranged to implement one or more columns of convolution kernels of a given layer of a CNN, and wherein the vector of input data represents neuron excitations to the given layer of the CNN presented one row at a time from the input data. The charge accumulated by the integrator from a set of integrators represents the output excitation according to the given layer of the CNN, which is converted and transmitted only after all rows of the convolution kernel are integrated.

[0007] In one or more embodiments of the present invention, the cross-point device is arranged to implement one or more rows of convolution kernels of a given layer of a CNN, and wherein the input data represents neuron excitations to the layer of the CNN presented one column at a time. The charge accumulated by the integrators from the integrator group represents the output excitation according to the given layer of the CNN, which is converted and transmitted only after all columns of the convolution kernel are integrated.

[0008] According to one or more embodiments of the present invention, an electronic circuit for performing calculations of a trained convolutional neural network (CNN) is described. The electronic circuit includes a crosspoint array and an output circuit, which in turn includes one or more integrators. The method also includes providing a crosspoint array, and providing an output circuit. The method also includes configuring a crosspoint array corresponding to a convolutional layer in the CNN by storing one or more convolution kernels of the convolutional layer in one or more crosspoint devices of the crosspoint array. The method also includes iterating a set of operations a predetermined number of times. The above method is used to perform training of the CNN.

[0009] According to one or more embodiments of the present invention, an electronic circuit is described that includes an array of resistive memory elements. The array provides a current output vector that is equal to the analog vector matrix product between (i) a voltage input vector for an array encoding an analog input value vector and (ii) an analog resistor weight matrix within the array. The electronic circuit also includes accumulation wires and circuitry that aggregate current from a dedicated subset of the resistive memory elements. The electronic circuit also includes integrating capacitors, each of which is electrically switchable to aggregate current from one of a plurality of accumulation wires during a single integration step. The electronic circuit also includes data output circuitry to allow the integrated charge from the subset of the integrating capacitors, accumulated over a plurality of integration steps, to be appropriately converted and transmitted as an analog duration or as a digital representation using binary digits, wherein the resistive memory elements are arranged to implement a vector of synaptic weight kernels for a given layer of a convolutional neural network.

[0010] According to one or more embodiments of the present invention, a method uses an electronic circuit to perform accumulation over multiple integration steps, which implements a multiplication-accumulation operation across multiple partial vectors of the weight kernel. The accumulation includes performing a calculation by a resistive memory element of a cross-point array by iterating a set of operations a predetermined number of times. The set of operations includes partitioning each vector of analog input values into multiple partial vectors. The set of operations also includes accumulating partial output stimuli corresponding to each partial vector of the multiple partial vectors in an analog memory. The set of operations also includes combining the partial output stimuli by routing the partial output stimuli to an integration capacitor that accumulates integrated charge. In addition, the accumulation also includes transferring the integrated charge on the integration capacitor, which represents the output stimuli.

[0011] In one or more embodiments of the present invention, the integrated charge on the integrating capacitor is locally pooled before being transferred. In one or more embodiments of the present invention, the resistive memory element is a non-volatile memory device. In one or more embodiments of the present invention, the subset of resistive memory elements corresponds to one or more columns of an array. In one or more embodiments of the present invention, the subset of resistive memory elements corresponds to one or more rows of an array.

[0012] In one or more embodiments of the invention, the cross-point device is arranged to implement one or more rows of convolution kernels of a given layer of a convolutional neural network, and wherein the input data represents neuron excitations to the layer of the convolutional neural network presented one column at a time.

[0013] In one or more embodiments of the invention, the cross-point device is arranged to implement one or more columns of convolution kernels of a given layer of a convolutional neural network, and wherein the vector of input data represents neuron excitations to the given layer of the convolutional neural network presented from the input data one row at a time.

[0014] It should be understood that the present technical solution is not limited in its application to the details of the construction and arrangement of the components set forth in the following description or shown in the accompanying drawings. In addition to the described embodiments, the technical solution can be practiced and implemented in various ways. Moreover, it should be understood that the wording and terminology used herein and the abstract are for descriptive purposes and should not be considered as limiting. Therefore, those skilled in the art will understand that the concepts on which this disclosure is based can be easily used as the basis for designing other structures, methods and systems for multiple purposes of performing the technical solutions currently described. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The examples described throughout this document will be better understood with reference to the following drawings and descriptions. The components in the figures are not necessarily to scale. In addition, in the drawings, the same reference numerals represent corresponding parts in different views.

[0016] Figure 1 A simplified diagram depicting the input and output connections of a mathematical neuron;

[0017] Figure 2 Depicts Figure 1 The simplified mathematical model of a neuron is shown in ;

[0018] Figure 3 Depicts the combination Figure 2 The simplified model of ANN of the mathematical neuron model shown;

[0019] Figure 4 A simplified block diagram of a representative CNN interpreting a sample input graph is shown.

[0020] Figure 5 shows an example convolutional layer in a CNN trained using training data including an input map and a convolution kernel;

[0021] Figure 6 Depicted is a system for performing matrix-matrix multiplication using a cross-point array according to one or more embodiments of the present invention;

[0022] Figure 7 Depicted is a two-dimensional (2D) crossbar system that performs forward matrix multiplication, reverse matrix multiplication, and weight update according to the present specification;

[0023] Figure 8 depicts an expanded view of a cross-point array according to one or more embodiments of the present invention;

[0024] Figure 9 Depicts the typical output circuitry in a crossbar system;

[0025] Figure 10 Depicts existing operations for performing forward inference operations using a crosspoint array;

[0026] Figure 11 Depicting performing forward reasoning operations using partial accumulation according to one or more embodiments of the present invention, wherein the partial accumulation is based on time partitioning;

[0027] Figure 12 Depicting performing forward inference operations using partial accumulation across multiple cross-point arrays in accordance with one or more embodiments of the present invention; and

[0028] Figure 13 Performing forward reasoning operations using partial accumulation according to one or more embodiments of the present invention is described, where the partial accumulation is based on spatial partitioning. DETAILED DESCRIPTION

[0029] The technical solutions described herein facilitate the implementation of deep learning techniques that use convolutional neural networks in a more efficient manner than existing techniques. Deep learning techniques are widely used for machine-based pattern recognition problems, such as image and speech recognition. Deep learning inherently exploits the availability of large-scale training datasets (which are enhanced by the use of big data) and computing power (which is expected to grow according to Moore's Law).

[0030] Embodiments of the present invention facilitate efficient workload mapping of convolutional neural networks (CNNs) to analog arrays when implementing analog artificial intelligence systems, such as artificial neural networks (ANNs) using crosspoint arrays. The prior art describes a "row-by-row" mapping of weights for CNN inference workloads, such that activations through each layer of the CNN are efficiently used and streamlined to limit storage requirements. However, there are technical challenges with the prior art because analog array area utilization using such "row-by-row" mapping techniques is low, and affects the scalability of such techniques. For example, mapping a large CNN, such as ResNet-50, may require a large number of analog arrays to implement the CNN, which may make the implementation inefficient, unwieldy, and cross-prohibitive.

[0031] Embodiments of the present invention address this technical challenge during the implementation of ANNs, particularly CNNs, by providing a flexible inter-array routing scheme that facilitates compact mapping of CNN layers for row-by-row mapping techniques. One or more embodiments of the present invention benchmark the number of analog arrays (tiles) required against existing row-by-row mapping techniques using general mapping techniques that do not streamline or reuse activations. Thus, embodiments of the present invention facilitate array utilization that is comparable for a wide range of CNNs while retaining the advantages of streamlining activations for row-by-row mapping.

[0032] It is understood in advance that although one or more embodiments are described in the context of biological neural networks, with particular emphasis on modeling brain structure and function, implementation of the teachings set forth herein is not limited to modeling a particular environment. Rather, embodiments of the present invention are capable of modeling any type of environment, including, for example, weather patterns, arbitrary data collected from the internet, etc., as long as the various inputs to the environment can be converted into vectors.

[0033] ANNs are typically implemented as so-called "neuromorphic" systems of interconnected processor elements that act as simulated "neurons" and exchange "messages" between each other in the form of electronic signals. Similar to the so-called "plasticity" of synaptic neurotransmitter connections that carry messages between biological neurons, the connections in ANNs that carry electronic messages between simulated neurons have numerical weights corresponding to the strength or weakness of a given connection. The weights can be adjusted and tuned based on experience so that the ANN adapts to the input and is able to learn. For example, an ANN for handwriting recognition is defined by a set of input neurons that can be activated by pixels of an input image. After being weighted and transformed by a function determined by the network designer, the activation of these input neurons is then passed to other downstream neurons, which are often called "hidden" neurons. This process is repeated until an output neuron is activated. The activated output neuron determines which character is read.

[0034] A crossbar array, also known as a crosspoint array, crossbar array, or resistive processing unit (RPU) array, is a high-density, low-cost circuit architecture used to form a variety of electronic circuits and devices, including artificial neural network architectures, neuromorphic microchips, and ultra-high-density non-volatile memories. A basic crosspoint array configuration includes a set of conductive row lines and a set of conductive column lines that intersect the row lines. The intersection between the two sets of wires is separated by a so-called crosspoint device, which can be formed from a thin-film material.

[0035] Cross-point devices are actually used as ANN weighted connections between neurons. Nanoscale two-terminal devices, such as memristors with "ideal" conductive state switching characteristics, are often used as cross-point devices to mimic synaptic plasticity with high energy efficiency. The conductive state (e.g., resistance) of the ideal memristor material can be changed by controlling the voltage applied between the individual lines of the row and column lines. Digital data can be stored by changing the conductive state of the memristor material at the intersection to achieve a high conductive state or a low conductive state. Memristor materials can also be programmed to maintain two or more different conductive states by selectively setting the conductive state of the material. The conductive state of the memristor material can be read by applying a voltage across the material and measuring the current through the target cross-point device.

[0036] To limit power consumption, crosspoint devices of ANN chip architectures are typically designed to utilize offline learning techniques, where the approximation of the objective function does not change once the initial training phase has been resolved. Offline learning allows the crosspoint devices of crossbar-type ANN architectures to be simplified, making them consume very little power.

[0037] Providing a simple crosspoint device that can implement cashbox inference of previously trained ANN networks with low power consumption, high computational throughput, and low latency would improve overall ANN performance and enable a wider range of ANN applications.

[0038] Although the present invention is directed to electronic systems, for ease of reference and explanation, neural terminology such as neurons, plasticity, and synapses are used to describe various aspects of the described electronic systems. It should be understood that for any discussion or description of electronic systems herein, the use of neural terminology or neural abbreviations is for ease of reference and is intended to encompass the neuromorphic, ANN equivalents of the described neural functions or neural components.

[0039] ANNs, also known as neuromorphic systems or synaptic systems, are computing systems that can estimate or approximate other functions or systems, including, for example, biological neural systems, the human brain, and brain-like functions such as image recognition, speech recognition, etc. ANNs combine knowledge from various disciplines, including neurophysiology, cognitive science / psychology, physics (statistical mechanics), control theory, computer science, artificial intelligence, statistics / mathematics, pattern recognition, computer vision, parallel processing, and hardware (e.g., digital / analog / VLSI / optical).

[0040] Instead of utilizing traditional digital models that manipulate zeros and ones, ANNs create connections between processing elements that are essentially functional equivalents of the core system functionality being estimated or approximated. For example, computer chips, which are central components of electronic neuromorphic machines, attempt to provide a form, function, and architecture similar to that of a mammalian brain. Although computer chips use the same basic transistor components as conventional computer chips, their transistors are configured to mimic the behavior of neurons and their synaptic connections. Computer chips process information using a network of just over a million simulated "neurons" that communicate with each other using electrical spikes similar to the synaptic communication between biological neurons. The architecture of such computer chips includes a configuration of processors (i.e., simulated "neurons") that read memory (i.e., simulated "synapses") and perform simple operations. The communication (pathways) between these processors are typically located in different cores and are performed by on-chip network routers.

[0041] As background, we will now refer to Figure 1 、 Figure 2 and Figure 3 A general description of how a typical ANN operates is provided.As described earlier in this article, a typical ANN is a mathematical model inspired by the human brain, which consists of about a billion interconnected cells called neurons. Figure 1 A simplified diagram of a mathematical neuron 102 is depicted having pathways 104, 106, 108, 110 connecting it to upstream inputs 112, 114, downstream outputs 116, and downstream "other" neurons 118, configured and arranged as shown. Each mathematical neuron 102 sends and receives electrical pulses through pathways 104, 106, 108, 110. The properties of these electrical pulses and how they are processed in biological neurons (not shown) are primarily responsible for overall brain function. Mimicking this function is the intent of mathematical ANNs constructed from mathematical neurons 102 organized in a network. Just as pathway connections between biological neurons can be strong or weak, pathways between mathematical neurons can also be strong or weak. When a given neuron receives an input pulse, the neuron processes the input according to the neuron's function and sends the result of the function to the downstream output and / or downstream "other" neurons.

[0042] Figure 2 In the example, the mathematical neuron 102 is modeled as a node 202 with a mathematical function f(x) given by Figure 2The equation shown describes that node 202 receives electrical signals from inputs 212, 214, multiplies each input 212, 214 by the strength of its respective connection path 204, 206, obtains the sum of the inputs, passes the sum through a function f(x), and generates a result 216, which can be the final output or an input to another node, or both. In this specification, an asterisk (*) is used to represent a multiplication, which can be a matrix multiplication. For example, a matrix multiplication can be used to perform a convolution operation between input data and one or more convolution kernels to generate an output graph. A weak input signal is multiplied by a very small connection strength number, so the effect of the weak input signal on the function is very low. Similarly, a strong input signal is multiplied by a higher connection strength number, so the effect of the strong input signal on the function is greater. The function f(x) is a design choice, and various functions can be used. A typical design choice for f(x) is the hyperbolic tangent function, which takes a function of the previous sum and outputs a number between negative one and positive one. An alternative design choice for f(x) is the rectified linear unit (ReLU), a function whose output matches the input for positive inputs and is zero otherwise.

[0043] Figure 3 A simplified ANN model 300 is depicted, organized as a weighted directed graph, where artificial neurons are nodes (e.g., 302, 308, 316), and where weighted directed edges (e.g., m1 to m20) connect the nodes. The ANN model 300 is organized such that nodes 302, 304, 306 are input layer nodes, nodes 308, 310, 312, 314 are hidden layer nodes, and nodes 316, 318 are output layer nodes. Each node is connected to each node in the adjacent layer through a connection path, which is in the Figure 3 , which are depicted as directed arrows with connection strengths m1 to m20. Although only one input layer, one hidden layer, and one output layer are shown, in practice, multiple input layers, hidden layers, and output layers may be provided.

[0044] In this attempt to mimic the function of the human brain, each input layer node 302, 304, 306 of the ANN 300 receives input x1, x2, x3 directly from a source (not shown) without connection strength adjustment and without node summation. Therefore, y1 = f(x1), y2 = f(x2), and y3 = f(x3), as shown in Figure 3As shown in the equations listed at the bottom. Each hidden layer node 308, 310, 312, 314 receives its input from all input layer nodes 302, 304, 306 according to the connection strength associated with the relevant connection path. Therefore, in the hidden layer node 308, y4 = f(m1*y1+m5*y2+m9*y3), where * represents multiplication. In one or more instances, the multiplication can be a matrix multiplication for performing a convolution operation. Similar connection strength multiplication and node summation are performed for the hidden layer nodes 310, 312, 314 and the output layer nodes 316, 318, as defined by Figure 3 The equations defining functions y5 to y9 are depicted at the bottom of .

[0045] The ANN model 300 processes one data record at a time, and it "learns" by comparing the initial arbitrary classification of the record with the known actual classification of the record. Using a training method known as "back propagation" (that is, "back propagation of error"), the error from the initial classification of the first record is fed back into the network and is used to modify the weighted connection of the network for the second time, and this feedback process continues for several iterations. In the training phase of the ANN, the correct classification of each record is known, and therefore the output node can be assigned a "correct" value, for example, a node value "1" (or 0.9) for the node corresponding to the correct classification, and a node value "0" (or 0.1) for other nodes. Therefore, the calculated value of the output node of the network can be compared with these "correct" values, and the error term (that is, "difference" rule) of each node is calculated. Then, these error terms are used to adjust the weights in the hidden layer so that in the next iteration, the output value will be closer to the "correct" value.

[0046] There are many types of neural networks, but the two most common categories are feedforward and feedback / recursive networks. The ANN model 300 is a non-recursive feedforward network with input, output, and hidden layers, where signals can only travel in one direction. Input data is passed to a layer of processing elements that perform calculations. Each processing element performs its calculation based on a weighted sum of its inputs. The new calculated value then becomes the new input value fed to the next layer. This process continues until it has passed through all layers and the output is determined. A threshold transfer function is sometimes used to quantize the output of neurons in the output layer.

[0047] Feedback / recursive networks include feedback paths, meaning that signals can propagate in both directions using loops. All possible connections between nodes are allowed. Because loops exist in this type of network, under certain conditions, it can become a nonlinear dynamic system that continuously changes until it reaches an equilibrium state. Feedback networks are often used in associative memory and optimization problems, where the network seeks the optimal arrangement of interconnected factors.

[0048] The speed and efficiency of machine learning in feedforward and recursive ANN architectures depends on how effectively the crosspoint devices of the ANN's crosspoint array perform the core operations of a typical machine learning algorithm. Although a precise definition of machine learning is difficult to formulate, the learning process in the context of an ANN can be viewed as a problem of updating the crosspoint device connection weights so that the network can effectively perform a specific task. The crosspoint devices typically learn the necessary connection weights from available training patterns. By iteratively updating the weights in the network, performance improves over time. Instead of following a set of rules specified by a human expert, an ANN "learns" the underlying rules (such as input-output relationships) from a given set of representative examples. Therefore, a learning algorithm can generally be defined as a process by which learning rules are used to update and / or adjust the relevant weights.

[0049] These three main learning algorithm paradigms are supervised, unsupervised and hybrid. In supervised learning or learning with a "teacher", the correct answer (output) for each input pattern is provided to the network. Weights are determined to allow the network to produce answers as close as possible to the known correct answer. Reinforcement learning is a variation of supervised learning, in which the network only provides judgments about the correctness of the network output, rather than the correct answer itself. On the contrary, unsupervised learning or learning without a teacher does not require the correct answer associated with each input pattern in the training data set. It explores the correlation between the underlying structure in the data or the patterns in the data, and organizes the patterns into categories based on these correlations. Hybrid learning combines supervised learning and unsupervised learning. Partial weights are usually determined by supervised learning, while other partial weights are obtained by unsupervised learning. Additional details of ANN and learning rules are described in Artificial Neural Networks: A Tutorial by Anil K. Jain, Jianchang Mao and KM Mohiuddin, IEEE, March 1996, which is incorporated herein by reference in its entirety.

[0050] In addition to applications for training ANNs, forward inference of trained networks includes applications ranging from the implementation of cloud-based services built on ANNs to smartphones, the Internet of Things (IoT), and other battery-constrained applications that require extremely low-power operation. In general, while training is an application that requires high throughput (in order to learn from many training examples), forward inference is an application that requires fast latency (so that any given new test example can be classified, recognized, or otherwise processed as quickly as possible).

[0051] In a CNN, kernels perform convolutions on overlapping regions, such as those in the field of view, and thus emphasize the importance of spatial location in feature detection. Computing the convolutional layers of a CNN typically accounts for over 90% of the computation time in neural network training and inference. Mapping a CNN into an analog array and ensuring efficient use of the power used in performing the mathematical operations of the convolutional layers, with minimal extraneous data movement or computation, is a technical challenge. The technical challenges include mapping the CNN for inference and maintaining the scalability of such a mapping so that even large CNNs such as ResNet-50 can be implemented. While existing solutions using row-by-row mapping assume certain limitations on the inter-array routing circuitry, one or more embodiments of the present invention facilitate flexible inter-array routing of data that enables compact mapping of CNN layers to a cross-point array for a row-by-row technique.

[0052] The technical solution achieved by embodiments of the present invention addresses such technical problems by providing very comparable array utilization for a wide range of CNN networks while retaining the advantages of streamlined activation for row-by-row mapping.

[0053] Figure 4 A simplified block diagram of a CNN is shown. In the depicted example, a CNN is used to interpret a sample input map 400, and in this particular example a handwritten letter "w" is used as the input map. However, it should be understood that other types of input maps are possible, and the technical solutions described herein are also applicable to CNNs that perform other operations, such as other types of feature detection. In the example shown, the input map 100 is used to create a set of values for an input layer 410 or "layer 1", for example, layer 1 can be generated by mapping pixels of the sample input map 400 directly to specific neurons in layer 1, such that the neuron shows a 1 or 0 depending on whether the pixel exhibits a specific property. Another example method of assigning values to neurons is discussed below with reference to convolutional neural networks. Depending on the variation of the neural network and the problem it is trying to solve, each layer of the network can have a different number of neurons, and these neurons can be related or unrelated to specific qualities of the input data.

[0054] refer to Figure 4 , the neurons in layer 1 410 are connected to the neurons in the next layer, layer 2 420, as previously described (see Figure 3 ). Figure 4 The neurons in the Figure 1As described. Thus, the neurons in layer 2 420 receive input values from each of the neurons in layer 1 410. The input values are then summed and the sum is compared to the bias value. If the value exceeds the bias for a particular neuron, then that neuron holds a value that can be used as input to a neuron in the next layer of neurons. This calculation continues through the various layers 430-450 of the CNN, including at least one FC layer 450, until it reaches the final layer 460, at which Figure 4 is called the "output". In some CNN networks, the "residuals" from earlier layers can be combined with the results of later layers, skipping the layers in between. In the example of a CNN for character recognition, each value in a layer is assigned to a specific character. When designed for a classification task, the network is configured to end with an output layer that has only one large positive value in one neuron, which then indicates which character the network has calculated is the most likely character of the handwritten input. In other scenarios, the network may have been designed so that the output neuron values can be used to estimate probability (likelihood), confidence, or other metrics of interest.

[0055] The data values at each layer in a CNN are typically represented using matrices (or tensors in some examples), and computations are performed as matrix computations. The indices (and / or sizes) of the matrices vary from layer to layer and from network to network, such as Figure 4 As shown, different implementations orient the matrix differently or map the matrix to computer memory differently. Figure 4 , in the example CNN shown, each level is a tensor of neuron values, as shown by the matrix dimensions of each layer of the neural network. At the input of the CNN, an example might be multiple input "planes", each of which is a two-dimensional image. For example, there might be a red plane, a green plane, and a blue plane derived from a full-color image. Going deeper into the CNN, layers can take in intermediate data in the form of many "planes" and produce a large number of output planes for the next layer. The values in the input tensor at one layer are multiplied by the connection strengths, which are in a transformation tensor called a filter. This matrix multiplication scales each value in the previous layer by the connection strengths and then sums the aggregate sum of these contributions. This basic operation is called a multiply-add operation. A bias matrix can then be added to the resulting product matrix to take into account the thresholds of each neuron in the next level. Additionally, an activation function is applied to each resulting value and the resulting value is placed in the output tensor to be applied to the next layer. In one example, the activation function can be a linear rectifier function, sigmoid or tanh(). Thus, as Figure 4 As shown, the connections between each layer, and therefore the connections between the entire network, can be represented as a series of matrices. Training a CNN consists of finding appropriate values for these matrices.

[0056] While fully connected neural networks are capable of recognizing input patterns such as notes or photos of family pets when properly trained, they do not exhibit shift invariance. In order for the network to recognize a cat's whiskers, it must be presented with cat images where the whiskers are located at multiple different 2-D locations within the image. Each different image location will result in a neuron value interacting with a different weight in such a fully connected network. In contrast, in a CNN, the connection strength is the convolution kernel. The convolution operation introduces shift invariance. Therefore, since multiple images of a cat with whiskers are presented, the 2-D location within the image is unimportant as long as the scale, color, and rotation of the whiskers remain constant from one image to the next. Therefore, during training, all examples of similar features work together to help learn that feature, regardless of the feature's location within the 2-D image. After training, a single or much smaller filter bank is sufficient to recognize such image features, allowing many filter banks (i.e., CNN layers) to then recognize many different features useful for distinguishing images (dog vs. cat, or even subtle differences representing different cat breeds).

[0057] Figure 5 An example convolutional layer 500 in a CNN trained using training data including an input map 510 and a convolution kernel 520 is shown. For simplicity, Figure 5 The bias matrix 525 is not shown. The input map 510 (also called the input plane) can include multiple input patterns, for example, D input maps. Each input map is a matrix of size N×M. Therefore, in this case, the total number of input neurons is N×M×D. As shown in the figure, the input map is convolved with F convolution kernels 520 of size k×k to produce a corresponding output map 530. Each output map can have dimensions N'×M'. In the case where the input map is a square matrix of size n, the size of the output map is n-k+1×n-k+1. Each convolution is a 3D convolution involving D input maps. A CNN can include multiple such layers, in which the output map 530 from the previous layer is used as the input map 510 for the subsequent layer. A backpropagation algorithm can be used to learn the k×k×D×F weight values of the filter.

[0058] For example, the input graph 510 is convolved with each filter group to generate a corresponding output graph. For example, when a CNN is trained to identify handwriting, the input graph 510 is combined with a filter group including a convolution kernel representing a vertical line. The resulting output graph identifies the vertical value lines present in the input graph 510. In addition, another filter group can include a convolution kernel representing a diagonal line, such as upward and rightward. The output graph generated by the convolution of the input graph 510 with the second filter group identifies a sample of the training data containing a diagonal line. The two output graphs illustrate different information of the characters while maintaining pixel adjacency. This can lead to more efficient character recognition.

[0059] Figure 6A system 600 is described in which a controller 610 is used to control a cross-point array 700 to perform matrix-matrix multiplication(s) and other operations, according to one or more embodiments of the present invention. For example, the controller 610 sends input data 510 to be multiplied by the cross-point array 700. In one or more examples, the controller 610 stores weight values, such as from a convolution kernel 520 in the cross-point array 700, and sends the input vector. In one or more examples, the controller 610 and the cross-point array 700 are coupled in a wired or wireless manner, or a combination thereof. The controller 610 also sends instructions / commands to the cross-point array 700 to initiate operation of one or more layers in a CNN. The controller 610 can also read output data 530 from the cross-point array 700 after receiving notification that a computation has been performed. The controller 610 can be a processing unit or computing system, such as a server, desktop computer, tablet computer, phone, etc. The controller 610 can include a memory device having computer-executable instructions stored therein that, when executed by the controller, cause a matrix-matrix computation.

[0060] Turning now to the overview of this specification, one or more embodiments are directed to a crosspoint array having a crosspoint device at each intersection of a crossbar switch conductor, the crosspoint array being used to implement a CNN. An example of a crosspoint device is a two-terminal programmable resistive crosspoint component, referred to herein as a resistive processing unit (RPU), which provides local data storage functionality and local data processing functionality. When performing data processing, the weighted contribution represented by each crosspoint device is contributed to a massively parallel multiplication-accumulation operation performed at the storage location of the data. This eliminates the need to move relevant data into and out of the processor and separate storage elements. Thus, implementing a machine learning CNN architecture with the described crosspoint device enables online machine learning capabilities that facilitate training CNNs, as well as subsequent use of the trained CNN model to perform reasoning. The described crosspoint device and resulting CNN architecture improve overall CNN performance and enable a wider range of practical CNN applications.

[0061] The described cross-point device can be implemented as a two-terminal resistive cross-point device. For example, the described cross-point device can be implemented using resistive random access memory (RRAM), phase change memory (PCM), programmable metallization cell (PMC) memory, nonlinear memristor systems, or any other device that provides a wide range of analog tunable non-volatile resistive memory states that are sufficiently stable over time.

[0062] Figure 7A two-dimensional (2D) crossbar system 700 for performing forward reasoning according to the present specification is described. Crossbar system 700 can be used to implement simple matrix multiplication, inverse matrix multiplication, and even in-situ weight updates according to the backpropagation algorithm. Crossbar system 700 includes a crosspoint array 705, input circuitry 710, and output circuitry 720, among other components. Input circuitry 710 and output circuitry 720 may be collectively referred to as peripheral circuitry, and crossbar system 700 may be a computer chip in one or more examples.

[0063] Figure 8 FIG2 shows an expanded view of a crosspoint array 705 according to one or more embodiments. The crosspoint array 705 is formed by a set of conductive row lines 802, 804, 806 and a set of conductive column lines 808, 810, 812, 814 that cross the set of conductive row lines 802, 804, 806. The intersections between the row lines and the column lines are separated by crosspoint devices. Figure 8 are shown as resistive elements, each with its own adjustable / updatable resistance weight, denoted as σ 11 , σ 21 , σ 31 , σ 41 , σ 12 , σ 22 , σ 32 , σ 42 , σ 13 , σ 23 , σ 33 and σ 43 For the sake of explanation, Figure 8 Only one cross-point device 820 is labeled with a reference numeral, and in the forward matrix multiplication, the conductive state of the cross-point device (ie, the stored weight) can be read by applying a voltage across the cross-point device and measuring the current through the cross-point device.

[0064] Input voltages V1, V2, V3 are applied to row lines 802, 804, 806, respectively. Each column line 808, 810, 812, 814 uses an integrator, such as a capacitor, to sum the currents I1, I2, I3, I4 generated by each cross-point device along that particular column line. For example, Figure 8 As shown, the current I4 generated by the column line 814 is given by the equation I4 = V1σ 41 +V2σ 42 +V3σ 43 Thus, the array 705 computes the forward matrix multiplication by multiplying the values stored in the cross-point devices by the row line inputs, which are defined by the voltages V1, V2, V3.

[0065] refer to Figure 7 In one or more examples, input circuitry 710 includes at least support circuitry 712, shared circuitry 714, and row circuitry 716. Row circuitry includes hardware components associated with each row line 802, 804, and 806. Input circuitry 710 supports providing input voltages to cross-point array 705.

[0066] Figure 9 A typical output circuit system 720 is depicted. The output circuit system includes integrators 908, 910, 912, and 914 corresponding to column lines 808, 8120, 812, and 814. In one or more examples, integrators 908, 910, 912, and 914 are capacitors. The output current along each column line is accumulated in the integrator and passed to the next layer of the CNN. As previously described, this arrangement of integrators makes the calculations of the FC layer very efficient. However, for convolution operations, using this arrangement of integrators results in significant additional overhead in data transmission, storage, organization, and subsequent data transmission. This operation requires additional resources, such as time, power, and additional circuit area, making the entire system inefficient.

[0067] Figure 10 Describes existing operations for performing forward reasoning operations using a crosspoint array. Figure 10 As shown, one image row (512, 514, and 516) of all input planes 510 is simultaneously presented as a column input to the array rows (802, 804, and 806) of the crosspoint array 705 of the crossbar switch system 700. The crosspoint device 820 at each crosspoint contains weight elements from the filter 525, each weight element causing the array row excitation X i and the stored weight W ij The voltages I1 and I2 are multiplied by Ohm's law (voltage times conductance equals current). The integrals of all these read current contributions are calculated along each array column and stored in the corresponding integrators (908, 910, 912, and 914) of the array columns (808, 810, 812, and 814). The calculation can be expressed as follows: the current I1 on column #1 (808) is stored on capacitor C1, I2 is stored on capacitor C2, I3 is stored on capacitor C2, and so on. In prior art solutions using such a cross-point array 705, the integrated charge on the capacitors (908, 910, 912, and 914) is treated as the output of a multiply-add and is converted into a digital number or pulse duration for transport to the next array 705.

[0068] In this way, at each time step (ie, each calculation performed by array 705 ), the values across all input planes 510 are integrated, resulting in an output for all output planes 530 .

[0069] Furthermore, each output from convolutional layer i must be combined with the outputs from other convolutional layers as part of pooling. The other convolutional layers from which the outputs will be pooled depend on the number of elements in the filter kernel 520. Additionally or alternatively, each output from layer i must be positioned at a different point in the input plane 510 for convolutional layer i+1. This organization of output values for the purpose of pooling may also require additional computational resources, such as read and write access, power, etc.

[0070] Thus, in the prior art system, at time step 1, system 700 integrates the results into capacitors 908, 910, 912, and 914, but does not immediately send the results to the next layer. This is because system 700 must direct read currents from several different columns to integrating capacitors 908, 910, 912, and 914. System 700 performs this manipulation of the results from other columns at subsequent time steps. In the same manner, system 700 takes k time steps to calculate every k-th output row. Therefore, the prior art technique using row-by-row mapping results in each output row taking k time steps to generate.

[0071] Figure 10 The operations performed by array 705 during forward reasoning according to the prior art are described. Figure 10 , time steps 1, 2, and 3 are described. At each time step, the input is mapped to a row in the cross-point array 705. At each time step, each of the integrators (908, 910, 912, and 914) receives contributions from k*p multiply-add terms, where P is the number of input planes 510. After k such time steps, the total charge of the integrator contains all k*k*p terms and is ready to be output to the next convolutional layer. Except during the first k or last k time steps, after each integration step, every k-th integrator from the output circuit 720 reaches this state and is therefore ready to generate all output pixels of one image row (512-A, 514-A, and 516-A) of the convolutional layer output. All other j-th integrators have different phases in their respective integration phases, depending on the value of j.

[0072] For example, Figure 10As shown, at time step 1 of the forward propagation, the first row of each input plane 512-A, 514-A, 516-A is input to the convolution layer. As shown, the crosspoint device 820 of the crosspoint array 705 is loaded with filters 520. Specifically, filter kernels 522-A and 522-B are loaded into the crosspoint device 820 to perform convolution with the first row of the first input plane 516-A. Similarly, filter kernels 524-A and 524-B from the second set of filter kernels 520 are convolved with the first row of the second input plane 514-A, and so on. The output controller 1110 forwards the results of the corresponding convolutions from the output circuit system 720 to one or more integrators (908, 910, 912, 914).

[0073] Output controller 1110 can be part of output circuitry 720 or an external controller coupled to output circuitry 720. Output controller 1110 directs the output of the multiply-add operation from each column in array 705 to a specific integrator in output circuitry 720. In one or more examples, output controller 1110 receives a mode signal that provides a selection of an integrator for each column at each time step. Alternatively, a mode signal is provided to output controller 1110 that indicates the selection of an integrator for each column until all convolutional layers are executed. In one or more examples, the mode signal can be a bit pattern that indicates the selected integrator for each column.

[0074] exist Figure 10 In the example of FIG, at time step #1, the outputs from column 808 and column 814 are stored in integrators 908 and 912, respectively. At time step #2, the second row 512-B, 514-B, and 516-B from input plane 510 are used as input to cross-point array 705. Cross-point device 820 remains as in time step #1 ( Figure 10 ) is loaded with kernel filter 520 as in time step 2. In time step 2, output controller 1110 selects the same integrators 908 and 912 for the outputs of columns 810 and 816 (different columns from time step 1). Therefore, in this case, integrators 908 and 912 (as well as other integrators) receive outputs from different columns at different time steps.

[0075] In time step 3, in a manner similar to the first two time steps, the third row 512-C, 514-C, and 516-C from input plane 510 are used as input to cross-point array 705. In time step 3, output controller 1110 selects the same integrators 908 and 912 for the outputs of columns 812 and 818 (different columns from time steps 1 and 2). Therefore, in this case, integrators 908 and 912 (as well as other integrators) receive outputs from different columns in different time steps. In this way, typically, after k time steps, the entire row in output plane 530 is calculated.

[0076] It should be noted that although only the calculation of the first two entries (A and B) from the first output row in output plane 530 is described above, in a similar manner, the other portions of output plane 530 are calculated in parallel by the other portions of cross-point array 705. In addition, as Figure 10 As shown, cross-point array 705 may use other integrators (910, 914, 916, and 918) to accumulate other output rows (C and D) at each time step.

[0077] Thus, as a result of the output controller 1110 directing the output of the crosspoint array 705, all inputs are in the form of complete and continuous image rows on all input planes. Furthermore, after the first k time steps for which any output is available, i.e., starting from the k+1 time step, a complete and continuous image row on all output planes is produced at each time step. Thus, the output map 530 produced by such an operation can be pipelined to subsequent convolutional layers without any intermediate storage of neuron excitations. Because pooling operations such as summation, averaging, and maximum can be performed incrementally on the data as it arrives, any pooling operation requires only temporary storage sufficient for the output image rows. These intermediate results are stored and updated as each set of neuron excitations arrives until the R times R pooling operation is complete, at which point the buffer of intermediate results is effectively the output of the pooling layer.

[0078] As previously discussed, a technical challenge of the prior art is that the number of crosspoint arrays required to implement a CNN may increase with the number of input data sets such as images, or with the type of network being implemented. Embodiments of the present invention facilitate row-by-row mapping by reducing the number of weight copies throughout the depth of the CNN. Thus, embodiments of the present invention facilitate load balancing to address changes in the CNN weight reuse factor. Furthermore, embodiments of the present invention facilitate row-by-row mapping with a more compact weight mapping using flexible data routing from the input circuit 710 to and through the crosspoint array 705 and to the output circuit 720.

[0079] In one or more embodiments of the present invention, the technical solutions described herein address such technical challenges of prior art solutions by facilitating row-by-row convolution with partial row inputs, where the input data is partitioned in time. In some other embodiments of the present invention, row-by-row convolution is facilitated with partial row inputs, where the input data is partitioned in space (cross-point array).

[0080] Figure 11 A row-by-row convolution mapping with partial row inputs according to one or more embodiments of the present invention is shown, where the input data is partitioned in time. Here, partial sums from different input row segments are stored on separate capacitor banks. In the depicted example, a first subset 1210 of input data from the first row is mapped to a first set of capacitors (or integrators) 1230; and a second subset 1220 of input data from the first row is mapped to a second set of capacitors 1240. For this mapping, the partitioning is determined using the following formula: L = D * (input image width / N + k-stride). Here, N is the number of copies of the capacitors used to help reuse the crosspoint array 705 for forward reasoning calculations. N can be predetermined based on the image width. For example, N can increase as the image size decreases to reduce the number of weight copies, thereby keeping the reuse factor the same. In a convolutional neural network (CNN), a weight kernel is convolved on the input image, that is, the same weights are reused multiple times with different parts of the input image to generate the output. The number of times the weights are reused is called the reuse factor.

[0081] In addition, the stride is a predetermined parameter that defines how much overlap there is within the subset of the first row. Overlap = (k-stride), where k is the kernel dimension. In the example shown, the two sets of capacitors 1230 and 1240 can reuse the weights stored in the cross-point array 705. In order to reuse the work, the input data is mapped to have row-by-row convolutions to be calculated by the cross-point device 820. For this mapping, the calculated L is the number of input data elements input to the cross-point array 705, where each sequential data element comes from a sequential input plane. For example, in the depicted example scenario with D = 3 input planes and L = 15 input planes, k = 3, stride = 1 and N = 2, L1 = D1 (1, 1), L2 = D2 (1, 1), L3 = D3 (1, 1). Here, the symbol D1 (1, 1) refers to the element from the first row and first column in D1. In the same manner, L4 = D1(1,2), L5 = D2(1,2), and L6 = D3(1,2). Cross-point array 705 is configured with N = 2 copies of the weights from kernel 520, shifted from each other by D*stride rows (or columns).

[0082] Once the data elements are input as described herein, the cross-point device 820 performs an in-memory calculation to determine the partial sums of the stored weights and the products of the data elements. The calculations are performed in an analog manner in memory. The resulting partial sums are stored in capacitors in groups 1230, 1240.

[0083] If the number of copies of the weights is reduced, the number of capacitors in each of capacitor banks 2130 and 2140 increases. In one or more embodiments of the present invention, in order to improve the efficiency of the capacitor area in output circuit 720, a partial sum is sent to the destination capacitor (the input side of the next layer cross-point array 705). Even if overlap results in redundant calculations, the improvement in the efficiency of cross-point array 705 achieved by reuse also leads to a reduction in the number of cross-point arrays 705 required to implement a CNN.

[0084] Figure 12 Depicted is a row-by-row convolutional mapping with full or partial row input in accordance with one or more embodiments of the present invention, where the input data is partitioned in time. The described mapping scheme facilitates an even more compact mapping by using multiple reused copies of weights across multiple cross point arrays 705. In one or more embodiments of the present invention, the number of output image channels is F, and the number of input image channels is D. Here, each weight group has a stride with a shift amount of (D*stride) across the (input image width*D) dimensions. After each such set of (output image width) weight copies, the next set is configured without any shift (D*stride). This shift is used within each group to separate the copies of the weights in that group. A set of such weight copies can span separate cross point arrays 705A and 705B. For example, in Figure 12 In the example shown, group 1280 has two weight copies 1282 and 1284 stored in the first cross-point array 705A, and a third weight copy 1286 stored in the second cross-point array 705B.

[0085] It should be understood that the dimensions in the examples shown in the figures and described herein may vary in one or more embodiments of the invention. In addition, the number of cross-point arrays 705 may also vary in one or more embodiments of the invention from the examples described herein.

[0086] Figure 13Another row-by-row convolution mapping with partial row input according to one or more embodiments of the present invention is shown, where the input data is spatially partitioned. Here, a single capacitor bank 1320 is used to calculate the resulting partial sum based on the kernel weights stored in the cross-point array 705. The input data elements are divided so that a subset of a single row is sent to a separate cross-point array 705 that implements a given CNN layer. The charge accumulated on the capacitor 1320 represents the partial sum, which is sent to the input circuit 710 of the system 700 that implements the next layer of the CNN.

[0087] Input circuit 710 combines the partial sums and organizes such input data for the weights of the next layer stored in separate cross-point array 705. For example, input circuit 710 delivers the output corresponding to input data element 1310 to the kernel weight in the next layer that is the same as the output corresponding to input data element 1320.

[0088] It can be seen that when (image size)*(#input channels) is large compared to the size of the kernel weights (kernel size*#input channels), the flexible routing provided by the input circuit 710 as described above facilitates more compact mapping of the weights of different output channels compared to existing solutions. The cost of storing and reordering the outputs from one layer during input to the next layer is lower than existing row-by-row mapping techniques. Therefore, one or more embodiments of the present invention facilitate configuring flexible signal routing schemes to improve the scalability of existing row-by-row mapping techniques. In one or more embodiments of the present invention, the CNN can be fine-tuned according to the specific operational details of each network. For example, the CNN kernel size or the number of CNN kernels can be adjusted to further optimize the mapping onto the analog crosspoint array.

[0089] It should be noted that the dimensions of the matrices shown in the figures herein are examples only, and in one or more examples, different dimensions may be used. Furthermore, it should be noted that during the forward reasoning operation, the CNN has been trained, and embodiments of the present invention are applicable regardless of the technique used to train the CNN.

[0090] In this way, embodiments of the present invention facilitate row-by-row mapping for forward inference of a trained CNN, where the mapping can be performed in a compact manner to reuse the cross-point array and supporting circuitry, thereby facilitating the implementation of CNNs of any size.

[0091] In one or more embodiments of the present invention, the described technical solution is implemented by an electronic circuit that includes a cross-point array of resistive memory elements. The array provides a current output vector equal to the analog vector matrix product between (i) the voltage input vector of the array that encodes the analog input value vector and (ii) the analog resistance weight matrix within the array. The electronic circuit 700 also includes support circuit systems 712, 722, as well as input circuit system 710 and output circuit system 720, which together include accumulation wires and circuits for collecting current from a dedicated subset of resistive memory elements. Support circuit system 722 includes integration capacitors, each of which is electrically switchable to collect current from one of the accumulation wires during a single integration step. Output circuit system 720 appropriately converts and transmits the integrated charge from the subset of integration capacitors as an analog duration or as a digital representation using binary digits, which is accumulated over a predetermined number of integration steps. The resistive memory elements are arranged to implement the columns (rows) of the synaptic weight kernel of a given layer of the convolutional neural network.

[0092] Accumulation over a predetermined number of integration steps implements multiply-add operations across multiple partial rows (columns) of the weight kernel, as the input neuron excitations to the layer of the convolutional neural network are presented one row (column) at each iteration of the integration. In one or more embodiments of the present invention, the input neuron excitations of the first layer are always presented one complete row (column) at a time, while subsequent layers of input neuron excitations can be partitioned into multiple partial rows (columns) and partially stored in local analog memory (e.g., capacitors) to be processed in the cross-point array over multiple integration cycles.

[0093] Only after all rows (columns) of the weight kernel are fully integrated, the integrated charge representing the complete and / or partial output excitation is appropriately converted and transmitted. The partial sums from multiple crossbar arrays are flexibly routed to be combined in one of the integrating capacitors, subsequently converted to the complete output excitation, and then transmitted after all partial sums are fully integrated. The integrated charge on the integrating capacitor represents the output excitation, which is appropriately converted. In addition, only after all relevant weight kernels are fully integrated, the appropriately pooled result (e.g., the maximum, sum, or average of the output excitation) is locally calculated and then transmitted.

[0094] The present invention may be a system, method, and / or computer program product at any possible level of technical detail integration. The computer program product may include one or more computer-readable storage media having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to perform various aspects of the present invention.

[0095] Computer readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer readable storage medium includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device such as a punch card or a raised structure in a groove on which instructions are recorded, and any suitable combination of the foregoing. As used herein, computer readable storage medium should not be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0096] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0097] The computer-readable program instructions for performing the operation of the present technical solution can be assembly instructions, instruction set architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, configuration data of integrated circuits, or source code or object code written in any combination of one or more programming languages including object-oriented programming languages (such as Smalltalk, C++, etc.) and procedural programming languages (such as " C " programming language or similar programming languages). The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or executed entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (such as, using an internet service provider through the internet). In certain embodiments, the electronic circuit system including, for example, a programmable logic circuit system, a field programmable gate array (FPGA) or a programmable logic array (PLA) can execute the computer-readable program instructions to personalize the electronic circuit system by utilizing the state information of the computer-readable program instructions, so as to perform the aspects of the present technical solution.

[0098] Various aspects of the present technical solution are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present technical solution. It will be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0099] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device create a device for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can direct the computer, programmable data processing device, and / or other equipment to operate in a specific manner, such that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0100] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0101] The flow charts and block diagrams in the accompanying drawings illustrate the possible architectures, functions and operations of the systems, methods and computer program products according to the various embodiments of the present technical solution. In this regard, each box in the flow chart or block diagram can represent a module, segment or part of an instruction, which includes one or more executable instructions for realizing (multiple) specified logical functions. In some alternative implementations, the functions noted in the box may not occur in the order noted in the figure. For example, the two boxes shown in succession can actually be performed substantially simultaneously, or these boxes can sometimes be performed in reverse order, depending on the functions involved. It will also be noted that each box in the block diagram and / or flow chart illustration and the combination of the boxes in the block diagram and / or flow chart illustration can be implemented by a dedicated hardware-based system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0102] A second action may be said to be "responsive to" a first action, regardless of whether the second action is caused directly or indirectly by the first action. A second action may occur at a substantially later time than the first action and still be responsive to the first action. Similarly, a second action may be said to be responsive to the first action even if intermediate actions occur between the first and second actions, and even if one or more of the intermediate actions directly cause the second action to be performed. For example, if a first action sets a flag, and a third action later initiates a second action whenever the flag is set, then the second action may be responsive to the first action.

[0103] To clarify the use and thereby provide notice to the public, the phrase " 、 ,…as well as <n> At least one of "or"< / n> 、 ,…and <n> At least one or a combination of "or"< / n> 、 , … and / or <n>” should be interpreted in the broadest sense, superseding any other implied definitions above or below, and unless explicitly stated to the contrary, means one or more elements selected from the group consisting of A, B, . . . and N. In other words, the phrase means any one element alone or one or more elements of the one element and other elements, which may also include additional elements not listed or in combination with additional elements.

[0104] It will also be understood that any module, unit, component, server, computer, terminal, or device illustrated herein that executes instructions may include or otherwise access a computer-readable medium, such as a storage medium, computer storage medium, or data storage device (removable and / or non-removable), such as, for example, a disk, optical disk, or tape. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Such computer storage media may be part of a device or accessible or connectable to a device. Any application or module described herein may be implemented using computer-readable / executable instructions, which may be stored or otherwise stored by such a computer-readable medium.

[0105] Descriptions of various embodiments of the technical features have been presented herein for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, to improve upon existing practical applications or technologies in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

[0106] In a preferred embodiment of the invention described herein, an electronic circuit is provided that includes: an array of resistive memory elements that provides a current output vector that is equal to an analog vector matrix product between (i) a voltage input vector to the array encoding an analog input value vector and (ii) an analog resistor weight matrix within the array; integrating capacitors, each of the integrating capacitors being electrically switchable to aggregate current from one of a plurality of accumulation conductors during a single integration step; accumulation conductors and circuitry that aggregate current from a dedicated subset of the resistive memory elements by routing a portion of the output stimulus to the integrating capacitors that accumulate the integrated charge; and data output circuitry that allows the integrated charge accumulated over the plurality of integration steps from the subset of the integrating capacitors to be appropriately converted and transmitted as an analog duration or as a digital representation using binary digits, wherein the resistive memory elements are arranged to implement a vector of synaptic weight kernels for a given layer of a convolutional neural network. The resistive memory elements are preferably non-volatile memory devices. The subset of the resistive memory elements may correspond to one or more columns of the array. The subset of the resistive memory elements may correspond to one or more rows of the array. In an embodiment of the invention described herein, a method is provided for performing computations of a trained convolutional neural network (CNN) using the circuit described above in this paragraph, the method comprising: performing computations by resistive memory elements of a cross-point array by iterating a set of operations a predetermined number of times, the set of operations comprising: partitioning each vector of analog input values into a plurality of partial vectors; accumulating partial output stimuli corresponding to each of the plurality of partial vectors in an analog memory; and combining the partial output stimuli by routing the partial output stimuli to integrating capacitors that accumulate integrated charge; and transmitting the integrated charge representing the plurality of output stimuli on the plurality of integrating capacitors. Prior to transmitting the integrated charge, the integrated charge on the plurality of integrating capacitors is preferably locally pooled. The cross-point device may be arranged to implement one or more rows of convolution kernels for a given layer of the convolutional neural network, and wherein the input data represents neuron excitations for the layer of the convolutional neural network, presented one column at a time. The cross-point device may be arranged to implement one or more columns of convolution kernels for a given layer of the convolutional neural network, and wherein the vector of input data represents neuron excitations for the given layer of the convolutional neural network presented one row at a time from the input data.< / n>

Claims

1. A computer-implemented method for implementing a convolutional neural network (CNN) using a cross-point array, the method comprising: Configuring the cross-point array by storing one or more convolution kernels of a convolution layer in the CNN in one or more cross-point devices of the cross-point array, the cross-point array corresponding to the convolution layer; Performing computation for the CNN via the cross-point array by iterating a set of operations a predetermined number of times, the set of operations comprising: transmitting a voltage pulse corresponding to a sub-portion of a vector of input data of the convolutional layer to the cross-point array; outputting a current representative of a multiplication operation performed at the one or more cross-point devices in the cross-point array, the current based on weight values stored by the cross-point devices and the voltage pulses from the input data; and accumulating charge based on output current from the cross-point device via a set of integrators; and After iterating the predetermined number of times, the accumulated charges are output by the group of integrators, and the accumulated charges represent the multiplication and addition results of the vector of input data and the one or more convolution kernels. 2 . The method of claim 1 , wherein outputting the accumulated charge in the set of integrators comprises pooling the accumulated charge.

3. The method of claim 1, wherein the sub-portion of each vector of input data is associated with the set of integrators.

4. The method of claim 1 , wherein the cross-point array is a plurality of cross-point arrays, and wherein a first sub-portion of the vector of input data is transmitted to a first cross-point array, and a second sub-portion of the vector of input data is transmitted to a second cross-point array.

5. The method of claim 4 , wherein accumulating the charge by the set of integrators comprises: The charges accumulated by the set of integrators of the second cross-point array are accumulated by the set of integrators of the first cross-point array.

6. A method according to claim 1, wherein the cross-point device is arranged to implement one or more columns of convolution kernels of a given layer of the CNN, and wherein the vector of input data represents neuron excitations to the given layer of the CNN presented from the input data one row at a time.

7. The method of claim 6 , wherein the charge accumulated by the integrators from the set of integrators represents an output excitation according to the given layer of the CNN, the output excitation being transformed and transmitted only after all rows of the convolution kernel are integrated.

8. A method according to claim 1, wherein the cross-point device is arranged to implement one or more rows of convolution kernels of a given layer of the CNN, and wherein the input data represents neuron excitations to the layer of the CNN presented one column at a time.

9. The method of claim 8, wherein the charge accumulated by the integrators from the set of integrators represents an output excitation according to the given layer of the CNN, the output excitation being transformed and transmitted only after all columns of the convolution kernel are integrated.

10. An electronic circuit for performing computations of a trained convolutional neural network (CNN), the circuit comprising: Crosspoint array; an output circuit, the output circuit comprising a group of integrators; configuring a circuit system of a cross-point array corresponding to the convolutional layer in the CNN by storing one or more convolution kernels of the convolutional layer in one or more cross-point devices of the cross-point array; as well as Circuitry for iterating a set of operations a predetermined number of times, the set of operations comprising: transmitting a voltage pulse corresponding to a sub-portion of a vector of input data of the convolutional layer to the cross-point array; outputting a current representative of a multiplication operation performed at the one or more cross-point devices in the cross-point array, the current based on weight values stored by the cross-point devices and the voltage pulses from the input data; and accumulating charge based on the output current from the cross-point device via the set of integrators; and After iterating the predetermined number of times, the accumulated charges are output by the group of integrators, and the accumulated charges represent the multiplication and addition results of the vector of input data and the one or more convolution kernels. 11 . The circuit of claim 10 , wherein outputting the accumulated charge in the set of integrators comprises pooling the accumulated charge.

12. The circuit of claim 10, wherein the sub-portion of each vector of input data is associated with the set of integrators.

13. The circuit of claim 10 , wherein the cross-point array is a plurality of cross-point arrays, and wherein a first sub-portion of the vector of input data is transmitted to a first cross-point array, and a second sub-portion of the vector of input data is transmitted to a second cross-point array.

14. The circuit of claim 13 , wherein accumulating the charge by the set of integrators comprises: The charges accumulated by the set of integrators of the second cross-point array are accumulated by the set of integrators of the first cross-point array.

15. The circuit of claim 10 , wherein the cross-point device is arranged to implement one or more columns of convolution kernels for a given layer of the CNN, and wherein the vector of input data represents neuron excitations to the given layer of the CNN presented from the input data one row at a time.

16. The circuit of claim 15 , wherein the charge accumulated by the integrators from the set of integrators represents an output excitation according to the given layer of the CNN, the output excitation being transformed and transmitted only after all rows of the convolution kernel are integrated.

17. The circuit of claim 10, wherein the cross-point devices are arranged to implement one or more rows of convolution kernels for a given layer of the CNN, and wherein the input data represents neuron excitations to the layer of the CNN presented one column at a time.

18. The circuit of claim 17 , wherein the charge accumulated by the integrators from the set of integrators represents an output excitation according to the given layer of the CNN, the output excitation being transformed and transmitted only after all columns of the convolution kernel are integrated.

Images