Pulse generation for updating crossbar arrays

Abstract

Embodiments of computer-implemented methods, systems, and computer program products for updating an analog crossbar array are provided. Embodiments include receiving a number used in a matrix multiplication to represent generating a pulse for a crossbar array, and receiving a bit length to represent the number. Embodiments also include selecting a pulse position in a pulse sequence having the bit length to represent the number, performing a computation using the selected pulse position in the pulse sequence, and updating the crossbar array using the computation.

Description

Background Technology

[0001] This invention generally relates to implementing cross arrays of neural networks, and more specifically, to training neural networks to update simulated cross arrays using compact, fixed-bit-length deterministic pulse generation.

[0002] Artificial neural networks (ANNs), also known as neuromorphic and synaptic systems, are computational systems that allow electronic systems to function substantially in a manner similar to biological neural systems such as the human brain. In this specific embodiment, any reference to ANNs is used in the broadest sense to cover a wide variety of computer and electronic systems that model biological neural networks, including but not limited to spiking neural networks (PNNs). Some implementations of ANNs do not typically use traditional numerical models that manipulate zeros and ones. Instead, some implementations of ANNs create connections between processing elements that are functionally roughly equivalent to neurons in a biological brain.

[0003] ANNs are typically implemented as relatively large parallel computing systems with multiple simple processors interconnected. Modeling biological neural systems using ANNs (or PNNs) can contribute to a better understanding of biological functions. Many ANN models attempt to utilize principles considered to be "organizing" principles used in the human brain. Modern computer hardware technologies (e.g., VLSI and optics) have made this modeling feasible. ANNs combine knowledge from various disciplines, including neurophysiology, cognitive science / psychology, physics (statistical mechanics), control theory, computer science, artificial intelligence, statistics / mathematics, pattern recognition, computer vision, parallel processing, and hardware (e.g., digital / analog / VLSI / optics).

[0004] Modern ANNs consist of networks of artificial neurons (also called "nodes"). These nodes are connected to each other, and the strength of these connections is assigned values. For example, a connection can be identified as exhibiting inhibition (maximum inhibition is -1.0) or excitation (maximum excitation is +1.0). A high magnitude value for a connection indicates a strong connection. A transfer function is also built into the design of each node. There are three types of neurons in a typical ANN: input nodes, hidden nodes, and output nodes.

[0005] Input nodes receive information that can be represented numerically. This information is represented as activation values, where each node is assigned a number, and numbers with higher amplitudes elicit larger activations. This information is then passed throughout the network. Activation values ​​are passed from node to node based on connection strength (weights), inhibition (or activation), and a transfer function. Each node sums the activation values ​​it receives. Then, each node modifies this value based on its transfer function. Activation flows through input nodes and hidden layers until it reaches the output node. The output node reflects the input back to the outside world in a meaningful way.

[0006] There are many types of neural networks, but the two most common categories are feedforward and feedback networks. A feedforward network is a non-recursive network with input, output, and hidden layers. Signals can only propagate in one direction. Input data is passed to layers of processing elements that perform computations. Each processing element performs its computation based on a weighted sum of its inputs. The new computed value then becomes the new input value fed to the next layer. This process continues until it has passed through all layers and determined the output. Sometimes a threshold transfer function is used to quantize the output of neurons in the output layer.

[0007] Feedback networks include feedback paths, meaning their signals can propagate in both directions using loops. This allows for all possible connections between neurons. Because loops exist in this type of network, under certain operations, it can become a nonlinear dynamic system that continuously changes until it reaches an equilibrium state. Feedback networks are frequently used in associative memory and optimization problems, where the network seeks the optimal arrangement of interconnected factors.

[0008] In an ANN, the spike generation function can be modeled by a component commonly referred to as a time encoder or spike generator. Therefore, the ability to gain further knowledge of the previously described biological nervous system depends on the development of an ANN that models how neural information is encoded in a recursive network of spike time encoders. A time encoder provides an alternative to both digital and analog encoding of information by asynchronously integrating the received signal and generating a subsequent set of asynchronous impulse signals. Certain configurations of the time encoder network allow for the extraction of information about the set of signals input to the network from the set of impulses or spikes output from the network. Summary of the Invention

[0009] Embodiments of the present invention relate to a computer-implemented method for updating pulse generation of a cross array. Non-limiting examples of the computer-implemented method include receiving a number used in matrix multiplication to represent pulse generation using a cross array, and receiving a bit length to represent the number. The computer-implemented method further includes selecting a pulse position in a pulse sequence having a bit length to represent the number, performing a calculation using the selected pulse position in the pulse sequence, and updating the cross array using the calculation.

[0010] Embodiments of the present invention relate to a system for generating pulses to update a cross array. A non-limiting example of the system includes a cross array having one or more memristor devices and a processor. The processor is configured to receive a number used in matrix multiplication to represent pulse generation for the cross array, and to receive a bit length to represent that number. The processor is further configured to select a pulse position in a pulse sequence having the bit length to represent the number, perform a calculation using the selected pulse position in the pulse sequence, and update the cross array using the calculation, wherein updating the cross array changes the weights of the one or more memristor devices.

[0011] Embodiments of the present invention relate to a computer program product for pulse generation to update a cross array, the computer program product comprising a computer-readable storage medium having program instructions. The program instructions are executable by a processor to cause the processor to perform a method. A non-limiting example of the method includes receiving a number used in matrix multiplication to represent pulse generation using the cross array, and receiving a bit length to represent the number. The method further includes selecting a pulse position in a pulse sequence having a bit length to represent the number, performing a calculation using the selected pulse position in the pulse sequence, and updating the cross array using the calculation.

[0012] Additional technical features and benefits are achieved through the technology of this invention. Embodiments and aspects of the invention are described in detail herein and are considered part of the claimed subject matter. For a better understanding, refer to the detailed description and accompanying drawings. Attached Figure Description

[0013] The details of the proprietary rights described herein are specifically pointed out and clearly claimed in the claims at the conclusion of the specification. Other features and advantages of the foregoing embodiments of the invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings, wherein:

[0014] Figure 1 A diagram illustrating the input and output connections of a biological neuron modeled using embodiments of the present invention is shown;

[0015] Figure 2 This illustrates embodiments that can be used in conjunction with the present invention. Figure 1 The model of a biological neuron shown;

[0016] Figure 3 It shows the incorporation Figure 2 The ANN model of the biological neuron model shown;

[0017] Figure 4 A block diagram is shown of a known weight update method that can be used in conjunction with embodiments of the present invention;

[0018] Figure 5 A block diagram of a computation method that can be used in one or more embodiments of the present invention is shown;

[0019] Figure 6 Known equations governing the operation of passive, two-terminal memristors that can be used in conjunction with embodiments of the present invention are shown;

[0020] Figure 7 A graphical comparison is shown between the known switching characteristics of a two-terminal memristor and the nonlinear switching characteristics of a two-terminal RPU that can be used in conjunction with embodiments of the present invention.

[0021] Figure 8 A cross array of two-terminal nonlinear RPU devices according to an embodiment of the present invention is shown, as well as a voltage sequence illustrating the operation of the RPU;

[0022] Figure 9 An algorithm for selecting bit positions in a bit stream according to one or more embodiments of the present invention is shown;

[0023] Figure 10 A diagram and a multiplication diagram are shown according to one or more embodiments of the present invention;

[0024] Figure 11 A diagram and a multiplication diagram are shown according to one or more embodiments of the present invention;

[0025] Figure 12 A diagram and a multiplication diagram are shown according to one or more embodiments of the present invention;

[0026] Figure 13 A flowchart of a method for generating pulses to update a cross array according to one or more embodiments of the present invention is shown; and

[0027] Figure 14 A block diagram illustrating an example of a processing system used to practice the teachings in this paper is provided.

[0028] The figures shown herein are illustrative. Numerous variations may be made to the figures or operations described therein without departing from the spirit of the invention. For example, actions may be performed in a different order, or actions may be added, deleted, or modified. Furthermore, the term "coupling" and its variations describe a communication path between two elements and do not imply a direct connection between the elements without any intermediate elements / connections between them. All such variations are considered part of the specification.

[0029] In the following detailed description of the accompanying drawings and embodiments of the invention, the various elements shown in the drawings have two or three numerical reference numerals. With minor exceptions, the leftmost numeral of each reference numeral corresponds to the figure in which the element is first shown. Detailed Implementation

[0030] Various embodiments of the invention are described herein with reference to the accompanying drawings. Alternative embodiments of the invention may be designed without departing from the scope of the invention. In the following description and drawings, various connections and positional relationships (e.g., above, below, adjacent, etc.) are illustrated between elements. Unless otherwise stated, these connections and / or positional relationships may be direct or indirect, and the invention is not intended to be limiting in this respect. Therefore, coupling of entities can refer to direct or indirect coupling, and positional relationships between entities can be direct or indirect positional relationships. Furthermore, the various tasks and process steps described herein may be incorporated into a more comprehensive procedure or process with additional steps or functionality not described in detail herein.

[0031] The following limitations and abbreviations are used to interpret the claims and specification. As used herein, the terms “comprising,” “including,” “having,” “containing,” or any other variation thereof are intended to cover non-exclusive inclusion. For example, a composition, mixture, process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but may include other elements not expressly listed or inherent to such compositions, mixtures, processes, methods, articles, or apparatus.

[0032] Additionally, the term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any embodiment of the invention described herein as "exemplary" is not necessarily to be construed as more preferred or advantageous than other embodiments of the invention. The terms "at least one" and "one or more" can be understood to include any integer greater than or equal to one, i.e., one, two, three, four, etc. The term "multiple" can be understood to include any integer greater than or equal to two, i.e., two, three, four, five, etc. The term "connection" can include both indirect "connection" and direct "connection."

[0033] The terms “about,” “substantially,” “approximately,” and variations thereof are intended to include the degree of error associated with a measurement based on a specific quantity of the device available at the time of filing of this application. For example, “about” may include a range of ±8%, 5%, or 2% of a given value.

[0034] For the sake of brevity, conventional techniques related to the manufacture and use of aspects of the present invention may or may not be described in detail herein. In particular, various aspects of the computing systems and specific computer programs used to implement the various technical features described herein are well known. Therefore, for the sake of brevity, many conventional implementation details are only briefly mentioned or omitted entirely herein, without providing well-known system and / or process details.

[0035] We now turn to an overview of the technology more specifically related to aspects of the present invention, which will now be referenced. Figure 1 ,2 Sections 3 and 4 are used to provide a description of how a typical ANN operates. As mentioned earlier in this article, a typical ANN models the human brain, which consists of approximately one billion interconnected cells called neurons. Figure 1 A simplified diagram depicts biological neurons 102 with pathways 104, 106, 108, and 110 connecting them to upstream inputs 112 and 114, downstream outputs 116, and downstream "other" neurons 118, configured and arranged as shown. Each biological neuron 102 sends and receives electrical impulses through pathways 104, 106, 108, and 110. The nature of these electrical impulses and how they are processed within the biological neurons 102 are primarily responsible for overall brain function. Pathway connections between biological neurons can be strong or weak. When a given neuron receives an input impulse, the neuron processes the input according to its function and sends the result of that function to the downstream output and / or downstream "other" neurons.

[0036] Biological neuron 102 in Figure 2 Node 202 is modeled as having a mathematical function f(x), which is derived from... Figure 2 The equation shown depicts node 202 receiving electrical signals from inputs 212 and 214, multiplying each input 212 and 214 by the strength of its corresponding connection path 204 and 206, obtaining the sum of the inputs, passing the sum through the function f(x), and generating a result 216, which can be the final output or input to another node, or both. In this specific embodiment, an asterisk (*) is used to indicate multiplication. Weak input signals are multiplied by very small connection strength numbers, so weak input signals have little effect on the function. Similarly, strong input signals are multiplied by higher connection strength numbers, so strong input signals have a greater effect on the function. The function f(x) is a design choice, and various functions can be used. A common design choice for f(x) is the hyperbolic tangent function, which takes the sum of the previous values ​​and outputs a number between negative and positive one.

[0037] Figure 3 A simplified ANN model 300, organized as a weighted directed graph, is depicted, where artificial neurons are nodes (e.g., 302, 308, 316), and weighted directed edges (e.g., m1 to m20) connect the nodes. The ANN model 300 is organized such that nodes 302, 304, and 306 are input layer nodes, nodes 308, 310, 312, and 314 are hidden layer nodes, and nodes 316 and 318 are output layer nodes. Each node is connected to every node in the adjacent layers via connection paths. Figure 3 The diagram is depicted as directional arrows with connection strengths m1 to m20. Although only one input layer, one hidden layer, and one output layer are shown, in reality, multiple input layers, hidden layers, and output layers can be provided.

[0038] Similar to the function of the human brain, each input layer node 302, 304, and 306 of ANN 300 directly receives inputs x1, x2, and x3 from the source (not shown), without connection strength adjustment or node summation. Therefore, y1 = f(x1), y2 = f(x2), and y3 = f(x3), as shown below. Figure 3 The equations listed at the bottom illustrate this. Each hidden layer node 308, 310, 312, 314 receives its input from all input layer nodes 302, 304, 306 based on the connection strength associated with the relevant connection path. Therefore, in hidden layer node 308, y4 = f(m1*y1 + m5*y2 + m9*y3), where * denotes multiplication. Similar connection strength multiplication and node summation are performed for hidden layer nodes 310, 312, 314 and output layer nodes 316, 318, as defined by the constraints. Figure 3 The equations depicting functions y5 to y9 at the bottom are shown.

[0039] An ANN model 300 processes one data record at a time, and it "learns" by comparing an initial arbitrary classification of a record with its known actual classification. Using a training method called "backpropagation" (i.e., "backpropagation of error"), the error from the initial classification of the first record is fed back into the network and used to modify the network's weighted connections a second time, and this feedback process continues iteratively. During the training phase of the ANN, the correct classification of each record is known, and therefore a "correct" value can be assigned to the output nodes. For example, for a node corresponding to the correct class, the node value is "1" (or 0.9), while for other nodes, the node value is "0" (or 0.1). Therefore, the network's computed values ​​for the output nodes can be compared with these "correct" values, and an error term (i.e., an "incremental" rule) is computed for each node. These error terms are then used to adjust the weights in the hidden layers so that in the next iteration, the output value will be closer to the "correct" value.

[0040] Despite the possibility of lower power consumption, performing offline training can be difficult and resource-intensive because it typically requires modifying a large number of adjustable parameters (e.g., weights) in the ANN model during training to match the input-output pairs of the training data. Figure 4 A simplified illustration of the typical read-process-write weight update operation is depicted, where the CPU / GPU core (i.e., the simulated "neuron") reads from memory (i.e., the simulated "synapse") and performs the weight update processing operation, then writes the updated weights back to memory. Therefore, the simplified cross-point device of the ANN architecture prioritizes energy efficiency and offline learning techniques, which typically means that training speed and efficiency are not optimized.

[0041] Random computation is a set of techniques for representing continuous values ​​through random bit streams, where complex computations can be performed by simple bitwise operations on these streams. Specifically, if there exist two random and independent bit streams S1 and S2 (i.e., Bernoulli processing) called random numbers, where the probability of a "1" in the first stream is p and the probability of a "1" in the second stream is q, then... Figure 5 The diagram shows the logical AND operation of these two streams. By observing enough output bits and measuring the frequency of the "ones," pq can be estimated with arbitrary precision. Due to the design of the "multiplication and accumulation" operation, it can be implemented with a few logic gates / transistors, and random computation is frequently used in the hardware design of neural networks.

[0042] However, in some stochastic techniques, the necessary weights for computation are supplied to the systolic array from an external location, and the array does not perform updates to the weights. This only addresses the acceleration of vector-matrix multiplication or matrix-matrix multiplication operations that are heavily used during neural network training. However, a systolic array without local storage cannot perform weight updates in parallel because the weights are stored at an external memory location. Accelerating weight updates is necessary to speed up the overall learning algorithm provided by embodiments of the techniques described herein.

[0043] Furthermore, training an ANN requires performing multiple computations. As the number of nodes increases, so does the complexity and number of computations, leading to further inefficiencies in training speed and accuracy. To compute the outer product used to adjust the weights, b is sent... 2 Pulses used to form an outer product are inefficient, where b represents the number of bits.

[0044] Turning now to an overview of various aspects of the invention, one or more embodiments of which address the aforementioned drawbacks of the prior art by providing an efficient neural network training scheme. The techniques of the embodiments of the invention described herein reduce the number of pulses (or bit length) used to represent each number used in matrix multiplication in outer product-based matrix updates. The reduced bit length determines the accuracy of the outer product-based matrix update. However, this reduction leads to information loss and must be optimized. The granularity in multiplications performed within outer product-based updates can be introduced by reducing the number of pulses, which can affect the accuracy of the neural network. Therefore, the selection of the pulse positions in the pulse sequence is critical and must be strategically chosen to maintain the contours presented in the true multiplication graph of the values.

[0045] Bit selection representing a pulse sequence or bitstream of numbers is mapped to an outer product used to train a neural network. The technical effects and benefits of embodiments of the invention described herein provide an accurate representation of bitwise multiplication performed in conventional neural network updates. The technical effects and benefits also provide an efficient way to encode multiplication with minimal error using only b bits of the pulse sequence. To optimize multiplication, algorithms according to various aspects of the invention provide a minimal representation sufficient to compute the outer product used to update one or more memristors in a cross array. Algorithms providing the generation of pulse positions for values ​​used in matrix multiplication can accelerate the speed and efficiency of training ANN architectures, improve overall ANN performance, and allow for a wider range of ANN applications.

[0046] The term "memristor" is used to describe a passive two-terminal electrical component where the resistance value of the device depends on the history of voltages previously applied to the device. The operation of a memristor is determined by... Figure 6 Equations [1] and [2] shown in the figure govern the current through the device, v is the voltage applied to the device, g is the conductance of the device (which is the reciprocal of the resistance), s is the internal state variable of the device that controls the conductance, and f is a function showing the time evolution of the internal state variable s.

[0047] Figure 7 The behavior of a cross array according to an embodiment of the invention is described below. This is to illustrate the difference between an ideal memristor and a non-ideal nonlinear memristor that can be used to implement the disclosed RPU. Figure 7 This is a graph comparing the voltage switching behavior of an ideal memristor with that of an RPU according to an embodiment of the invention. The vertical axis of the graph represents the device state change at a specific voltage, while the horizontal axis represents the applied voltage. In ideal memristor operation, the change in resistance is linearly proportional to the voltage applied to the device. Therefore, once the memristor sees any voltage, its resistance state changes. This is illustrated in Figure 702, which shows that the change in state is significant even at low voltages.

[0048] For a nonlinear RPU device as shown in Figure 704, there is a well-defined set voltage VSET that the device needs to pass through to change its internal resistance state. A bias voltage of 0.5VSET will not change the internal resistance state. In embodiments of the invention, this nonlinear characteristic of the RPU device is utilized to perform multiplication locally. Assuming f1 is small, when only 0.5VSET is applied, the device will not change its internal state. It is worth noting that... Figure 7 The positive voltage and positive change in the resistance state of the RPU device are shown; however, a similar relationship exists between the negative voltage and negative resistance change.

[0049] Now we turn to a more detailed description of various aspects of the invention. Figure 8This is a diagram of a two-dimensional (2D) cross array 800 performing forward matrix multiplication, backward matrix multiplication, and weight update according to an embodiment of the present invention. The cross array 800 is formed by a set of conductive row lines 802, 804, 806 and a set of conductive column lines 808, 810, 812, 814 intersecting the set of conductive row lines 802, 804, 806. The intersection points between the row lines and the column lines are separated by an RPU (Radio Partition Object). Figure 8 The elements are shown as resistive elements, each with its own adjustable / update resistance weights, described as σ11, σ21, σ31, σ41, σ12, σ22, σ32, σ42, σ13, σ23, σ33, and σ43, respectively. For ease of illustration, in Figure 8 Only one RPU 820 is labeled with a reference number. In forward matrix multiplication, the conduction state of the RPU (i.e., the stored weights) can be read by applying a voltage across the RPU and measuring the current through the RPU.

[0050] Input voltages V1, V2, and V3 are applied to row lines 802, 804, and 806, respectively. Each column line 808, 810, 812, and 814 sums the currents I1, I2, I3, and I4 generated by each RPU along that specific column line. For example, as... Figure 8 As shown, the current I4 generated by column line 814 is determined according to the equation I4 = V1σ41 + V2σ42 + V3σ43. Therefore, array 800 calculates the forward matrix multiplication by multiplying the value stored in the RPU by the row line input, which is defined by voltages V1, V2, and V3. The backward matrix multiplication is very similar. In the reverse matrix multiplication, voltages are applied to column lines 808, 810, 812, and 814, and then read from row lines 802, 804, and 806. For the weight update described in more detail below, voltages are applied to both column and row lines simultaneously, and the conductance values ​​836 stored in the relevant RPU devices are all updated in parallel. Therefore, using the RPU devices themselves plus the relevant row or column lines of array 800, the multiplication and addition operations required for weight updates are performed locally in each RPU 820 of array 800. Therefore, according to an embodiment of the invention, no read-update-write cycle is required in array 800. Figure 4 (as shown in the image).

[0051] continue Figure 8According to one or more embodiments of the invention, a positive weight update method for RPU 820 and its operation at the corresponding weight σ33 at the intersection of conductive row line 806 and conductive column line 812 will now be provided. The nonlinear characteristics of RPU 820 are used to perform multiplication locally on RPU 820 using the calculations described below. More specifically, the disclosed method uses the nonlinear switching characteristics of RPU 820 and bitstreams 830, 832 to perform multiplication locally on RPU 820 and the necessary weight updates without requiring other circuit elements. An update generator circuit (not shown) is provided at the periphery of cross array 800 and is used as a peripheral "converter" to generate the necessary voltage pulses (e.g., 830, 832) in the form of bitstreams that will be applied in parallel to all RPUs of 2D cross array 800.

[0052] Brief reference Figure 7 The diagram, assuming Figure 8 The f1 of the RPU 820 shown is very small (e.g., f1 = 0), meaning that its internal state remains unchanged when only 0.5VSET is applied to the RPU 820. The row voltage sequence or bitstream 830 applied to row line 806 is shown as a weighted voltage pulse sequence with zero voltage or +0.5VSET voltage. The column voltage sequence or bitstream 832 applied to column line 814 is also shown as a weighted voltage pulse sequence with zero voltage or -0.5VSET voltage. Figure 8 In the example, 4 / 8 is encoded by row voltage sequence 830, and 6 / 8 is encoded by column voltage sequence 832. Example voltage sequences 830 and 832 represent the positive resistance change phase of the weight update. After performing a positive weight update, separate sets of sequences with reversed polarities of the corresponding voltages can be used to update those weights requiring this correction in the negative direction.

[0053] Voltage sequence 834 is the voltage applied to RPU 820 generated by the difference between row voltage sequence 830 and column voltage sequence 832. Voltage sequence 834 will have three voltage steps at 0V, 0.5VSET, and VSET. However, because the resistance σ43 of RPU 820 only changes when the device voltage reaches VSET, a single pulse sent through a column or row line is insufficient to change the resistance state of RPU 820. When a 0.5VSET voltage is sent to the column line and a -0.5VSET voltage is sent to the row line, the resulting VSET pulse applied to the relevant RPU will cause an incremental change in device resistance. Therefore, the voltage pulse applied to RPU 820 utilizes the non-linear switching characteristics of RPU 820 to perform a bit-by-bit AND operation locally at RPU 820 (e.g., as shown in the image). Figure 5(As shown). Therefore, the final change in the weights of the stored RPU (e.g., σ43) is proportional to the product of the two numbers (4 / 8 * 6 / 8 = 3 / 8) "converted" by the update generator circuitry surrounding the cross array 800.

[0054] According to one or more embodiments of the present invention, Figure 9 Algorithm 900 is shown for performing multiplication with overlapping pulses. Specifically, algorithm 900 generates pulse positions in a pulse sequence for the δ value to optimize the weights used to update the cross array. Variables used in algorithm 900 include variable b, which is the number of distinct x and δ values ​​in matrix w; compression, which is the compression factor; ∈, which is the error tolerance; and T, which ranges from 1 to min(b). 2 b 2 The counter variable for / compression. When the b-bit resolution equals b 2 At that time, the uncompressed set is obtained. Any set smaller than b... 2 The value corresponds to how bits are represented during compression (because information loss will occur). Algorithm 900 aims to perform compression with the minimum penalty.

[0055] Weight updates are performed by calculating a vector-vector cross product over the weight matrix (as shown above), similar to multiplication. After the calculation, incremental weight updates are applied at each processing component using the updated computation. The above method determines the pulse positions of the δ values ​​of matrix w in the pulse sequence. A first pulse sequence representing x values ​​is multiplied by a second pulse sequence representing δ values. The pulse positions of the δ values ​​can be represented as shown in Figure 1004 (reference). Figure 10 (Further discussion) In the graph. Using the pulse sequence from Algorithm 900, the multiplication is simplified and the total number of computations required to perform the weight update is reduced.

[0056] Figure 10 A series of graphs and diagrams are shown illustrating an exemplary implementation of pulse generation for updating a cross array according to one or more embodiments of the present invention. In this non-limiting example, there are b intervals, where the b intervals indicate that the x and δ values ​​of the weight matrix w can each take 100 different values. Here, b intervals are 100, and the bit length (BL) is 100 bits. Conventionally, to represent the different numbers of x and δ, b... 2 The computation count is used to update the weights used in the neural network, as element-wise multiplication updates are performed for each element (x and δ). Figure 9 The method described in the text compresses b 2 The number of computations is reduced to b computations to perform weight updates on the neural network, which exhibits results similar to element-wise multiplication.

[0057] exist Figure 10 In the diagram, graph 1002 represents the x-values ​​in the weight matrix used when performing updates to the neural network. The x-axis of graph 1002 represents the pulse position in the pulse sequence or the bit position in the bit stream. The y-axis of graph 1002 represents the number of x-values ​​that will be represented by the pulse sequence. The pulse sequence or bit stream can be represented by a series of ones and zeros. Region 1020 represents one (logic high), while region 1030 represents zero (logic low). For example, in graph 1002, the row corresponding to the x-value 80 indicates the position in the pulse sequence, and the number 80 is represented by the pulses provided at the positions shown on the x-axis. In another example, the bottom row of graph 1002 corresponding to the number zero is represented by 100 zero pulses in the pulse sequence, and the top row corresponding to one is represented by 100 ones in the pulse sequence.

[0058] The pulse sequences generated for x and δdelta values ​​are completely different. Therefore, the determination of the pulse sequence for the δ value differs from that for the x value, and a reference is used. Figure 9 The algorithm 900 discussed is determined in Figure 10 In Figure 1004, the pulse position in the pulse sequence for the δ value is shown. Similar to Figure 1002, the x-axis of Figure 1004 represents the pulse position, and the y-axis represents the number of δ values ​​to be used in the calculation for weight updates.

[0059] Still referencing Figure 10 Multiplication graph 1006 represents a true multiplication graph, where, for each element (x) in the weight matrix... i δ j Element-wise multiplication is performed, where true multiplication diagram 1006 provides the outline corresponding to element-wise multiplication. In this example, 10,000 pulses / bits are used to represent each number used to perform matrix multiplication to update the weights of the simulated cross array, where x is 100 bits and δ is 100 bits. As the number of bits increases, the computational complexity and the numbers increase rapidly.

[0060] Figure 10 A heatmap 1008 is also shown, generated by taking the outer product of the number of x values ​​represented by the first pulse sequence and the number of δ values ​​represented by the second pulse sequence, wherein the second pulse sequence of δ values ​​is generated by... Figure 9 The algorithm shown in Heatmap 1008 allows for an intuitive analysis of the approximation of the element-wise multiplication of the outer product of the first and second pulse sequences with each combination of x and δ.

[0061] As shown in heatmap 1008, when performing true multiplication for each element, the outer product allows the element-wise multiplication representation to be shown using only 100 bits instead of 10,000 bits. The error can be calculated by comparing multiplication graph 1006 with heatmap 1008 and taking the difference between corresponding values. Figure 10 As shown, error graph 1010 is calculated and displayed. The contour of heatmap 1008 is similar to the contour shown in multiplication graph 1006, and the similarity indicates that the representation will use the outer product to provide efficient updates and accurate classification for the neural network.

[0062] As the number of representations decreases from the true multiplication graph, heatmap 1006 becomes increasingly granular, which can lead to inaccurate results. However, optimal simplification can be performed to obtain a bit representation that maintains the integrity of the results from the neural network, as when updating the neural network using a true multiplication graph. In one or more embodiments of the invention, optimal simplification is achieved when BL is set to equal b.

[0063] exist Figure 11 In the configuration with intervals of 100 and 50 bits, and Figure 10 Similarly, Figure 1102 shows the pulse sequence for each x value, and Figure 1104 shows the pulse sequence for each δ value, which will be used to update the neural network. When compared with... Figure 10 When comparing, BL is 50, which is less than... Figure 10 Since more values ​​of x and δ are mapped to the true multiplication graph 1006 with a shorter BL, a certain granularity is introduced into the heatmap 1108 when the number of representations exceeds the BL used for updating. Additionally, the error graph 1110 indicates a larger difference between the true multiplication graph 1006 and the heatmap 1008.

[0064] refer to Figure 12 The BL used to represent the x and δ values ​​of the weight matrix w is further reduced to BL = 10, while... Figure 11 Compared to the interval that remains at 100, Figure 1202 represents the pulse sequence for each x value, and Figure 1204 represents the pulse sequence for each δ value. As can be seen by comparing the true multiplication graph 1206 with the heatmap 1208, more granularity is introduced into the heatmap 1208, which indicates a decrease in the accuracy of the update. Figure 12 An example is also shown where the granularity increases as the BL used to represent each value decreases.

[0065] Figure 13 A flowchart of a method 1300 for updating pulse generation of a cross array according to one or more embodiments of the present invention is shown. Method 1300 may use methods such as... Figure 4 The processor or reference shown Figure 14The processor discussed is used for execution; it should be understood that any other computing device can be used. Method 1300 begins at block 1302 and proceeds to block 1304, which provides a number received by the processor for matrix multiplication, representing the number generated using pulses for a cross array. Block 1306 receives a bit length to represent the number. The bit length provided to the processor is known from a neural network. Block 1308 selects a pulse position in a pulse sequence having a bit length to represent the number. In one or more embodiments of the invention, the pulse position is used... Figure 9 The pulse positions in the pulse sequence, determined by the algorithm provided, are determined for the δ value. Box 1310 uses the selected pulse positions in the pulse sequence to perform a calculation. A first pulse sequence for the δ value of the weight matrix and a second pulse sequence for the x value are determined to form an outer product. The result is used to add or subtract an increment to adjust the weights used to train the neural network. Box 1312 uses this calculation to update the cross array. In one or more embodiments of the invention, the result is used to adjust the conductance of one or more memristor devices in the cross array. Method 1300 ends at box 1314. It should be understood that method 1300 may include steps in a different order or additional steps, and is not intended to be affected by... Figure 13 The steps shown are limited.

[0066] refer to Figure 14 A processing system 1400 for implementing aspects of the invention according to the teachings described herein is shown. The processing system can be implemented, for example... Figure 4 In the processor (CPU / GPU core) shown, in this embodiment, system 1400 has one or more central processing units (processors) 1401a, 1401b, 1401c, etc. (collectively or generally referred to as processors 1401). In one embodiment, each processor 1401 may include a Reduced Instruction Set Computer (RISC) microprocessor. Processor 1401 is coupled to system memory 1414 and various other components via system bus 1413. Read-only memory (ROM) 1402 is coupled to system bus 1413 and may include a Basic Input / Output System (BIOS) that controls certain basic functions of system 1400.

[0067] Figure 14Input / output (I / O) adapter 1407 and network adapter 1406 coupled to system bus 1613 are also described. I / O adapter 1407 may be a Small Computer System Interface (SCSI) adapter that communicates with hard disk 1403 and / or tape storage drive 1405 or any other similar component. I / O adapter 1407, hard disk 1403, and tape storage device 1405 are collectively referred to herein as mass storage 1404. Operating system 1420 for execution on processing system 1400 may be stored in mass storage 1404. Network adapter 1406 interconnects bus 1413 with external network 1416, enabling data processing system 1400 to communicate with other such systems. Screen (e.g., display monitor) 1415 is connected to system bus 1413 via display adapter 1412, which may include graphics adapters and video controllers for improving performance in graphics-intensive applications. In one embodiment, adapters 1407, 1406, and 1412 may be connected to one or more I / O buses, which are connected to system bus 1413 via an intermediate bus bridge (not shown). Suitable I / O buses for connecting peripheral devices such as hard disk controllers, network adapters, and graphics adapters typically include common protocols such as Peripheral Component Interconnect (PCI). Additional input / output devices are shown connected to system bus 1413 via user interface adapter 1408 and display adapter 1412. Keyboard 1409, mouse 1410, and speaker 1411 are all interconnected to bus 1413 via user interface adapter 1408, which may include, for example, a super I / O chip integrating multiple device adapters into a single integrated circuit.

[0068] In an exemplary embodiment, the processing system 1400 includes a graphics processing unit 1430. The graphics processing unit 1430 is a dedicated electronic circuit designed to manipulate and modify memory to accelerate the generation of a predetermined image in a frame buffer for display. Typically, the graphics processing unit 1430 is highly efficient in manipulating computer graphics and image processing and has a highly parallel architecture that makes it more efficient than a general-purpose CPU for algorithms in which large blocks of data are processed in parallel.

[0069] Therefore, as Figure 14 The system 1400 configured therein includes processing power in the form of a processor 1401, storage capacity including system memory 1414 and mass storage 1404, input devices such as a keyboard 1409 and a mouse 1410, and output capacity including a speaker 1411 and a display 1415. In one embodiment, a portion of the system memory 1414 and mass storage 1404 jointly stores the operating system for coordination. Figure 14 The functions of each component are shown below.

[0070] The embodiments of the invention described herein improve upon the prior art by determining the bit positions within a bitstream to represent each digit of a multiplication used for neural network training. Furthermore, embodiments of the technology described herein provide acceleration for training neural networks using these methods by providing initial values ​​to the neural network before receiving any weight updates.

[0071] This invention can be a system, method, and / or computer program product at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0072] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or recessed structures with instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0073] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the respective computing / processing device.

[0074] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages, such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, to perform aspects of this invention, electronic circuits, including, for example, programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions to personalize the electronic circuits by utilizing the status information of the computer-readable program instructions.

[0075] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0076] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to operate in a particular manner, such that the computer-readable storage medium in which the instructions are stored includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0077] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0078] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative embodiments, the functions indicated in the blocks may occur in a non-consecutive order as shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0079] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Numerous modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or improvements to existing technologies on the market, or to enable others skilled in the art to understand the embodiments described herein.

Claims

1. A computer-implemented method for pulse generation for updating an analog cross array, the computer-implemented method comprising: The processor receives numbers used in matrix multiplication, generated using pulses applied to the cross array. The processor receives the bit length used to represent the number; The processor selects the pulse position in the pulse sequence, the pulse sequence having the bit length used to represent the number; The processor performs calculations using the selected pulse positions in the pulse sequence; and The processor uses the computation to update the cross array. The selected pulse position includes at least one of a first pulse sequence or a second pulse sequence; and There is pulse overlap between the first pulse sequence and the second pulse sequence.

2. The computer-implemented method of claim 1, wherein the calculation includes performing an outer product operation using the number used in the matrix multiplication and a second number used in the matrix multiplication.

3. The computer-implemented method of claim 1, wherein the selected pulse position in the pulse sequence is at least partially based on the bit length.

4. The computer-implemented method of claim 1, wherein the bit length is equal to the number of distinct elements for the weight matrix.

5. The computer-implemented method of claim 1, further comprising generating a heatmap, wherein the generated heatmap is at least partially based on the pulse overlap between the first pulse sequence and the second pulse sequence.

6. The computer-implemented method of claim 1, wherein at least one of the following is true: the first pulse sequence corresponds to the number, and the second pulse sequence corresponds to the second number.

7. The computer-implemented method of claim 1, further comprising comparing a true multiplication graph with a pulse coincidence using the selected pulse position, wherein the true multiplication graph performs element-wise multiplication for each element.

8. The computer-implemented method according to claim 7, further comprising: Determine the error between the true multiplication graph and the pulse overlap; as well as This displays the error between the true multiplication diagram and the pulse overlap.

9. The computer-implemented method of claim 1, wherein updating the cross array comprises updating the conductance values ​​of one or more memristor devices of the cross array based at least in part on the calculation.

10. A system for implementing pulse generation for updating a cross array, the system comprising: A cross array comprising one or more memristor devices; Processor, the processor being configured to: Receive numbers used in matrix multiplication to represent pulses generated for the cross array; Receive the bit length used to represent the number; Select the pulse position in the pulse sequence, the pulse sequence having the bit length used to represent the number; The calculation is performed using the selected pulse position in the pulse sequence; and The calculation is used to update the cross array, wherein updating the cross array changes the weights of the one or more memristor devices. The selected pulse position includes at least one of a first pulse sequence or a second pulse sequence; and There is pulse overlap between the first pulse sequence and the second pulse sequence.

11. The system of claim 10, wherein the calculation includes performing an outer product operation using the number used in the matrix multiplication and a second number used in the matrix multiplication.

12. The system of claim 10, wherein the selected pulse position in the pulse sequence is at least partially based on the bit length.

13. The system of claim 10, wherein the bit length is equal to the number of distinct elements for the weight matrix.

14. The system of claim 10, wherein the calculation includes determining the outer product of a first vector and a second vector for the neural network.

15. The system of claim 10, wherein the processor is further configured to generate a heatmap, wherein the generated heatmap is based at least in part on the pulse overlap between the first pulse sequence and the second pulse sequence.

16. The system of claim 10, further comprising comparing a true multiplication graph with a pulse coincidence using the selected pulse position, wherein the true multiplication graph performs element-wise multiplication for each element.

17. The system of claim 16, wherein the processor is further configured to: Determine the error between the true multiplication graph and the pulse overlap; and This displays the error between the true multiplication diagram and the pulse overlap.

18. The system of claim 10, wherein updating the cross array comprises updating the conductance values ​​of the one or more memristor devices of the cross array based at least in part on the calculation.

19. A computer program product for updating pulse generation of an analog cross array, the computer program product comprising: A computer-readable storage medium having first program instructions stored thereon, the first program instructions being executable by a processor to cause the processor to: Receive numbers used in matrix multiplication to represent pulses generated for the cross array; Receive the bit length used to represent the number; Select the pulse position in the pulse sequence, the pulse sequence having the bit length used to represent the number; The calculation is performed using the selected pulse position in the pulse sequence; and The calculation is used to update the cross array, wherein updating the cross array includes updating the conductance values ​​of one or more memristor devices in the cross array, at least in part, based on the calculation. The selected pulse position includes at least one of a first pulse sequence or a second pulse sequence; and There is pulse overlap between the first pulse sequence and the second pulse sequence.

20. The computer program product of claim 19, wherein the calculation comprises performing an outer product operation using the number used in the matrix multiplication and a second number used in the matrix multiplication.

21. The computer program product of claim 19, wherein the bit length is equal to the number of distinct elements for the weight matrix.

22. The computer program product of claim 19, wherein the instructions are further executable by a processor to cause the processor to determine the outer product of a first pulse sequence and a second pulse sequence for a neural network.

23. The computer program product of claim 19, wherein the instructions are further executable by a processor to cause the processor to generate a heatmap, wherein the generated heatmap is based at least in part on the pulse overlap between the first pulse sequence and the second pulse sequence.

24. The computer program product of claim 19, wherein the instructions are further executable by a processor to cause the processor to: The true multiplication graph is compared with the pulse coincidence using the selected pulse position, wherein the true multiplication graph performs element-wise multiplication for each element.

25. The computer program product of claim 19, wherein the instructions are further executable by a processor to cause the processor to: Determine the error between the true multiplication graph and the pulse overlap; and This displays the error between the true multiplication diagram and the pulse overlap.

Images