Weight duplication on the RPU crossbar array

Abstract

A method for training an artificial neural network is proposed. The method includes storing weight values ​​in an array of resistive processing unit (RPU) devices, wherein the array of RPU devices represents a weight matrix; defining the weight matrix to have an output dimension that is smaller than an input dimension, such that the weight matrix has a rectangular configuration; and converting the weight matrix from a rectangular configuration to a more square configuration by repeating or cascading the rectangular configuration of the weight matrix to increase the signal strength of a backward pass signal by: replicating the inputs of the repeated weight elements during a forward recurrent pass; summing the output calculations from the repeated weight elements; updating each of the repeated weight elements according to a backward propagated error; or alternatively, updating only one of the repeated weight elements by setting all forward values ​​except one to zero during an update pass.

Description

Background Art

[0001] The present invention generally relates to artificial neural networks (ANNs) having analog cross-point arrays of resistive processing unit (RPU) devices, and more particularly to increasing signal strength through weight repetition across the RPU cross-point array.

[0002] Machine learning is used broadly to describe the main functions of electronic systems that learn from data. In machine learning and cognitive science, ANN is a family of statistical learning models inspired by biological neural networks (particularly the brain). ANN can be used to estimate or approximate systems and functions that depend on a large number of inputs and are generally unknown. ANN is typically implemented as a so-called "neuromorphic" system of interconnected processor elements that act as simulated "neurons" and exchange messages between each other in the form of electronic signals. Similar to the so-called "plasticity" of synaptic neurotransmitter connections that carry information between biological neurons, the connections in ANNs that carry electronic information between simulated neurons have digital weights corresponding to the strength (strength) or weakness (weakness) of a given connection. The weights can be adjusted and tuned based on experience so that the ANN adapts to the input and is able to learn. Summary of the Invention

[0003] According to one embodiment, a method for training an artificial neural network (ANN) is provided. The method includes: storing weight values ​​in an array of resistive processing unit (RPU) devices, wherein the array of RPU devices represents a weight matrix W of an ANN having m rows and n columns by storing the weight values ​​of the weight matrix W as the resistance values ​​of the RPU devices in the array; defining the weight matrix W to have an output dimension smaller than the input dimension, such that the weight matrix W has a rectangular configuration; and converting the weight matrix W from a rectangular configuration to a more square configuration by repeating or cascading the rectangular configuration of the weight matrix W to increase the signal strength of a backward pass signal by: replicating the inputs of the repeated weight elements during a forward recurrent pass; computing and summing the outputs from the repeated weight elements to obtain one output per row; and updating each of the repeated weight elements according to a backward propagated error or, in an alternative embodiment, updating only one of the repeated weight elements by setting all forward values ​​except one to zero during an update pass.

[0004] A non-transitory computer-readable storage medium including a computer-readable program is presented for use in artificial neural network (ANN) training, wherein when executed on a computer, the computer-readable program causes the computer to perform the following steps: storing weight values ​​in an array of resistive processing unit (RPU) devices, wherein the array of RPU devices represents a weight matrix W of the ANN having m rows and n columns by storing the weight values ​​of the weight matrix W as resistance values ​​of the RPU devices in the array; defining the weight matrix W to have an output dimension that is smaller than the input dimension so that the weight matrix W has a rectangular configuration; and converting the weight matrix W from a rectangular configuration to a more square configuration by repeating or cascading the rectangular configuration of the weight matrix W to increase the signal strength of a backward pass signal by the following steps: replicating the inputs of the repeated weight elements during a forward recurrent pass; summing the outputs from the repeated weight elements to obtain one output per row; and updating each of the repeated weight elements according to a backward propagated error or, in the alternative, updating only one of the repeated weight elements by setting all forward values ​​except one to zero during an update pass.

[0005] A system for training an artificial neural network (ANN) is provided. The system includes an array of resistive processing unit (RPU) devices for storing weight values, wherein the array of RPU devices represents a weight matrix W of an ANN having m rows and n columns by storing the weight values ​​of the weight matrix W as resistance values ​​of the RPU devices in the array, and a processor for controlling voltages across the RPU devices in the array, wherein the weight matrix W is defined to have an output dimension smaller than an input dimension such that the weight matrix W has a rectangular configuration, and the weight matrix W is converted from the rectangular configuration to a more square configuration by repeating or cascading the rectangular configuration of the weight matrix W to increase the signal strength of a backward-passed signal.

[0006] It should be noted that exemplary embodiments are described with reference to different subject matters. In particular, some embodiments are described with reference to method-type claims, while other embodiments are described with reference to apparatus-type claims. However, a person skilled in the art will appreciate from the above and following description that, unless otherwise indicated, any combination of features relating to different subject matters, in addition to any combination of features belonging to one type of subject matter, any combination of features relating to different subject matters, in particular any combination of features of method-type claims with features of apparatus-type claims, is also considered to be described in this document.

[0007] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments of the invention, which is to be read in connection with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present invention will be provided in detail in the following description of preferred embodiments with reference to the following drawings, in which:

[0009] Figure 1 is a diagram illustrating an artificial neural network (ANN) implemented in an analog cross-point array of a resistive processing unit (RPU) device according to an embodiment of the present invention;

[0010] Figure 2 shows an exemplary simulated vector-matrix multiplication on an RPU array according to an embodiment of the present invention;

[0011] Figure 3 is a diagram illustrating an exemplary rectangular RPU array in a forward pass according to an embodiment of the present invention, wherein rectangular sub-regions of the RPU array are employed;

[0012] Figure 4 is a diagram illustrating an exemplary square RPU array in a forward pass according to an embodiment of the present invention, wherein the entire square RPU array is employed;

[0013] Figure 5 is a diagram illustrating an exemplary rectangular RPU array in a backward pass according to an embodiment of the present invention, wherein rectangular sub-regions of the RPU array are employed;

[0014] Figure 6 is a diagram illustrating an exemplary rectangular RPU array in a backward pass according to an embodiment of the present invention, wherein the entire square RPU array is employed, and wherein the output dimension m is half the input dimension such that each column is repeated exactly once;

[0015] Figure 7 is a block diagram / flowchart of exemplary equations employed in the forward pass and backward pass according to an embodiment of the present invention;

[0016] Figure 8 A block diagram illustrating components of a computing system including a computing device operative to perform weight repetition on an RPU crossbar array according to an embodiment of the present invention; and

[0017] Figure 9 is an exemplary block diagram / flowchart of a method for increasing signal strength through weight repetition on an RPU crossbar array according to an embodiment of the present invention.

[0018] Throughout the drawings, the same or similar reference numerals denote the same or similar elements. DETAILED DESCRIPTION

[0019] An exemplary embodiment according to the present invention provides increased signal strength of a backward pass signal by weight repetition on a resistive processing unit (RPU) crossbar array. In particular, the weight matrix is ​​modified by repeating or duplicating columns and / or rows of the RPU crossbar array. This is achieved by repeating some weight elements (weight elements) on the resistive device array and accumulating the results of the repeated weight elements (i.e., rows and / or columns) in the digital periphery to increase the output signal strength during the backward pass cycle, which improves accuracy and reduces the requirements for high-resolution analog-to-digital converters (ADCs). As a result, the use of lower-precision ADCs helps increase the power efficiency of analog hardware chips, which provides better noise and boundary management, which in turn improves neural network training performance.

[0020] A crossbar array, also known as a crosspoint array or crossbar array, is a high-density, low-cost circuit architecture used to form a variety of electronic circuits and devices, including artificial neural network architectures, neuromorphic microchips, and ultra-high-density non-volatile memories. The basic crossbar array configuration consists of a set of conductive row lines and a set of conductive column lines that intersect the row lines. The intersection between these two sets of lines is separated by a so-called crosspoint device, which can be formed from a thin-film material.

[0021] Cross-point devices are actually used as ANN weighted connections between neurons. Nanoscale two-terminal devices, such as memristors with "ideal" conductive state switching characteristics, are often used as cross-point devices to simulate synaptic plasticity with high energy efficiency. The conductive state (e.g., resistance) of the ideal memristor material can be changed by controlling the voltage applied between the individual lines of the row and column lines. Digital data can be stored by changing the conductive state of the memristor material at the intersection to achieve a high conductive state or a low conductive state. Memristor materials can also be programmed to maintain two or more different conductive states by selectively setting the conductive state of the material. The conductive state of the memristor material can be read by applying a voltage across the material and measuring the current through the target cross-point device.

[0022] However, ANN training using analog resistive crossbar arrays, such as analog RPU arrays, can be difficult due to analog noise. Furthermore, the training process is limited by the bounded range of the analog-to-digital converters (ADCs) and digital-to-analog converters (DACs) used in the RPU arrays. The ADCs and DACs are used to convert digital inputs to the RPUs into analog signals and to convert outputs from the RPUs back into digital signals, respectively. Analog noise can be reduced by noise management methods that involve augmenting the output signals of rectangular weight matrices.

[0023] Exemplary embodiments of the present invention disclose a method and system that advantageously manages analog noise by repeatedly encoding the weights of a deep neural network (DNN) on a physical analog crossbar array to increase the output signal of the backward-passed signal. Exemplary embodiments of the present invention also disclose a method and system that uses n times a rectangular weight matrix to make the rectangular weight matrix closer to a square, thereby better fitting into a physical square crossbar array. Exemplary embodiments of the present invention also disclose a method and system that repeatedly encodes the weights of a DNN (e.g., a convolutional neural network (CNN)) network layer onto a single analog crossbar array, and then sums (averages) or distributes (replicates) the outputs of the repeated rows and / or columns in the digital periphery to maintain the correct DNN / CNN network architecture. Exemplary embodiments of the present invention also disclose a method and system that repeatedly encodes the weights of a DNN / CNN network layer onto a single analog crossbar array, and then sums (averages) or distributes (replicates) the outputs of the repeated rows and / or columns in the digital periphery to maintain the correct DNN / CNN network architecture, and simultaneously or concurrently updates only one (e.g., randomly selected) or all of the repeated weights.

[0024] Thus, the exemplary methods and systems provide improved neural network training performance. In particular, the weak output signal of the backward pass signal is significantly increased, which reduces the requirements for a precise ADC. In addition, due to repetition, the noise of the analog components is averaged, thereby improving training accuracy.

[0025] It should be understood that the present invention will be described in terms of a given illustrative architecture; however, other architectures, structures, substrate materials, and process features and steps / blocks may vary within the scope of the present invention. It should be noted that for clarity, some features may not be shown in all drawings. This should not be interpreted as limiting any particular embodiment or the scope of the description or claims.

[0026] Various illustrative embodiments of the present invention are described below. For the sake of clarity, not all features of an actual implementation are described in this specification. Of course, it should be understood that in developing any such actual embodiment, many implementation-related decisions must be made to achieve the developer's specific goals, such as complying with system-related and business-related constraints, which will vary from implementation to implementation. Furthermore, it should be understood that such development work can be complex and time-consuming, but is a routine undertaking for those of ordinary skill in the art having the benefit of this disclosure.

[0027] Figure 1 is a diagram illustrating an artificial neural network (ANN) implemented in an analog cross-point array of a resistive processing unit (RPU) device according to an embodiment of the present invention.

[0028] like Figure 1 As shown, each parameter (weight W ij ) is mapped to a single RPU device (RPU ij ), that is, the physical cross-point array 12 of the RPU device. The cross-point array 12 has a series of conductive row lines 14 and a series of conductive column lines 16, and the series of conductive column lines 16 are oriented orthogonal to and intersect the conductive row lines 14, and the intersections of the conductive row lines 14 and the column lines 16 are separated by RPU devices 18, forming a cross-point array 12 of RPU devices 18. Each RPU device 18 may include a first terminal, a second terminal and an active area. The conductive state of the active area identifies the weight value of the RPU device 18, which can be updated / adjusted by applying a signal to the first / second terminal. In addition, a three-terminal (or even more terminal) device can be effectively used as a two-terminal resistive memory device by controlling the additional terminals.

[0029] A matrix W having M rows and N columns is typically mapped to an RPU array having M columns and N rows, so the integration in the figure occurs along the columns of the RPU array, while the summation occurs mathematically along the rows of W. Therefore, the mapping of the mathematical W to the RPU array is actually a transpose. As a result, for ease of display, the mathematical rows of the matrix W are shown as columns of the RPU array. For example, from the top to the bottom of the crosspoint array 12, and from left to right, the RPU device 18 at the intersection of the first conductive row line 14 and the first conductive column line 16 is designated as RPU 11 , the RPU device 18 at the intersection of the first conductive row line 14 and the second conductive column line 16 is designated as RPU 12 , and so on. The mapping of weight parameters in the weight matrix 10 to the RPU devices 18 in the cross-point array 12 follows a similar convention, although it is common practice to interchange the columns and rows of the RPU array for display purposes. For example, the weight W of the weight matrix 10 is i1 RPU mapped to crosspoint array 12 1i , the weight W of the weight matrix 10 i2 RPU mapped to crosspoint array 12 2i ,etc.

[0030] The RPU devices 18 of the crosspoint array 12 serve as weighted connections between neurons in the ANN. The resistance of the RPU devices 18 can be changed by controlling the voltage applied between each conductive row line 14 and the conductive column line 16, and the change in resistance is based on how data is stored in the RPU device 18, such as a high resistance state or a low resistance state. The resistance state of the RPU device 18 is read by applying a voltage and measuring the current through the target RPU device 18. All operations involving weights are performed by the RPU devices 18 completely in parallel.

[0031] In machine learning and cognitive science, ANN-based models are a family of statistical learning models inspired by biological neural networks of animals, particularly the brain. These models can be used to estimate or approximate system and cognitive functions that rely on a large number of inputs and weights of connections that are usually unknown. ANNs are typically implemented as so-called "neuromorphic" systems of interconnected processor elements that act as simulated "neurons" ( Figure 8 The connections in an ANN, which carry electronic messages between simulated neurons, are provided with digital weights corresponding to the strength or weakness of a given connection ( Figure 8 These numerical weights can be adjusted and tuned based on experience, allowing the ANN to adapt to the input and learn. For example, an ANN for handwriting recognition is defined by a set of input neurons that can be activated by pixels from an input image. After being weighted and transformed by a function determined by the network designer, the activation of these input neurons is then passed on to other downstream neurons. This process is repeated until an output neuron is activated. The activated output neuron determines which character is read.

[0032] ANNs can be trained using an incremental or stochastic gradient descent (SGD) process, where backpropagation is used to calculate the weights W ij ) of the error gradient. Backward propagation is performed in three loops, namely the forward loop, the backward loop, and the weight update loop, which are repeated multiple times until the convergence criteria are met. DNN-based models include multiple processing layers that learn data representations with multiple levels of abstraction. For a single processing layer with N input neurons connected to M output neurons, the forward loop involves computing a vector-matrix multiplication (y=Wx), where the vector x of length N represents the activity of the input neurons and the matrix W of size M×N stores the weight values ​​between each pair of input and output neurons. The resulting vector y of length M is further processed by performing nonlinear activation on each resistive storage element and then passed to the next layer.

[0033] Once the information reaches the final output layer, the backward pass loop involves computing an error signal and propagating it backward through the ANN. The backward pass loop on a single layer involves performing a multiplication of the weight matrices (z = W T δ), where the vector δ of length M represents the error calculated by the output neuron, and the vector z of length N is further processed using the derivative of the neuron's nonlinearity and then passed down to the previous layer.

[0034] Finally, in the weight update loop, the weight matrix W is updated by performing the outer product of the two vectors used in the forward and backward pass loops. The outer product of these two vectors is usually expressed as W←W+η(δx T ), where η is the global learning rate.

[0035] During this backward propagation process, all operations performed on the weight matrix W can be implemented using the cross-point array 12 of the RPU device 18 having a corresponding number of m rows and n columns, where the conductance values ​​stored in the cross-point array 12 form the matrix W. In the forward loop, the input vector x is transmitted as a voltage pulse through each of the conductive column lines 16, and the resulting vector y is read as a current output from the conductive row lines 14. Similarly, when a voltage pulse is provided from the conductive row lines 14 as an input to the backward pass loop, the transpose of the weight matrix W is obtained. T Finally, in the update cycle, voltage pulses representing vectors x and δ are provided simultaneously from the conductive column lines 16 and the conductive row lines 14. Thus, each RPU device 18 performs a local multiplication and summation operation by processing the voltage pulses from the corresponding conductive column lines 16 and conductive row lines 14, thereby achieving incremental weight updates.

[0036] The resistance values ​​of RPU devices are restricted to a bounded range with limited and finite state resolution, which limits the range of weights that can be used for ANN training. In addition, the operations performed on the RPU array are analog in nature and are therefore prone to various noise sources. When the input values ​​of the RPU array are small (such as for the backward pass), the output signal y may be masked by noise, resulting in incorrect results. During the training phase, ANN training can involve an SGD process with backpropagation.

[0037] CNN training is performed using batches. Therefore, a batch of input data is selected to be used for training. Using the input map and the convolution kernel, an output map is generated. Generating the output map is generally referred to as a "forward pass". In addition, the method includes using the output map to determine how close or far away the expected character recognition is from the CNN. Such as using gradient descent to determine the error associated with each of the matrices comprising the CNN. Determining the relative error is called a "backward pass". The method also includes modifying or updating the matrix to adjust for the error. Adjusting the convolution kernel based on the output error information and using it to determine the modification of each neural network matrix is ​​called an "update pass".

[0038] Figure 2 An exemplary simulated vector-matrix multiplication on an RPU array according to an embodiment of the present invention is shown.

[0039] The analog vector matrix multiplication 100 involves a set of digital input values ​​(δ) 110, wherein each of the digital input values ​​(δ) 110 is represented by a corresponding analog signal pulse width 120. The analog signal pulse width 120 is provided as an input to the array, and the resulting current signal is input to an operational amplifier (op-amp) integration circuit 130 having an operational amplifier 131, wherein a capacitor (C int ) 132 is connected (across) the inverting input of op-amp 131 and the output of op-amp 131. The non-inverting input of op-amp 131 is grounded. The output of op-amp 131 is also connected to the input of analog-to-digital converter (ADC) 140. ADC 140 outputs signal y1, which represents the (digitized) result of analog vector-matrix multiplication 100 on the RPU array.

[0040] For the full integration time, analog noise accumulates at op amp 131. When the input value (δ) 110 becomes too small (e.g., such as for the backward pass), the output signal is masked by the noise integrated over the cycle (SNR ~ 0) and produces incorrect results.

[0041] Even though the actual pulse duration is much shorter than the full integration time, ADC 140 waits for a full cycle to evaluate the analog output from op-amp 131. It is desirable to reduce analog noise. Figures 3 to 6 Methods and systems are shown for managing noise by providing symmetrical signal strength between forward and reverse pass signals.

[0042] Figure 3 is a diagram illustrating an exemplary rectangular RPU array in a forward pass according to an embodiment of the present invention, wherein rectangular sub-regions of the RPU array are employed.

[0043] Each circle 202 represents a separate digital input x to the RPU hardware system 200. For example, in the forward loop pass, the digital input x (or 202) is provided to the mth row of the matrix W, and when received by the RPU array 200, the digital input 202 is designated as the digital RPU input x' (or 204). The digital RPU input 204 is fed into the noise / boundary management unit or component 210. The vector-matrix multiplication performed on the RPU array 225 is analog in nature and is therefore prone to various noise sources. Therefore, the noise / boundary management unit or component 210 performs noise reduction operations. The digital-to-analog converter (labeled "DA converter 212") provides the digital RPU input x' (or 204) as input to the RPU array 225 as an analog pulse width 215. The RPU array 225 includes a first region 230 and a second region 235. The first region 230 is a rectangular region that is being used, while the second region 235 is an unused region. The term "used" means that the RPU is loaded with a conductance corresponding to its weight. The (analog) output 240 from the RPU array 230 is converted into a vector of digital RPU outputs y' (or 260) by an analog-to-digital converter (labeled "AD converter 250"). The digital RPU output 260 is fed to another noise / boundary management unit or component 270. Moreover, the result of the vector-matrix multiplication is an analog voltage, so the result is bounded by the signal limits imposed by the circuit. Therefore, the noise / boundary management unit or component 270 performs noise reduction operations to ensure that the result at the output of the RPU array 230 is always within the range of an acceptable voltage swing.

[0044] As a result, the output capacitors of the resistive crossbar elements have a finite size (resulting in a finite output bound b), and due to the finite ADC resolution, analog output signals close to zero are set to zero. Therefore, if the analog output of the RPU array 230 is too small, the digital output can be all zero. This effect is undesirable when the ADC resolution is small (e.g., when the output bound b remains constant while the ADC bin size becomes larger). If the weight matrix encoded onto the RPU array 230 is not square (e.g., Figure 3 This effect is particularly undesirable if the forward and backward directions have, on average, very different average signal strengths. For example, in a 10-class classification network, the last fully connected layer is typically of the order of 1000×10. Therefore, on average, there is at least times the signal. If the backward signal is too small (e.g., smaller than the minimum ADC bin size), the error is set to zero and learning fails. For a symmetric RPU (e.g., the same hardware specifications for forward and backward, such as ADC resolution and output bounds), this effect is undesirable. Figure 4Solutions that alleviate these problems are shown.

[0045] Figure 4 is a diagram illustrating an exemplary square RPU array in a forward pass according to an embodiment of the present invention, where the entire square RPU array is employed.

[0046] For the sake of clarity, the Figure 3 Similar components. RPU hardware system 200' and RPU hardware system 200 ( Figure 3 ) is that the weight matrix W (225) has been replicated k times so that additional weight elements are employed in order to convert or modify the rectangular configuration of the weight matrix W (225) into a close or approximately or substantially square configuration. Note that the RPU array (225) is always physically square. However, if the weight matrix has a rectangular shape and assuming that the number of columns of the weight matrix fits into the input dimensions of the RPU array, only a rectangular sub-region of the RPU array that is physically square in shape is used. As mentioned above, the term "used" means that the RPU is loaded with a conductance corresponding to its weights. Therefore, in Figure 3 In the example, the second region 235 (including several rows and / or columns of RPUs) is not used. Figure 4 In the embodiment of the present invention, the size of the weight matrix is ​​increased by adding more rows and / or columns of the RPU 405 to convert or modify the rectangular configuration of the weight matrix W (225) into a substantially or approximately square configuration. The duplicated or repeated rows and / or columns 405 make the rectangular weight matrix 225 more square-like. The duplicated or repeated rows and / or columns are designated as 280 after the noise / boundary management unit 270 and are summed (282), resulting in one output per row. All or only one of the repeated weights 280 is updated. If only one is updated, the repeated weights 290 can be randomly selected or sequentially selected. Therefore, in the second embodiment, only a subset of all repeated weights are updated simultaneously or concurrently.

[0047] The system knows how many rows / columns to add to achieve an approximately or substantially square configuration by counting the number of repetitions of all weight matrices. To calculate the number of repetitions of all weight matrices, the system can simply divide the input dimension N (number of columns) of the matrix W by the output dimension M (number of rows) of the matrix W and take the largest integer, such as r = floor(N / M). This is the number of how often all rows / columns are repeated. For example, in the case of M = 250 and N = 512, then r = 2, and the resulting weight matrix is ​​of size N = 512, and M = 250*r = 500. This method only makes the weight matrix approximately square.

[0048] Therefore, in order to solve the above Figure 3For the problems mentioned above, row / column 405 is added to provide symmetry for the signal strength of the forward pass and the backward pass. Note that the signal strength of the backward pass does not need to exactly match that of the forward pass. However, if it is assumed that the input dimension is similar to the size of the RPU array, the signal strength of the backward pass is maximized. This will prevent the system from adding more rows than necessary, since the physical RPU layout is substantially or approximately square.

[0049] Specifically, assuming a square-sized RPU crossbar array, the exemplary embodiment enables replicating rows and / or columns to make the rectangular weight matrix W more square. In other words, physically, the RPU array is always square. However, as Figure 3 shown, only a rectangular sub-region of the RPU array is actually used. Therefore, the exemplary embodiment can use more available crosspoints and then increase the output / input processing digitally by adding the repeated columns together.

[0050] For example, assuming the size of W is m×n, in many DNN networks, m << n. For example, the output dimension is much smaller than the input dimension.

[0051] The exemplary embodiment constructs or forms a larger matrix of size km×n and For example, W is replicated k times and cascaded.

[0052] In the forward pass: Then, for example, the replicated outputs are added together (e.g., in the digital periphery after the ADC 250).

[0053] In the backward process, as will be described in detail below with reference to Figure 5 and Figure 6 :

[0054] where where j = 0,..., k - 1.

[0055] Therefore, the corresponding new incremental inputs are copied from the original delta.

[0056] Generally, the number of replications k is selected such that mk ≈ n (while not exceeding the physical size limit of the crossbar array). In addition, ensure that k is less than b / w max , where b is the output boundary (about 12) and w max is the maximum weight (about 0.6). Therefore, k is maximized to achieve mk ≈ n, but not greater than b / w max (about 20).

[0057] In the update pass, do not change (except for possible learning rate adaptation), or optionally, change the error Random score Set to zero.

[0058] Figure 5 is a diagram illustrating an exemplary rectangular RPU array in a backward pass according to an embodiment of the present invention, wherein rectangular sub-regions of the RPU array are used.

[0059] In the RPU hardware system 500, the vector-matrix multiplications performed on the RPU array 525 are analog in nature and therefore susceptible to various noise sources. Therefore, the noise / boundary management unit 510 performs noise reduction operations. The digital-to-analog converter (labeled "DA converter 512") converts the digital RPU input x' (or 560; Figure 6 ) is provided as input to the RPU array 525 as an analog pulse width 515. The RPU array 525 includes a first region 530 and a second region 535. The first region 530 is a rectangular region that is being used, while the second region 535 is an unused region. The term "used" means that the RPU is loaded with a conductance corresponding to its weight. The (analog) output 240 from the RPU array 530 is converted into a vector of digital RPU outputs y' (or 260) by an analog-to-digital converter (labeled "AD converter 250"). The digital RPU output 260 is fed to another noise / boundary management unit 270. Moreover, the result of the vector-matrix multiplication is an analog voltage, and therefore, the result is bounded by the signal limits imposed by the circuit. Therefore, the noise / boundary management unit 270 performs a noise reduction operation to ensure that the result at the output of the RPU array 530 is always within an acceptable range of voltage swings. Digital output 272 is the output from the noise / boundary management unit 270.

[0060] As described above, the output capacitors of the resistive crossbar elements have a finite size (resulting in a finite output bound b), and due to the finite ADC resolution, analog output signals close to zero are set to zero. Therefore, if the analog output of the RPU array 530 is too small, the digital output can be all zero. This effect is undesirable when the ADC resolution is small (e.g., when the ADC bin size becomes larger while the output bound b remains constant). If the weight matrix encoded onto the RPU array 530 is not square (e.g., Figure 5 This effect is particularly undesirable if the forward and backward directions have very different average signal strengths, as shown. Then, on average, the forward and backward directions have very different average signal strengths. If the backward signal is too small (e.g., smaller than the smallest ADC bin size), the error is set to zero and learning fails. For symmetric RPUs (e.g., identical hardware specifications for forward and backward directions, such as ADC resolution and output bounds), this effect is undesirable. Figure 6Solutions that alleviate these problems are shown.

[0061] Figure 6 is a diagram illustrating an exemplary rectangular RPU array in a backward pass according to an embodiment of the present invention, wherein the entire square RPU array is employed, and wherein the output dimension m is half the input dimension such that each column is repeated exactly once.

[0062] In the backward loop pass, the digital input x (or 550) is provided to the n columns of the matrix W (525). The RPU hardware system 500' is connected to the RPU hardware system 500 ( Figure 5 ) is that the weight matrix W (525) has been replicated k times so that additional weight elements are employed in order to convert or modify the rectangular configuration of the weight matrix W (525) into a substantially or approximately square configuration. Note that the weight matrix W (525) is always physically square. However, only rectangular sub-regions of the RPU array that are physically square in shape are used. As mentioned above, the term "used" means that the RPUs are loaded with conductances corresponding to their weights. Therefore, in Figure 5 In the example, the second region 535 (including several rows and / or columns of RPUs) is not used. Figure 6 In the embodiment of the present invention, the size of the weight matrix is ​​increased by adding more rows and / or columns of the RPU 605 to convert or modify the rectangular configuration of the weight matrix W (525) into a substantially or approximately square configuration. The duplicated or repeated rows and / or columns 605 make the rectangular weight matrix 525 more square-like. The duplicated or repeated rows and / or columns are designated as 560 before the noise / boundary management unit 510.

[0063] Figure 7 is a block diagram / flowchart of an exemplary equation 700 employed in the forward pass and the backward pass according to an embodiment of the present invention.

[0064] In summary, exemplary embodiments disclose a method and system that copies / repeats / concatenates a rectangular weight matrix of a DNN onto a physical analog crossbar and copies the inputs to the repeated weight elements during a forward pass. Exemplary embodiments also disclose a method and system that copies / repeats / concatenates a rectangular weight matrix of a DNN onto a physical analog crossbar and averages or sums the output calculations from the repeated weight elements to obtain one output per row of the original weight matrix. Exemplary embodiments also disclose a method and system that copies / repeats / concatenates a rectangular weight matrix of a DNN onto a physical analog crossbar and updates each repeated weight based on the backpropagated error. Exemplary embodiments also disclose a method or system that copies / repeats / concatenates a rectangular weight matrix of a DNN onto a physical analog crossbar and updates only one of the repeated weight elements by setting all backward increments (or forward values) except one to zero during an update pass.

[0065] Figure 8 A block diagram of components of a system 900 including a computing device 905 is depicted. It should be understood that Figure 8 This merely provides an illustration of one implementation and does not imply any limitation with respect to the environments in which different embodiments may be implemented.Many modifications to the described environments are possible.

[0066] Computing device 905 includes a communications fabric 902 that provides communications between a computer processor 904, memory 906, persistent storage 908, a communications unit 910, and input / output (I / O) interfaces 912. Communications fabric 902 can be implemented using any architecture designed to transfer data and / or control information between a processor (such as a microprocessor, communications and network processors, etc.), system memory, peripheral devices, and any other hardware components within the system. For example, communications fabric 902 can be implemented using one or more buses.

[0067] Memory 906, cache memory 916, and permanent storage 908 are computer-readable storage media. In this embodiment, memory 906 includes random access memory (RAM) 914. In general, memory 906 can include any suitable volatile or non-volatile computer-readable storage media.

[0068] In some embodiments of the present invention, the deep learning program 925 is included and operated by the neuromorphic chip 922 as a component of the computing device 905. In other embodiments, the deep learning program 925 is stored in a persistent storage device 908 for execution by the neuromorphic chip 922 via one or more memories of the memory 906 in conjunction with one or more of the corresponding computer processors 904. In this embodiment, the persistent storage device 908 includes a magnetic hard drive. As an alternative to or in addition to the magnetic hard drive, the persistent storage device 908 may include a solid-state hard drive, a semiconductor memory device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0069] The media used by persistent storage 908 also may be removable. For example, a removable hard drive may be used for persistent storage 908. Other examples include optical and magnetic disks, thumb drives, and smart cards, which are inserted into a drive to transfer to another computer-readable storage medium that is also part of persistent storage 908.

[0070] In some embodiments of the invention, neuromorphic chip 922 is included in computing device 905 and connected to communication fabric 902 .

[0071] In these examples, communication unit 910 provides communication with other data processing systems or devices, including resources of a distributed data processing environment. In these examples, communication unit 910 includes one or more network interface cards. Communication unit 910 can provide communication using one or both of physical and wireless communication links. Deep learning program 925 can be downloaded to persistent storage device 908 via communication unit 910.

[0072] The I / O interface 912 allows for input and output of data with other devices that may be connected to the computing system 900. For example, the I / O interface 912 may provide a connection to external devices 918, such as a keyboard, a keypad, a touch screen, and / or some other suitable input device. External devices 918 may also include portable computer-readable storage media, such as, for example, a thumb drive, a portable optical or magnetic disk, and a memory card.

[0073] Display 920 provides a mechanism for displaying data to a user and may be, for example, a computer monitor.

[0074] Figure 9 is an exemplary block diagram / flowchart of a method for increasing signal strength through weight repetition on an RPU crossbar array according to an embodiment of the present invention.

[0075] At block 1010 , a rectangular weight matrix of a DNN is copied or repeated or concatenated onto a physical analog crossbar.

[0076] At block 1020 , the input is copied to repeated weight elements during a forward pass.

[0077] At block 1030, the output calculations from the repeated weight elements are averaged or summed, resulting in one output per row of the original weight matrix.

[0078] At block 1040, each repeated weight is updated according to the back-propagated error, or in the alternative, only one of the repeated weight elements is updated by setting all backward increments (or forward values) except one to zero during the update pass.

[0079] As used herein, the terms "data," "content," "information," and similar terms may be used interchangeably to refer to data that can be captured, sent, received, displayed, and / or stored in accordance with various example embodiments. Thus, the use of any such terms should not be taken to limit the spirit and scope of the present disclosure. Furthermore, where a computing device is described herein as receiving data from another computing device, the data may be received directly from the other computing device or may be received indirectly via one or more intermediate computing devices, such as one or more servers, repeaters, routers, network access points, base stations, and the like.

[0080] To provide for interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) that the user can use to provide input to the computer. Other types of devices may also be used to provide for interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input.

[0081] The present invention may be a system, method, and / or computer program product. The computer program product may include (one or more) computer-readable storage media having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to perform various aspects of the present invention.

[0082] Computer readable storage medium can be a tangible device that can retain and store the instructions used by the instruction execution device.Computer readable storage medium can be, for example, but not limited to, electronic storage device, magnetic storage device, optical storage device, electromagnetic storage device, semiconductor storage device or any suitable combination of the foregoing.A non-exhaustive list of more specific examples of computer readable storage medium includes the following: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device such as a punch card or a raised architecture in a groove with instructions recorded thereon, and any suitable combination of the foregoing.Computer readable storage medium as used herein should not be interpreted as a temporary signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by lines.

[0083] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0084] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, wherein the programming language includes object-oriented programming languages, such as Smalltalk, C++, and conventional procedural programming languages, such as "C" programming language or similar programming languages. The computer-readable program instructions can be performed completely on the user's computer, partly on the user's computer, performed as an independent software package, partly on the user's computer and partly on a remote computer, or performed completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network (including local area network (LAN) or wide area network (WAN)), or can be connected to an external computer (for example, using an Internet service provider through the Internet). In certain embodiments, in order to perform various aspects of the present invention, the electronic circuit comprising for example programmable logic circuit, field programmable gate array (FPGA) or programmable logic array (PLA) can perform computer-readable program instructions to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions.

[0085] Aspects of the present invention are described herein with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present invention. It will be understood that each block of the flowcharts and / or block diagrams and combinations of blocks in the flowcharts and / or block diagrams can be implemented by computer-readable program instructions.

[0086] These computer-readable program instructions can be provided to at least one processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing apparatus create a device for implementing the functions / actions specified in one or more blocks or modules of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can direct the computer, programmable data processing apparatus, and / or other device to operate in a specific manner, such that the computer-readable storage medium having the instructions stored therein comprises an article of manufacture, which includes instructions for implementing aspects of the functions / actions specified in one or more blocks or modules of the flowchart and / or block diagram.

[0087] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device, so that a series of operation blocks / steps are performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more blocks or modules of the flowchart and / or block diagram.

[0088] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flow chart or block diagram can represent a module, segment or portion of an instruction, which includes one or more executable instructions for realizing a specified logical function. In some alternative implementations, the functions noted in the frame may not occur in the order noted in the figure. For example, the two frames shown in succession can actually be performed substantially simultaneously, or these frames can sometimes be performed in reverse order, depending on the functions involved. It will also be noted that each block of the block diagram and / or flow chart illustration and the combination of the blocks in the block diagram and / or flow chart illustration can be implemented by a dedicated hardware-based system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0089] References in the specification to "one embodiment" or "an embodiment" and other variations of the present principles mean that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment of the present principles. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" and any other variations in various places throughout the specification are not necessarily all referring to the same embodiment.

[0090] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," the use of any of the following " / ," "and / or," and "at least one of" is intended to encompass selection of only the first listed option (A), or only the second listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to include selection of only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended to many of the listed items, as will be apparent to one of ordinary skill in this and related arts.

[0091] Having described preferred embodiments of the system and method for increasing signal strength by weight repetition on an RPU crosspoint array (which are intended to be illustrative and not limiting), it is noted that modifications and variations may be made by those skilled in the art in light of the above teachings. Therefore, it should be understood that changes may be made in the specific embodiments described that are within the scope of the invention as outlined by the appended claims. Having thus described various aspects of the invention with the details and particularity required by the patent laws, what is claimed and desired to be protected by Letters Patent is set forth in the appended claims.

Claims

1. A method for training an artificial neural network (ANN), the method comprising: Storing the weight values ​​in an array of resistive processing unit (RPU) devices, wherein the array of RPU devices represents a weight matrix W of an ANN having m rows and n columns by storing the weight values ​​of the weight matrix W as resistance values ​​of the RPU devices in the array; defining the weight matrix W to have an output dimension smaller than an input dimension such that the weight matrix W has a rectangular configuration; as well as The weight matrix W is converted from a rectangular configuration to a more square configuration by repeating or cascading the rectangular configuration of the weight matrix W to increase the signal strength of the backward passed signal by the following steps: Copy the inputs of repeated weight elements during the forward recurrent pass; summing the outputs from the repeated weight elements to obtain one output per row; as well as Each of the repeated weight elements is updated according to the back-propagated error.

2. The method according to claim 1, wherein The weight elements are m rows and n columns of the weight matrix W.

3. The method according to claim 1, wherein The repeated weight elements added to the weight matrix W provide symmetrical signal strength between the forward pass signal and the backward pass signal.

4. The method according to claim 1, wherein In the forward loop pass, digital RPU input is fed into a first noise / margin measurement component and digital-to-analog converter (DAC) before being received by the array of RPU devices.

5. The method according to claim 4, wherein In the forward loop pass, the output calculations from the repeated weight elements are summed after being processed by an analog-to-digital converter (ADC).

6. The method according to claim 5, wherein: In the forward loop pass: as well as in is the immediate output signal vector from the modified RPU array / weight matrix, is the modified weight matrix stored on the RPU array, x is the input signal vector, y i is the i-th element of the original output vector, and k is the number of copies.

7. The method according to claim 1, wherein In the backward loop pass: in where j = 0, ..., k-1 in is the transposed modified weight matrix stored on the RPU array, is the modified error signal vector, which is input to the modified RPU array / weight matrix during the backward pass, and is the i+jkth element of the modified error signal.

8. A non-transitory computer-readable storage medium comprising a computer-readable program for artificial neural network (ANN) training, wherein: The computer readable program, when executed on a computer, causes the computer to perform the following steps: Storing the weight values ​​in an array of resistive processing unit (RPU) devices, wherein the array of RPU devices represents a weight matrix W of an ANN having m rows and n columns by storing the weight values ​​of the weight matrix W as resistance values ​​of the RPU devices in the array; defining the weight matrix W to have an output dimension smaller than an input dimension such that the weight matrix W has a rectangular configuration; as well as The weight matrix W is converted from a rectangular configuration to a more square configuration by repeating or cascading the rectangular configuration of the weight matrix W to increase the signal strength of the backward passed signal by the following steps: Copy the inputs of repeated weight elements during the forward recurrent pass; summing the outputs from the repeated weight elements to obtain one output per row; as well as Each of the repeated weight elements is updated according to the back-propagated error.

9. The non-transitory computer-readable storage medium of claim 8, wherein: The weight elements are m rows and n columns of the weight matrix W.

10. The non-transitory computer-readable storage medium of claim 8, wherein: The repeated weight elements added to the weight matrix W provide symmetrical signal strength between the forward pass signal and the backward pass signal.

11. The non-transitory computer-readable storage medium of claim 8, wherein: In the forward loop pass, digital RPU input is fed into a first noise / margin measurement component and digital-to-analog converter (DAC) before being received by the array of RPU devices.

12. The non-transitory computer-readable storage medium of claim 11, wherein: In the forward loop pass, the output calculations from the repeated weight elements are summed after being processed by an analog-to-digital converter (ADC).

13. The non-transitory computer-readable storage medium of claim 12, wherein: In the forward loop pass: as well as in is the immediate output signal vector from the modified RPU array / weight matrix, is the modified weight matrix stored on the RPU array, x is the input signal vector, y i is the i-th element of the original output vector, and k is the number of copies.

14. The non-transitory computer-readable storage medium of claim 8, wherein: In the backward loop pass: in where j = 0, ..., k-1 in is the transposed modified weight matrix stored on the RPU array, is the modified error signal vector, which is input to the modified RPU array / weight matrix during the backward pass, and is the i+jkth element of the modified error signal.

15. A system for artificial neural network (ANN) training, the system comprising: an array of resistive processing unit (RPU) devices for storing weight values, wherein the array of RPU devices represents a weight matrix W of an ANN having m rows and n columns by storing the weight values ​​of the weight matrix W as resistance values ​​of the RPU devices in the array; as well as a processor configured to control a voltage across the RPU devices in the array, wherein the processor: defining the weight matrix W to have an output dimension smaller than an input dimension, such that the weight matrix W has a rectangular configuration; and The weight matrix W is converted from a rectangular configuration to a more square configuration by repeating or cascading the rectangular configuration of the weight matrix W to increase the signal strength of the backward passed signal.

16. The system according to claim 15, wherein: Increase the signal strength by following these steps: Copy the inputs of repeated weight elements during the forward recurrent pass; summing the outputs from the repeated weight elements to obtain one output per row; as well as Each of the repeated weight elements is updated according to the back-propagated error, or alternatively only one of the repeated weight elements is updated by setting all forward values ​​except one to zero during an update pass.

17. The system according to claim 16, wherein: The repeated weight elements added to the weight matrix W provide symmetrical signal strength between the forward pass signal and the backward pass signal.

18. The system according to claim 17, wherein: In the forward loop pass, digital RPU input is fed into a first noise / margin measurement component and digital-to-analog converter (DAC) before being received by the array of RPU devices.

19. The system according to claim 18, wherein: In the forward loop pass, the output calculations from the repeated weight elements are summed after being processed by an analog-to-digital converter (ADC).

20. The system of claim 16, wherein: Randomly select a repeated weight element to update.

21. A method for training an artificial neural network (ANN), the method comprising: Storing the weight values ​​in an array of resistive processing unit (RPU) devices, wherein the array of RPU devices represents a weight matrix W of an ANN having m rows and n columns by storing the weight values ​​of the weight matrix W as resistance values ​​of the RPU devices in the array; defining the weight matrix W to have an output dimension smaller than an input dimension such that the weight matrix W has a rectangular configuration; as well as The weight matrix W is converted from a rectangular configuration to a more square configuration by repeating or cascading the rectangular configuration of the weight matrix W to increase the signal strength of the backward passed signal by the following steps: Copy the inputs of repeated weight elements during the forward recurrent pass; summing the outputs from the repeated weight elements to obtain one output per row; as well as Only one of the repeated weight elements is updated by setting all forward values ​​except one to zero during an update pass.

22. The method according to claim 21, wherein The repeated weight elements added to the weight matrix W provide symmetrical signal strength between the forward pass signal and the backward pass signal.

23. The method according to claim 21, wherein In the forward loop pass, digital RPU input is fed into a first noise / margin measurement component and digital-to-analog converter (DAC) before being received by the array of RPU devices.

24. The method according to claim 23, wherein In the forward loop pass, the output calculations from the repeated weight elements are summed after being processed by an analog-to-digital converter (ADC).

25. A non-transitory computer-readable storage medium comprising a computer-readable program for artificial neural network (ANN) training, wherein: The computer readable program, when executed on a computer, causes the computer to perform the following steps: Storing the weight values ​​in an array of resistive processing unit (RPU) devices, wherein the array of RPU devices represents a weight matrix W of an ANN having m rows and n columns by storing the weight values ​​of the weight matrix W as resistance values ​​of the RPU devices in the array; defining the weight matrix W to have an output dimension smaller than an input dimension such that the weight matrix W has a rectangular configuration; as well as The weight matrix W is converted from a rectangular configuration to a more square configuration by repeating or cascading the rectangular configuration of the weight matrix W to increase the signal strength of the backward passed signal by the following steps: Copy the inputs of repeated weight elements during the forward recurrent pass; summing the outputs from the repeated weight elements to obtain one output per row; as well as Only one of the repeated weight elements is updated by setting all forward values ​​except one to zero during an update pass.

Images