Worst-Case Noise and Bounce Management for Crossbar Arrays

Abstract

A technique for noise and bound management for DNN training on an RPU crossbar array using scaling factors based on a worst-case scenario is provided. The noise and bound management method includes obtaining input vector values ​​x for an analog crossbar array of an RPU device, where a weight matrix is ​​mapped to the analog crossbar array of an RPU device; scaling the input vector values ​​x based on a worst-case scenario to provide scaled input vector values ​​x' for use as input to the analog crossbar array of an RPU device, where the worst-case scenario includes the maximum expected weight of the weight matrix multiplied by the sum of absolute values ​​from the input vector values ​​x.

Description

[Technical field]

[0001] The present invention relates to training deep neural networks (DNNs) with analog crossbar arrays of resistive processing unit (RPU) devices, and more particularly to techniques for noise and bounds management of DNN training in RPU crossbar arrays using scaling factors based on worst-case scenarios that result in effective run-time improvements. [Background technology]

[0002] Deep Neural Networks (DNNs) can be embodied in analog crossbar arrays of memory devices, such as resistive processing units (RPUs). DNN-based models have been used for a variety of different cognitive-based tasks, such as object and speech recognition, and natural language processing. In performing these tasks, training of neural networks is necessary to provide a high level of accuracy. However, the computations performed in the RPU array are analog in nature and are therefore prone to various noise sources. When the input values ​​to the RPU array are small (e.g., backward cycle pass), the output signal can be swamped by noise, resulting in inaccurate results.

[0003] In addition, digital-to-analog converters (DACs) and analog-to-digital converters (ADCs) are used to convert the digital inputs to the RPU array into analog signals and the outputs from the RPU array back into digital signals, respectively. Thus, the training process is also limited by the bounded ranges of the DACs and ADCs used for the array.

[0004] For DNN training in an RPU array, bound management is important, especially when setting weights according to automatic weight scaling. In automatic weight scaling, the available resistance state resources of the RPU devices in the RPU array are optimally mapped to weight ranges (resistance values) that are useful for DNN training by scaling the bounded weight ranges of the RPU devices according to the size of the RPU array.

[0005] The conventional approach is to identify the maximum value (m) of the input vector and scale the input values ​​to the RPU array by that maximum value (m) to get the best analog noise performance (noise management). Saturation at the output of the RPU array is eliminated to manage the bounds (bound management) by reducing the values ​​at which the input signals to the RPU array are shaped.

[0006] However, beyond the bound, repeated computations are required using downscaled inputs to reach the output threshold. Although very effective in solving the problem of increased test error due to automatic weight scaling, this iterative downscaling approach undesirably incurs a cost, namely variable run-time.

[0007] Therefore, effective noise and bounds management techniques with improved execution times are desirable. Summary of the Invention [Means for solving the problem]

[0008] The present invention provides techniques for noise and bounds management for deep neural network (DNN) training in a crossbar array of a resistive processing unit (RPU) using scaling factors based on a worst case scenario that provides effective runtime improvement. In one aspect of the present invention, a method of noise and bounds management is provided, the method comprising obtaining input vector values ​​x of an analog crossbar array of an RPU device, where a weight matrix is ​​mapped to the analog crossbar array of an RPU device; and scaling the input vector values ​​x based on a worst case scenario to provide scaled input vector values ​​x' for use as input to the analog crossbar array of an RPU device, where the worst case scenario is 、 Assumed maximal weight of the weight matrix (ω) multiplied by the total input variable (s) calculated by calculating the absolute values ​​of all the input vector values, ωs Includes.

[0009] For example, the absolute maximum input value x mx can be calculated from the input vector values ​​x, and a suggested scaling factor σ is

number

[0010] A more complete understanding of the present invention, as well as further features and advantages of the present invention, will be obtained by reference to the following detailed description and drawings. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating a deep neural network (DNN) implemented in an analog crossbar array of a resistive processing unit (RPU) device in accordance with one embodiment of the present invention. [Diagram 2] FIG. 2 illustrates an exemplary methodology for noise and bounds management in accordance with one embodiment of the present invention. [Diagram 3] FIG. 3 is a diagram illustrating an exemplary implementation of the noise and bounds management technique of the present invention in a forward cycle operation, in accordance with one embodiment of the present invention. [Figure 4] FIG. 4 is a diagram illustrating an exemplary implementation of the noise and bounds management technique of the present invention in a backward cycle operation, in accordance with one embodiment of the present invention. [Diagram 5] FIG. 5 is a diagram illustrating an exemplary alternative methodology for noise and bounds management in accordance with one embodiment of the present invention. [Figure 6] FIG. 6 illustrates an exemplary apparatus for performing one or more of the methodologies presented herein in accordance with one embodiment of the present invention. [Figure 7] FIG. 7 is a diagram illustrating a cloud computing environment according to one embodiment of the present invention. [Figure 8] FIG. 8 is a diagram illustrating abstraction model layers according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] As provided above, existing noise and bound management techniques include using the maximum value (m) in the input vector to scale the input value to the RPU array to obtain the best analog noise performance (noise management). To manage the bounds, the value at which the input signal to the RPU array is formed is repeatedly decreased until an output threshold is reached, thereby eliminating saturation at the output of the RPU array (bound management). However, doing so undesirably results in run-time delays.

[0013] Advantageously, noise and bounds management techniques for deep neural network (DNN) training in an analog RPU crossbar array are provided herein when the dynamic range of the (noisy) analog system is limited and run time must be minimized. That is, as described in detail below, the inventive approach is to scale the input signal to the RPU array by considering the worst case scenario (i.e., the maximum possible output of any weight matrix given a particular input), which is used to estimate the scaling factor that is required to fit the input signal into the limited dynamic range of the analog crossbar system.

[0014] As shown in FIG. 1, the DNN can be embodied in an analog crossbar array of an RPU device, where each parameter (weight W ij ) is a single RPU device on the hardware (RPU ij), i.e., the physical crossbar array 104 of the RPU device 110. The crossbar array 104 has a (first) set of conductive row wires 106 and a (second) set of conductive column wires 108 oriented orthogonal and crossing to the set of conductive row wires 106 (see FIG. 1). The crossing points of the set of conductive row wires 106 and the set of conductive column wires 108 are separated by RPU devices 110 forming the crossbar array 104 of the RPU device 110. Each RPU device 110 may comprise an active area between two electrodes (i.e., a two-terminal device). The conductive state of the active area identifies the weight value of the RPU device 110, which can be updated / adjusted by application of programming signals to the electrodes.

[0015] Each RPU device 110 (RPU ij ) is uniquely identified based on its location, i.e., its position in the i-th row and j-th column of the crossbar array 104. For example, going from top to bottom and left to right across the crossbar array 104, the RPU device 110 at the intersection of the first conductive row wire 106a and the first conductive column wire 108a is the RPU 11 and the RPU device 110 at the intersection of the first conductive row wire 106a and the second conductive column wire 108b is designated as an RPU 12 The mapping between the weight parameters of the weight matrix 102 and the RPU device 110 in the crossbar array 104 follows the same rule. For example, the weights W i1 is a crossbar array of 104 RPUs. i1 and the weights W i2 is a crossbar array of 104 RPUs. i2 , and so on.

[0016] The RPU device 110 of the crossbar array 104 functions as a weighted connection between neurons in the DNN. The resistance of the RPU device 110 can be changed by controlling the voltage applied between individual wires from the multiple sets of conductive row wires 106 and conductive column wires 108. Changing the resistance value of the RPU device 110 is how data is stored in the crossbar array 104, for example, based on the high resistance state or low resistance state of the RPU device 110. The resistance state (high or low) of the RPU device 110 is read by applying (reading) a voltage to the corresponding wire from the multiple sets of conductive row wires 106 and conductive column wires 108 and measuring the current passing through the target RPU device 110. All operations involving weights are performed by the RPU device 110 in full parallel.

[0017] In machine learning and cognitive science, DNN-based models are a family of statistical learning models inspired by biological neural networks of animals, especially the brain. These models can be used to estimate or approximate systems and cognitive functions that depend on a large number of inputs and connection weights that are generally unknown. DNNs are often embodied as so-called "neuromorphic" systems of interconnected processor elements that act as simulated "neurons" that exchange "messages" with each other in the form of electronic signals. The connections in a DNN that transmit electronic messages between the simulated neurons are provided with numerical weights that correspond to the strength or weakness of a given connection. These numerical weights can be adjusted and tuned based on experience, so that the DNN can adapt to the inputs and learn. For example, a DNN for image classification is defined by a set of input neurons that are activated by pixels of the input image. After being weighted and transformed by a function determined by the network's designer, the activations of these input neurons are passed on to other downstream neurons. This process is repeated until an output neuron is activated. The activated output neurons determine the classification of the image.

[0018] DNN training involves the weights W ij This can be done using a process, such as stochastic gradient descent (SGD), in which the error gradient of the weights is calculated using backpropagation. Backpropagation is done in three cycles: a forward cycle, a backward cycle, and a weight update cycle, which are repeated multiple times until a convergence criterion is met.

[0019] DNN-based models consist of multiple processing layers that learn representations of data with multiple levels of abstraction. In the case of a single processing layer with N input neurons connected to M output neurons, the forward cycle consists of vector-matrix multiplications (y = Wx ), where a vector x of length N represents the activity of the input neurons, and a matrix W of size M×N stores the weight values ​​between each pair of input and output neurons. The resulting vector y of length M is further processed by performing a nonlinear activation for each of the resistive memory elements and then passed to the next layer.

[0020] Once the information reaches the final output layer, the backward cycle involves computing the error signal and backpropagating the error signal through the DNN. The backward cycle in a single layer also involves computing the weight matrix (z=W t The method involves vector-matrix multiplication on the transpose (swapping each row with its corresponding column) of n (δ), where the length M vector δ represents the error computed by the output neuron, and the length N vector z is further processed with the derivative of the neuron's nonlinearity and then passed to the previous layer.

[0021] Finally, in the weight update cycle, the weight matrix W is updated by taking the cross product of the two vectors used in the forward and backward cycles. This cross product of the two vectors is often

number

[0022] As highlighted above, a digital-to-analog converter (DAC) and an analog-to-digital converter (ADC) are used to convert the digital input to the RPU device 110 of the crossbar array 104 to an analog signal and the output from the RPU device 110 of the crossbar array 104 back to a digital signal, respectively. For noise and bounds management for DNN training in an analog RPU crossbar array:

number

number

number

number

[0023] In contrast, in accordance with the techniques of the present invention, the inputs of the crossbar array 104 to the RPU device 110 are scaled based on a worst case scenario to mitigate the risk of clipped output values ​​due to the limited dynamic range of the analog crossbar array 104 of the RPU device 110. As used herein, the term "worst case scenario" refers to the maximum possible output (i.e., maximum weight) from a weight matrix (e.g., weight matrix 102) given a particular input vector. The physical conductances representing the weights in the RPU device are physically limited. Thus, as used herein, the weights in the weight matrix are scaled in the range -wmax ...w max where w max is g max (i.e., corresponds to the maximum achievable conductance of the RPU device). As described in detail below, in the technique of the present invention, the assumption of absolute sum of input signals and constant maximum weight is used to calculate input and output noise and bound management scaling factor alpha (α) in the digital periphery (before DAC or after ADC) to bring the input within the dynamic range of the RPU device 110 of the crossbar array 104. Advantageously, the worst-case management process of the present invention does not add variable runtime since the results do not need to be recalculated when the bounds are clipped. Because by taking the worst-case situation as the basis for determining the scaling factor, it is guaranteed that the output bounds are never clipped.

[0024] 2 illustrates an example methodology 200 for noise and bounds management that scales the inputs of crossbar array 104 to RPU device 110 with respect to a worst-case scenario. As described above, each forward and backward cycle performed on weight matrix 102 includes a vector-matrix multiplication operation. In analog crossbar array 104 of RPU device 110, this vector-matrix multiplication operation scales each input vector value (see below) to a corresponding weight value W (in the corresponding row) of weight matrix 102. ijand summing the results. This process is also referred to herein as a "multiply-and-accumulate" operation of the analog crossbar array 104 of the RPU device 110. For each multiply-and-accumulate cycle, the steps of the methodology 200 are performed as digital pre-computations to determine the scaling factor α used for the computation of the analog crossbar array 104 of the RPU device 110. It should be noted that, according to an exemplary embodiment, one or more steps of the methodology 200, such as computing the noise and bound management scaling factor α, scaling / rescaling the input / output values ​​from the crossbar array 104 of the RPU device 110 by α (see below), are performed outside the RPU array hardware, such as by an apparatus, such as the apparatus 600 described in connection with the description of FIG. 6 below. In addition, one or more elements of the techniques of the present invention can optionally be provided as a service in a cloud environment. For example, by way of example only, the training data used for the input vector values ​​(see below) can reside remotely on a cloud server. Additionally, any of the steps of methodology 200 may be executed on dedicated cloud servers to take advantage of high performance CPUs and GPUs, and then the results are sent back to the local device.

[0025] In step 202, an input vector is obtained. The input vector includes digital values ​​x. The digital values ​​x in the input vector are also referred to herein as "input vector values ​​x." According to an exemplary embodiment, the input vector values ​​x include data from a training dataset. By way of example only, the training dataset can be obtained from a database or other repository of DNN training data.

[0026] In step 204, the absolute maximum value of the input vector values ​​x mx is calculated by the following formula:

number

[0027] Here, in this specification, x mx is also referred to as the absolute maximum input value. Next, in step 206, the absolute maximum input value x mx is designated as the scaling factor α.

[0028] At this stage in the process, the weight values ​​that will be used for the vector-matrix multiplication operations performed on the analog crossbar array 104 (see above) of the RPU device 110 are unknown a priori. However, as emphasized above, the worst-case assumption is that all analog weights are maximally positive for all positive input vector values ​​and maximally negative for all negative input vector values. Then, in step 208, the sum of all the absolute values ​​of the input vector values ​​x is calculated. In step 210, the sum is assigned to the total input variable s, i.e.,

number

[0029] In the above example, the input vector value x (both negative and positive) is provided as an input to the analog crossbar array 104 of the RPU device 110 in one pass. Alternatively, in another exemplary embodiment, the negative input value and the positive input value of the input vector value x are provided as an input to the analog crossbar array 104 of the RPU device 110 in two separate passes, where each other input value (negative or positive) of the input vector is set to zero, and the outputs of the two passes are sign-corrected and added accordingly to obtain a final result. For example, if a negative input value is provided as an input to the analog crossbar array 104 of the RPU device 110 in a first pass (whereas the positive input value is set to zero), a positive input value is provided as an input to the analog crossbar array 104 of the RPU device 110 in a second pass (whereas the negative input value is set to zero), or vice versa. In that case, a corresponding worst case situation can be applied in which all positive (or negative) input vector values ​​hit the maximum positive (or negative) weight and all other weights are assumed not to contribute to the output (and therefore can be assumed to be zero) since the corresponding input values ​​are set to zero for each path. The greater amount of either the positive or negative input vector values ​​is then assigned to the total input variable s.

[0030] That is, according to this alternative embodiment, in step 212, the positive input vector values ​​(s p ) is calculated using the following formula:

number

[0031] Here, I(true)=1 and I(false)=0 indicate whether the condition is true. In step 214, negative input vector values ​​(s n ) is calculated using the following formula:

number

[0032] Here, I(true)=1 and I(false)=0 indicate whether the condition is true or not.

[0033] In step 216, two quantities s p ands n is assigned to be the total input variable s. That is, in this exemplary embodiment, the total input variable s is set by the following formula:

number

[0034] Omega (ω) in the following equation is the maximum weight of the weight matrix 102, i.e., ω is the maximum weight expected. However, as explained in detail below, ω can be reduced (e.g., to 50% of the maximum weight expected) because the worst-case scenario occurs very rarely. That is, although the output may indeed be clipped by the output bound for an impossible input vector, in most cases this will not happen, and therefore the results of DNN training and inference will not change significantly.

[0035] Considering that omega (ω) is the maximum weight expected, the expected worst-case total output is omega times the total input variable s, i.e., ωs. As emphasized above, it is desirable for this expected worst-case total output to be less than the output bound b. Thus, in step 218, a proposed scaling factor sigma (σ) for the input vector values ​​(in the worst-case scenario) is calculated as the product of omega (ω) and the total input variable s divided by the output bound b, as follows:

number

[0036] As provided above, ω is the expected maximum weight. According to an exemplary embodiment, ω is a user-defined value based on a mapping of what the maximum conductance value of the RPU device 110 corresponds to in terms of the DNN's "mathematical" weight value. For example, by way of example only, a suitable value for the expected maximum weight ω may be 0.6. Therefore, based on equation (7), if the actual total output from the analog crossbar array 104 of the RPU device 110 is in fact as large as the expected worst-case total output (ωs), then sigma (σ)=1 and the total input variable s does not need to be scaled. In this way, the scaling ensures that the output bound b is never hit, even in the worst-case scenario.

[0037] However, because the worst-case scenario is very rare (i.e., the actual total output from the analog crossbar array 104 of the RPU device 110 is unlikely to be as large as the expected worst-case total output), it may be desirable to adjust the proposed scaling factor sigma (σ) by reducing the expected maximum weight ω, e.g., to 50% of the expected maximum weight, and then recalculate the proposed scaling factor in step 218 (see FIG. 2 (i.e., Adjust ω)). In that case, the expected worst-case total output bound will actually be hit, but most cases will have an output below the output bound b. Adjusting σ in this way generally produces a larger signal-to-noise ratio (SNR), which is desirable.

[0038] As highlighted above, a digital-to-analog converter (DAC) will be used to convert the scaled digital input vector values ​​(according to methodology 200) to analog signals for use in performing vector-matrix multiplication operations on the analog crossbar array 104 of the RPU device 110. Specifically, the DAC will convert the scaled digital input vector values ​​to analog pulse widths. Note, however, that there may be limited DAC resolution. That is, the proposed scaling factor σ calculated according to equation (7) above may actually be so large that the possible values ​​of the input vector values ​​divided by the proposed scaling factor σ may be smaller than the smallest bin of the digital-to-analog conversion. In that case, the input to the analog crossbar array 104 of the RPU device 110 will be all zeros after DAC conversion.

[0039] To avoid this occurrence, in step 220, the value calculated from equation (7) above, or the absolute maximum input value x mx Multiply by the variable (ρ) to get the quantization bin width (γ DAC ) is calculated by the following formula:

number

number

[0040] For an input range of (-1,...,1), the total range is 2. This total range is divided by the DAC into n steps, where n is the number of quantization steps (e.g., for an 8-bit DAC, n=256), to arrive at the quantization bin width (or simply "bin width"). Thus, in this example,

number

[0041] In step 222, the noise and bounds management scaling factor (α) is calculated as x mx (from equation (3) above) or the proposed scaling factor value σ (from equation (8) above), whichever is greater, by

number

[0042] In doing so, one skilled in the art would avoid having the maximum value of the scaled input vector value (see step 224 described below) be greater than 1 (which is assumed to be the maximum input vector value range of the arbitrarily set DAC, (-1,...,1)) since one would not want the input values ​​to be clipped.

[0043] As highlighted above, the above described process is used to pre-calculate the noise and bound management scaling factor α for each multiplication and accumulation cycle performed in the analog crossbar array 104 of the RPU device 110. Thus, in step 224, each of the digital input vector values ​​x is pre-calculated with the noise and bound management scaling factor (α) (calculated in step 222) according to the following equation:

number

[0044] Similarly, in step 228, each of the analog output vector values ​​obtained from the analog crossbar array 104 of the RPU device 110 is converted to a digital signal via an analog-to-digital converter (ADC) to provide a digital output vector value y'. In step 230, each of the digital output vector values ​​y' is scaled by a noise and bound management scaling factor (α) (calculated in step 222) according to the following equation:

number

[0045] An exemplary embodiment of the technique of the present invention will now be described by reference to Figures 3 and 4. That is, Figure 3 is a schematic diagram illustrating a forward cycle operation being performed in the analog crossbar array 304 of the RPU device 310. As shown in Figure 3, digital input vector values ​​x (see "digital input x") are provided as inputs to the analog crossbar array 304 of the RPU device 310. However, first, a noise and bound management scaling factor (α) is calculated as described in conjunction with the description of the methodology 200 of Figure 2 above (see "Noise / bound management calculation α"). Next, each of the digital input vector values ​​x is scaled by the noise and bound management scaling factor (α), i.e., x←x / α, to provide a scaled digital input vector value x' (see "Scaled digital RPU input x'").

[0046] The scaled digital input vector values ​​x' are converted to analog signals via a digital-to-analog converter (see "DA converter"). The analog signals are provided as analog pulse widths 320 to the analog crossbar array 304 of the RPU device 310, where the analog computation is performed in the analog crossbar array 304 of the RPU device 310. As highlighted above, this analog computation includes performing a vector-matrix multiplication operation in the analog crossbar array 304 of the RPU device 310 by multiplying each of the scaled input vector values ​​x' with a corresponding weight value in a corresponding weight matrix (not shown). The mapping of the weight matrix to the analog crossbar array of the RPU device was described in conjunction with the description of FIG. 1 above.

[0047] As shown in FIG. 3, the analog output vector values ​​obtained from the operations performed in the analog crossbar array 304 of the RPU device 310 are input to the inverting input of an operational amplifier 324 and to the output of the operational amplifier 324 (V out) connected across a capacitor (C int ) is provided to an integrated circuit 322 having an operational amplifier 324 with a non-inverting input connected to ground. The output of the operational amplifier 324 (V out ) is also connected to the input of an analog-to-digital converter (see "AD converter").

[0048] An analog-to-digital converter (AD converter) converts each of the analog output vector values ​​obtained from the analog crossbar array 304 of the RPU device 310 into a digital signal to provide a digital output vector value y' (see "Digital RPU Output y'"). Each of the digital output vector values ​​y' is then rescaled by a noise and bound management scaling factor (α) (see "Noise / Bound Management with Alpha"), i.e., y←yα, to provide a rescaled digital output vector value y (see "Rescaled Digital Output y"). As provided above, processing such as computing the noise and bound management scaling factor α, scaling / rescaling input / output values ​​from the crossbar array of the RPU device by α, etc., can be performed outside the RPU array hardware, for example by an apparatus, such as apparatus 600 described in connection with the description of FIG. 6 below.

[0049] FIG. 4 is a schematic diagram showing the backward cycle operation being performed in the analog crossbar array 404 of the RPU device 410. The process is generally the same as the forward cycle operation described in conjunction with the description of FIG. 3 above, except that a transposed analog RPU array 404 is used for the backward cycle path. "Transposed" means that the inputs and outputs are swapped, i.e., the previous outputs now become the inputs, and the previous inputs become the outputs. This is (effectively) calculating x=W'y, where W' is the transpose of the matrix W. As shown in FIG. 4, a digital input vector value x (see "Digital Input x") is provided as an input to the analog crossbar array 404 of the RPU device 410. However, first, a noise and bound management scaling factor (α) is calculated as described in conjunction with the description of the methodology 200 of FIG. 2 above (see "Noise / Bound Management Calculation α"). That is, as provided above, for each forward and backward cycle, the steps of methodology 200 are performed as digital pre-computations to determine a scaling factor α to be used for operations in the analog crossbar array of the RPU device. Then, each of the digital input vector values ​​x is scaled by a noise and bound management scaling factor (α), i.e., x←xα, to provide a scaled digital input vector value x′ (see “Scaled Digital RPU Input x′”).

[0050] The scaled digital input vector values ​​x' are converted to analog signals via a digital-to-analog converter (see "DA converter"). The analog signals are provided as analog pulse widths 420 to the analog crossbar array 404 of the RPU device 410, where an analog computation is performed in the analog crossbar array 404 of the RPU device 410. As highlighted above, this analog computation includes performing a vector-matrix multiplication operation in the analog crossbar array 404 of the RPU device 410 by multiplying each of the scaled input vector values ​​x' with a corresponding weight value in a corresponding weight matrix (not shown). The mapping of the weight matrix to the analog crossbar array of the RPU device was described in conjunction with the description of FIG. 1 above.

[0051] As shown in FIG. 4, the analog output vector values ​​obtained from the operations performed in the analog crossbar array 404 of the RPU device 410 are connected to the inverting input of an operational amplifier 424 and the output of the operational amplifier 424 (V out ) connected across a capacitor (C int BACKWORD ) is provided to an integrated circuit 422 that includes an operational amplifier 424 having a non-inverting input connected to ground. The non-inverting input of the operational amplifier 424 is connected to ground. The output (Vout) of the operational amplifier 424 is also connected to the input of an analog-to-digital converter (see "AD converter").

[0052] An analog-to-digital converter (AD converter) converts each of the analog output vector values ​​obtained from the analog crossbar array 404 of the RPU device 410 into a digital signal to provide a digital output vector value y' (see "Digital RPU Output y'"). Each of the digital output vector values ​​y' is then rescaled by a noise and bound management scaling factor (α) (see "Noise / Bound Management with Alpha"), i.e., y←yα, to provide a rescaled digital output vector value y (see "Rescaled Digital Output y"). As provided above, processing such as computing the noise and bound management scaling factor α, scaling / rescaling input / output values ​​from the crossbar array of the RPU device by α, etc., can be performed outside the RPU array hardware, for example by an apparatus, such as apparatus 600 described in connection with the description of FIG. 6 below.

[0053] As provided above, the technique of the present invention minimizes run time by scaling the input vector values ​​x to the analog crossbar array of the RPU device based on the worst case scenario. In the above-described embodiment, by taking the worst case situation as the basis for determining the scaling factor, it is guaranteed that the output bounds are never clipped.

[0054] According to an alternative embodiment, the above-described methodology 200 is executed only if the bound is exceeded (i.e., scale / rescale the input / output values ​​from the crossbar array of the RPU device by α to calculate the worst-case noise and bound management scaling factor α, etc.). See, for example, the exemplary methodology 500 of FIG. 5.

[0055] In step 502, an input vector of digital values ​​x, i.e., "input vector values ​​x," is obtained. According to an exemplary embodiment, the input vector values ​​x include data from a training data set. By way of example only, the training data set can be obtained from a database or other repository of DNN training data.

[0056] In step 504, the absolute maximum value of the input vector values ​​x mx (also referred to herein as the absolute maximum input value) is calculated according to equation (3) above. In step 506, the absolute maximum input value x mx is the noise and bounds management scaling factor α, i.e., α=x mx , is designated.

[0057] In step 508, each of the digital input vector values ​​x is scaled by the noise and bound management scaling factor α=x mx Thus, the following equation is obtained:

number

[0058] In step 514, a plurality of digital output vector values ​​y'initial A determination is then made as to whether any of the digital output vector values ​​y' have been clipped (bounds management). initial The determination of whether any of the bounds are clipped can be made by sensing saturation at the output of the op-amp. There are circuit methods to determine the clipping of the bounds. However, a simple method is to simply use the maximum and minimum output values ​​of the ADC because when the input to the ADC exceeds the ADC range the output saturates to these values. So, for example, if any digital output is 255 (which is the highest output value of an 8-bit DAC) or 0 (which is the lowest output value of an 8-bit DAC), the bounds are determined to be clipped and the calculation is repeated.

[0059] In step 514, the digital output vector value y' initial If it is determined that none of the digital output vector values ​​y′ have been clipped (NO), then in step 516 the digital output vector value y′ initial Each of these is a noise and bounds management scaling factor α=x mx Thus, the following equation (13) is obtained, that is,

number

number

[0060] Next, in step 520, each of the digital input vector values ​​x is multiplied by the worst-case noise and bounds management scaling factor

number

number

[0061] Similarly, in step 524, each of the analog output vector values ​​obtained from the analog crossbar array 104 of the RPU device 110 is converted to a digital signal via an ADC to provide a digital output vector value y'. In step 526, each of the digital output vector values ​​y' is scaled to a digital value y' based on the worst case scenario noise and bounds management scaling factor.

number

number

[0062] The present invention may be a system, method, or computer program product, or any combination thereof. The computer program product may include one or more computer readable storage media having computer readable program instructions for causing a processor to perform aspects of the present invention.

[0063] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer diskette; (登録商標), hard disk, random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or raised structures in grooves in which instructions are recorded, or any suitable combination of these. As used herein, computer-readable storage medium should not be interpreted as being a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a wave guide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0064] The computer readable program instructions described herein can be downloaded from a computer readable storage medium to an individual computing device / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may be comprised of copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing device / processing device receives the computer readable program instructions from the network and transmits the computer readable program instructions to the respective computing device / processing device for storage in a computer readable storage medium.

[0065] The computer readable program instructions for carrying out the operations of the present invention may be either assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, such as object-oriented programming languages, such as Smalltalk, C++, and the like, conventional procedural programming languages ​​(e.g., the "C" programming language or similar programming languages). The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any kind of network, such as a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry to perform aspects of the invention.

[0066] Aspects of the present invention are described herein with reference to flowchart illustrations or block diagrams, or combinations thereof, of methods, apparatus (systems), and computer program products or computer programs according to embodiments of the present invention. It will be understood that each block of the flowchart illustrations or block diagrams, or combinations thereof, and combinations of blocks in the flowchart illustrations or block diagrams, or combinations thereof, can be implemented by computer readable program instructions.

[0067] These computer readable program instructions may be provided to a processor of a computer or other programmable data processing apparatus such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / operations identified in one or more blocks of the flowchart illustrations or block diagrams, or combinations thereof, to produce a machine. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer programmable data processing apparatus or other device, or combinations thereof, to function in a particular manner, such that a computer readable storage medium having stored thereon includes an article of manufacture including instructions that implement aspects of the functions / operations identified in one or more blocks of the flowchart illustrations or block diagrams, or combinations thereof.

[0068] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device such that the instructions, which execute on a computer, other programmable data processing apparatus, or other device, implement the functions / acts identified in one or more blocks of the flowchart illustrations or block diagrams, or combinations thereof, to cause the computer, other programmable apparatus, or other device to perform a series of operational steps to generate a computer-implemented process.

[0069] The flowchart diagrams and block diagrams in the drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products or computer programs according to various embodiments of the present invention. In this regard, each block in the flowchart diagrams or block diagrams may represent a module, segment, or portion of an instruction, which includes one or more executable instructions for implementing one or more specified logical functions. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the drawings. For example, two blocks shown in succession may actually be accomplished as one step performed simultaneously, substantially simultaneously, partially or fully in a time-overlapping manner, depending on the functions involved, or the blocks may be performed in the reverse order. It should be noted that each block of the block diagrams or flowchart diagrams or combinations thereof, as well as combinations of multiple blocks of the block diagrams or flowchart diagrams or combinations thereof, may be implemented by a special purpose hardware-based system that performs the specified functions or operations, or may execute a combination of special purpose hardware and computer instructions.

[0070] As provided above, according to an exemplary embodiment, one or more steps of the methodology 200, including the noise and bound management scaling factor α, scaling / rescaling of input / output values ​​from the crossbar array of the RPU device, etc., can be performed outside the RPU array hardware, for example, by an apparatus, such as the apparatus 600 shown in FIG. 6. FIG. 6 is a block diagram of the apparatus 600 for implementing one or more of the methodologies presented herein. By way of example only, the apparatus 600 can be configured to implement one or more of the steps of the methodology 200 of FIG.

[0071] Apparatus 600 includes a computer system 610 and a removable media 650. Computer system 610 includes a processor device 620, a network interface 625, memory 630, a media interface 635, and an optional display 640. Network interface 625 allows computer system 610 to connect to a network, and media interface 635 allows computer system 610 to interact with media, such as a hard drive or removable media 650.

[0072] The processor device 620 can be configured to implement the methods, steps, and functions disclosed herein. The memory 630 can be distributed or local, and the processor device 620 can be distributed or stand-alone. The memory 630 can be implemented as an electrical, magnetic, or optical memory, or any combination of these or other types of storage devices. Moreover, the term "memory" should be interpreted broadly enough to encompass any information that can be read from or written to an address in an addressable space accessed by the processor device 620. In this definition, information on a network accessible through the network interface 625 is still in the memory 630 because the processor device 620 can obtain the information from the network. It should be noted that each distributed processor that constitutes the processor device 620 typically includes its own addressable memory space. It should also be noted that part or all of the computer system 610 can be incorporated into an application specific integrated circuit or an application general integrated circuit.

[0073] Optional display 640 is any type of display suitable for interacting with a human user of device 600. Typically, display 640 is a computer monitor or other similar display.

[0074] 7 and 8, although the present disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings described herein is not limited to a cloud computing environment. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0075] Cloud computing is a model of service delivery for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal administrative effort or interaction with the service provider. The cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0076] The features are as follows.

[0077] On-demand self-service: A cloud consumer can unilaterally provision computing capacity, such as server time and network storage, automatically as needed, without requiring human interaction with the provider of the service.

[0078] Broad Network Access: Capabilities are available over the network and accessed via standard mechanisms that facilitate use by heterogeneous thin or thick client platforms (e.g., cell phones, laptops, and PDAs).

[0079] Resource Pooling: Provider computing resources are pooled to serve multiple consumers using a multi-tenant model, and various physical and virtual resources are dynamically allocated and reallocated according to demand. Consumers generally have no control or knowledge of the exact location of the resources provided to them, but are said to be location-independent in that they may be able to specify a location at a higher level of abstraction (e.g., country, state, or data center).

[0080] Rapid Elasticity: Capabilities can be provisioned quickly and elastically, sometimes automatically, quickly scaled out, quickly released and quickly scaled in. To the consumer, the capabilities available for provisioning are often unlimited and can be purchased in any quantity at any time.

[0081] Measured Services: Cloud systems automatically control and optimize resource usage by using metering capabilities at some level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of the services being utilized.

[0082] The service model is as follows:

[0083] Software as a Service (SaaS): The capability offered to consumers to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through a thin-client interface, e.g., a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application functions, with the possible exception of limited user-specific application configuration settings.

[0084] Platform as a Service (PaaS): A capability offered to a consumer to deploy consumer-created or acquired applications, created using programming languages ​​and tools supported by a provider, onto a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, e.g., including networks, servers, operating systems, or storage, but does have control over the deployed applications and, potentially, the application-hosting environment configuration.

[0085] Infrastructure as a Service (IaaS): The capability offered to consumers to provision processing, storage, network, and other basic computing resources on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, deployed applications, and in some cases, limited control over selecting network components (e.g., host firewalls).

[0086] The deployment models are as follows:

[0087] Private Cloud: Cloud infrastructure is operated exclusively for an organization. The cloud infrastructure may be managed by the organization or a third party, and may be on-premises or off-premises.

[0088] Community Cloud: Cloud infrastructure is shared by several organizations and supports a particular community with common concerns (e.g., mission, security requirements, policies, and compliance considerations). The cloud infrastructure may be managed by the organizations or a third party, and may reside on-premise or off-premise.

[0089] Public Cloud: The cloud infrastructure is available to the general public or large industry groups and is owned by an organization that sells cloud services.

[0090] Hybrid Cloud: The cloud infrastructure remains a unique entity, but is a hybrid of two or more clouds (private, community, or public) brought together by standardized or specialized technologies that enable data and application portability (e.g., cloud bursting for load balancing between clouds).

[0091] A cloud computing environment is service oriented with a focus on statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0092] 7, an exemplary cloud computing environment 50 is illustrated. As illustrated, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or mobile phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or combinations thereof, may communicate. The cloud computing nodes 10 may communicate with each other. They may be physically or virtually grouped in one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or combinations thereof, as described herein (not shown). This allows the cloud computing environment 50 to provide infrastructure, platform, or software, or combinations thereof, as a service without the cloud consumer having to maintain resources on the local computing device. It will be understood that the types of computing devices 54A-54N shown are intended to be illustrative only, and that computing node 10 and cloud computing environment 50 communicate with any type of computerized device (e.g., using a web browser) via any type of network or network-addressable connection or combination thereof.

[0093] Referring now to Figure 8, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 7) is shown. It should be understood that the components, layers and functions shown in Figure 8 are intended to be merely exemplary, and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0094] The hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframes 61, RISC (Reduced Instruction Set Computer) architecture based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, the software components include network application server software 67 and database software 68.

[0095] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities may be provided: virtual servers 71; virtual storage 72; virtual networks 73; virtual networks 73, such as those mentioned above including virtual private networks; virtual applications and operating systems 74; and virtual clients 75.

[0096] In one example, the management layer 80 may provide multiple functions described below. Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, and protection for data and other resources. User portal 83 provides access to the cloud computing environment for consumers and system administrators. Service level management 84 provides allocation and management of cloud computing resources such that required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides pre-provisioning and procurement of cloud computing resources where future requirements are predicted according to SLAs.

[0097] The workload tier 90 provides examples of functions for which the cloud computing environment may be utilized. Examples of workloads and functions that may be provided from this tier include mapping and navigation 91; software development and lifecycle management 92; virtual classroom instruction delivery 93; data analytics processing 94; transaction processing 95; and scaling factor calculations and input / output scaling / rescaling 96.

[0098] Although exemplary embodiments of the present invention have been described herein, it is to be understood that the invention is not limited to those precise embodiments, and that various other changes and modifications may be made by those skilled in the art without departing from the scope of the present invention.

Claims

1. 1. A method for noise and bounds management comprising: obtaining an input vector value x for an analog crossbar array of a resistive processing unit (RPU) device, where a weight matrix is ​​mapped to the analog crossbar array of the RPU device; and scaling the input vector values ​​x based on a worst case scenario to provide scaled input vector values ​​x' for use as input to the analog crossbar array of the RPU device, where the worst case scenario includes ωs multiplied by a maximum possible weight (ω) of a weighting matrix and a total input variable (s) that is calculated by calculating the absolute values ​​of all of the input vector values; The method comprising:

2. Absolute maximum input value x mx from said input vector values ​​x; The proposed scaling factor σ is [0010] where ω is the maximum expected weight in the weight matrix, s is the total input variables, and b is the output bound of the analog crossbar array of the RPU device; A noise and bound management scaling factor α is defined as the absolute maximum input value x mx or the proposed scaling factor σ, whichever is larger; and Scaling the input vector values ​​x using the noise and bounds management scaling factor α. The method of claim 1 further comprising:

3. The absolute maximum input value x mx but, [0025] The method of claim 2, wherein the calculated

4. calculating the sum of all the absolute values ​​of the input vector values ​​x; and assigning said sum of all said absolute values ​​of said input vector values ​​x to said sum input variable s. The method of claim 2 further comprising:

5. Positive input vector values ​​(s p ) to calculate the sum of all the absolute values ​​of only Negative input vector values ​​(s n ) ) ; and s p ands n The larger of the two is assigned to the total input variable s. The method of claim 2 further comprising: 【Request 6】 【Number 3】 or [0045] where ρ is a variable, and γ DAC is the bin width of the digital-to-analog quantization, The method of claim 2 further comprising:

7. Reducing the maximum possible weight ω of the weighting matrix; and Recalculating the proposed scaling factor σ The method of claim 2 further comprising:

8. converting the scaled input vector values ​​x' to an analog signal; and performing a vector-matrix multiplication operation on the analog crossbar array of the RPU device; The method of claim 2 further comprising:

9. performing the vector-matrix multiplication operation, multiplying each of said scaled input vector values ​​x' by a corresponding weight value in said weight matrix; The method of claim 8 , comprising:

10. converting a plurality of analog output vector values ​​obtained from the analog crossbar array of the RPU device into a single digital signal to provide a digital output vector value y′; and rescaling the digital output vector values ​​y' using the noise and bounds management scaling factor α to provide rescaled digital output vector values ​​y. The method of claim 8 further comprising:

11. Absolute maximum input value x mx from said input vector values ​​x; The absolute maximum input value x mx to be the scaling factor α; Scale the input vector value x using the scaling factor α to obtain a scaled input vector value x′ for use as an input to an analog crossbar array of an RPU device. initial To provide: The scaled input vector values ​​x' initial converting the signal into an analog signal; performing a vector-matrix multiplication operation on the analog crossbar array of an RPU device; Converting the multiple analog output vector values ​​obtained from the analog crossbar array of the RPU device into one digital signal to generate multiple digital output vector values ​​y' initial To provide: said plurality of digital output vector values ​​y' initial is clipped; and said plurality of digital output vector values ​​y' initial Scaling the input vector values ​​x based on the worst case scenario when at least one of The method of claim 1 further comprising:

12. An apparatus for noise and bounds management, the apparatus comprising a processor coupled to a memory, the processor comprising: Obtain an input vector value x for an analog crossbar array of an RPU device, where a weight matrix is ​​mapped to the analog crossbar array of an RPU device; and Scaling the input vector values ​​x based on a worst case scenario to provide scaled input vector values ​​x' for use as input to the analog crossbar array of the RPU device, where the worst case scenario includes ωs multiplied by the maximum possible weight (ω) of a weighting matrix and a total input variable (s) that is calculated by calculating the absolute values ​​of all of the input vector values; The apparatus is operable to:

13. The processor, Absolute maximum input value x mx from the input vector values ​​x; The proposed scaling factor σ is [0050] where ω is the maximum expected weight in the weight matrix, s is the total input variables, and b is the output bound of the analog crossbar array of the RPU device; A noise and bound management scaling factor α is defined as the absolute maximum input value x mx or the proposed scaling factor σ, whichever is larger; and Scale the input vector values ​​x using the noise and bounds management scaling factor α. The apparatus of claim 12 , further operable to:

14. The processor, Calculate the sum of all the absolute values ​​of the input vector values ​​x; and assigning said sum of all said absolute values ​​of said input vector values ​​x to said sum input variable s; The apparatus of claim 13 further operable to:

15. The processor, Positive input vector values ​​(s p ) to calculate the sum of all the absolute values ​​of only Negative input vector values ​​(s n ) only; and s p ands n The larger of these is assigned to the total input variable s. The apparatus of claim 13 further operable to:

16. The processor, [006] or [0070] ρ is a variable, and γ is a non-negative value. DAC is the bin width of the digital-to-analog quantization The apparatus of claim 13 further operable to:

17. 1. A non-transitory computer program for noise and bounds management, comprising: Obtaining an input vector value x for an analog crossbar array of an RPU device, where a weight matrix is ​​mapped to the analog crossbar array of an RPU device; and scaling the input vector values ​​x based on a worst case scenario to provide scaled input vector values ​​x' for use as input to the analog crossbar array of the RPU device, where the worst case scenario includes ωs multiplied by a maximum possible weight (ω) of a weighting matrix and a total input variable (s) that is calculated by calculating the absolute values ​​of all of the input vector values; The computer program causes a computer to execute each step of the method comprising the steps of:

18. Absolute maximum input value x mx from said input vector values ​​x; The proposed scaling factor σ is [0080] where ω is the maximum expected weight in the weight matrix, s is the total input variables, and b is the output bound of the analog crossbar array of the RPU device; A noise and bound management scaling factor α is defined as the absolute maximum input value x mx or the proposed scaling factor σ, whichever is larger; and Scaling the input vector values ​​x using the noise and bounds management scaling factor α. The computer program product of claim 17 , further comprising:

19. calculating the sum of all the absolute values ​​of the input vector values ​​x; and assigning said sum of all said absolute values ​​of said input vector values ​​x to said sum input variable s. The computer program product of claim 18 , further comprising:

20. Positive input vector values ​​(s p ) to calculate the sum of all the absolute values ​​of only Negative input vector values ​​(s n ) ) ; and s p ands n The larger of the two is assigned to the total input variable s. The computer program product of claim 18 , further comprising:

Images