Non-volatile memory-based activation functions

Abstract

An analog memory-based activation function for an artificial neural network may be provided. The apparatus may include at least two non-volatile memory devices connected in parallel, such that a current can flow through one of the two non-volatile memory devices depending on the voltage level driving the current. To control which branch the input current flows through, each of the two non-volatile memory devices may be connected to a circuit element that can function as a switch, such as a diode, such as a semiconductor diode, a transistor, or another circuit element. Such an apparatus may implement an analog memory-based activation function, for example, for an analog memory-based artificial neural network.

Description

[Technical Field]

[0001] This application relates generally to analog memory-based artificial neural networks, and more particularly to analog or non-volatile memory-based activation functions that may be tunable for deep learning. [Background technology]

[0002] Analog memory crossbar arrays that implement multiply-accumulate (MAC) operations can facilitate the performance of deep learning neural networks or deep neural networks (DNNs). The activation functions that convert the linear outputs of these MAC operations as neuron outputs are typically implemented in digital circuits, for example, using analog-to-digital converters. Summary of the Invention

[0003] The summary of the present disclosure is provided to aid in understanding analog memory-based or non-volatile memory-based activation functions, for example, for deep learning, and is not intended to limit the present disclosure or the present invention. It should be understood that various aspects and features of the present disclosure may be advantageously used separately in some cases or in other cases in combination with other aspects and features of the present disclosure. Accordingly, variations and modifications may be made to computer systems and / or their operating methods to achieve various advantages.

[0004] In one aspect, the apparatus may include a first non-volatile memory connected to the positive terminal of a first diode. The first non-volatile memory may be programmed to store a first parameter of an activation function associated with an artificial neural network. The apparatus may also include a second non-volatile memory connected to the negative terminal of a second diode. The second non-volatile memory may be programmed to store a second parameter of the activation function associated with the artificial neural network. The first non-volatile memory and the second non-volatile memory may be connected in parallel to an input line, and the negative terminal of the first diode and the positive terminal of the second diode may be connected in parallel to an output line.

[0005] Advantageously, implementing analog memory-based activation functions can improve energy efficiency in analog memory-based neural network implementations.

[0006] In one aspect, the activation function may include a leaky rectified linear unit (ReLU), a first gradient of the leaky ReLU determined by the first parameter stored on the first non-volatile memory, and a second gradient of the leaky ReLU determined by the second parameter stored on the second non-volatile memory, the first parameter being proportional to the conductance of the first non-volatile memory, and the second parameter being proportional to the conductance of the second non-volatile memory.

[0007] Advantageously, implementing an analog memory-based leaky rectified linear unit (ReLU) activation function can improve energy efficiency in analog memory-based neural network implementations.

[0008] In another aspect, the activation function may include a rectified linear unit (ReLU), a first gradient of the ReLU may be determined by the first parameter stored on the first non-volatile memory, and a second gradient of the ReLU may be determined by the second parameter stored on the second non-volatile memory, the first parameter may be proportional to the conductance of the first non-volatile memory, and the second parameter may be proportional to the conductance of the second non-volatile memory.

[0009] Advantageously, implementing an analog memory-based rectified linear unit (ReLU) activation function can improve energy efficiency in analog memory-based neural network implementations.

[0010] In another aspect, an apparatus may include a first non-volatile memory coupled to a first field effect transistor (FET). The first non-volatile memory may be programmed to store first parameters of an activation function associated with an artificial neural network. The apparatus may also include a second non-volatile memory coupled to a second field effect transistor (FET). The second non-volatile memory may be programmed to store second parameters of the activation function associated with the artificial neural network. The first non-volatile memory and the second non-volatile memory may be coupled in parallel to an input line. The first field effect transistor and the second field effect transistor may be coupled in parallel to an output line, a source terminal of the first field effect transistor may be coupled to the first non-volatile memory, a source terminal of the second field effect transistor may be coupled to the second non-volatile memory, and a drain terminal of the first field effect transistor and the drain terminal of the second field effect transistor may be coupled to the output line.

[0011] Advantageously, implementing analog memory-based activation functions can improve energy efficiency in analog memory-based neural network implementations.

[0012] In one aspect, a method may include tuning a first non-volatile memory connected to a column output of a crossbar array of memristive elements so that the first non-volatile memory stores a first parameter of an activation function of an artificial neural network. The method may also include tuning a second non-volatile memory connected to the column output of the crossbar array of memristive elements so that the second non-volatile memory stores a second parameter of the activation function of the artificial neural network. The method may also include maintaining the first parameter and the second parameter fixed as hyperparameters of the artificial neural network, while updating synaptic weights stored in the memristive elements of the crossbar array during training of the artificial neural network.

[0013] Advantageously, implementing analog memory-based activation functions can improve energy efficiency in analog memory-based neural network implementations.

[0014] A computer-readable storage medium storing a program of instructions executable by a machine to perform one or more of the methods described herein may also be provided.

[0015] Further features, as well as the structure and operation of various embodiments, are described in detail below with reference to the accompanying drawings, where like reference numbers indicate identical or functionally similar elements. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 illustrates an analog memory-based neural network in one embodiment.

[0017] [Figure 2] FIG. 2A illustrates an exemplary leaky rectified linear unit (ReLU) activation function.

[0018] FIG. 2B illustrates an exemplary rectified linear unit (ReLU) activation function.

[0019] [Figure 3] FIG. 10 illustrates an analog memory-based neural network with a non-volatile memory-based activation function in another embodiment.

[0020] [Figure 4] FIG. 1 is a flow diagram illustrating a method in one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0021] Analog memory-based neural networks may utilize the storage capabilities and physical properties of memory devices, such as non-volatile memory devices, to implement artificial neural networks. This type of in-memory computing hardware offers increased speed and energy efficiency, potentially improving performance. For example, rather than moving data from dynamic random access memory (DRAM) to a processor, such as a central processing unit (CPU), to perform calculations, analog neural network chips perform calculations in the same location where the data is stored. Without data movement, tasks can be performed faster and require less energy. In one or more embodiments, an analog memory-based or non-volatile memory-based activation function is provided.

[0022] An implementation of an artificial neural network can include successive layers of interconnected neurons, such that the output signal of a neuron in one layer is weighted and sent to a neuron in the next layer. A neuron N in a given layer can be connected to one or more neurons N in the next layer, and a different weight w can be associated with each neuron-neuron connection N-N to weight the signal sent from N to N. A neuron N generates an output signal depending on its accumulated input applied to an activation function, and the weighted signal can be propagated through subsequent layers of the network, from the input to the output neuron layer. Briefly, the activation function determines whether a neuron should be activated or the level of activation for a neuron, e.g., for the neuron's output. An artificial neural network machine learning model can undergo a training phase in which a set of weights associated with each neuron layer is determined. In an iterative training scheme, in which weights are repeatedly updated as the network "learns" from the training data, the network is exposed to a set of training data. The resulting trained model, with weights determined by the training operation, can be applied to perform tasks based on new data.

[0023] FIG. 1 illustrates an analog memory-based neural network, according to one embodiment. The analog memory-based neural network includes a non-volatile memory-based activation function. For example, circuit elements including non-volatile memory may implement the activation function of the neural network. Briefly, non-volatile memory (NVM) devices (also referred to as memristive devices or technologies) can retain values ​​stored therein even when power is turned off. In a neural network implementation, these values ​​may represent synaptic weights or neural network weights.

[0024] A computational memory based on a crossbar array (shown at 102 for simplicity) using electronic devices including resistive NVM devices 104 labeled as resistive processing units (RPUs) can be used as an inference accelerator for artificial neural network (ANN) computations, such as for training deep neural networks (DNNs), and / or for inference using such networks. For example, in deep learning or neural network inference, data propagation through multiple layers of a neural network involves a series of matrix multiplications. Each layer can be represented as a matrix of synaptic weights. These weights can be stored in the conductance states of NVM devices 104. These NVM devices 104 can be arranged in a crossbar array to create an artificial neural network that performs all matrix multiplications in-place in an analog fashion. For example, an on-chip array can directly relate to the synaptic layers of the neural network. In one embodiment, for example, a crossbar array can represent a layer of the neural network.

[0025] For example, a multiply accumulate (MAC) device may include NVM devices arranged in a crossbar configuration, such as crossbar array 102, and may perform analog matrix-vector multiplication in a single step. Such MAC devices may be used to implement hardware or analog memory-based neural networks.

[0026] Examples of such memory devices may include, but are not limited to, resistive random access memory (RRAM or ReRAM), electrochemical random-access memory (ECRAM), ferroelectric random access memory (FeRAM), phase-change memory (PCM), conductive-bridging RAM (CBRAM), NOR flash, and magnetic RAM (MRAM).

[0027] In one embodiment, the analog memory-based devices may implement a hardware neural network. For example, in a system including a crossbar array with analog memory-based devices, e.g., as coprocessors or accelerators, one or more digital processors may communicate with the coprocessors in performing their operations or functions for various applications.

[0028] In one embodiment, such a device may be a coprocessor or accelerator including multiply-accumulate (MAC) hardware having a crossbar structure or array, such as that shown at 102. While the diagram shows one crossbar array, many of these arrays may be integrated on the coprocessor. As an example, an analog multiply-accumulate device may include electronic devices including memory elements 104 located at the intersections of the crossbar array 102. For example, at each intersection or junction of the crossbar structure or crossbar array, there may be at least one electronic device 104 including a resistive memory or memristive element. In one embodiment, such resistive memory elements may be programmed to store synaptic weights of an artificial neural network (ANN). Each array 102 may represent a layer of the ANN. For example, the coprocessor may include one of the resistive memory elements 104 at each intersection, connecting each of the input lines 106 to each of the output lines 108. The array 102 may be a regular array, with a constant distance between its intersections in the horizontal and vertical dimensions on the surface of the substrate. Each crossbar array 102 may perform vector-matrix multiplication. By way of example, the array 102 may include peripheral circuits such as pulse-width modulators and analog-to-digital converters (ADCs).

[0029] As an example, a device such as a coprocessor including the crossbar array 102 may be interfaced to hardware including another processor, such as a field programmable gate array (FPGA). There may also be a digital-to-analog converter that may provide power, voltage, and current to the coprocessor. For example, a digital-to-analog converter 132 may convert digital input data (referred to as input x) 130 into an analog signal (e.g., 110) that is input to the crossbar array 102. A processor such as a field programmable gate array (FPGA) may implement digital logic to interface with the electronics of the coprocessor and the digital-to-analog converter.

[0030] An electrical pulse or voltage signal 110 may be input (or applied) to the input lines 106 of the crossbar structure 102. An output current 112 may be obtained from the output lines 108 of the crossbar structure and generated, for example, according to a multiply-and-accumulate operation based on the input pulse or voltage signal 110 applied to the input lines 106 and the values ​​(synaptic weights) stored in the resistive memory elements 104. An output 114 may be read out in a readout circuit 116.

[0031] To implement an analog memory-based activation function, additional circuit elements including at least a non-volatile memory device (collectively referred to as analog-based activation function devices for purposes of illustration only) may be connected to the output line 108. For example, an output from a column of the crossbar array, representing a multiply-and-accumulate (linear operation) performed on neurons of a neural network, may be connected to the input of a circuit element performing the analog memory-based activation function. In one embodiment, the circuit element may include two non-volatile memory devices 118, 120 and diodes 122, 124 connected to each of the two non-volatile memory devices. The diodes may be used to control or select the direction of current flow in the analog-based activation function devices (column outputs of the crossbar array). The diodes may be semiconductor diodes having at least a positive terminal and a negative terminal, such as pn junction diodes. The diode 122 (referred to as the first diode for purposes of illustration) may be connected to the non-volatile memory device 118 (referred to as the first non-volatile memory device for purposes of illustration) such that the positive terminal of the first diode is closer to the first non-volatile memory device 118. The diode 124 (referred to as the second diode for purposes of explanation) may be connected to the nonvolatile memory device 120 (referred to as the second nonvolatile memory device) such that the negative terminal of the second diode is closer to the second nonvolatile memory device 120. The first nonvolatile memory device 118 and the first diode 122 are connected in series with each other; the second nonvolatile memory device 120 and the second diode 124 are connected in series with each other. The two sets of nonvolatile memory devices and diodes are connected in parallel: for example, the first nonvolatile memory device 118 and the first diode 122 are connected in parallel to the second nonvolatile memory device 120 and the second diode 124. The output 108 or 114 goes to both of the parallel paths.

[0032] The two NVMs 118, 120 and diodes 122, 124 implement an activation function such as a leaky rectified linear unit (ReLU). Another example of an activation function that the two NVMs 118, 120 and diodes 122, 124 can represent includes a ReLU activation function. FIG. 2A illustrates a leaky rectified linear unit (ReLU) activation function. FIG. 2B illustrates a rectified linear unit (ReLU) activation function. The output of the multiply-accumulate operation in each column 108 can be input to the activation function implemented by the two NVMs 118, 120 and diodes 122, 124, where nonlinear operations in neural networks occur. The leaky ReLU has a gradient 202 for positive values ​​210 of the output 108 and a smaller gradient 204 for negative values ​​212 of the output 108 instead of the flat gradient that a ReLU has (ReLU is shown in FIG. 2B). For example, in Figure 2B, which shows the ReLU activation function, the gradient 208 in the negative region 216 is flat or zero, while there is a non-zero gradient 206 in the positive region 214. In Figures 2A and 2B, the x-axis represents the output values ​​108 (of the MAC operation) and the y-axis represents the activation values.

[0033] In one embodiment, the gradient coefficients are determined before training the neural network, i.e., they are not learned during training of the weights stored in the non-volatile memory device 104 in the crossbar array. For example, in one embodiment, the gradient values ​​may be stored as hyperparameters in the two NVMs 118, 120. These values ​​remain fixed during training of the neural network.

[0034] In one embodiment, the first non-volatile memory device 118 may store a gradient 202 of a leaky ReLU to be applied when the output value 108 is positive (e.g., greater than 1); the second non-volatile memory device 120 may store a gradient 204 of a leaky ReLU to be applied when the output value 108 is negative (e.g., less than 1).

[0035] In another embodiment, to implement a ReLU activation function, the first non-volatile memory device 118 may store a ReLU gradient 206 to be applied when the output value 108 is positive (e.g., greater than 1); and the second non-volatile memory device 120 may store a ReLU substantially flat gradient (or substantially zero value) 208 to be applied when the output value 108 is negative (e.g., less than 1).

[0036] Examples of non-volatile memory devices 118, 120 used to implement the activation function may include similar devices used in the RPU 104. Various types of non-volatile memory based on different physical mechanisms may be used: for example, phase change memory, which uses a phase change material sandwiched between two electrodes to control conductance or resistivity by changing between amorphous or crystalline states; resistive random access memory, which uses a filament between structures to control conductance or resistivity; and ferroelectric memory, which uses polarization to control conductance or resistivity. Generally, resistance or conductance can be fine-tuned to store information and perform the activation function.

[0037] As described above, different rows 106 of the crossbar array represent different inputs, and different columns 108 of the crossbar array represent different outputs. The crossbar array can be used to perform linear matrix operations and collect the outputs on the columns. For example, a multiply-accumulate operation on the crossbar array 102 is followed by a non-volatile memory-based activation. The conductance of the non-volatile memory is related to the gradient of the leaky ReLU. Applying the outputs to an activation function device 126 introduces the non-linear aspects of the neural network. In one embodiment, each of the output columns 108 can be connected to a respective readout circuit and activation function device 126.

[0038] One embodiment of activation function device 126 may operate as follows: Based on the voltage per column (the voltage from each column 108 or 114), one of diodes 122 and 124 may be turned on. The leakage ReLU indicates that based on whether the output voltage is positive or negative, current will go to a different diode branch, e.g., 118 / 122 or 120 / 124. For example, a positive voltage applied to device 126 (e.g., Vout 114 is positive) will turn on diode 122 because its positive terminal connects to the column output of the crossbar array (via the NVM of 118), thereby allowing current to flow, e.g., as in a forward biased state, while current flow in diode 124 will be restricted, as in a reverse biased state. Similarly, a negative voltage applied to device 126 (e.g., Vout 114 is negative) will turn on diode 124 because the negative terminal of diode 124 connects (via NVM 120) to the column output of the crossbar array, allowing current to flow through diode 124 as in the forward bias state, while restricting current flow in diode 122 as in the reverse bias state. The conductance or resistance values ​​of memories 118 and 120 are different, thereby representing different slopes of the leaky ReLU. Similarly, to implement a ReLU activation function, the value of NVM 118 will be set to some (predetermined) slope, while the value of NVM 120 will be set to a substantially zero value (substantially flat slope). Activation function device 126 outputs or generates activation function output 128. Activation output 128 may be further processed. For example, the activation output 128 may be converted to digital form by an analog-to-digital converter 134 for further processing, for example, by a digital computer. In another embodiment, the activation output 128 may be input to and processed by another crossbar array.For example, there may be one or more analog-to-digital converters such as those shown at 134 as peripheral circuitry to the crossbar array, but there may be no need for analog-to-digital converters to perform the activation function (e.g., in a digital computer), which may further save power because the activation function calculation can be performed entirely within an analog-based memory device.

[0039] An apparatus may be provided that implements an analog memory-based activation function. Such an apparatus may include, for example, two nonvolatile memory devices connected in parallel, referred to as a first nonvolatile memory and a second nonvolatile memory, such that a current can flow through one of the two nonvolatile memory devices depending on a voltage level driving the current. To control which branch the current will flow through, each of the two nonvolatile memory devices may be connected to a switch or a circuit element that can function as a switch, such as a diode, such as a semiconductor diode, a transistor, or another circuit element. Such an apparatus may implement, for example, an analog memory-based activation function for an analog memory-based artificial neural network.

[0040] In one embodiment, the device may include a first non-volatile memory connected to the positive terminal of the first diode. The first non-volatile memory may be programmed or tuned to store a first parameter of an activation function associated with the artificial neural network. The device may also include a second non-volatile memory connected to the negative terminal of the second diode, where the second non-volatile memory may be programmed or tuned to store a second parameter of an activation function associated with the artificial neural network. The first non-volatile memory and the second non-volatile memory may be connected in parallel to the input line, and the negative terminal of the first diode and the positive terminal of the second diode may be connected in parallel to the output line. In this way, for example, a signal entering the input line may pass through one of two non-volatile memory paths or branches depending on the value of the signal (e.g., voltage level), be transformed into an activation function output according to the activation function (e.g., having the parameter), and be output via the output line.

[0041] In one embodiment, the activation function may include a leaky rectified linear unit (ReLU), where a first gradient of the leaky ReLU may be determined by a first parameter stored on a first non-volatile memory, and a second gradient of the leaky ReLU may be determined by a second parameter stored on a second non-volatile memory. In one aspect, the first parameter may be proportional to the conductance of the first non-volatile memory, and the second parameter may be proportional to the conductance of the second non-volatile memory.

[0042] In another embodiment, the activation function may include a rectified linear unit (ReLU), where a first gradient of the ReLU may be determined by a first parameter stored on a first non-volatile memory, and a second gradient of the ReLU may be determined by a second parameter stored on a second non-volatile memory, where the second gradient is substantially flat, e.g., zero or a value representing substantially zero. For example, the second gradient may be significantly smaller than the first gradient, e.g., 100 times smaller, for example.

[0043] In one embodiment, the apparatus may also include a crossbar array of memristive elements configured to perform a multiply-accumulate operation, where at least one column output line of the crossbar array is connected to the input lines. For example, each of the column output lines of the crossbar array may be connected to a set of circuit elements (e.g., at least two diode-connected non-volatile memory devices) that implement an analog memory-based activation function.

[0044] In one embodiment, the first parameter and the second parameter remain fixed as hyperparameters of the artificial neural network during training of the artificial neural network, while the values ​​stored on the memristive elements of the crossbar array are updated as part of the training.

[0045] Non-volatile memory retains information even when power is turned off. In one aspect, non-volatile memory with a diode or diode structure provides efficiency for DNN acceleration. The presence of NVM elements in addition to the diode allows for tuning of leaky ReLU parameters.

[0046] FIG. 3 is a diagram showing an analog memory-based neural network having a non-volatile memory-based activation function in another embodiment. The crossbar array 102 and its components, and its peripheral circuits, such as 132, 134, also operate in the same manner as described above with reference to FIG. 1. In this embodiment, the analog-based activation function device 302 can be implemented using a pair of elements, a pair including at least a non-volatile memory device and a transistor device (e.g., 304, 308; e.g., 306, 310). For example, the diodes 122, 124 shown in FIG. 1 may be replaced by transistors, such as field effect transistors (FETs) like nFETs and pFETs, where the source is directed to the column output 114 and the drain is directed to the activation function output 128. For example, an FET has three terminals, and when the third terminal (e.g., the gate shown as Vg) is driven by an appropriate signal, a current can flow between the two terminals (source and drain). For example, the gate Vg can be synchronized with the reading of the column output 108 or the column output 114. The FET has a threshold voltage (Vt), which is the voltage between the gate and the source required to create a conductive path between the source and drain terminals. In an nFET, when Vg is greater than or equal to Vt (Vg >= Vt), the nFET turns on, and when Vg is less than Vt (Vg < Vt), the nFET turns off. In a pFET, when Vg < Vt, the pFET turns on, and when Vg >= Vt, the pFET turns off. For example, saying Vg is high or low is relative to the corresponding Vt of the FET. When Vg is higher than the nFET Vt, the nFET 308 turns on, and since Vg is higher than the pFET Vt, the pFET 310 turns off. Similarly, when Vg is lower than the nFET Vt, the nFET 308 turns off, and since this Vg is also lower than the pFET Vt, the pFET 310 turns on.In one embodiment where activation function device 302 represents a leaky ReLU activation function, the non-volatile memory of 304 stores or is tuned to the first gradient of the leaky ReLU, i.e., the gradient (e.g., 202) in the positive region (e.g., 210); the non-volatile memory of 306 stores or is tuned to the second gradient of the leaky ReLU, i.e., the gradient (e.g., 204) in the negative region (e.g., 212). In another embodiment where activation function device 302 represents a ReLU activation function, the non-volatile memory of 304 stores or is tuned to the first gradient of the ReLU, i.e., the gradient (e.g., 206) in the positive region (e.g., 214); and the non-volatile memory of 306 stores or is tuned to the second gradient of the ReLU, i.e., the gradient (e.g., 208) in the negative region (e.g., 216) or zero. Thus, for example, when Vg is high, the nFET is turned on, the pFET is turned off, and the combination of the gradient value stored in the NVM 304 and the column output 114 is output as the activation function output 128; when Vg is low, the nFET is turned off, the pFET is turned on, and the combination of the gradient value stored in the NVM 306 and the column output 114 is output as the activation function output 128.

[0047] In another embodiment, an apparatus implementing an analog memory-based activation function may include a first non-volatile memory coupled to a first field effect transistor (FET). The first non-volatile memory may be programmed or tuned to store a first parameter of an activation function associated with the artificial neural network. The apparatus may also include a second non-volatile memory coupled to a second field effect transistor (FET), where the second non-volatile memory may be programmed or tuned to store a second parameter of an activation function associated with the artificial neural network. The first non-volatile memory and the second non-volatile memory may be coupled in parallel to an input line 312, and the first field effect transistor and the second field effect transistor may be coupled in parallel to an output line. For example, a source terminal of the first field effect transistor may be coupled to the first non-volatile memory, and a source terminal of the second field effect transistor may be coupled to the second non-volatile memory; a drain terminal of the first field effect transistor and a drain terminal of the second field effect transistor may be coupled to an output line 314. In this way, a signal entering the input line can be converted into an activation function output according to the activation function (e.g., a current flowing through one of two paths constructed by two non-volatile memories) and output via the output line.

[0048] In one embodiment, the activation function may include a leaky rectified linear unit (ReLU), where a first gradient of the leaky ReLU may be determined by a first parameter stored on a first non-volatile memory, and a second gradient of the leaky ReLU may be determined by a second parameter stored on a second non-volatile memory. In one aspect, the first parameter may be proportional to the conductance of the first non-volatile memory, and the second parameter may be proportional to the conductance of the second non-volatile memory.

[0049] In another embodiment, the activation function may include a rectified linear unit (ReLU), where a first gradient of the ReLU may be determined by a first parameter stored on a first non-volatile memory, and a second gradient of the ReLU may be determined by a second parameter stored on a second non-volatile memory, where the second gradient is substantially flat, e.g., zero or a value representing substantially zero. For example, the second gradient may be significantly smaller than the first gradient, e.g., 100 times smaller, for example.

[0050] In one embodiment, the apparatus may also include a crossbar array of memristive elements configured to perform a multiply-and-accumulate operation, where at least one column output line of the crossbar array is connected to gate voltages of a first FET and a second FET. For example, each of the column output lines of the crossbar array may be connected to a set of circuit elements (e.g., at least two non-volatile memory devices connected to FETs) that implement an analog memory-based activation function. In one embodiment, the first FET may be an n-channel field effect transistor (nFET) and the second FET may be a p-channel field effect transistor (pFET).

[0051] In one embodiment, the first parameter and the second parameter remain fixed as hyperparameters of the artificial neural network during training of the artificial neural network, while the values ​​stored on the memristive elements of the crossbar array are updated as part of the training.

[0052] As described above, non-volatile memory devices can be used to implement analog memory-based activation functions. Such analog memory-based activation functions can be used, for example, to calculate the activation of neurons in a neural network implemented on an analog memory-based crossbar array. There are at least two non-volatile memory devices, each of which can store a parameter value associated with an activation function, such as a ReLU, a leaky ReLU, or the gradient of another. There can also be circuit elements connected to the non-volatile memory devices for controlling the selection of current flow through the paths of the two non-volatile memory devices. An analog memory-based crossbar array with integrated capacitors for deep learning can be integrated with the analog memory-based activation function devices.

[0053] Figure 4 is a flow diagram illustrating a method in one embodiment. The method may be implemented, for example, to provide an analog memory-based activation function for an analog memory-based artificial neural network (e.g., as shown in Figures 1 and 3). In this way, for example, an analog signal representing the result of a multiply-and-accumulate operation performed by a crossbar array implementing the artificial neural network does not need to be converted to a digital signal for activation function calculation. At 402, a first non-volatile memory is tuned to store first parameters of the activation function of the artificial neural network, where the first non-volatile memory is connected or coupled to a column output of a crossbar array of memristive elements (e.g., resistive processing units shown in Figures 1 and 3) implementing the artificial neural network.

[0054] At 404, a second non-volatile memory coupled to a column output of the crossbar array of memristive elements is tuned to store a second parameter of an activation function of the artificial neural network. The first non-volatile memory and the second non-volatile memory may be coupled in parallel to the column output of the crossbar array.

[0055] At 406, the synaptic weights stored in the memristive elements of the crossbar array may be updated during training of the artificial neural network, while the first and second parameters may be maintained fixed as hyperparameters of the artificial neural network. For example, these values ​​do not change during training of the artificial neural network, during which the values ​​of the memristive elements are updated. The first and second non-volatile memories implement an analog memory-based activation function of the analog memory-based artificial neural network.

[0056] In one embodiment, the activation function may be a leaky rectified linear unit (ReLU). A first gradient of the leaky ReLU may be determined by a first parameter stored on a first non-volatile memory. A second gradient of the leaky ReLU may be determined by a second parameter stored on a second non-volatile memory.

[0057] In another embodiment, the activation function may be a rectified linear unit (ReLU), where a first gradient of the ReLU may be determined by a first parameter stored on a first non-volatile memory, and a second gradient of the ReLU may be determined by a second parameter stored on a second non-volatile memory, where the second gradient is substantially flat (e.g., zero or a value substantially representing zero). For example, the second gradient may be significantly smaller than the first gradient, such as, for example, 100 times smaller.

[0058] In one embodiment, for example as shown in FIG. 1, a first non-volatile memory is connected to the positive terminal of a first diode and a second non-volatile memory is connected to the negative terminal of a second diode.

[0059] In another embodiment, for example as shown in FIG. 3, a first non-volatile memory is connected to an n-channel field effect transistor (nFET) and a second non-volatile memory is connected to a p-channel field effect transistor (pFET), wherein at least one column output line of the crossbar array is connected to the gate voltages of the nFET and the pFET.

[0060] In one embodiment, only one non-volatile memory is used, as the current will pass through one branch of two parallel connected non-volatile memories depending on whether a given voltage (e.g., a column output of a crossbar array) is high or low (e.g., positive or negative).

[0061] In the above description, both memory elements NVM1 (118) and NVM2 (120) shown in FIG. 1 are non-volatile memory elements that can be tuned to obtain the first and second gradients. However, in other embodiments, one of the memory elements can be a tunable non-volatile memory element, while the other can be a fixed element, such as a fixed resistor. Similarly, both memory elements NVM1 (304) and NVM2 (306) shown in FIG. 3 are shown as tunable non-volatile memory elements. Again, however, in other embodiments, one of the memory elements can be a tunable non-volatile memory element, while the other can be a fixed element, such as a fixed resistor. For example, one of the NVMs can be substituted with a fixed resistor. For example, in implementing the ReLU shown in FIG. 2B, the fixed element can be substituted for or replaced with a non-volatile memory element that stores a value representing, for example, a substantially flat gradient.

[0062] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. As used herein, the term "or" is an inclusive operator and can mean "and / or" unless the context explicitly or clearly indicates otherwise. It will be further understood that as used herein, the terms "comprise," "comprises," "comprising," "include," "includes," "including," and / or "having" may specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the phrase "in an embodiment" does not necessarily refer to the same embodiment, but may. As used herein, the phrase "in one embodiment" does not necessarily refer to the same embodiment, but may. As used herein, the phrase "in another embodiment" does not necessarily refer to a different embodiment, but may. Furthermore, embodiments and / or components of embodiments may be freely combined with each other unless they are mutually exclusive.

[0063] The corresponding structure, material, acts, and equivalents of all means or step plus function elements in the following claims are intended to include, if present, any structure, material, or acts for performing that function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the invention to the form disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the invention. The embodiments have been chosen and described to best explain the principles and practical applications of the invention and to enable others skilled in the art to understand the invention in various embodiments with various modifications suited to the particular uses contemplated.

Claims

1. a first non-volatile memory connected to the positive terminal of the first diode, the first non-volatile memory programmed to store first parameters of an activation function associated with the artificial neural network; and a second non-volatile memory connected to the negative terminal of the second diode, the second non-volatile memory programmed to store second parameters of the activation function associated with the artificial neural network; Equipped with The first non-volatile memory and the second non-volatile memory are connected in parallel to an input line, and the negative terminal of the first diode and the positive terminal of the second diode are connected in parallel to an output line.

2. 2. The apparatus of claim 1, wherein the activation function comprises a leaky normalized linear unit (ReLU), a first slope of the leaky ReLU determined by the first parameter stored on the first non-volatile memory, a second slope of the leaky ReLU determined by the second parameter stored on the second non-volatile memory, the first parameter being proportional to a conductance of the first non-volatile memory, and the second parameter being proportional to a conductance of the second non-volatile memory.

3. 2. The apparatus of claim 1, wherein the activation function comprises a rectified linear unit (ReLU), a first slope of the ReLU determined by the first parameter stored on the first non-volatile memory, a second slope of the ReLU determined by the second parameter stored on the second non-volatile memory, the first parameter being proportional to a conductance of the first non-volatile memory, and the second parameter being proportional to a conductance of the second non-volatile memory.

4. 10. The apparatus of claim 1, further comprising a crossbar array of memristive elements configured to perform multiply-accumulate operations, at least one column output line of the crossbar array connected to the input line.

5. 5. The apparatus of claim 4, wherein the first parameter and the second parameter remain fixed as hyperparameters of the artificial neural network during training of the artificial neural network, and wherein values ​​stored on the memristive elements of the crossbar array are updated as part of the training.

6. The apparatus of claim 1 , wherein the apparatus implements an analog memory-based activation function of an analog memory-based artificial neural network.

7. a first non-volatile memory coupled to a first field effect transistor (FET), the first non-volatile memory being programmed to store first parameters of an activation function associated with the artificial neural network; and a second non-volatile memory coupled to a second field effect transistor (FET), the second non-volatile memory programmed to store second parameters of the activation function associated with the artificial neural network; Equipped with a first non-volatile memory connected in parallel to an input line, a second field effect transistor connected in parallel to an output line, a source terminal of the first field effect transistor connected to the first non-volatile memory, a source terminal of the second field effect transistor connected to the second non-volatile memory, and a drain terminal of the first field effect transistor and a drain terminal of the second field effect transistor connected to the output line.

8. 8. The apparatus of claim 7, wherein the activation function comprises a leaky normalized linear unit (ReLU), a first slope of the leaky ReLU determined by the first parameter stored on the first non-volatile memory, a second slope of the leaky ReLU determined by the second parameter stored on the second non-volatile memory, the first parameter being proportional to a conductance of the first non-volatile memory, and the second parameter being proportional to a conductance of the second non-volatile memory.

9. 8. The apparatus of claim 7, wherein the activation function comprises a rectified linear unit (ReLU), a first slope of the ReLU determined by the first parameter stored on the first non-volatile memory, a second slope of the ReLU determined by the second parameter stored on the second non-volatile memory, the first parameter being proportional to a conductance of the first non-volatile memory, and the second parameter being proportional to a conductance of the second non-volatile memory.

10. 8. The apparatus of claim 7, further comprising a crossbar array of memristive elements configured to perform a multiply-accumulate operation, at least one column output line of the crossbar array connected to gate voltages of the first FET and the second FET.

11. 11. The apparatus of claim 10, wherein the first FET comprises an n-channel field effect transistor (nFET) and the second FET comprises a p-channel field effect transistor (pFET).

12. 11. The apparatus of claim 10, wherein the first parameter and the second parameter remain fixed as hyperparameters of the artificial neural network during training of the artificial neural network, and wherein values ​​stored on the memristive elements of the crossbar array are updated as part of the training.

13. 8. The apparatus of claim 7, wherein the apparatus implements an analog memory-based activation function of an analog memory-based artificial neural network.

14. tuning a first non-volatile memory coupled to a column output of the crossbar array of memristive elements such that said first non-volatile memory stores a first parameter of an activation function of an artificial neural network; tuning a second non-volatile memory coupled to the column output of the crossbar array of memristive elements such that the second non-volatile memory stores a second parameter of the activation function of the artificial neural network; maintaining the first and second parameters fixed as hyperparameters of the artificial neural network, while updating synaptic weights stored in the memristive elements of the crossbar array during training of the artificial neural network. A method comprising:

15. 15. The method of claim 14, wherein the activation function comprises a leaky rectified linear unit (ReLU), a first gradient of the leaky ReLU determined by the first parameters stored on the first non-volatile memory, and a second gradient of the leaky ReLU determined by the second parameters stored on the second non-volatile memory.

16. 15. The method of claim 14, wherein the activation function comprises a rectified linear unit (ReLU), a first gradient of the ReLU determined by the first parameters stored on the first non-volatile memory, and a second gradient of the ReLU determined by the second parameters stored on the second non-volatile memory.

17. 15. The method of claim 14, wherein the first non-volatile memory is connected to a positive terminal of a first diode and the second non-volatile memory is connected to a negative terminal of a second diode.

18. 15. The method of claim 14, wherein the first non-volatile memory is connected to an n-channel field effect transistor (nFET), the second non-volatile memory is connected to a p-channel field effect transistor (pFET), and at least one column output line of the crossbar array is connected to gate voltages of the nFET and the pFET.

19. 15. The method of claim 14, wherein the first non-volatile memory and the second non-volatile memory implement an analog memory-based activation function of an analog memory-based artificial neural network.

20. 2. The device of claim 1, wherein the first non-volatile memory and the second non-volatile memory comprise at least one selected from the group consisting of resistive random access memory (RRAM), electrochemical random access memory (ECRAM), ferroelectric random access memory (FeRAM), phase change memory (PCM), conductive bridge RAM (CBRAM), NOR flash, and magnetic RAM (MRAM).

Images