Non-volatile memory-based activation function

Abstract

Analog memory-based activation functions for artificial neural networks may be provided. An apparatus may include at least two non-volatile memory devices connected in parallel such that a current may flow through one of the two non-volatile memory devices according to a voltage level of a drive current. To control which branch the input current flows through, each of the two non-volatile memory devices may be connected to a circuit element that may function as a switch, for example, a diode such as a semiconductor diode, a transistor, or other circuit element. Such an apparatus may implement analog memory-based activation functions, e.g., for analog memory-based artificial neural networks.

Description

Background Art

[0001] This application generally relates to artificial neural networks based on analog memories, and more particularly to adjustable analog or non-volatile memory-based activation functions for deep learning.

[0002] An analog memory crossbar array that implements multiply and accumulate operations can accelerate the performance of deep learning neural networks or deep neural networks (DNNs). Activation functions that convert the linear output of such MAC operations into neuron outputs are generally implemented in digital circuits (e.g., using analog-to-digital converters). Summary of the Invention

[0003] An overview of the present disclosure is given to aid in understanding analog memory or non-volatile memory-based activation functions (e.g., activation functions for deep learning), without intending to limit the present disclosure or the invention. It should be understood that various aspects and features of the present disclosure may be advantageously used alone in some cases, or in combination with other aspects and features of the present disclosure in other cases. Accordingly, variations and modifications may be made to the computer system and / or its method of operation to achieve different effects.

[0004] In one aspect, a device may include a first non-volatile memory connected to the positive terminal of a first diode. The first non-volatile memory may be programmed to store a first parameter of an activation function associated with an artificial neural network. The device may also include a second non-volatile memory connected to the negative terminal of a second diode. The second non-volatile memory may be programmed to store a second parameter of an activation function associated with an artificial neural network. The first non-volatile memory and the second non-volatile memory may be connected in parallel to an input line, and the negative terminal of the first diode and the positive terminal of the second diode may be connected in parallel to an output line.

[0005] Advantageously, implementing an analog memory-based activation function can provide energy efficiency in an analog memory-based neural network implementation.

[0006] In one aspect, the activation function may include a leaky rectified linear unit (ReLU), where a first slope of the leaky ReLU may be defined by a first parameter stored on the first non-volatile memory, and a second slope of the leaky ReLU may be defined by a second parameter stored on the second non-volatile memory. The first parameter may be proportional to the conductance of the first non-volatile memory, and the second parameter may be proportional to the conductance of the second non-volatile memory.

[0007] Advantageously, implementing a leaky rectified linear unit (ReLU) activation function based on analog memory can provide energy efficiency in an analog memory-based neural network implementation.

[0008] In another aspect, the activation function can include a rectified linear unit (ReLU), where a first slope of the ReLU can be defined by a first parameter stored on a first non-volatile memory, and a second slope of the ReLU can be defined by a second parameter stored on a second non-volatile memory. The first parameter can be proportional to the conductance of the first non-volatile memory, and the second parameter can be proportional to the conductance of the second non-volatile memory.

[0009] Advantageously, implementing a rectified linear unit (ReLU) activation function based on analog memory can provide energy efficiency in an analog memory-based neural network implementation.

[0010] In another aspect, a device can include a first non-volatile memory connected to a first field-effect transistor (FET). The first non-volatile memory can be programmed to store a first parameter of an activation function associated with an artificial neural network. The device can also include a second non-volatile memory connected to a second field-effect transistor (FET). The second non-volatile memory can be programmed to store a second parameter of an activation function associated with the artificial neural network. The first non-volatile memory and the second non-volatile memory can be connected in parallel to an input line. The first field-effect transistor and the second field-effect transistor can be connected in parallel to an output line, where a source terminal of the first field-effect transistor can be connected to the first non-volatile memory, and a source terminal of the second field-effect transistor can be connected to the second non-volatile memory, and where a drain terminal of the first field-effect transistor and a drain terminal of the second field-effect transistor can be connected to the output line.

[0011] Advantageously, implementing an activation function based on analog memory can provide energy efficiency in an analog memory-based neural network implementation.

[0012] In one aspect, a method can include adjusting a first non-volatile memory connected to a column output of a crossbar array of memristive elements to cause the first non-volatile memory to store a first parameter of an activation function of an artificial neural network. The method can also include adjusting a second non-volatile memory connected to a column output of the crossbar array of memristive elements to cause the second non-volatile memory to store a second parameter of the activation function of the artificial neural network. The method can further include keeping the first parameter and the second parameter fixed as hyperparameters of the artificial neural network when updating synaptic weights stored in the memristive elements of the crossbar array during artificial neural network training.

[0013] Advantageously, implementing an analog-memory-based activation function can provide energy efficiency in an analog-memory-based neural network implementation.

[0014] A computer-readable storage medium can also be provided that stores an instruction program executable by a machine to perform one or more of the methods described herein.

[0015] Further features as well as structures and operations of various embodiments will be described in detail below with reference to the drawings. In the drawings, the same reference numerals indicate the same or functionally similar elements. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic diagram of an analog-memory-based neural network in an exemplary embodiment.

[0017] Figure 2A illustrates an example leaky rectified linear unit (ReLU) activation function.

[0018] Figure 2B illustrates an example rectified linear unit (ReLU) activation function.

[0019] Figure 3 is a schematic diagram of an analog-memory-based neural network with a non-volatile-memory-based activation function in another exemplary embodiment.

[0020] Figure 4 is a flowchart of a method in an exemplary embodiment. DETAILED DESCRIPTION

[0021] An analog-memory-based neural network can utilize the storage capabilities and physical characteristics of a memory device such as a non-volatile memory device to implement an artificial neural network. This type of in-memory computing hardware improves speed and energy efficiency, providing potential performance improvements. For example, instead of moving data from dynamic random access memory (DRAM) to a processor such as a central processing unit (CPU) to perform calculations, an analog neural network chip performs calculations at the same location where the data is stored. Since there is no movement of data, tasks can be performed faster and require less energy. In one or more embodiments, an analog-memory-based or non-volatile-memory-based activation function is provided.

[0022] The implementation of an artificial neural network can include a series of neuron layers that are interconnected such that the output signals of the neurons in one layer are weighted and transmitted to the neurons in the next layer. A neuron Ni in a given layer can be connected to one or more neurons Nj in the next layer, and different weights wij can be associated with each neuron-neuron connection Ni-Nj for weighting the signal transmitted from Ni to Nj. The neuron Nj generates an output signal based on its cumulative input applied to an activation function, and the weighted signals can propagate through the series of layers of the network from the input neuron layer to the output neuron layer. In short, the activation function determines whether a neuron should be activated or the level of activation of a neuron, such as the output of a neuron. An artificial neural network machine learning model can undergo a training phase in which a set of weights associated with the respective neuron layers is determined. In an iterative training scheme, the network is exposed to a training data set in which the weights are repeatedly updated as the network "learns" from the training data. The resulting trained model with weights defined via the training operation can be applied to perform tasks based on new data.

[0023] Figure 1 FIG. is a schematic diagram of a neural network based on analog memory in an exemplary embodiment. The neural network based on analog memory includes an activation function based on non-volatile memory. For example, a circuit element including non-volatile memory can implement the activation function of a neural network. In short, a non-volatile memory (NVM) device (also referred to as a memristive device or technology) can maintain the values stored thereon even when the power is turned off. In a neural network implementation, these values can represent the synaptic weights or weights of a neural network.

[0024] An electronic device using a resistive NVM device 104 (labeled as a resistive processing unit (RPU)) and a computational memory based on a crossbar array (the crossbar array is shown at 102 in the figure for illustrative purposes) can be used for artificial neural network (ANN) computations, such as for training a deep neural network (DNN) and / or as an inference accelerator for inferring using such a network. For example, in deep learning or neural network inference, the propagation of data through multiple layers of a neural network involves a series of matrix multiplications. Each layer can be represented as a synaptic weight matrix. These weights can be stored in the conductance states of the NVM devices 104. These NVM devices 104 can be arranged in a crossbar array to create an artificial neural network in which all matrix multiplications are performed in-situ in an analog manner. For example, an array on a chip can be directly related to the synaptic layer of a neural network. In an embodiment, for example, the crossbar array can represent a layer of a neural network.

[0025] For example, a multiply-accumulate (MAC) device may include NVM devices arranged in a crossbar configuration (e.g., as crossbar array 102), and may perform analog matrix-vector multiplication in a single time step. Such MAC devices can be used to implement neural networks based on hardware or analog memory.

[0026] Examples of such memory devices may include, but are not limited to, resistive random access memory (RRAM or ReRAM), electrochemically random access memory (ECRAM), ferroelectric random access memory (FeRAM), phase change memory (PCM), conductive bridge RAM (CBRAM), NOR flash memory, magnetic RAM (MRAM).

[0027] In an embodiment, an analog memory-based device may implement a hardware neural network. For example, in a system including a crossbar array with an analog memory-based device (e.g., as a coprocessor or accelerator), one or more digital processors may communicate with the coprocessor while performing their operations or functions for various applications.

[0028] In an embodiment, such a device may be a coprocessor or accelerator that includes multiply-accumulate (MAC) hardware having a crossbar structure or array, such as shown in 102. Although one crossbar array is shown in the figure, many such arrays may be integrated on the coprocessor. By way of example, an analog multiply-accumulate device may include an electronic device containing memory elements 104 that are arranged at the intersections of the crossbar array 102. For example, at each intersection or node of the crossbar structure or crossbar array, there may be at least one electronic device 104 containing a resistive memory or memristive element. In an embodiment, such resistive memory elements may be programmed to store the synaptic weights of an artificial neural network (ANN). Each array 102 may represent a layer of the ANN. For example, the coprocessor may include one of the resistive memory elements 104 at each intersection that connects a corresponding one of the input lines 106 to a corresponding one of the output lines 108. The array 102 may be a regular array having a constant distance between intersections in the horizontal and vertical dimensions of the substrate surface. Each crossbar array 102 may perform vector-matrix multiplication. By way of example, the array 102 may include peripheral circuits such as a pulse width modulator and peripheral circuits such as an analog-to-digital converter (ADC).

[0029] For example, a device including a co-processor, such as the crossbar array 102, can interface with hardware including another processor (e.g., a field programmable gate array (FPGA)). There can also be a digital-to-analog converter, which can supply power, voltage, and current to the co-processor. For example, the digital-to-analog converter 132 can convert digital input data (referred to as input x) 130 into an analog signal (e.g., 110) that is input to the crossbar array 102. A processor such as a field programmable gate array (FPGA) can implement digital logic to interface with the electronics of the co-processor and the digital-to-analog converter.

[0030] An electrical pulse or voltage signal 110 can be input (or applied) to the input lines 106 of the crossbar structure 102. An output current 112 can be obtained from the output lines 108 of the crossbar structure. For example, based on a multiply-accumulate operation, the output current 112 is generated based on the input pulse or voltage signal 110 applied to the input lines 106 and the values (synaptic weights) stored on the resistive memory elements 104. The output 114 can be read out at the read circuit 116.

[0031] Additional circuit elements, at least including a non-volatile memory device (collectively referred to as an analog-based activation function device for ease of explanation), can be connected to the output line 108 to implement an activation function based on analog memory. For example, the output from a column of the crossbar array, which represents a multiply-accumulate operation (linear operation) performed on a neuron of a neural network, can be connected to the input of a circuit element that implements an activation function based on analog memory. In an embodiment, the circuit element can include two non-volatile memory devices 118, 120 and diodes 122, 124 connected to each of the two non-volatile memory devices. The diodes can be used to control or select the direction of current (the column output of the crossbar array) in the analog-based activation function device. The diodes can be semiconductor diodes having at least a positive terminal and a negative terminal, such as a p-n junction diode. The diode located at 122 (referred to as the first diode for ease of explanation) can be connected to the non-volatile memory device located at 118 (referred to as the first non-volatile memory device for ease of explanation), where the positive terminal of the first diode is closer to the first non-volatile memory device 118. The diode located at 124 (referred to as the second diode for ease of explanation) can be connected to the non-volatile memory device located at 120 (referred to as the second non-volatile memory device), where the negative terminal of the second diode is closer to the second non-volatile memory device 120. The first non-volatile memory device 118 and the first diode 122 are connected in series with each other; the second non-volatile memory device 120 and the second diode 124 are connected in series with each other. The two sets of non-volatile memory devices and diodes are connected in parallel: for example, the first non-volatile memory device 118 and the first diode 122 are connected in parallel with the second non-volatile memory device 120 and the second diode 124. The output 108 or 114 flows to both parallel paths.

[0032] The two NVMs 118, 120 and the diodes 122, 124 implement an activation function such as a leaky rectified linear unit (ReLU). Another example of an activation function that the two NVMs 118, 120 and the diodes 122, 124 can represent includes the ReLU activation function. Figure 2A A leaky rectified linear unit (ReLU) activation function is shown. Figure 2BShows a rectified linear unit (ReLU) activation function. The output of the multiply-accumulate operation on each column 108 can be input into an activation function implemented by two NVMs 118, 120 and diodes 122, 124, where non-linear operations in the neural network occur. The leaky ReLU has a slope 202 for positive values 210 of the output 108 and a smaller slope 204 for negative values 212 of the output 108, rather than the flat slope of the ReLU (as shown by the ReLU Figure 2B shown). For example, in the ReLU activation function shown in Figure 2B , the slope 208 of the negative region 216 is flat or zero, while there is a non-zero slope 206 in the positive region 214. In Figure 2A and Figure 2B , the x-axis represents the output value 108 (of the MAC operation), and the y-axis represents the activation value.

[0033] In an embodiment, the slope coefficient is determined before neural network training, i.e., during the training of the weights stored in the non-volatile memory devices 104 in the crossbar array, the slope coefficient is not learned. For example, in an embodiment, the slope values can be stored as hyperparameters in the two NVMs 118, 120. During the training of the neural network, these values remain fixed.

[0034] In an embodiment, the first non-volatile memory device 118 can store the slope 202 of the leaky ReLU to be applied when the output value 108 is positive (e.g., greater than 1); the second non-volatile memory device 120 can store the slope 204 of the leaky ReLU to be applied when the output value 108 is negative (e.g., less than 1).

[0035] In another embodiment, to implement the ReLU activation function, the first non-volatile memory device 118 can store the slope 206 of the ReLU to be applied when the output value 108 is positive (e.g., greater than 1); the second non-volatile memory device 120 can store the substantially flat slope (or substantially zero value) 208 of the ReLU to be applied when the output value 108 is negative (e.g., less than 1).

[0036] Examples of non - volatile memory devices 118, 120 for implementing activation functions may include devices similar to those used for the RPU 104. Different types of non - volatile memory based on different physical mechanisms can be used: for example, phase - change memory, where a phase - change material sandwiched between two electrodes can be used to control conductance or resistivity by changing between an amorphous or crystalline state; resistive random - access memory, where filaments between structures can be used to control conductance or resistivity; ferroelectric memory, where polarization can be used to control conductance or resistivity. Generally, the resistance or conductance can be finely tuned to store information and perform activation functions.

[0037] As described above, different rows 106 of the crossbar array represent different inputs, and different columns 108 of the crossbar array represent different outputs. Using the crossbar array, linear matrix operations can be performed and the outputs can be collected on the columns. For example, the activation operation based on non - volatile memory follows a multiply - accumulate operation on the crossbar array 102. The non - volatile memory conductance is related to the slope of the leaky ReLU. Applying the output to the activation function device 126 introduces the non - linear aspect of the neural network. In an embodiment, each output column 108 can be connected to a corresponding readout circuit and activation function device 126.

[0038] An embodiment of the activation function device 126 may operate as follows. Based on the voltage of each column (the voltage from each of columns 108 or 114), one of the diodes 122 and 124 may be turned on. Leaky ReLU is represented, as based on whether the output voltage is positive or negative, current will flow into different diode branches, such as 118 / 122 or 120 / 124. For example, applying a positive voltage to device 126 (e.g., Vout 114 is positive) will turn on the diode at 122, as the positive terminal of this diode 122 is connected to the column output of the crossbar array (via the NVM at 118), allowing current to flow in a forward-biased state, while the current flow in the diode at 124 will be restricted and in a reverse-biased state. Similarly, applying a negative voltage to device 126 (e.g., Vout 114 is negative) will turn on the diode at 124, as the negative terminal of this diode 124 is connected to the column output of the crossbar array (via the NVM at 120), allowing current to flow in a forward-biased state via diode 124, while the current flow in the diode at 122 will be restricted and in a reverse-biased state. The conductance or resistance values of the memories 118 and 120 will be different, representing different slopes of the leaky ReLU. Similarly, to implement the ReLU activation function, the value of the NVM at 118 will be set to a certain (predetermined) slope, while the value of the NVM at 120 will be set to a value that is essentially zero (an essentially flat slope). The activation function device 126 outputs or generates an activation function output 128. The activation output 128 may be further processed. For example, the activation output 128 may be converted to digital form by an analog-to-digital converter 134, for further processing by a digital computer, for example. On the other hand, the activation output 128 may be input to another crossbar array for processing by that other crossbar array. Although there may be one or more analog-to-digital converters as shown at 134 (e.g., as peripheral circuitry of the crossbar array), the need for an analog-to-digital converter for performing the activation function (e.g., in a digital computer) may be eliminated, which may further save power as the activation function calculation may be performed entirely in the analog-based memory device.

[0039] An apparatus for implementing an activation function based on an analog memory may be provided. Such an apparatus may include two non-volatile memory devices connected in parallel (e.g., referred to as a first non-volatile memory and a second non-volatile memory, respectively), such that current may flow through one of the two non-volatile memory devices based on the voltage level of the driving current. To control which branch the current will flow through, each of the two non-volatile memory devices may be connected to a switch or a circuit element that can act as a switch, such as a diode like a semiconductor diode, a transistor, or another circuit element. Such an apparatus may implement an activation function based on an analog memory, for example, for an artificial neural network based on an analog memory.

[0040] In an embodiment, the apparatus may include a first non-volatile memory connected to the positive terminal of a first diode. The first non-volatile memory may be programmed or adjusted to store a first parameter of an activation function associated with an artificial neural network. The apparatus may further include a second non-volatile memory connected to the negative terminal of a second diode, where the second non-volatile memory may be programmed or adjusted to store a second parameter of the activation function associated with the artificial neural network. The first non-volatile memory and the second non-volatile memory may be connected in parallel to an input line, and the negative terminal of the first diode and the positive terminal of the second diode may be connected in parallel to an output line. In this way, for example, a signal entering the input line may pass through one of two non-volatile memory paths or branches according to the value of the signal (e.g., voltage level), and may be converted to an activation function output according to the activation function (e.g., using its parameters) and output via the output line.

[0041] In an embodiment, the activation function may include a leaky rectified linear unit (ReLU), where a first slope of the leaky ReLU may be defined by a first parameter stored on the first non-volatile memory, and a second slope of the leaky ReLU may be defined by a second parameter stored on the second non-volatile memory. In one aspect, the first parameter may be proportional to the conductance of the first non-volatile memory, and the second parameter may be proportional to the conductance of the second non-volatile memory.

[0042] In another embodiment, the activation function may include a rectified linear unit (ReLU), where a first slope of the ReLU may be defined by a first parameter stored in the first non-volatile memory, and a second slope of the ReLU may be defined by a second parameter stored in the second non-volatile memory, where the second slope is substantially flat, e.g., zero or a value substantially representing zero. For example, the second slope may be significantly less than the first slope, e.g., 100 times smaller for example.

[0043] In an embodiment, the apparatus may further include a crossbar array of memristive elements configured to perform multiply-accumulate operations, where at least one column output line of the crossbar array is connected to the input line. For example, each column output line of the crossbar array may be connected to a set of circuit elements implementing an activation function based on an analog memory (e.g., at least two non-volatile memory devices connected to diodes).

[0044] In an embodiment, during the training of the artificial neural network, the first parameter and the second parameter remain fixed as hyperparameters of the artificial neural network, while the values stored on the memristive elements of the crossbar array are updated as part of the training.

[0045] Non-volatile memory retains information even when the power is turned off. In one aspect, non-volatile memory with diodes or diode structures provides efficiency for DNN acceleration. The presence of NVM elements other than diodes enables the adjustment of leaky ReLU parameters.

[0046] Figure 3 FIG. illustrates a schematic diagram of an analog memory-based neural network having an activation function based on non-volatile memory in another embodiment. The crossbar array 102 and its components, as well as its peripheral circuits (such as 132, 134), operate similarly to those described above with reference to Figure 1 The operations are performed. In this embodiment, the analog-based activation function device 302 can be implemented using pairs of elements, the element pairs including at least non-volatile storage (such as: 304, 308; such as: 306, 310) devices and transistor devices. For example, Figure 1The diodes 122, 124 shown in [FIGURE] can be replaced by transistors (e.g., field effect transistors (FETs), such as nFETs and pFETs), where the source faces the column output 114 and the drain faces the activation function output 128. For example, an FET has three terminals, and when the third terminal (gate, e.g., shown as Vg) is driven by an appropriate signal, current can conduct between the two terminals (source and drain). For example, the gate Vg can be synchronized with the readout of the column output 108 or the column output 114. The FET has a threshold voltage (Vt), i.e., the voltage required between the gate and the source to form a conduction path between the source terminal and the drain terminal. For an nFET, if Vg is greater than or equal to Vt (Vg >= Vt), the nFET conducts; if Vg is less than Vt (Vg < Vt), the nFET is cutoff. For a pFET, if Vg < Vt, the pFET conducts; if Vg >= Vt, the pFET is cutoff. For example, the high or low of Vg is relative to the corresponding Vt of the FET. When Vg is higher than the Vt of the nFET, the nFET 308 conducts, and since Vg is higher than the Vt of the pFET, the pFET 310 is cutoff. Similarly, when Vg is lower than the Vt of the nFET, the nFET 308 is cutoff, and since this Vg is also lower than the Vt of the pFET, the pFET 310 conducts. In an embodiment where the activation function device 302 represents a leaky ReLU activation function, the non-volatile memory at 304 stores the first slope of the leaky ReLU (i.e., the slope (e.g., 202) of the positive region (e.g., 210)), or is adjusted with this first slope; the non-volatile memory at 306 stores the second slope of the leaky ReLU (i.e., the slope (e.g., 204) of the negative region (e.g., 212)), or is adjusted with this second slope. In another embodiment where the activation function device 302 represents a ReLU activation function, the non-volatile memory at 304 stores the first slope of the ReLU (i.e., the slope (e.g., 206) of the positive region (e.g., 214)), or is adjusted with this second slope; the non-volatile memory at 306 stores the second slope of the ReLU (i.e., the slope (e.g., 208) or zero of the negative region (e.g., 216)), or is adjusted with this second slope. Thus, for example, when Vg is high, the nFET conducts and the pFET is cutoff, and the combination of the slope value stored in the NVM 304 and the column output 114 is output as the activation function output 128; when Vg is low, the nFET is cutoff and the pFET conducts, and the combination of the slope value stored in the NVM 306 and the column output 114 is output as the activation function output 128.

[0047] In another embodiment, an apparatus for implementing an activation function based on an analog memory may include a first non-volatile memory connected to a first field-effect transistor (FET). The first non-volatile memory may be programmed or adjusted to store a first parameter of an activation function associated with an artificial neural network. The apparatus may also include a second non-volatile memory connected to a second field-effect transistor (FET), where the second non-volatile memory may be programmed or adjusted to store a second parameter of the activation function associated with the artificial neural network. The first non-volatile memory and the second non-volatile memory may be connected in parallel to an input line 312, and the first field-effect transistor and the second field-effect transistor may be connected in parallel to an output line. For example, the source terminal of the first field-effect transistor may be connected to the first non-volatile memory, and the source terminal of the second field-effect transistor may be connected to the second non-volatile memory; the drain terminals of the first field-effect transistor and the second field-effect transistor may be connected to the output line 314. In this way, a signal entering the input line may be converted into an activation function output according to the activation function (e.g., a current flowing through one of two paths formed by two non-volatile memories) and output via the output line.

[0048] In an embodiment, the activation function may include a leaky rectified linear unit (ReLU), where a first slope of the leaky ReLU may be defined by a first parameter stored in the first non-volatile memory, and a second slope of the leaky ReLU may be defined by a second parameter stored in the second non-volatile memory. In one aspect, the first parameter may be proportional to the conductance of the first non-volatile memory, and the second parameter may be proportional to the conductance of the second non-volatile memory.

[0049] In another embodiment, the activation function may include a rectified linear unit (ReLU), where a first slope of the ReLU may be defined by a first parameter stored in the first non-volatile memory, and a second slope of the ReLU may be defined by a second parameter stored in the second non-volatile memory, where the second slope is substantially flat (e.g., zero or a value substantially representing zero). For example, the second slope may be significantly less than the first slope, say, 100 times smaller.

[0050] In an embodiment, the apparatus may further include a crossbar array of memristive elements configured to perform multiply-accumulate operations, wherein at least one column output line of the crossbar array is connected to the gate voltages of the first FET and the second FET. For example, each column output line of the crossbar array may be connected to a group of circuit elements that implement an activation function based on an analog memory (e.g., at least two non-volatile memory devices connected to FETs). In an embodiment, the first FET may be an n-channel field effect transistor (nFET), and the second FET may be a p-channel field effect transistor (pFET).

[0051] In an embodiment, during the training of an artificial neural network, the first parameter and the second parameter are kept fixed as hyperparameters of the artificial neural network, while the values stored on the memristive elements of the crossbar array are updated as part of the training.

[0052] As described above, non-volatile memory devices may be used to implement an activation function based on an analog memory. Such an activation function based on an analog memory may be used to compute, for example, the activation of a neuron in a neural network implemented on an analog memory-based crossbar array. There may be at least two non-volatile memory devices, each storing parameter values associated with the activation function. For example, the slope of ReLU, leaky ReLU, or another function. There may also be circuit elements connected to the non-volatile memory devices to control the selection of the path for current to flow through the two non-volatile memory devices. An analog memory-based crossbar array with integrated capacitors for deep learning may be integrated with an activation function device based on an analog memory.

[0053] Figure 4 is a flowchart of the method in an exemplary embodiment. The method may be executed for an analog memory-based artificial neural network (e.g., as Figure 1 and Figure 3 shown), for example, to provide an activation function based on an analog memory. In this way, for example, an analog signal representing the result of a multiply-accumulate operation performed by a crossbar array implementing an artificial neural network does not need to be converted to a digital signal for performing the activation function calculation. At 402, a first non-volatile memory is adjusted to store a first parameter of an activation function of the artificial neural network, wherein the first non-volatile memory is connected or coupled to a column output of a crossbar array of memristive elements (e.g., a resistive processing unit as Figure 1 and Figure 3 shown) implementing the artificial neural network.

[0054] At 404, a second non-volatile memory that adjusts the column output of the crossbar array connected to the memristive element is configured to store a second parameter of the activation function of the artificial neural network. The first non-volatile memory and the second non-volatile memory can be connected in parallel to the column output of the crossbar array.

[0055] At 406, when updating the synaptic weights stored in the memristive elements of the crossbar array during the training of the artificial neural network, the first parameter and the second parameter can be held fixed as hyperparameters of the artificial neural network. For example, during the training of the artificial neural network, these values do not change while the values of the memristive elements are updated during this period. The first non-volatile memory and the second non-volatile memory implement a simulation-memory-based activation function of the simulation-memory-based artificial neural network.

[0056] In an embodiment, the activation function can be a leaky rectified linear unit (ReLU). The first slope of the leaky ReLU can be defined by the first parameter stored on the first non-volatile memory. The second slope of the leaky ReLU can be defined by the second parameter stored on the second non-volatile memory.

[0057] In another embodiment, the activation function can be a rectified linear unit (ReLU), where the first slope of the ReLU can be defined by the first parameter stored on the first non-volatile memory, and the second slope of the ReLU can be defined by the second parameter stored on the second non-volatile memory, where the second slope is substantially flat (e.g., zero or a value that substantially represents zero). For example, the second slope can be significantly less than the first slope, e.g., 100 times smaller for example.

[0058] In an embodiment, for example, as Figure 1 shown, the first non-volatile memory is connected to the positive terminal of the first diode, and the second non-volatile memory is connected to the negative terminal of the second diode.

[0059] In another embodiment, for example, as Figure 3 shown, the first non-volatile memory is connected to an n-channel field effect transistor (nFET), and the second non-volatile memory is connected to a p-channel field effect transistor (pFET), where at least one column output line of the crossbar array is connected to the gate voltages of the nFET and the pFET.

[0060] In an embodiment, only one non-volatile memory is used because, depending on whether a given voltage (e.g., the column output of the crossbar array) is high or low (e.g., positive or negative), current will pass through one branch of the two non-volatile memories connected in parallel.

[0061] In the above description, Figure 1The memory elements NVM1 (118) and NVM2 (120) shown can both be non-volatile memory elements that can be adjusted to obtain a first slope and a second slope. However, in other embodiments, one of the memory elements can be an adjustable non-volatile memory element while the other can be a fixed element, such as a fixed resistor. Similarly, Figure 3 The memory elements NVM1 (304) and NVM2 (306) shown are both shown as adjustable non-volatile memory elements. However, similarly, in other embodiments, one of the memory elements can be an adjustable non-volatile memory element while the other can be a fixed element, such as a fixed resistor. For example, one of the NVMs can be replaced with a fixed resistor. For example, in implementing Figure 2B the ReLU shown, the fixed element can replace or substitute for the non-volatile memory element (e.g., storing a value representative of a near-flat slope).

[0062] The terms used herein are for the purpose of describing particular embodiments only and are not intended to limit the invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. As used herein, the term "or" is an inclusive operator and can mean "and / or", unless the context clearly or expressly dictates otherwise. It should also be understood that when the terms "comprises", "comprising", "contains" and / or "having" are used herein, the presence of the stated features, integers, steps, operations, elements and / or components can be indicated, but one or more other features, integers, steps, operations, elements, components and / or groups thereof are not precluded from the presence or addition. As used herein, the phrase "in an embodiment" does not necessarily refer to the same embodiment, although it may. As used herein, the phrase "in one embodiment" does not necessarily refer to the same embodiment, although it may. As used herein, the phrase "in another embodiment" does not necessarily refer to a different embodiment, although it may. Additionally, unless mutually exclusive, embodiments and / or components of embodiments can be freely combined with each other.

[0063] All corresponding structures, materials, acts, and equivalents of means or steps plus function elements (if any) in the following claims are intended to include any structure, material, or act that combines with other claimed elements to perform the function as specifically claimed. The description of the invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or to limit the invention to the disclosed form. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope of the invention. Embodiments were chosen and described to best explain the principles of the invention and its practical application, and to enable others of ordinary skill in the art to understand the invention in various embodiments, along with the various modifications suitable for the particular uses contemplated.

Claims

1. A device, comprising: A first non-volatile memory, the first non-volatile memory being connected to the positive terminal of a first diode, the first non-volatile memory being programmed to store a first parameter of an activation function associated with an artificial neural network; And A second non-volatile memory, the second non-volatile memory being connected to the negative terminal of a second diode, the second non-volatile memory being programmed to store a second parameter of the activation function associated with the artificial neural network, The first non-volatile memory and the second non-volatile memory are connected in parallel to an input line, and the negative terminal of the first diode and the positive terminal of the second diode are connected in parallel to an output line.

2. The device according to claim 1, wherein the activation function includes a leaky rectified linear unit (ReLU), wherein a first slope of the leaky ReLU is defined by the first parameter stored in the first non-volatile memory, and a second slope of the leaky ReLU is defined by the second parameter stored in the second non-volatile memory, the first parameter being proportional to the conductance of the first non-volatile memory, and the second parameter being proportional to the conductance of the second non-volatile memory.

3. The device according to claim 1, wherein the activation function includes a rectified linear unit (ReLU), wherein a first slope of the ReLU is defined by the first parameter stored in the first non-volatile memory, and a second slope of the ReLU is defined by the second parameter stored in the second non-volatile memory, the first parameter being proportional to the conductance of the first non-volatile memory, and the second parameter being proportional to the conductance of the second non-volatile memory.

4. The device according to claim 1, further comprising a crossbar array of memristive elements configured to perform multiplication and accumulation operations, wherein at least one column output line of the crossbar array is connected to the input line.

5. The device according to claim 4, wherein during training of the artificial neural network, the first parameter and the second parameter remain fixed as hyperparameters of the artificial neural network, while values stored on the memristive elements of the crossbar array are updated as part of the training.

6. The device according to claim 1, wherein the device implements an analog-memory-based activation function of an analog-memory-based artificial neural network.

7. A device, comprising: A first non-volatile memory, the first non-volatile memory being connected to a first field-effect transistor (FET), the first non-volatile memory being programmed to store a first parameter of an activation function associated with an artificial neural network; And A second non-volatile memory, the second non-volatile memory being connected to a second field-effect transistor (FET), the second non-volatile memory being programmed to store a second parameter of the activation function associated with the artificial neural network, The first non-volatile memory and the second non-volatile memory are connected in parallel to an input line, and the first field-effect transistor and the second field-effect transistor are connected in parallel to an output line, wherein a source terminal of the first field-effect transistor is connected to the first non-volatile memory, and a source terminal of the second field-effect transistor is connected to the second non-volatile memory, and wherein a drain terminal of the first field-effect transistor and the drain terminal of the second field-effect transistor are connected to the output line.

8. The apparatus according to claim 7, wherein the activation function comprises a leaky rectified linear unit (ReLU), wherein a first slope of the leaky ReLU is defined by the first parameter stored on the first non-volatile memory, and a second slope of the leaky ReLU is defined by the second parameter stored on the second non-volatile memory, the first parameter being proportional to the conductance of the first non-volatile memory, and the second parameter being proportional to the conductance of the second non-volatile memory.

9. The apparatus according to claim 7, wherein the activation function comprises a rectified linear unit (ReLU), wherein a first slope of the ReLU is defined by the first parameter stored on the first non-volatile memory, and a second slope of the ReLU is defined by the second parameter stored on the second non-volatile memory, the first parameter being proportional to the conductance of the first non-volatile memory, and the second parameter being proportional to the conductance of the second non-volatile memory.

10. The apparatus according to claim 7, further comprising a crossbar array of memristive elements configured to perform multiplication and accumulation operations, wherein at least one column output line of the crossbar array is connected to gate voltages of the first FET and the second FET.

11. The apparatus according to claim 10, wherein the first FET comprises an n-channel field-effect transistor (nFET), and the second FET comprises a p-channel field-effect transistor (pFET).

12. The apparatus according to claim 10, wherein the first parameter and the second parameter are held fixed as hyperparameters of the artificial neural network during training of the artificial neural network, while values stored on the memristive elements of the crossbar array are updated as part of the training.

13. The apparatus according to claim 7, wherein the apparatus implements an analog-memory-based activation function of an analog-memory-based artificial neural network.

14. A method, comprising: Adjusting a first non-volatile memory connected to a column output of a crossbar array of memristive elements to cause the first non-volatile memory to store a first parameter of an activation function of an artificial neural network; Adjusting a second non-volatile memory connected to the column output of the crossbar array of memristive elements to cause the second non-volatile memory to store a second parameter of the activation function of the artificial neural network; Keep the first parameter and the second parameter fixed as hyperparameters of the artificial neural network, while the synaptic weights stored on the memristive elements of the crossbar array are updated during the training of the artificial neural network.

15. The method according to claim 14, wherein the activation function includes a leaky rectified linear unit (ReLU), wherein the first slope of the leaky ReLU is defined by the first parameter stored on the first non-volatile memory, and the second slope of the leaky ReLU is defined by the second parameter stored on the second non-volatile memory.

16. The method according to claim 14, wherein the activation function includes a rectified linear unit (ReLU), wherein the first slope of the ReLU is defined by the first parameter stored on the first non-volatile memory, and the second slope of the ReLU is defined by the second parameter stored on the second non-volatile memory.

17. The method according to claim 14, wherein the first non-volatile memory is connected to the positive terminal of a first diode, and the second non-volatile memory is connected to the negative terminal of a second diode.

18. The method according to claim 14, wherein the first non-volatile memory is connected to an n-channel field effect transistor (nFET), and the second non-volatile memory is connected to a p-channel field effect transistor (pFET), wherein at least one column output line of the crossbar array is connected to the gate voltages of the nFET and the pFET.

19. The method according to claim 14, wherein the first non-volatile memory and the second non-volatile memory implement an analog-memory-based activation function of an analog-memory-based artificial neural network.

20. The apparatus according to claim 1, wherein the first non-volatile memory and the second non-volatile memory include at least one selected from the group consisting of resistive random access memory (RRAM), electrochemical random access memory (ECRAM), ferroelectric random access memory (FeRAM), phase change memory (PCM), conductive bridge RAM (CBRAM), NOR flash memory, and magnetic RAM (MRAM).

Images