Precision programming circuits for simulating neural memory in deep learning artificial neural networks

Abstract

Various embodiments of a high voltage generating circuit, a high voltage operational amplifier, an adaptive high voltage source, an adjustable high voltage incrementer, an adjustable reference power supply, and a reference circuit are disclosed. These circuits can optionally be used to program a non-volatile memory cell in an analog neural memory to store one of many possible values.

Description

[0001] Priority Declaration

[0002] This application claims priority to U.S. Provisional Patent Application No. 62 / 798,394, filed on January 29, 2019, entitled “Precision Programming Circuit For Analog Neural Memory In Deep Learning Artificial Neural Network,” and U.S. Patent Application No. 16 / 360,733, filed on March 21, 2019, entitled “Precision Programming Circuit For Analog Neural Memory In Deep Learning Artificial Neural Network.” Technical Field

[0003] The present invention discloses a number of sophisticated programming circuits for programming non-volatile memory cells in an analog neural memory to store one of many possible values. Background Art

[0004] Artificial neural networks simulate biological neural networks (the central nervous system of animals, especially the brain) and are used to estimate or approximate functions that may depend on a large number of inputs and are generally unknown. Artificial neural networks typically consist of layers of interconnected "neurons" that exchange messages with each other.

[0005] Figure 1 An artificial neural network is shown, where circles represent inputs or layers of neurons. Connections (called synapses) are represented by arrows and have numerical weights that can be adjusted based on experience. This allows the neural network to adapt to the input and learn. Typically, a neural network includes multiple layers of inputs. There are typically one or more intermediate layers of neurons, and an output layer of neurons that provide the output of the neural network. Neurons at each level make decisions based on the data received from the synapses, either individually or collectively.

[0006] One of the main challenges in developing artificial neural networks for high-performance information processing is the lack of adequate hardware technology. In fact, practical neural networks rely on a large number of synapses to achieve high connectivity between neurons, that is, very high computational parallelism. In principle, such complexity can be achieved using digital supercomputers or clusters of dedicated graphics processing units. However, in addition to being high-cost, these approaches are also mediocre in energy efficiency compared to biological networks, which consume less energy mainly due to the low-precision analog calculations they perform. CMOS analog circuits have been used in artificial neural networks, but due to the large number of neurons and synapses required, the synapses of most CMOS implementations are too large.

[0007] Applicant previously disclosed an artificial (simulated) neural network utilizing one or more nonvolatile memory arrays as synapses in U.S. patent application Ser. No. 15 / 594,439 (published as U.S. Patent Publication No. 2017 / 0337466), which is incorporated herein by reference. The nonvolatile memory array operates as a simulated neuromorphic memory. A neural network device includes a first plurality of synapses configured to receive a first plurality of inputs and generate a first plurality of outputs therefrom, and a first plurality of neurons configured to receive the first plurality of outputs. The first plurality of synapses includes a plurality of memory cells, wherein each of the memory cells includes: spaced-apart source and drain regions formed in a semiconductor substrate, wherein a channel region extends between the source and drain regions; a floating gate disposed over and insulated from a first portion of the channel region; and a non-floating gate disposed over and insulated from a second portion of the channel region. Each of the plurality of memory cells is configured to store a weight value corresponding to a plurality of electrons on the floating gate. The plurality of memory cells is configured to multiply the first plurality of inputs by the stored weight values ​​to generate a first plurality of outputs.

[0008] Each nonvolatile memory cell used in an analog neuromorphic memory system must be erased and programmed to maintain a very specific and precise amount of charge (i.e., number of electrons) in the floating gate. For example, each floating gate must maintain one of N different values, where N is the number of different weights that can be represented by each cell. Examples of N include 16, 32, 64, 128, and 256.

[0009] One challenge in a VMM system is the ability to accurately program the memory cells of the VMM, as the floating gates of selected cells will need to be programmed to hold a very specific and precise amount of charge so that each floating gate can hold one of different N values.

[0010] What is needed is an improved system for accurately programming memory cells within a VMM. Summary of the Invention

[0011] The present invention discloses a number of sophisticated programming circuits for programming non-volatile memory cells in a VMM in an emulated neural memory to store one of many possible values. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 FIG. 1 is a schematic diagram showing an artificial neural network of the prior art.

[0013] Figure 2 A prior art split-gate flash memory cell is shown.

[0014] Figure 3 Another prior art split-gate flash memory cell is shown.

[0015] Figure 4 Another prior art split-gate flash memory cell is shown.

[0016] Figure 5 Another prior art split-gate flash memory cell is shown.

[0017] Figure 6 Another prior art split-gate flash memory cell is shown.

[0018] Figure 7 A prior art stacked gate flash memory cell is shown.

[0019] Figure 8 A schematic diagram illustrating different levels of an exemplary artificial neural network using one or more non-volatile memory arrays.

[0020] Figure 9 A block diagram showing a vector-matrix multiplication system.

[0021] Figure 10 is a block diagram illustrating an exemplary artificial neural network using one or more vector-matrix multiplication systems.

[0022] Figure 11 Another embodiment of a vector-matrix multiplication system is shown.

[0023] Figure 12 Another embodiment of a vector-matrix multiplication system is shown.

[0024] Figure 13 Another embodiment of a vector-matrix multiplication system is shown.

[0025] Figure 14 Another embodiment of a vector-matrix multiplication system is shown.

[0026] Figure 15 Another embodiment of a vector-matrix multiplication system is shown.

[0027] Figure 16 A prior art long short-term memory system is shown.

[0028] Figure 17 An exemplary cell used in a long short-term memory system is shown.

[0029] Figure 18 Show Figure 17 An embodiment of an exemplary unit of .

[0030] Figure 19 Show Figure 17 Another embodiment of an exemplary unit of .

[0031] Figure 20 A prior art gate-controlled recursive cell system is shown.

[0032] Figure 21 An exemplary cell for use in a gate-controlled recursive cell system is shown.

[0033] Figure 22 Show Figure 21 An embodiment of an exemplary unit of .

[0034] Figure 23 Show Figure 21 Another embodiment of an exemplary unit of .

[0035] Figure 24 Another embodiment of a vector-matrix multiplication system is shown.

[0036] Figure 25 Another embodiment of a vector-matrix multiplication system is shown.

[0037] Figure 26 Another embodiment of a vector-matrix multiplication system is shown.

[0038] Figure 27 Another embodiment of a vector-matrix multiplication system is shown.

[0039] Figure 28 Another embodiment of a vector-matrix multiplication system is shown.

[0040] Figure 29 Another embodiment of a vector-matrix multiplication system is shown.

[0041] Figure 30 Another embodiment of a vector-matrix multiplication system is shown.

[0042] Figure 31 Another embodiment of a vector-matrix multiplication system is shown.

[0043] Figure 32A VMM system is shown.

[0044] Figure 33 Shown is a simulated neural memory system.

[0045] Figure 34 Shown is a high voltage generation block for use with a vector multiplier matrix system.

[0046] Figure 35 The charge pump and charge pump regulation circuit are shown.

[0047] Figure 36 A high voltage generation block with current compensation circuitry is shown.

[0048] Figure 37 Another high voltage generating block with a current compensation circuit is shown.

[0049] Figure 38 Another high voltage generating block is shown.

[0050] Figure 39 A dummy bit line is shown for providing current compensation.

[0051] Figure 40 A high voltage decoder is shown.

[0052] Figure 41 Shows the high voltage test circuit.

[0053] Figure 42 A high voltage generation block is shown.

[0054] Figure 43 Another high voltage generating block is shown.

[0055] Figure 44 Another high voltage generating block is shown.

[0056] Figure 45 A high voltage operational amplifier is shown.

[0057] Figure 46 Another high voltage operational amplifier is shown.

[0058] Figure 47 An adaptive high voltage source is shown.

[0059] Figure 48 Column drivers are shown.

[0060] Figure 49 Column sense amplifiers are shown.

[0061] Figure 50 The read reference circuit is shown.

[0062] Figure 51 Another read reference circuit is shown.

[0063] Figure 52 An adaptive high voltage source is shown.

[0064] Figure 53 Another adaptive high voltage source is shown.

[0065] Figure 54 Another adaptive high voltage source is shown.

[0066] Figure 55 Another adaptive high voltage source is shown.

[0067] Figure 56 Another adaptive high voltage source is shown.

[0068] Figure 57 Shown is a single reference verification algorithm.

[0069] Figure 58 A dual reference verification algorithm is shown.

[0070] Figure 59 An adjustable reference voltage source is shown.

[0071] Figure 60 Shown in Figure 59 The subcircuit used in the adjustable reference voltage source. DETAILED DESCRIPTION

[0072] The artificial neural network of the present invention utilizes a combination of CMOS technology and non-volatile memory arrays.

[0073] Non-volatile memory cells

[0074] Digital non-volatile memory is well known. For example, U.S. Patent No. 5,029,130 ​​(“the '130 patent”), which is incorporated herein by reference, discloses an array of split-gate non-volatile memory cells, which is a type of flash memory cell. Such a memory cell 210 is Figure 2 . Each memory cell 210 includes a source region 14 and a drain region 16 formed in a semiconductor substrate 12, with a channel region 18 therebetween. A floating gate 20 is formed over and insulated from a first portion of the channel region 18 (and controls its electrical conductivity), and is formed over a portion of the source region 14. A wordline terminal 22 (which is typically coupled to a wordline) has a first portion disposed over and insulated from a second portion of the channel region 18 (and controls its electrical conductivity), and a second portion extending upward and over the floating gate 20. The floating gate 20 and wordline terminal 22 are insulated from the substrate 12 by a gate oxide. A bitline 24 is coupled to the drain region 16.

[0075] Memory cell 210 is erased (where electrons are removed from the floating gate) by placing a high positive voltage on wordline terminal 22, which causes the electrons on floating gate 20 to tunnel through the intervening insulator via Fowler-Nordheim tunneling from floating gate 20 to wordline terminal 22.

[0076] Memory cell 210 is programmed by placing a positive voltage on wordline terminal 22 and a positive voltage on source region 14 (where electrons are placed on the floating gate). Electron current will flow from source region 14 to drain region 16. When the electrons reach the gap between wordline terminal 22 and floating gate 20, they will accelerate and become heated. Due to the electrostatic attraction from floating gate 20, some of the heated electrons will be injected through the gate oxide onto floating gate 20.

[0077] Memory cell 210 is read by placing a positive read voltage across drain region 16 and wordline terminal 22 (which turns on the portion of channel region 18 below the wordline terminal). If floating gate 20 is positively charged (i.e., electrons are erased), the portion of channel region 18 below floating gate 20 is also turned on, and current will flow through channel region 18, which is sensed as an erased state or a "1" state. If floating gate 20 is negatively charged (i.e., programmed by electrons), the portion of the channel region below floating gate 20 is mostly or completely turned off, and no current (or very little current) will flow through channel region 18, which is sensed as a programmed state or a "0" state.

[0078] Table 1 shows typical voltage ranges that may be applied to the terminals of the memory cell 110 for performing read, erase, and program operations:

[0079] Table 1: Figure 2 Operation of the flash memory unit 210

[0080] WL BL SL Read 2-3V 0.6-2V 0V Erase About 11-13V 0V 0V programming 1-2V 1-3μA 9-10V

[0081] Figure 3 Memory cell 310 is shown, which is connected to Figure 2 The memory cell 210 is similar to the memory cell 210 of FIG, but with the addition of a control gate (CG) 28. The control gate 28 is biased at a high voltage (e.g., 10V) during programming, at a low voltage or negative voltage (e.g., 0V / -8V) during erasing, and at a low voltage or medium voltage (e.g., 0V / 2.5V) during reading. The other terminals are similar to Figure 2 That's biased.

[0082] Figure 4A quad-gate memory cell 410 is shown, comprising a source region 14, a drain region 16, a floating gate 20 over a first portion of a channel region 18, a select gate 22 (typically coupled to a word line WL) over a second portion of the channel region 18, a control gate 28 over the floating gate 20, and an erase gate 30 over the source region 14. This configuration is described in U.S. Patent 6,747,310, which is incorporated herein by reference for all purposes. Here, except for the floating gate 20, all gates are non-floating, meaning they are electrically connected or capable of being electrically connected to a voltage source. Programming is performed by heated electrons from the channel region 18 that inject themselves into the floating gate 20. Erasing is performed by electrons tunneling from the floating gate 20 to the erase gate 30.

[0083] Table 2 shows typical voltage ranges that may be applied to the terminals of the memory cell 310 for performing read, erase, and program operations:

[0084] Table 2: Figure 4 Operation of the flash memory unit 410

[0085] WL / SG BL CG EG SL Read 1.0-2V 0.6-2V 0-2.6V 0-2.6V 0V Erase -0.5V / 0V 0V 0V / -8V 8-12V 0V programming 1V 1μA 8-11V 4.5-9V 4.5-5V

[0086] Figure 5 Memory cell 510 is shown, except that it does not include an erase gate EG. Memory cell 510 is similar to Figure 4 Erasing is performed by biasing the substrate 18 to a high voltage and biasing the control gate CG28 to a low or negative voltage. Alternatively, erasing is performed by biasing the word line 22 to a positive voltage and biasing the control gate 28 to a negative voltage. Programming and reading are similar to Figure 4 Like that.

[0087] Figure 6 A tri-gate memory cell 610 is shown, which is another type of flash memory cell. Figure 4 The memory cell 410 is identical to the memory cell 610, except that the memory cell 610 does not have a separate control gate. Except that no control gate bias is applied, the erase operation (thus erasing by using the erase gate) and the read operation are the same as Figure 4 The programming operation is also completed without a control gate bias, and as a result, a higher voltage must be applied to the source line during the programming operation to compensate for the lack of control gate bias.

[0088] Table 3 shows typical voltage ranges that may be applied to the terminals of the memory cell 610 for performing read, erase, and program operations:

[0089] Table 3: Figure 6 Operation of the flash memory unit 610

[0090] WL / SG BL EG SL Read 0.7-2.2V 0.6-2V 0-2.6V 0V Erase -0.5V / 0V 0V 11.5V 0V programming 1V 2-3μA 4.5V 7-9V

[0091] Figure 7 A stacked gate memory cell 710 is shown, which is another type of flash memory cell. Figure 2 Memory cell 210 is similar to that of FIG1 , except that floating gate 20 extends over the entire channel region 18, and control gate 22 (which here will be coupled to a word line) extends over floating gate 20, separated by an insulating layer (not shown). Erase, program, and read operations operate in a manner similar to that previously described for memory cell 210.

[0092] Table 4 shows typical voltage ranges that may be applied to the terminals of the memory cell 710 and substrate 12 for performing read, erase, and program operations:

[0093] Table 4: Figure 7 Operation of the flash memory unit 710

[0094] CG BL SL substrate Read 2-5V 0.6–2V 0V 0V Erase -8 to -10V / 0V FLT FLT 8-10V / 15-20V programming 8-12V 3-5V 0V 0V

[0095] In order to utilize a memory array comprising one of the above-described types of nonvolatile memory cells in an artificial neural network, two modifications were made. First, the circuitry was configured so that each memory cell could be individually programmed, erased, and read without adversely affecting the memory states of other memory cells in the array, as explained further below. Second, continuous (analog) programming of the memory cells was provided.

[0096] Specifically, the memory state (i.e., charge on the floating gate) of each memory cell in the array can be changed continuously from a fully erased state to a fully programmed state independently and with minimal disturbance to other memory cells. In another embodiment, the memory state (i.e., charge on the floating gate) of each memory cell in the array can be changed continuously from a fully programmed state to a fully erased state, and vice versa, independently and with minimal disturbance to other memory cells. This means that the cell storage device is analog, or at least can store one of many discrete values ​​(such as 16 or 64 different values), which allows very precise and individual tuning of all cells in the memory array, and makes the memory array ideal for storing and fine-tuning the synaptic weights of neural networks.

[0097] The methods and apparatus described herein can be applied to other non-volatile memory technologies, such as, but not limited to, SONOS (silicon-oxide-nitride-oxide-silicon, charge trapped in nitride), MONOS (metal-oxide-nitride-oxide-silicon, metal charge trapped in nitride), ReRAM (resistive RAM), PCM (phase change memory), MRAM (magnetic RAM), FeRAM (ferroelectric RAM), OTP (two-layer or multi-layer one-time programmable), and CeRAM (correlated electron RAM). The methods and apparatus described herein can be applied to volatile memory technologies used in neural networks, such as, but not limited to, SRAM, DRAM, and / or volatile synaptic cells.

[0098] Neural Networks Using Nonvolatile Memory Cell Arrays

[0099] Figure 8 A non-limiting example of a neural network using a non-volatile memory array according to the present embodiment is conceptually illustrated. This example uses the non-volatile memory array neural network for a facial recognition application, but any other suitable application may also be implemented using a non-volatile memory array-based neural network.

[0100] For this example, S0 is the input layer, which is a 32×32 pixel RGB image with 5 bits of precision (i.e., three 32×32 pixel arrays, one for each color R, G, and B, with 5 bits of precision per pixel). Synapse CB1 from input layer S0 to layer C1 applies different sets of weights in some cases and shared weights in other cases, and scans the input image with a 3×3 pixel overlapping filter (kernel), shifting the filter by 1 pixel (or more than 1 pixel as dictated by the model). Specifically, the values ​​of 9 pixels in the 3×3 portion of the image (i.e., called the filter or kernel) are provided to synapse CB1, where these 9 input values ​​are multiplied by the appropriate weights, and after summing the outputs of this multiplication, a single output value is determined and provided by the first synapse of CB1 for use in generating a feature map for one of the pixels of layer C1. The 3×3 filter is then shifted one pixel to the right within the input layer S0 (i.e., a column of three pixels on the right is added and a column of three pixels on the left is released), whereby the nine pixel values ​​in this newly positioned filter are provided to the synapse CB1, where they are multiplied by the same weights and a second single output value is determined by the associated synapse. This process continues until the 3×3 filter has scanned all three colors and all bits (precision values) across the entire 32×32 pixel image of the input layer S0. This process is then repeated using different sets of weights to generate different feature maps for C1 until all feature maps for layer C1 are calculated.

[0101] At layer C1, in this example, there are 16 feature maps, each with 30×30 pixels. Each pixel is a new feature pixel extracted from the product of the input and the kernel, so each feature map is a two-dimensional array, so in this example, layer C1 is composed of a two-dimensional array of 16 layers (remember that the layers and arrays referred to in this article are logical relationships, not necessarily physical relationships, that is, arrays do not have to be oriented to physical two-dimensional arrays). Each of the 16 feature maps in layer C1 is generated by one of sixteen different sets of synaptic weights applied to the filter scan. The C1 feature maps can all relate to different aspects of the same image features, such as edge recognition. For example, a first map (generated using a first set of weights, shared for all scans used to generate the first map) can identify circular edges, a second map (generated using a second set of weights different from the first) can identify rectangular edges, or the aspect ratio of certain features, and so on.

[0102] Before passing from layer C1 to layer S1, an activation function P1 (pooling) is applied, which pools the values ​​from consecutive non-overlapping 2×2 regions in each feature map. The purpose of the pooling function is to average the values ​​of adjacent positions (or a max function can also be used) to, for example, reduce dependencies on edge positions and reduce the size of the data before entering the next stage. At layer S1, there are 16 15×15 feature maps (i.e., sixteen different arrays of 15×15 pixels each). The synapse CB2 from layer S1 to layer C2 scans the map in S1 using a 4×4 filter, where the filter is shifted by 1 pixel. At layer C2, there are 22 12×12 feature maps. Before passing from layer C2 to layer S2, an activation function P2 (pooling) is applied, which pools the values ​​from consecutive non-overlapping 2×2 regions in each feature map. At layer S2, there are 22 6×6 feature maps. An activation function (pooling) is applied to the synapse CB3 from layer S2 to layer C3, where each neuron in layer C3 is connected to each map in layer S2 via a corresponding synapse on CB3. At layer C3, there are 64 neurons. Synapse CB4 from layer C3 to output layer S3 completely connects C3 to S3, that is, every neuron in layer C3 is connected to every neuron in layer S3. The output at S3 includes 10 neurons, where the highest output neuron determines the class. For example, this output can indicate the recognition or classification of the content of the original image.

[0103] The synapses at each layer are implemented using an array or a portion of an array of non-volatile memory cells.

[0104] Figure 9 A block diagram of an array that can be used for this purpose is shown in FIG. The vector-matrix multiplication (VMM) array 32 includes non-volatile memory cells and serves as a synapse between one layer and the next (such as Figure 6CB1, CB2, CB3, and CB4 in FIG. 1 ). Specifically, the VMM array 32 includes a nonvolatile memory cell array 33, an erase gate and word line gate decoder 34, a control gate decoder 35, a bit line decoder 36, and a source line decoder 37, which decode the corresponding inputs of the nonvolatile memory cell array 33. The inputs to the VMM array 32 can come from the erase gate and word line gate decoder 34 or from the control gate decoder 35. In this example, the source line decoder 37 also decodes the output of the nonvolatile memory cell array 33. Alternatively, the bit line decoder 36 can decode the output of the nonvolatile memory cell array 33.

[0105] The non-volatile memory cell array 33 serves two purposes. First, it stores weights to be used by the VMM array 32. Second, the non-volatile memory cell array 33 effectively multiplies the inputs by the weights stored in the non-volatile memory cell array 33, and each output line (source line or bit line) adds them together to produce an output, which will serve as the input to the next layer or the final layer. By performing multiplication and addition functions, the non-volatile memory cell array 33 eliminates the need for separate multiplication and addition logic circuits and is also highly power-efficient due to its in-situ memory calculations.

[0106] The output of the non-volatile memory cell array 33 is provided to a differential summer (such as a summing operational amplifier or a summing current mirror) 38, which sums the output of the non-volatile memory cell array 33 to create a single value for the convolution. The differential summer 38 is arranged to perform the summation of positive and negative weights.

[0107] The output values ​​of the difference summer 38 are then summed and provided to the activation function circuit 39, which corrects the output. The activation function circuit 39 can provide a sigmoid, tanh, ReLU function or any other nonlinear function. The corrected output value of the activation function circuit 39 becomes the next layer (for example, Figure 8 The elements of the feature map of layer C1 in the image processing unit are then applied to the next synapse to produce the next feature map layer or the final layer. Thus, in this example, the non-volatile memory cell array 33 constitutes a plurality of synapses (which receive their inputs from existing neuron layers or from an input layer such as an image database), and the summer 38 and activation function circuit 39 constitute a plurality of neurons.

[0108] Figure 9The inputs to the VMM array 32 (WLx, EGx, CGx, and optionally BLx and SLx) can be analog levels, binary levels, digital pulses (in which case a pulse-to-analog converter PAC may be required to convert the pulses to appropriate input analog levels), or digital bits (in which case a DAC is provided to convert the digital bits to appropriate input analog levels); the outputs can be analog levels, binary levels, digital pulses, or digital bits (in which case an output ADC is provided to convert the output analog levels into digital bits).

[0109] Figure 10 FIG. 1 is a block diagram illustrating the use of multiple layers of VMM arrays 32 (labeled here as VMM arrays 32a, 32b, 32c, 32d, and 32e). Figure 10 As shown, the input (denoted as Inputx) is converted from digital to analog by a digital-to-analog converter 31 and provided to the input VMM array 32a. The converted analog input can be a voltage or a current. The first level of input D / A conversion can be accomplished by using a function or LUT (lookup table) that maps the input Inputx to the appropriate analog levels of the matrix multiplier of the input VMM array 32a. The input conversion can also be accomplished by an analog-to-analog (A / A) converter to convert the external analog input into a mapped analog input to the input VMM array 32a. The input conversion can also be accomplished by a digital-to-digital pulse (D / P) converter to convert the external digital input into one or more digital pulses that are mapped to the input VMM array 32a.

[0110] The output generated by the input VMM array 32a is provided as input to the next VMM array (hidden level 1) 32b, which in turn generates an output that is provided as input to the next VMM array (hidden level 2) 32c, and so on. The layers of the VMM array 32 serve as different layers of synapses and neurons of a convolutional neural network (CNN). Each VMM array 32a, 32b, 32c, 32d, and 32e can be a separate physical non-volatile memory array, or multiple VMM arrays can utilize different portions of the same non-volatile memory array, or multiple VMM arrays can utilize overlapping portions of the same physical non-volatile memory array. Each VMM array 32a, 32b, 32c, 32d, and 32e can also be time-division multiplexed for different portions of its array or neurons. Figure 10 The example shown includes five layers (32a, 32b, 32c, 32d, 32e): one input layer (32a), two hidden layers (32b, 32c), and two fully connected layers (32d, 32e). Those skilled in the art will appreciate that this is merely exemplary and that, on the contrary, the system may include more than two hidden layers and more than two fully connected layers.

[0111] Vector-Matrix Multiplication (VMM) Array

[0112] Figure 11 A neuron VMM array 1100 is shown, which is particularly suitable for Figure 3 The memory cells 310 shown are used as synapses and components for neurons between the input layer and the next layer. The VMM array 1100 includes a memory array 1101 of nonvolatile memory cells and a reference array 1102 of nonvolatile reference memory cells (at the top of the array). Alternatively, another reference array can be placed at the bottom.

[0113] In VMM array 1100, control gate lines (such as control gate line 1103) extend in the vertical direction (so reference array 1102 is orthogonal to control gate line 1103 in the row direction), and erase gate lines (such as erase gate line 1104) extend in the horizontal direction. Here, the inputs of VMM array 1100 are provided on control gate lines (CG0, CG1, CG2, CG3), and the outputs of VMM array 1100 appear on source lines (SL0, SL1). In one embodiment, only even-numbered rows are used, and in another embodiment, only odd-numbered rows are used. The current placed on each source line (SL0, SL1, respectively) performs a summation function of all currents from the memory cells connected to that particular source line.

[0114] As described herein for neural networks, the non-volatile memory cells of VMM array 1100 (ie, the flash memory of VMM array 1100) are preferably configured to operate in the sub-threshold region.

[0115] Biasing the nonvolatile reference memory cell and the nonvolatile memory cell described herein in weak inversion:

[0116] Ids=Io*e (Vg-Vth) / kVt =w*Io*e (Vg) / kVt ,

[0117] where w = e (-Vth) / kVt

[0118] For an I-to-V logarithmic converter that uses a memory cell (such as a reference memory cell or a peripheral memory cell) or a transistor to convert the input current to an input voltage:

[0119] Vg=k*Vt*log[Ids / wp*Io]

[0120] Here, wp is w of a reference memory cell or a peripheral memory cell.

[0121] For a memory array used as a vector matrix multiplier VMM array, the output current is:

[0122] Iout=wa*Io*e (Vg) / kVt ,Right now

[0123] Iout=(wa / wp)*Iin=W*Iin

[0124] W=e (Vthp-Vtha) / kVt

[0125] Here, wa = w for each memory cell in the memory array.

[0126] The word line or control gate may be used as the input to the memory cell for the input voltage.

[0127] Alternatively, the flash memory cells of the VMM array described herein may be configured to operate in the linear region:

[0128] Ids=β*(Vgs-Vth)*Vds;β=u*Cox*W / L

[0129] W=α(Vgs-Vth)

[0130] The word line or control gate or bit line or source line can serve as the input of the memory cell operating in the linear region. The bit line or source line can serve as the output of the memory cell.

[0131] For an I to V linear converter, a memory cell (eg, a reference memory cell or a peripheral memory cell) or a transistor or a resistor operating in a linear region may be used to linearly convert an input / output current into an input / output voltage.

[0132] U.S. Patent Application 15 / 826,345 describes Figure 9 Other embodiments of the VMM array 32 of the present invention are incorporated herein by reference. As described herein, the source line or the bit line can be used as the neuron output (current summing output). Alternatively, the flash memory cells of the VMM array described herein can be configured to operate in the saturation region:

[0133] Ids=α 1 / 2*β*(Vgs-Vth) 2 β=u*Cox*W / L

[0134] W=α(Vgs-Vth) 2

[0135] The word line, control gate, or erase gate can be used as the input of a memory cell operating in the saturation region. The bit line or source line can be used as the output of an output neuron.

[0136] Alternatively, the flash memory cells of the VMM arrays described herein may be used in all regions or a combination thereof (subthreshold, linear, or saturation regions).

[0137] Figure 12 A neuron VMM array 1200 is shown, which is particularly suitable for Figure 2 Memory cell 210 is shown and serves as a synapse between the input layer and the next layer. VMM array 1200 includes a memory array 1203 of nonvolatile memory cells, a reference array 1201 of first nonvolatile reference memory cells, and a reference array 1202 of second nonvolatile reference memory cells. Reference arrays 1201 and 1202, arranged along the columns of the array, are used to convert current inputs flowing into terminals BLR0, BLR1, BLR2, and BLR3 into voltage inputs WL0, WL1, WL2, and WL3. In practice, the first and second nonvolatile reference memory cells are diode-connected via a multiplexer 1214 (only partially shown), with the current input flowing therein. The reference cells are tuned (e.g., programmed) to a target reference level. The target reference level is provided by a reference microarray matrix (not shown).

[0138] The memory array 1203 serves two purposes. First, it stores the weights that the VMM array 1200 will use on its corresponding memory cells. Second, the memory array 1203 effectively multiplies the inputs (i.e., the current inputs provided in terminals BLR0, BLR1, BLR2, and BLR3, which the reference arrays 1201 and 1202 convert into input voltages to be provided to the word lines WL0, WL1, WL2, and WL3) by the weights stored in the memory array 1203, and then adds all the results (memory cell currents) to produce an output on the corresponding bit lines (BL0-BLN), which will be the input to the next layer or the final layer. By performing multiplication and addition functions, the memory array 1203 eliminates the need for separate multiplication and addition logic circuits and is also highly power-efficient. Here, the voltage inputs are provided on the word lines (WL0, WL1, WL2, and WL3), and the outputs appear on the corresponding bit lines (BL0-BLN) during a read (inference) operation. The current placed on each of the bit lines BL0-BLN performs a summing function of the currents from all of the nonvolatile memory cells connected to that particular bit line.

[0139] Table 5 shows the operating voltages for VMM array 1200. The columns in the table indicate the voltages placed on the word line for a selected cell, the word line for an unselected cell, the bit line for a selected cell, the bit line for an unselected cell, the source line for a selected cell, and the source line for an unselected cell. The rows indicate read, erase, and program operations.

[0140] Table 5: Figure 12 Operation of the VMM array 1200

[0141] WL WL-Not selected BL BL-Not selected SL SL-Not selected Read 1-3.5V -0.5V / 0V 0.6-2V(Ineuron) 0.6V-2V / 0V 0V 0V Erase About 5-13V 0V 0V 0V 0V 0V programming 1-2V -0.5V / 0V 0.1-3uA Vinh about 2.5V 4-10V 0-1V / FLT

[0142] Figure 13 A neuron VMM array 1300 is shown, which is particularly suitable for Figure 2 Memory cell 210 is shown and serves as a synapse and component for neurons between the input layer and the next layer. VMM array 1300 includes a memory array 1303 of nonvolatile memory cells, a reference array 1301 of first nonvolatile reference memory cells, and a reference array 1302 of second nonvolatile reference memory cells. Reference arrays 1301 and 1302 extend in the row direction of VMM array 1300. The VMM array is similar to VMM 1000, except that in VMM array 1300, the word lines extend in the vertical direction. Here, inputs are provided on word lines (WLA0, WLB0, WLA1, WLB2, WLA2, WLB2, WLA3, WLB3), and outputs appear on source lines (SL0, SL1) during a read operation. The current placed on each source line performs a summing function of all currents from the memory cells connected to that particular source line.

[0143] Table 6 shows the operating voltages for VMM array 1300. The columns in the table indicate the voltages placed on the word line for a selected cell, the word line for an unselected cell, the bit line for a selected cell, the bit line for an unselected cell, the source line for a selected cell, and the source line for an unselected cell. The rows indicate read, erase, and program operations.

[0144] Table 6: Figure 13 Operation of the VMM array 1300

[0145] WL WL-Not selected BL BL-Not selected SL SL-Not selected Read 1-3.5V -0.5V / 0V 0.6-2V 0.6V-2V / 0V About 0.3-1V (Ineuron) 0V Erase About 5-13V 0V 0V 0V 0V SL-Prohibit (about 4-8V) programming 1-2V -0.5V / 0V 0.1-3uA Vinh about 2.5V 4-10V 0-1V / FLT

[0146] Figure 14 A neuron VMM array 1400 is shown, which is particularly suitable for Figure 3Memory cell 310 is shown and serves as a synapse and component of neurons between the input layer and the next layer. VMM array 1400 includes a memory array 1403 of nonvolatile memory cells, a reference array 1401 of first nonvolatile reference memory cells, and a reference array 1402 of second nonvolatile reference memory cells. Reference arrays 1401 and 1402 are used to convert current inputs flowing into terminals BLR0, BLR1, BLR2, and BLR3 into voltage inputs CG0, CG1, CG2, and CG3. In practice, the first nonvolatile reference memory cell and the second nonvolatile reference memory cell are diode-connected via a multiplexer 1412 (only partially shown), with the current input flowing into them through BLR0, BLR1, BLR2, and BLR3. Multiplexers 1412 each include a respective multiplexer 1405 and a cascode transistor 1404 to ensure a constant voltage on a bit line (such as BLR0) of each of the first and second nonvolatile reference memory cells during a read operation. The reference cells are tuned to a target reference level.

[0147] Memory array 1403 serves two purposes. First, it stores weights to be used by VMM array 1400. Second, memory array 1403 effectively multiplies the inputs (current inputs provided to terminals BLR0, BLR1, BLR2, and BLR3, which reference arrays 1401 and 1402 convert into input voltages to be provided to control gates CG0, CG1, CG2, and CG3) by the weights stored in the memory array, and then adds all the results (cell currents) to produce an output, which appears at BL0-BLN and will be the input to the next layer or the final layer. By performing multiplication and addition functions, the memory array eliminates the need for separate multiplication and addition logic circuits and is also highly power-efficient. Here, the inputs are provided on the control gate lines (CG0, CG1, CG2, and CG3), and the outputs appear on the bit lines (BL0-BLN) during a read operation. The current placed on each bit line performs a summing function of all the currents from the memory cells connected to that particular bit line.

[0148] The VMM array 1400 implements unidirectional tuning for the nonvolatile memory cells in the memory array 1403. That is, each nonvolatile memory cell is erased and then partially programmed until the desired charge on the floating gate is reached. This can be performed, for example, using the precision programming techniques described below. If too much charge is placed on the floating gate (causing an incorrect value to be stored in the cell), the cell must be erased and the sequence of partial programming operations must be restarted. As shown, two rows sharing the same erase gate (such as EG0 or EG1) need to be erased together (which is called a page erase), and thereafter, each cell is partially programmed until the desired charge on the floating gate is reached.

[0149] Table 7 shows the operating voltages for VMM array 1400. The columns in the table indicate the voltages applied to the word line for a selected cell, the word line for an unselected cell, the bit line for a selected cell, the bit line for an unselected cell, the control gate for a selected cell, the control gate for an unselected cell in the same sector as the selected cell, the control gate for an unselected cell in a different sector from the selected cell, the erase gate for a selected cell, the erase gate for an unselected cell, the source line for a selected cell, and the source line for an unselected cell. The rows indicate read, erase, and program operations.

[0150] Table 7: Figure 14 Operation of the VMM array 1400

[0151]

[0152] Figure 15 A neuron VMM array 1500 is shown, which is particularly suitable for Figure 3 Memory cells 310 are shown and serve as synapses and components for neurons between the input layer and the next layer. VMM array 1500 includes a memory array 1503 of nonvolatile memory cells, a reference array 1501 of first nonvolatile reference memory cells, and a reference array 1502 of second nonvolatile reference memory cells. EG lines EGR0, EG0, EG1, and EGR1 extend vertically, while CG lines CG0, CG1, CG2, and CG3 and SL lines WL0, WL1, WL2, and WL3 extend horizontally. VMM array 1500 is similar to VMM array 1400, except that VMM array 1500 implements bidirectional tuning, where each individual cell can be fully erased, partially programmed, and partially erased as needed to achieve a desired charge on the floating gate due to the use of separate EG lines. As shown, reference arrays 1501 and 1502 convert input currents in terminals BLR0, BLR1, BLR2, and BLR3 into control gate voltages CG0, CG1, CG2, and CG3 to be applied to the memory cells in the row direction (through the action of the reference cells connected via the diodes of multiplexer 1514). The current outputs (neurons) are in bit lines BL0-BLN, where each bit line sums all currents from the nonvolatile memory cells connected to that particular bit line.

[0153] Table 8 shows the operating voltages for VMM array 1500. The columns in the table indicate the voltages applied to the word line for a selected cell, the word line for an unselected cell, the bit line for a selected cell, the bit line for an unselected cell, the control gate for a selected cell, the control gate for an unselected cell in the same sector as the selected cell, the control gate for an unselected cell in a different sector from the selected cell, the erase gate for a selected cell, the erase gate for an unselected cell, the source line for a selected cell, and the source line for an unselected cell. The rows indicate read, erase, and program operations.

[0154] Table 8: Figure 15 Operation of the VMM array 1500

[0155]

[0156] Figure 24 A neuron VMM array 2400 is shown, which is particularly suitable for Figure 2 The memory cell 210 shown is used as a synapse and component of neurons between the input layer and the next layer. In the VMM array 2400, the input INPUT0..., INPUT N On bit lines BL0,...BL N The signals are received on the source lines SL0, SL1, SL2 and SL3, and outputs OUTPUT1, OUTPUT2, OUTPUT3 and OUTPUT4 are generated on the source lines SL0, SL1, SL2 and SL3, respectively.

[0157] Figure 25 A neuron VMM array 2500 is shown, which is particularly suitable for Figure 2 The memory cell 210 is shown and serves as a synapse and component of the neurons between the input layer and the next layer. In this example, inputs INPUT0, INPUT1, INPUT2 and INPUT3 are received on source lines SL0, SL1, SL2 and SL3 respectively, and outputs OUTPUT0, ... OUTPUT N On bit lines BL0,…,BL N Generate on.

[0158] Figure 26 A neuron VMM array 2600 is shown, which is particularly suitable for Figure 2 The memory unit 210 shown is used as a synapse and a component of the neurons between the input layer and the next layer. In this example, the inputs INPUT0, ..., INPUT M On word lines WL0,…,WL M is received and output OUTPUT0,…OUTPUT N On bit lines BL0,…,BL N Generate on.

[0159] Figure 27 A neuron VMM array 2700 is shown, which is particularly suitable for Figure 3 The memory unit 310 shown is used as a synapse and a component of the neurons between the input layer and the next layer. In this example, the inputs INPUT0, ..., INPUT M On word lines WL0,…,WL M is received and output OUTPUT0,…OUTPUT N On bit lines BL0,…,BL N Generate on.

[0160] Figure 28 A neuron VMM array 2800 is shown, which is particularly suitable for Figure 4 The memory unit 410 shown is used as a synapse and a component of the neurons between the input layer and the next layer. In this example, the inputs INPUT0, ..., INPUT n On the control gate lines CG0,…,CG N The signals are received on the source lines SL0 and SL1, and outputs OUTPUT1 and OUTPUT2 are generated on the source lines SL0 and SL1.

[0161] Figure 29 A neuron VMM array 2900 is shown, which is particularly suitable for Figure 4 The memory unit 410 shown in FIG. 4 is used as a synapse and a component of a neuron between an input layer and a next layer. In this example, inputs INPUT0 to INPUT N are received at the gates of bit line control gates 2901-1, 2901-2 to 2901-(N-1), and 2901-N, which are coupled to bit lines BL0 to BL1, respectively. N Exemplary outputs OUTPUT1 and OUTPUT2 are generated on source lines SL0 and SL1.

[0162] Figure 30 A neuron VMM array 3000 is shown, which is particularly suitable for Figure 3 The memory unit 310 shown, Figure 5 The storage unit 510 and Figure 7 The storage unit 710 shown is used as a synapse and a component of the neurons between the input layer and the next layer. In this example, the inputs INPUT0, ..., INPUT M On word lines WL0,…,WL M is received and output OUTPUT0,…,OUTPUT N On bit lines BL0,…,BLN Generate on.

[0163] Figure 31 A neuron VMM array 3100 is shown, which is particularly suitable for Figure 3 The memory unit 310 shown, Figure 5 The memory cell 510 and Figure 7 The memory unit 710 shown in FIG. 1 is used as a synapse and a component of a neuron between an input layer and the next layer. In this example, inputs INPUT0 to INPUT M On the control gate lines CG0 to CG M OUTPUT0 is output to OUTPUT N On the source lines SL0 to SL N On the generation, each source line SL i Coupled to the source line terminals of all memory cells in column i.

[0164] Figure 32 A VMM system 3200 is shown. The VMM system 3200 includes a VMM array 3201 (which may be based on any of the previously discussed VMM designs, such as VMMs 900, 1000, 1100, 1200, and 1320, or other VMM designs), a low voltage row decoder 3202, a high voltage row decoder 3203, a reference cell low voltage column decoder 3204 (shown in the column direction, meaning it provides input to output conversion in the row direction), a bit line multiplexer 3205, control logic 3206, analog circuitry 3207, a neuron output block 3208, an input VMM circuit block 3209, a pre-decoder 3210, a test circuit 3211, erase-program control logic EPCTL 3212, analog and high voltage generation circuitry 3213, a bit line PE driver 3214, redundant arrays 3215 and 3216, an NVR sector 3217, and a reference sector 3218. The input circuit block 3209 serves as an interface for external input to the input terminal of the memory array. The neuron output block 3208 serves as an interface for output from the memory array to the external interface.

[0165] Low-voltage row decoder 3202 provides bias voltages for read and program operations and provides decoding signals for high-voltage row decoder 3203. High-voltage row decoder 3203 provides high-voltage bias signals for program and erase operations. Optional reference cell low-voltage column decoder 3204 provides decoding functionality for reference cells. Bitline PE driver 3214 provides control functionality for the bit lines during programming, verifying, and erasing operations. Analog and high-voltage generation circuit 3213 is a shared bias block that provides multiple voltages required for various programming, erasing, program verification, and read operations. Optional redundant arrays 3215 and 3216 provide array redundancy for replacing defective array portions. Optional NVR (non-volatile register, also known as information sector) sector 3217 is a sector of the array used to store, but not limited to, user information, device ID, passwords, security keys, trim bits, configuration bits, and manufacturing information.

[0166] Figure 33 An analog neural memory system 3300 is shown. The analog neural memory system 3300 includes macroblocks 3301a, 3301b, 3301c, 3301d, 3301e, 3301f, 3301g, and 3301h; neuron output blocks (such as summer circuits and sample and hold (S / H) circuits) 3302a, 3302b, 3302c, 3302d, 3302e, 3302f, 3302g, and 3302h; and input circuit blocks 3303a, 3303b, 3303c, 3303d, 3303e, 3303f, 3303g, and 3304h. Each of the macroblocks 3301a, 3301b, 3301c, 3301d, 3301e, and 3301f is a VMM subsystem that includes a VMM array. The neural memory subsystem 3333 includes a macro block 3301, an input block 3303, and a neuron output block 3302. The neural memory subsystem 3333 may have its own digital control block.

[0167] The simulated neural memory system 3300 also includes a system control block 3304 , an simulated low voltage block 3305 , and a high voltage block 3306 .

[0168] The system control block 3304 may include a microcontroller core (such as an ARM / MIPS / RISC_V core) to process general control functions and arithmetic operations. The system control block 3304 may also include a SIMD (single instruction multiple data) unit to operate on multiple data using a single instruction. The system control block may include a DSP core. The system control block may include hardware or software for executing functions such as, but not limited to, pooling, averaging, minimum, maximum, softmax, addition, subtraction, multiplication, division, logarithm, antilogarithm, ReLu, sigmoid, tanh, and data compression. The system control block may include hardware or software for executing functions such as activating an approximator / quantizer / normalizer. The system control block may include the ability to execute functions such as input data approximator / quantizer / normalizer. The system control block may include hardware or software for executing functions such as activating an approximator / quantizer / normalizer. The control block of the neural memory subsystem 3333 may include similar elements of the system control block 3304, such as a microcontroller core, a SIMD core, a DSP core, and other functional units.

[0169] In one embodiment, neuron output blocks 3302a, 3302b, 3302c, 3302d, 3302e, 3302f, 3302g, and 3302h each include a buffered (e.g., op amp) low-impedance output type circuit that can drive long and configurable interconnects. In one embodiment, input circuit blocks 3303a, 3303b, 3303c, 3303d, 3303e, 3303f, 3303g, and 3303h each provide a summed high-impedance current output. In another embodiment, neuron output blocks 3302a, 3302b, 3302c, 3302d, 3302e, 3302f, 3302g, and 3302h each include an activation circuit, in which case an additional low-impedance buffer is required to drive the output.

[0170] In another embodiment, neuron output blocks 3302a, 3302b, 3302c, 3302d, 3302e, 3302f, 3302g, and 3302h each include an analog-to-digital conversion block that outputs digital bits rather than analog signals. In this embodiment, input circuit blocks 3303a, 3303b, 3303c, 3303d, 3303e, 3303f, 3303g, and 3303h each include a digital-to-analog conversion block that receives digital bits from the corresponding neuron output block and converts the digital bits into analog signals.

[0171] Thus, neuron output blocks 3302a, 3302b, 3302c, 3302d, 3302e, 3302f, 3302g, and 3302h receive output current from macroblocks 3301a, 3301b, 3301c, 3301d, 3301e, and 3301f and optionally convert the output current into an analog voltage, a digital bit, or one or more digital pulses, where the width of each pulse or the number of pulses varies in response to the value of the output current. Similarly, input circuit blocks 3303a, 3303b, 3303c, 3303d, 3303e, 3303f, 3303g, and 3303h optionally receive an analog current, an analog voltage, a digital bit, or digital pulses, wherein the width of each pulse or the number of pulses varies in response to the value of the output current, and provide the analog current to macro blocks 3301a, 3301b, 3301c, 3301d, 3301e, and 3301f. Input circuit blocks 3303a, 3303b, 3303c, 3303d, 3303e, 3303f, 3303g, and 3303h optionally include a voltage-to-current converter, an analog or digital counter for counting the number of digital pulses in the input signal or the width of the digital pulses in the input signal, or a digital-to-analog converter.

[0172] Long Short-Term Memory

[0173] Prior art includes a concept known as long short-term memory (LSTM). LSTM cells are commonly used in neural networks. LSTM allows neural networks to remember information for a predetermined, arbitrary time interval and use that information in subsequent operations. A typical LSTM cell consists of a cell, an input gate, an output gate, and a forget gate. These three gates regulate the flow of information into and out of the cell, as well as the time interval over which information is remembered within the LSTM. VMMs are particularly useful in LSTM cells.

[0174] Figure 16An exemplary LSTM 1600 is shown. LSTM 1600 in this example includes units 1601, 1602, 1603, and 1604. Unit 1601 receives an input vector x0 and generates an output vector h0 and a unit state vector c0. Unit 1602 receives an input vector x1, the output vector (hidden state) h0 from unit 1601, and the unit state c0 from unit 1601, and generates an output vector h1 and a unit state vector c1. Unit 1603 receives an input vector x2, the output vector (hidden state) h1 from unit 1602, and the unit state c1 from unit 1602, and generates an output vector h2 and a unit state vector c2. Unit 1604 receives an input vector x3, the output vector (hidden state) h2 from unit 1603, and the unit state c2 from unit 1603, and generates an output vector h3. Additional units may be used, and an LSTM with four units is merely an example.

[0175] Figure 17 Shown available for Figure 16 FIG1 is an exemplary implementation of an LSTM unit 1700 for units 1601, 1602, 1603, and 1604 in FIG1. ​​LSTM unit 1700 receives an input vector x(t), a cell state vector c(t-1) from the previous unit, and an output vector h(t-1) from the previous unit, and generates a cell state vector c(t) and an output vector h(t).

[0176] LSTM unit 1700 includes sigmoid function devices 1701, 1702, and 1703, each of which applies a number between 0 and 1 to control the amount of each component in the input vector that is allowed to pass through to the output vector. LSTM unit 1700 also includes tanh devices 1704 and 1705 for applying a hyperbolic tangent function to the input vector, multiplier devices 1706, 1707, and 1708 for multiplying two vectors together, and an addition device 1709 for adding two vectors together. The output vector h(t) can be provided to the next LSTM unit in the system, or it can be accessed for other purposes.

[0177] Figure 18LSTM unit 1800 is shown, which is an example of a specific implementation of LSTM unit 1700. For the convenience of the reader, the same numbering is used in LSTM unit 1800 as in LSTM unit 1700. Sigmoid function devices 1701, 1702, and 1703, and tanh device 1704 each include multiple VMM arrays 1801 and activation circuit blocks 1802. Therefore, it can be seen that VMM arrays are particularly useful in LSTM units used in certain neural network systems. Multiplier devices 1706, 1707, and 1708, and adder device 1709 can be implemented digitally or in analog form. Activation function block 1802 can be implemented digitally or in analog form.

[0178] An alternative form of LSTM cell 1800 (and another example of a specific implementation of LSTM cell 1700) is Figure 19 In Figure 19 In the example, sigmoid function devices 1701, 1702, and 1703 and tanh device 1704 share the same physical hardware (VMM array 1901 and activation function block 1902) in a time-division multiplexing manner. The LSTM unit 1900 also includes a multiplier device 1903 that multiplies two vectors together, an addition device 1908 that adds two vectors together, a tanh device 1705 (which includes an activation circuit block 1902), a register 1907 that stores the value i(t) when the value i(t) is output from the sigmoid function block 1902, a register 1904 that stores the value f(t)*c(t-1) when the value is output from the multiplier device 1903 through the multiplexer 1910, a register 1905 that stores the value i(t)*u(t) when the value is output from the multiplier device 1903 through the multiplexer 1910, a register 1906 that stores the value o(t)*c~(t) when the value is output from the multiplier device 1903 through the multiplexer 1910, and a multiplexer 1909.

[0179] LSTM unit 1800 includes multiple sets of VMM arrays 1801 and corresponding activation function blocks 1802, while LSTM unit 1900 includes only one set of VMM arrays 1901 and activation function blocks 1902, which are used to represent multiple layers in the implementation of LSTM unit 1900. LSTM unit 1900 will require less space than LSTM 1800 because LSTM unit 1900 only requires 1 / 4 of its space for VMM and activation function blocks compared to LSTM unit 1800.

[0180] It is also understood that an LSTM cell will typically include multiple VMM arrays, each of which requires functionality provided by certain circuit blocks outside the VMM array (such as summer and activation circuit blocks and high-voltage generation blocks). Providing a separate circuit block for each VMM array would require a significant amount of space within the semiconductor device and would be somewhat inefficient.

[0181] Gate-controlled recursive unit

[0182] The simulated VMM implementation can be used for a Gated Recurrent Unit (GRU) system. A GRU is a gate-controlled mechanism in recurrent neural networks. A GRU is similar to an LSTM, except that a GRU cell typically contains fewer components than an LSTM cell.

[0183] Figure 20 An exemplary GRU 2000 is shown. GRU 2000 in this example includes units 2001, 2002, 2003, and 2004. Unit 2001 receives an input vector x0 and generates an output vector h0. Unit 2002 receives an input vector x1, an output vector h0 from unit 2001, and generates an output vector h1. Unit 2003 receives an input vector x2 and an output vector (hidden state) h1 from unit 2002, and generates an output vector h2. Unit 2004 receives an input vector x3 and an output vector (hidden state) h2 from unit 2003, and generates an output vector h3. Additional units may be used, and a GRU with four units is merely an example.

[0184] Figure 21 Shown available for Figure 20 2001, 2002, 2003, and 2004 of a GRU unit 2100. The GRU unit 2100 receives an input vector x(t) and an output vector h(t-1) from a previous GRU unit and generates an output vector h(t). The GRU unit 2100 includes sigmoid function devices 2101 and 2102, each of which applies a number between 0 and 1 to components from the output vector h(t-1) and the input vector x(t). The GRU unit 2100 also includes a tanh device 2103 for applying a hyperbolic tangent function to the input vector, a plurality of multiplier devices 2104, 2105, and 2106 for multiplying two vectors together, an addition device 2107 for adding two vectors together, and a complement device 2108 for subtracting the input from 1 to generate an output.

[0185] Figure 222 shows a GRU unit 2200, which is an example of a specific implementation of the GRU unit 2100. For the convenience of the reader, the same numbering is used in the GRU unit 2200 as in the GRU unit 2100. Figure 22 As shown, sigmoid function devices 2101 and 2102 and tanh device 2103 each include multiple VMM arrays 2201 and activation function blocks 2202. Therefore, it can be seen that VMM arrays are particularly useful in GRU units used in certain neural network systems. Multiplier devices 2104, 2105, and 2106, addition device 2107, and complement device 2108 are implemented digitally or in analog form. Activation function block 2202 can be implemented digitally or in analog form.

[0186] An alternative form of GRU unit 2200 (and another example of a specific implementation of GRU unit 2300) is Figure 23 As shown in Figure 23 In , the GRU unit 2300 utilizes a VMM array 2301 and an activation function block 2302, which, when configured as a sigmoid function, applies a number between 0 and 1 to control the amount of each component in the input vector that is allowed to reach the output vector. Figure 23 In the example, sigmoid function devices 2101 and 2102 and tanh device 2103 share the same physical hardware (VMM array 2301 and activation function block 2302) in a time-division multiplexing manner. The GRU unit 2300 also includes a multiplier device 2303 that multiplies two vectors together, an addition device 2305 that adds two vectors together, a complement device 2309 that subtracts the input from 1 to generate an output, a multiplexer 2304, a register 2306 that holds the value h(t-1)*r(t) when it is output from the multiplier device 2303 through the multiplexer 2304, a register 2307 that holds the value h(t-1)*z(t) when it is output from the multiplier device 2303 through the multiplexer 2304, and a register 2308 that holds the value h^(t)*(1-z(t)) when it is output from the multiplier device 2303 through the multiplexer 2304.

[0187] The GRU unit 2200 includes multiple sets of VMM arrays 2201 and activation function blocks 2202, while the GRU unit 2300 includes only one set of VMM arrays 2301 and activation function blocks 2302, which are used to represent multiple layers in the implementation of the GRU unit 2300. The GRU unit 2300 will require less space than the GRU unit 2200 because the GRU unit 2300 only requires 1 / 3 of its space for VMMs and activation function blocks compared to the GRU unit 2200.

[0188] It will also be appreciated that a GRU system will typically include multiple VMM arrays, each of which requires functionality provided by certain circuit blocks outside the VMM array (such as summer and activation circuit blocks and high-voltage generation blocks). Providing a separate circuit block for each VMM array would require a significant amount of space within the semiconductor device and would be somewhat inefficient.

[0189] The inputs to the VMM array can be analog levels, binary levels, or digital bits (in which case a DAC is required to convert the digital bits to the appropriate input analog levels), and the outputs can be analog levels, binary levels, or digital bits (in which case an output ADC is required to convert the output analog levels to digital bits).

[0190] For each memory cell in the VMM array, each weight w can be implemented by a single memory cell, a differential cell, or two hybrid memory cells (the average of the two cells). In the case of a differential cell, two memory cells are required to implement the weight w as a differential weight (w = w + – w -). In the case of two hybrid memory cells, two memory cells are required to implement the weight w as the average of the two cells.

[0191] High voltage generation circuits and other circuits

[0192] Figure 34 1 shows a block diagram of a VMM system 3400. The VMM system 3400 includes a VMM array 3408, a row decoder 3407, a high voltage decoder 3409, a column decoder 3410, and a bit line driver 3411. The VMM system 3400 also includes a high voltage generation block 3412, which includes a charge pump 3401, a charge pump regulator 3402, and a high voltage level generator 3403. The VMM system 3400 also includes an algorithm controller 3404, an analog circuit 3405, and control logic 3406.

[0193] Figure 35Further details are provided regarding the charge pump 3401 and the charge pump regulator 3402. The charge pump 3401 is controlled by an enable signal 3501. When the enable signal 3501 is not active, the charge pump 3401 continues to increase the voltage of its output. When the enable signal 3501 is active, the charge pump 3401 maintains the voltage level of its output. The charge pump regulator 3402 includes a voltage divider structure comprising a series connection of diodes 3504, 3506, and 3508 and resistors 3505, 3507, and 3509, each resistor being coupled to the cathode of a respective one of the diodes 3504, 3506, and 3508. The divided voltage node within the structure is input to a comparator 3503, which receives another input comprising a voltage reference. When the voltage output from charge pump 3401 is sufficient to activate diodes 3504, 3506, and 3508 so that current will flow, and the voltage from the voltage divider voltage node exceeds the voltage reference, the enable signal will take effect. Therefore, charge pump regulator 3402 controls charge pump 3401 until the desired voltage level is achieved, which is based on the characteristics of diodes 3504, 3506, and 3508 and resistors 3505, 3507, and 3509. As an example of a voltage divider structure, three diodes and three resistors are shown, but more than three diodes and three resistors are typically required. Alternatively, capacitors can be implemented instead of diodes and resistors to produce the desired voltage ratio, thereby providing input to comparator 3503. Alternatively, an appropriately ratioed capacitor can be connected in parallel with the diodes and resistors to accelerate the response of the voltage divider structure.

[0194] Figure 36 VMM system 3600 is shown, which is one embodiment of VMM system 3400. VMM system 3600 includes a high voltage buffer 3601 and an adjustable current sink 3602. High voltage generation block 3412 generates a voltage provided to high voltage buffer 3601, which in turn provides the voltage to high voltage decoder 3409 and adjustable current sink (programming compensation current Icomp) 3602. The current Icomp drawn from high voltage buffer 3601 by adjustable current sink 3602 can be adjusted, for example, to cause a compensation voltage drop within high voltage buffer 3601, thereby compensating for the number of memory cells to be programmed (e.g., dVout 1 / 2 / ... / 32 voltage drops for 1 / 2 / ... / 32 IOs to be programmed) and reducing the temperature of high voltage buffer 3601. For example, Icomp = (number of memory cells to be programmed) * Iprog * M, where Iprog = cell programming current and M = a multiplication factor due to memory cell hot carrier effects during programming operations. Compensation Icomp is applied to maintain a constant high voltage output over varying output loads.

[0195] Figure 37 One embodiment of a VMM system 3700 is shown for use with a high voltage buffer 3701 and an adjustable current sink 3702. A high voltage generator 3412 generates a voltage that is provided to the high voltage buffer 3701, which in turn provides the voltage to the high voltage decoder 3409. The current (compensation current) Icomp 3702 drawn from the high voltage decoder 3409 by the adjustable current sink can be adjusted to, for example, reduce the current drop within the high voltage decoder 3409 as a function of the number of memory cells to be programmed and / or reduce the temperature of the high voltage decoder 3409. For example, Icomp = (number of memory cells to be programmed) * Iprog * M. Iprog = cell programming current, M = a multiplication factor due to hot carrier effects in the memory cells during programming operations. Compensation Icomp is applied to maintain a constant high voltage output over varying output loads.

[0196] Figure 38 A VMM system 3800 is shown in conjunction with a high-voltage buffer 3801, which is an operational amplifier. A high-voltage generator 3412 generates a voltage that is provided to the high-voltage buffer 3801, which in turn provides the voltage to a high-voltage decoder 3409. The output from the high-voltage decoder 3409 (e.g., a feedback indicator of the HV voltage in the decoder) is provided as an input to the high-voltage buffer 3801, which then operates as a closed-loop operational amplifier. Closed-loop compensation is applied to maintain a constant high-voltage output over varying output loads.

[0197] Figure 39 A programming current compensation block 3900 is shown for use in conjunction with VMM systems 2400, 2600, 2700, or 2800, for example, as a supplement to the VMM array in each VMM system. Here, dummy programming bit lines (programmable dummy arrays) are provided with groups of 32 bit lines. For example, group 3901 includes dummy bit line 3903, and group 3902 includes dummy bit line 3904. These dummy bit lines 3903 and 3904 can be turned on (to provide bit line programming current) when one or more other bits in groups 3901 and 3902, respectively, are not being programmed. This allows the current drawn during programming operations to remain more constant compared to when dummy bit lines 3903 and 3904 are not used. A programmed dummy array compensation scheme is employed to maintain a constant high voltage output across varying output loads.

[0198] Figure 40An example of a high voltage decoder block 4000 that can be used to implement the high voltage decoder 3409 is shown. Here, source line 4005 is coupled to one or two rows in the VMM array 3408. NMOS transistors 4001, 4002, 4003, and 4004 are coupled to source line 4005 as shown. The HV supply 4010 is provided by an HV buffer (such as HV buffer 3601, 3701, or 3801), and the HV comp signal 4011 is provided by a CMOS controller such as the CMOS controller 3408. Figure 38 As shown in .

[0199] Figure 41 Test circuit 4100 is shown. Test circuit 4100 includes a high-voltage transmitter 4101 that receives an enable signal EN. The high-voltage transmitter provides a high-voltage enable signal to an NMOS transistor 4102 and an NMOS cascode transistor 4103. One terminal of NMOS transistor 4102 is connected to an external test pad, and one terminal of NMOS transistor 4103 is coupled to an internal node within VMM system 3400. This circuit can be used, for example, during a voltage calibration process.

[0200] Figure 42 An embodiment of a high voltage generation block 3412 is shown, which here includes a high voltage generation circuit 4200, a control logic block 4201, an analog circuit block 4202, and a test block 4203. High voltage generation circuit 4200 includes a charge pump and regulator 4204, a high voltage incrementer 4205, and a high voltage operational amplifier 4206. The voltage at the output of high voltage incrementer 4205 can be controlled based on a trim signal sent to a transistor in high voltage incrementer 4205, as will be further described below. Control logic block 4201 receives a control logic input and generates a control logic output. Analog circuit block 4202 includes a current bias generator 4207, which receives a reference voltage Vref and generates a current that can be used to apply a bias signal iBias for use elsewhere. Analog circuit block 4202 also includes a voltage generator 4208, which receives a set of trim bits TRBIT_WL and generates a voltage applied to the word line during various operations. Test block 4203 receives signals on test pad MONHV_PAD and outputs various signals for monitoring during testing.

[0201] Figure 43Another embodiment of a high voltage generation block 3412 is shown. Here, the high voltage generation block includes a charge pump and regulator 4301, a high voltage (HV) increaser 4303, and a high voltage operational amplifier 4302. The voltage of the output of the high voltage increaser 4303 can be controlled based on a signal sent to the gate of a transistor in the high voltage increaser 4303. The HV increaser 4303 includes a resistor string 4315i connected in series from ground to the output of the charge pump 4301. A network of switches 4310a, 4310b, 4310z is used to multiplex the voltage levels along the output string in an incremental manner. The gates of the transistors are enabled / disabled by a high voltage level shifter (HVLS) 4320, which is in turn enabled / disabled by a digital control input. The HVLS 4320 is used to convert a digital voltage level (e.g., 1V) to a high voltage level (e.g., 12V). For example, resistor string 4315i will provide voltage levels from 3V to 10V, with voltage increments of 10mV (the voltage across one resistor). Therefore, the resistor string output, VHVROUT, will have values ​​from 3V to 10V, with 10mV increments. High Voltage (HV) op amp 4302 is used to buffer this VHVROUT increment voltage. Because the HV voltage is required at the highest voltage (e.g., 10V), leakage associated with near-breakdown (BV) conditions occurs when the PMOS switch is at or near this value. This affects the accuracy of small incremental voltages (e.g., 10mV). Therefore, improvements are needed to overcome this BV leakage. First, the power supply for the HVLS circuit is tapped based on the position of the PMOS switch. For example, for a PMOS switch in a 4V-6V string position, the high power supply for the HVLS circuit is 6V (instead of the more typical 12V supply). Additionally, the low power supply can be 4V (instead of the more typical ground value). This reduces the voltage stress across the PMOS switch at this resistor string position. For the PMOS switch connected to VHVROUT, two PMOS switches in series with adaptive HV bias are required to disable the corresponding multiplexing path to avoid BV stress, for example, PMOS 4310e / 4310f and 4310y / 4310z are shown. PMOS 4310f and 4310z are used to disable the multiplexing path. For example, the gates of PMOS 4310f and 4310z are at a high voltage (e.g., 10V) to disable the 10V multiplexing path. In the off condition, the gate of PMOS 4310e is preferably at about 6V for common source and common gate to reduce BV leakage, and the gate of PMOS 4310f is at 10V. In the on state, the gate of PMOS 4310f may be at about 6V and the gate of PMOS 4310f may be at <6V to pass an 8V-10V (as an example) multiplexing path from the corresponding string to VHVROUT.In the off condition, the gate of PMOS 4310y is at approximately 6 V for cascode to reduce BV leakage, and the gate of PMOS 4310z is at 10 V. In the on condition, the gate of PMOS 4310y may be at 0 V and the gate of PMOS 4310z may be at 0 V to pass a 3 V-5 V (as an example) multiplexing path from the string to VHVROUT.

[0202] Figure 44 Another embodiment of a high voltage generation block 3412 is shown. High voltage generation block 3412 includes a high voltage operational amplifier 4403, an SC (switched capacitor) network 4402, and an SC network 4401. SC network 4402 includes an adjustable capacitor 4404. SC network 4401 includes switches 4405, 4407, 4408, and 4409, and an adjustable capacitor 4406. A high voltage level shifter (HVLS) circuit operating at a high voltage level (e.g., 10V-13V) is required to adjust capacitor 4404 of SC network 4402. SC network 4401 requires an IO voltage (e.g., 1.8V, 2.5V) or core voltage (e.g., 1.2V) switching circuit.

[0203] Figure 45 A high voltage operational amplifier 4500 is shown, which can be used to Figure 44 High voltage operational amplifier 4403 in FIG. High voltage operational amplifier 4500 includes the components shown in the illustrated arrangement. HV cascode bias nodes VCASP, VCASN1, and VCASN2 are implemented in an adaptive manner so that the voltage value depends on the output voltage VOUT to minimize the maximum stress voltage drop across the transistor. For example, when node voltage VOUT is high, VCASN2 is high and VCASN1 is low.

[0204] Figure 46 A high voltage operational amplifier 4600 is shown, which can be used to Figure 44 High voltage operational amplifier 4403 in FIG. High voltage operational amplifier 4600 includes the components shown in the illustrated arrangement. HV cascode bias nodes VCASN2A and VCASN2B are implemented so that the voltage value depends on the output voltage VOUT to minimize the maximum voltage drop across the transistors. For example, when node voltage VOUT is high, VCASN1B and VCAS2B are high.

[0205] Figure 47 An adaptive high voltage source 4700 is shown, which can be used to Figure 44 The high voltage operational amplifier 4403 in the CMOS circuit provides an adaptive high voltage cascode bias. The adaptive high voltage source 4700 includes the components shown in the shown arrangement.

[0206] Figure 48 Column driver 4800 is shown, which can be used for each bit line driver in bit line driver 3411. In the configuration shown, column driver 4800 includes latch 4801, inverter 4802, NOR gate 4803, PMOS transistor 4804, NMOS transistors 4805 and 4806, and sense amplifier 4807. As shown, the VCASA level tracks VIN at a higher level, i.e., =~VIN+2*VT_PMOS. The VCASB level tracks VIN at a lower level, i.e., =~VIN-V*VT_NMOS. Other values ​​are possible for different MOS transistors and different I*R voltage drops (such as by inserting resistor R in the current path).

[0207] Figure 49 Sense amplifier 4900 is shown, which can be used to Figure 48 Sense amplifier 4907 in FIG. In the configuration shown, sense amplifier 4900 includes an adjustable current reference source 4901, a switch 4902, an NMOS transistor 4903, a capacitor 4904, a switch 4905, a current source 4906, and an inverter 4907. Sense amplifier 4907 is coupled to memory cell 4908 in VMM array 3408.

[0208] Figure 50 5. The reference array circuit 5000 is shown. The reference array circuit includes a bit line reference decoder 5001 and reference cells 50010 to 5002. N .

[0209] Figure 51 The reference array circuit 5100 includes a bit line reference decoder 5101 and reference cells 51020 to 5100. N .

[0210] Precision programmed circuits and algorithms

[0211] Figure 52 The adaptive high voltage source 5200 is shown to provide voltage for the HV operational amplifier 4403. The adaptive high voltage source includes an operational amplifier 5201, a resistor 5203 and a variable resistor 5202 (which may be a low voltage domain variable resistor). The adaptive high voltage source 5200 receives an input V IN And generate a high voltage signal HV OUT, where the gain can be adjusted by adjusting the resistance of the variable resistor 5202 by a resistor trimming circuit network (not shown). In one embodiment, the resistor 5203 is in a low voltage domain (e.g., 1V or 1.8V) and uses low voltage devices and operational amplifier 5201, and the variable resistor 5202 with the trimming circuit network is in a high voltage domain (e.g., 12V) and uses high voltage devices. Then, the HV OUT Programming nonvolatile memory cells.

[0212] Figure 53 The adaptive high voltage source 5300 is shown for providing voltage to the HV operational amplifier 3403. The adaptive high voltage source includes an operational amplifier 5301, a variable resistor 5303 and a resistor 5302. The adaptive high voltage source 5300 receives an input V IN And generate a high voltage signal HV OUT , where the gain can be adjusted by adjusting the resistance of the variable resistor 5303 through a resistor trimming circuit network (not shown). In one embodiment, the variable resistor 5303 and the trimming circuit network are in a low voltage domain (e.g., 1V or 1.8V) and use low voltage devices, and the operational amplifier 5301 and the resistor 5302 are in a high voltage domain (e.g., 12V) and use high voltage devices. Then, the HV OUT Programming nonvolatile memory cells.

[0213] Figure 54 FIG4 shows an adaptive high voltage source 5400 for providing voltage to the HV operational amplifier 4403, which includes an operational amplifier 5401, a resistor 5402, a resistor 5403, an operational amplifier 5404, and an adjustable voltage divider 5405. The adjustable voltage divider 5405 receives a voltage source V S The voltage output by the adjustable voltage divider 5405 (which is also the input voltage on the non-inverting terminal of the operational amplifier 5404 and is the input voltage V IN ) will vary depending on which of the switches 5407i is closed. IN , generating a high voltage signal HV OUT Here, V can be adjusted by the adjustable voltage divider 5405. IN In one embodiment, the adjustable voltage divider 5405 and the operational amplifier 5404 are in the low voltage domain (e.g., 1V or 1.8V) and use low voltage devices, and the operational amplifier 5401 is in the high voltage domain (e.g., 12V) and uses high voltage devices. Then, the HV OUT Programming nonvolatile memory cells.

[0214] Figure 55 An adaptive high voltage source 5500 is shown for providing voltage to the HV operational amplifier 4403. The adaptive high voltage source includes an adjustable voltage divider 5505 and a thin resistor HV network 5580. The thin resistor HV network 5580 includes a buffer 5501, a buffer 5502, an adjustable voltage divider 5503, and the adjustable voltage divider includes a set of j resistors 5504j and switches 5504j. The adjustable voltage divider 5505 receives a voltage source V S And includes resistor 4408 and i groups of resistors 5506i and switch network 5507i. The voltage output by adjustable voltage divider 5505 will change depending on which switch of switch network 5507i is closed. Adjustable voltage divider 5503 receives high voltage HV_COARSE 1 and HV_COARSE 2. High voltage HV_COARSE 1 is further the first output of adaptive high voltage source 5500. The voltage output by adjustable voltage divider 5503 (which is HV_FINE) will change depending on HV_COARSE and which switch of switch network 5504j is closed. Here, the magnitude of HV_COARSE1 / 2 can be adjusted by changing the switch of switch network 5507i closed in adjustable voltage divider 5505. The magnitude of HV_FINE can be adjusted by changing the switch of switch network 5504j closed in adjustable voltage divider 5503. As a numerical example, the adjustable voltage divider 5505 can provide 200mV per step (i.e., voltage increment, voltage across one resistor 5506), the voltage across HV_COARSE 1 and HV_COARSE 2 is 600mV, and the adjustable voltage divider 5503 can provide 5mV per step (i.e., voltage increment, voltage across one resistor 5504j). These high voltages can be used to program non-volatile memory cells.

[0215] Figure 56 An adaptive high voltage source 5600 for providing voltage to the HV operational amplifier 4403 is shown, which includes a coarse SC (switched capacitor) network 5650 and a fine resistor HV network 5680. The coarse SC network 5650 includes an operational amplifier 5601, an SC network 5609, and an SC network 5608. The SC network 5609 includes an adjustable capacitor 5602 of capacitance CFB. The SC network 5608 includes an adjustable capacitor 5603 of capacitance CIN and switches 5604, 5605, 5606, and 5607. Here, the HV OUT =V*(CIN / CFB). The thin resistor HV network 5680 is similar to Figure 55 The coarse SC network 5650 provides a coarse adjustable level (e.g., 200 mV steps), and the fine resistor HV network 5680 provides a fine level (e.g., 5 mV steps).OUT Can be used to program non-volatile memory cells.

[0216] like Figure 52 、 Figure 43 、 Figures 52 to 56 The adaptive HV power supply of the HV operational amplifier 5403 is shown to optimize power according to the output voltage. VHVOPA = VOUT + dV, for example, VHVOPA = 6V, where VOUT = 4V and dV = 2V. Basically, the HVOPA 4403 does not need to be always provided with the maximum HV voltage (e.g., 12V).

[0217] Figure 57 A modified SAR (successive approximation) sequential verification algorithm 5700 is shown that uses only a single-level reference implementation, which simplifies the hardware implementation. The figure shows a 4-bit verification algorithm for converting the cell output into 4 output digital bits to compare with the 4 input digital bits. The most significant bit DOUT3 is first converted by applying an intermediate reference value on the reference line and comparing the cell output with the reference level. The next significant bit DOUT2 is then converted by applying the intermediate reference value in the upper half (i.e., from L8 to L15) and comparing the cell output with the intermediate reference level, and then applying the intermediate reference value in the lower half (i.e., from L7 to L0) and comparing the cell output with the intermediate reference level. The next digital bits are converted in a similar manner. For a 4-bit output, this method requires 15 conversion pulses (steps). The cell output is a current or voltage converted from the weights stored in the memory cell.

[0218] Figure 58 A modified SAR sequential verification algorithm 5800 is shown using two reference lines and halving the number of conversion pulses. The most significant bit is completed as described above using a single reference line. The subsequent sequential conversion steps use two reference lines. For each reference line, the conversion steps are similar to those above. For a 4-bit output, this method would require 8 steps.

[0219] The verification algorithm described above can be used to convert neuron currents (output currents from the VMM array) into digital bits.

[0220] Figure 59An adjustable 2D thermometer code reference current source 5900 is shown for use during a verification operation following a programming operation of a non-volatile memory cell. The 2D thermometer code reference current source 5900 includes a buffer mirror 5901 (the buffer mirror includes an operational amplifier 5902 and a PMOS transistor 5903), an adjustable bias source 5904, and a 2D array 5905 that includes an array of i rows and j columns of devices 5906, where a particular device 5906 is labeled by the label 5906-(row)(column). Here, various combinations of devices 5906 can be activated so that the amount of reference current output by the buffer mirror 5901 can be adjusted. As shown, there are 16 current mirrors (devices 5906) in the 2D array 5905. The adjustable 2D thermometer code reference current source 5900 essentially converts a 4-digital input code into a reference current bias having a value between 1 and 16 times the Ibiasunit provided from the bias source 5904. For example, these values ​​correspond to values ​​such as Figure 58 The 16 levels of memory cells in the VMM array are shown.

[0221] For example, bias source 5904 can provide a 1 nA current Ibiasunit, which is mirrored into device 5906. Here, the first row consists of devices 5906-11 through 5906-1j and are enabled sequentially from left to right, one device 5906 at a time. The next row is then enabled in a sequential manner from left to right to add to the first row, meaning 5, then 6, then 7, then 8 devices 5906 are enabled. Thus, by enabling devices 5906 sequentially, transistor mismatch conditions associated with conventional binary decoding are avoided. The sum of the enabled devices 5906 is then mirrored by buffer mirror 5901 and output as an adjustable current that can be used to Figure 39 The adjustable current reference source 3901 in FIG. The bias source 5904 can provide a fine-tunable unit bias range, such as 50pA / 100pA / 200pA / … / 100nA. The adjustable 4×4 2D thermometer code reference current source 5900 shown can be any other size, such as 32×32 or 8×32.

[0222] Figure 60 A reference subcircuit 6000 is shown, which can be used to Figure 59 Device 5906 in. Reference subcircuit 6000 includes NMOS transistors 6001 and 6002 configured as shown. Transistor 6002 is a current mirror bias transistor, and transistor 6001 is an enable transistor (to enable bias transistor 6002 to be connected to output node OUTPUT).

[0223] It should be noted that, as used herein, the terms "above" and "on" both inclusively include "directly on" (no intervening material, element, or space disposed therebetween) and "indirectly on" (intervening material, element, or space disposed therebetween). Similarly, the term "adjacent" includes "directly adjacent" (no intervening material, element, or space disposed therebetween) and "indirectly adjacent" (intervening material, element, or space disposed therebetween), "mounted to" includes "directly mounted to" (no intervening material, element, or space disposed therebetween) and "indirectly mounted to" (intervening material, element, or space disposed therebetween), and "electrically coupled to" includes "directly electrically coupled to" (no intervening material or element electrically connecting the elements together) and "indirectly electrically coupled to" (intervening material or element electrically connecting the elements together). For example, forming an element "above a substrate" may include forming the element directly on the substrate without an intervening material / element therebetween, as well as forming the element indirectly on the substrate with one or more intervening materials / elements therebetween.

Claims

1. An adjustable programming circuit configured to generate one of a plurality of different voltages to program a non-volatile memory cell so as to store one of the plurality of different voltages on a floating gate of the non-volatile memory cell for use in a neural network, the adjustable programming circuit comprising: an operational amplifier, the operational amplifier comprising a first input terminal, a second input terminal, and an output terminal, the first input terminal receiving a reference voltage; a first resistor including a first terminal coupled to the second input terminal of the operational amplifier and a second terminal receiving an input voltage; a second resistor coupled to the second input terminal of the operational amplifier and the output terminal of the operational amplifier; wherein one of the first resistor and the second resistor is a variable resistor; as well as Resistor trimming circuit network; The output terminal outputs a programming voltage generated by applying a gain to the input voltage, wherein the gain changes in response to adjustment of the resistance of the variable resistor by the resistance trimming circuit network and is used to program the nonvolatile memory cell.

2. The adjustable programmable circuit according to claim 1, wherein the first resistor is the variable resistor.

3. The adjustable programmable circuit according to claim 1, wherein the second resistor is the variable resistor. 4 . The adjustable programming circuit of claim 1 , wherein the first resistor is the variable resistor operating in a low voltage domain. 5 . The adjustable programming circuit of claim 1 , wherein the second resistor is the variable resistor operating in a high voltage domain.

6. The adjustable programming circuit of claim 1, wherein a high voltage source for the operational amplifier is adjustable based on an output voltage of the operational amplifier.

7. The adjustable programming circuit according to claim 1, wherein the non-volatile memory cell is a stacked gate memory cell.

8. The adjustable programming circuit of claim 1, wherein the nonvolatile memory cell is a split-gate memory cell.

9. An adjustable programming circuit for generating one of a plurality of different voltages to program a non-volatile memory cell, thereby storing one of the plurality of different voltages on a floating gate of the non-volatile memory cell for use in a neural network, the adjustable programming circuit comprising: a first operational amplifier, the first operational amplifier comprising a first input terminal, a second input terminal, and an output terminal, the first input terminal receiving a reference voltage; a second operational amplifier, the second operational amplifier comprising a first input terminal, a second input terminal, and an output terminal, the first input terminal of the second operational amplifier being directly coupled to the output terminal of the second operational amplifier; a first resistor coupled to the second input terminal of the first operational amplifier and the output terminal of the first operational amplifier; a second resistor coupled between the second input terminal of the first operational amplifier and the output terminal of the second operational amplifier; an adjustable voltage divider coupled to the second input terminal of the second operational amplifier; The output terminal of the first operational amplifier outputs a programming voltage, and the programming voltage changes in response to the voltage output by the adjustable voltage divider.

10. The adjustable programming circuit of claim 9, wherein the adjustable voltage divider and the second operational amplifier operate in a low voltage domain.

11. The adjustable programmable circuit of claim 9, wherein the first operational amplifier operates in a high voltage domain.

12. The adjustable programming circuit of claim 9, wherein a high voltage source for the first operational amplifier is adjustable based on the programming voltage.

13. The adjustable programming circuit according to claim 9, wherein the non-volatile memory cell is a stacked gate memory cell.

14. The adjustable programming circuit of claim 9, wherein the non-volatile memory cell is a split-gate memory cell.

15. An adjustable programming circuit for generating one of a plurality of different voltages to program a non-volatile memory cell, thereby storing one of the plurality of different voltages on a floating gate of the non-volatile memory cell for a neural network, the adjustable programming circuit comprising: a first adjustable voltage divider comprising a first output terminal and a second output terminal; a second adjustable voltage divider; a first operational amplifier, the first operational amplifier comprising a first input terminal, a second input terminal, and an output terminal, wherein the first input terminal of the first operational amplifier is coupled to the output terminal of the first operational amplifier, and the second input terminal of the first operational amplifier is coupled to the first output terminal of the first adjustable voltage divider, and the output terminal of the first operational amplifier provides a first programming voltage; and a second operational amplifier comprising a first input terminal, a second input terminal, and an output terminal, wherein the first input terminal of the second operational amplifier is coupled to the output terminal of the second operational amplifier, and the first input terminal of the second operational amplifier is coupled to the second output terminal of the first adjustable voltage divider; wherein the second adjustable voltage divider is coupled to the output terminal of the second operational amplifier and the output terminal of the first operational amplifier and provides a second programming voltage; The first programming voltage varies in response to the first adjustable voltage divider, and the second programming voltage varies in response to the first adjustable voltage divider and the second adjustable voltage divider.

16. The adjustable programming circuit of claim 15, wherein the first adjustable voltage divider provides coarse-step adjustability and the second adjustable voltage divider provides fine-step adjustability.

17. The adjustable programming circuit of claim 15, wherein a high voltage source for the first operational amplifier is adjustable based on the programming voltage.

18. The adjustable programming circuit of claim 15, wherein the non-volatile memory cell is a stacked gate memory cell.

19. The adjustable programming circuit of claim 15, wherein the non-volatile memory cell is a split-gate memory cell.

Images