Hybrid system comprising analog computation-in-memory engine and digital computation-in-memory engine to perform operations in a neural network and operating method thereof

Abstract

In one example disclosed herein, a system includes: an analog memory computation engine for performing operations in a first layer of a neural network; and a digital memory computation engine for performing operations in a second layer of the neural network, different from the first layer. The system optionally includes a dynamic weight engine for performing operations in a third layer of the neural network, different from both the first and second layers.

Description

Technical Field

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 458,439, filed on April 10, 2023, and entitled “Neural Network Comprising Analog Computation-in-Memory Engine and Digital Computation-in-Memory Engine,” and U.S. Patent Application No. 18 / 218,368, filed on July 5, 2023, and entitled “Analog Computation-In-Memory Engine And Digital Computation-In-Memory Engine to Perform Operations in a Neural Network.”

[0002] Various embodiments of a system including one or more analog in-memory compute engines and one or more digital in-memory compute engines for performing neural network operations are disclosed. Prior Art

[0003] Artificial neural networks (ANNs) simulate biological neural networks (the central nervous system of animals, specifically the brain) and are used to estimate or approximate functions that may depend on a large number of inputs and are usually unknown. ANNs typically consist of layers of interconnected "neurons" that exchange messages with each other.

[0004] Figure 1 illustrates an artificial neural network, where circles represent inputs or layers of neurons. Connections (called synapses) are represented by arrows and have numerical weights that can be tuned based on experience. This allows the neural network to adapt to the input and learn. Typically, a neural network includes multiple input layers. There are usually one or more layers of intermediate neurons and a layer of output neurons that provide the neural network's output. Neurons at each level make decisions individually or collectively based on the data received from the synapses.

[0005] One of the major challenges in developing artificial neural networks for high-performance information processing is the lack of adequate hardware technology. Practical neural networks rely on extremely large numbers of synapses to achieve high connectivity between neurons, which translates to extremely high computational parallelism. In principle, this complexity could be achieved using clusters of digital supercomputers or graphics processing cells. However, in addition to high cost, these approaches suffer from moderate energy efficiency compared to biological networks, primarily because biological networks perform low-precision analog computations and therefore consume much less energy. CMOS analog circuits have been used in artificial neural networks, but given the large number of neurons and synapses, the synapses in most CMOS implementations are too large.

[0006] Applicants previously disclosed an artificial (analog) neural network utilizing one or more non-volatile memory arrays as synapses in U.S. Patent Application Publication 2017 / 0337466A1, which is incorporated by reference. The non-volatile memory array operates as an analog neural memory and includes non-volatile memory cells arranged in rows and columns. The neural network includes a first plurality of synapses configured to receive a first plurality of inputs and generate a first plurality of outputs therefrom, and a first plurality of neurons configured to receive the first plurality of outputs. The first plurality of synapses includes a plurality of memory cells, each of which includes: spaced-apart source and drain regions formed in a semiconductor substrate, wherein a channel region extends between the source and drain regions; a floating gate disposed over and insulated from a first portion of the channel region; and a non-floating gate disposed over and insulated from a second portion of the channel region. Each of the plurality of memory cells stores a weight value corresponding to the number of electrons on the floating gate. The plurality of memory cells multiply the first plurality of inputs by the stored weight value to generate a first plurality of outputs. Non-volatile memory cells

[0007] Non-volatile memory is well known. For example, U.S. Patent No. 5,029,130 ​​(the "'130 patent"), incorporated herein by reference, discloses a split-gate non-volatile memory cell array, a type of flash memory cell. Such a memory cell 210 is shown in FIG. Each memory cell 210 includes a source region 14 and a drain region 16 formed in a semiconductor substrate 12, with a channel region 18 located between the source and drain regions. A floating gate 20 is formed over and insulated from a first portion of the channel region 18 (and controls the conductivity of the first portion), and is also formed over a portion of the source region 14. A wordline terminal 22 (which is typically coupled to a wordline) has a first portion disposed over and insulated from a second portion of the channel region 18 (and controls the conductivity of the second portion), and a second portion extending upward and past the floating gate 20. The floating gate 20 and the word line terminal 22 are insulated from the substrate 12 by a gate oxide. The bit line 24 is coupled to the drain region 16.

[0008] The memory cell 210 is erased (where electrons are removed from the floating gate) by placing a high positive voltage on the word line terminal 22, which causes the electrons on the floating gate 20 to tunnel from the floating gate 20 through the intermediate insulating member to the word line terminal 22 via Fowler-Nordheim (FN) tunneling.

[0009] Memory cell 210 is programmed using source-side injection (SSI) of hot electrons (where electrons are placed on the floating gate) by applying a positive voltage to wordline terminal 22 and a positive voltage to source region 14. Electron current will flow from drain region 16 toward source region 14. Electrons will accelerate and heat up when they reach the gap between wordline terminal 22 and floating gate 20. Some of the heated electrons will be injected through the gate oxide onto floating gate 20 due to the attractive electrostatic force from floating gate 20.

[0010] Memory cell 210 is read by applying a positive read voltage to drain region 16 and wordline terminal 22 (this turns on the portion of channel region 18 beneath the wordline terminal). If floating gate 20 is positively charged (i.e., electrons are erased), the portion of channel region 18 beneath floating gate 20 is also turned on, and current will flow through channel region 18, which is sensed as an erased or "1" state. If floating gate 20 is negatively charged (i.e., programmed with electrons), the portion of the channel region beneath floating gate 20 is mostly or completely off, and no current (or very little current) will flow through channel region 18, which is sensed as a programmed or "0" state.

[0011] Table 1 describes typical voltage and current ranges that may be applied to the terminals of the memory cell 210 for performing read, erase, and program operations: Table 1: Operations of the flash memory cell 210 in FIG. 2 WL BL SL Read 2-3V 0.6-2V 0V Erase ~11-13V 0V 0V Programming 1-2V 10.5-3μA 9-10V

[0012] Other split-gate memory cell configurations are known and are other types of flash memory cells. For example, FIG3 depicts a quad-gate memory cell 310 comprising a source region 14, a drain region 16, a floating gate 20 above a first portion of a channel region 18, a select gate 22 (typically coupled to a word line WL) above a second portion of the channel region 18, a control gate 28 above the floating gate 20, and an erase gate 30 above the source region 14. This configuration is described in U.S. Patent 6,747,310, which is incorporated herein by reference for all purposes. Here, all gates except floating gate 20 are non-floating, meaning they are electrically connected or connectable to a voltage source. Programming is performed by heated electrons from the channel region 18 injecting themselves into the floating gate 20. Erasing is performed by tunneling electrons from the floating gate 20 to the erase gate 30 .

[0013] Table 2 describes typical voltage and current ranges that may be applied to the terminals of the memory cell 310 for performing read, erase, and program operations: Table 2: Operations of the flash memory cell 310 in FIG. 3 WL / SG BL CG EG SL Read 1.0-2V 0.6-2V 0-2.6V 0-2.6V 0V Erase -0.5V / 0V 0V 0V / -8V 8-12V 0V Programming 1V 0.1-1μA 8-11V 4.5-9V 4.5-5V

[0014] Figure 4 depicts a tri-gate memory cell 410, another type of flash memory cell. Memory cell 410 is identical to memory cell 310 in Figure 3, except that it does not have a separate control gate. Erase operations (in which erase is performed using the erase gate) and read operations are similar to those in Figure 3, except that no control gate bias is applied. Programming operations are also performed without a control gate bias, and therefore, a higher voltage is applied to the source line during programming operations to compensate for the lack of a control gate bias.

[0015] Table 3 describes typical voltage and current ranges that may be applied to the terminals of the memory cell 410 for performing read, erase, and program operations: Table 3: Operations of the flash memory cell 410 in FIG. 4 WL / SG BL EG SL Read 0.7-2.2V 0.6-2V 0-2.6V 0V Erase -0.5V / 0V 0V 11.5V 0V Programming 1V 0.2-3μA 4.5V 7-9V

[0016] FIG5 depicts a stacked-gate memory cell 510, another type of flash memory cell. Memory cell 510 is similar to memory cell 210 of FIG2 , except that floating gate 20 extends entirely over channel region 18, and control gate 22 (which will be coupled to the word line here) extends over floating gate 20, separated from the control gate by an insulating layer (not shown). Erasing is performed by FN tunneling of electrons from the FG to the substrate, while programming is performed by channel hot electron (CHE) injection between channel region 18 and drain region 16, with electrons flowing from source region 14 toward drain region 16. Reading is similar to the read operation for memory cell 210 with a higher control gate voltage.

[0017] Table 4 describes typical voltage ranges that may be applied to the terminals of the memory cell 510 and the substrate 12 for performing read, erase, and program operations: FIG4 : Operation of the flash memory cell 510 of FIG5 CG BL SL substrate Read 2-5V 0.6-2V 0V 0V Erase -8 to -10V / 0V FLT FLT 8-10V / 15-20V Programming 8-12V 3-5V 0V 0V

[0018] The methods and approaches described herein can be applied to other non-volatile memory technologies, such as FINFET split-gate flash or stacked-gate flash, NAND flash, silicon-oxide-nitride-oxide-silicon (SONOS, charge trapping in nitride), metal-oxide-nitride-oxide-silicon (MONOS, metal charge trapping in nitride), resistive RAM (ReRAM), phase-change memory (PCM), magnetic RAM (MRAM), ferroelectric RAM (FeRAM), charge trapping (CT) memory, carbon nanotube (CN) memory, dual-level or multi-level one-time programmable (OTP), and correlated electron RAM (CeRAM).

[0019] To utilize a memory array containing one of the non-volatile memory cell types described above in artificial neural networks, two modifications are made. First, the circuit is configured so that each memory cell can be individually programmed, erased, and read without adversely affecting the memory state of other memory cells in the array, as explained further below. Second, continuous (analog) programming of the memory cells is provided.

[0020] Specifically, the memory state (i.e., the charge on the floating gate) of each memory cell in the array can be changed independently and continuously from a fully erased state to a fully programmed state, and vice versa, with minimal disturbance to other memory cells. This means that the cell effectively stores an analogy or at least can store one of many discrete values ​​(e.g., 16 or 64 different values), which allows for extremely precise and individual tuning of all memory cells in the memory array, and this makes the memory array ideal for storing and fine-tuning the synaptic weights of neural networks. Neural network using non-volatile memory cell array

[0021] Figure 6 conceptually illustrates a non-limiting example of a neural network utilizing a non-volatile memory array according to an embodiment of the present invention. This example utilizes a non-volatile memory array neural network for facial recognition applications, but any other suitable application can be implemented using a non-volatile memory array-based neural network. The non-volatile memory array and associated circuitry used in the neural network is a compute-in-memory (CIM) engine or a vector-matrix multiplication (VMM) system.

[0022] S0 is the input layer, which for this example is a 32×32 pixel RGB image with 5-bit precision (i.e., three 32×32 pixel arrays, one for each color R, G, and B, with 5 bits of precision per pixel). Synapse CB1, which travels from input layer S0 to layer C1, applies different sets of weights in some cases and shares weights in others, and scans the input image with a 3×3 pixel overlapping filter (kernel), shifting the filter by 1 pixel (or more than 1 pixel, as dictated by the model). Specifically, the values ​​of 9 pixels in the 3×3 portion of the image (i.e., called the filter or kernel) are provided to synapse CB1, where these 9 input values ​​are multiplied by the appropriate weights. After summing the outputs of the multiplications, a single output value is determined and provided by the first synapse of CB1 for use in generating a pixel in one of the feature maps for layer C1. The 3×3 filter is then shifted right by one pixel within input layer S0 (i.e., a row of three pixels is added on the right and a row of three pixels is dropped on the left). The nine pixel values ​​in this newly positioned filter are then provided to synapse CB1, where they are multiplied by the same weights and a second single output value is determined by connecting the synapses. This process continues for all three colors and for all bits (precision values) until the 3×3 filter has scanned across the entire 32×32 pixel image of input layer S0. This process is then repeated using different sets of weights to generate different feature maps for layer C1 until all feature maps for layer C1 have been calculated. In this example, there are 16 feature maps in layer C1, each with 30×30 pixels. Each pixel is a new feature pixel extracted by multiplying the input by the kernel, and therefore each feature map is a two-dimensional array. Therefore, in this example, layer C1 comprises 16 layers of a two-dimensional array (it should be noted that the terms layer and array are logically related, not necessarily physically related, i.e., the array is not necessarily oriented as a physical two-dimensional array). Each of the 16 feature maps in layer C1 is generated by one of sixteen different sets of synaptic weights applied to the filter scan. The C1 feature maps can each target different aspects of the same image feature, such as edge recognition. For example, a first map (generated using a first set of weights, common to all scans used to generate this first map) can identify circular edges, while a second map (generated using a second set of weights, different from the first) can identify rectangular edges or the aspect ratio of certain features.

[0023] Activation function P1 (pooling) is applied before entering layer S1 from layer C1. This activation function pools values ​​from consecutive, non-overlapping 2×2 regions within each feature map. The purpose of pooling function P1 is to average nearby locations (or use a maximum function), for example, to reduce dependencies at edge locations and reduce data size before entering the next stage. At layer S1, there are 16 15×15 feature maps (i.e., 16 different arrays of 15×15 pixels each). Synapse CB2 from layer S1 to layer C2 scans the map in layer S1 using a 4×4 filter with a filter shift of 1 pixel. At layer C2, there are 22 12×12 feature maps. Activation function P2 (pooling) is applied before entering layer S2 from layer C2. This activation function pools values ​​from consecutive, non-overlapping 2×2 regions within each feature map. At layer S2, there are 22 6×6 feature maps. An activation function (pooling) is applied at synapse CB3 from layer S2 into layer C3, where each neuron in layer C3 is connected to a graph in layer S2 via a respective synapse CB3. Layer C3 has 64 neurons. Synapse CB4 from layer C3 into output layer S3 fully connects C3 to S3, meaning that every neuron in layer C3 is connected to every neuron in layer S3. The output from S3 consists of 10 neurons, with the highest output neuron determining the class. This output can, for example, indicate the recognition or classification of the content of the original image.

[0024] Each synaptic layer is implemented using an array or portion of an array of non-volatile memory cells.

[0025] Figure 7 is a block diagram of an array that can be used for this purpose. The vector-matrix multiplication (VMM) array 32 includes non-volatile memory cells and serves as a synapse between one layer and the next (e.g., CB1, CB2, CB3, and CB4 in Figure 6). Specifically, the VMM array 32 includes a non-volatile memory cell array 33, an erase gate and wordline gate decoder 34, a control gate decoder 35, a bitline decoder 36, and a source line decoder 37, which decode the respective inputs of the non-volatile memory cell array 33. The inputs to the VMM array 32 can come from either the erase gate and wordline gate decoder 34 or the control gate decoder 35. In this example, the source line decoder 37 also decodes the output of the non-volatile memory cell array 33. Alternatively, the bitline decoder 36 can decode the output of the non-volatile memory cell array 33.

[0026] Non-volatile memory cell array 33 serves two purposes. First, it stores weights to be used by VMM array 32. Second, non-volatile memory cell array 33 efficiently multiplies inputs by the weights stored in non-volatile memory cell array 33 and adds the results along an output line (source line or bit line) to produce an output, which will be the input to the next layer or the final layer. By performing multiplication and addition functions, non-volatile memory cell array 33 eliminates the need for separate multiplication and addition logic circuits and is also power-efficient due to its in-memory computation.

[0027] The output of the non-volatile memory cell array 33 is supplied to a differential summer (such as a summing operational amplifier or a summing current mirror) 38, which sums the output of the non-volatile memory cell array 33 to produce a single value for the convolution. The differential summer 38 is configured to perform the summation of positive and negative weights.

[0028] The total output value of the differential summer 38 is then supplied to the activation function block 39, which rectifies the output. The activation function block 39 can provide a sigmoid, hyperbolic tangent (tanh), or ReLU function. The rectified output value of the activation function block 39 becomes an element of the feature map for the next layer (e.g., C1 in FIG6 ) and is then applied to the next synapse to generate the next feature map layer or the final layer. Thus, in this example, the non-volatile memory cell array 33 constitutes a plurality of synapses (which receive inputs from the previous neuron layer or from an input layer such as an image database), and the summing operational amplifier 38 and activation function block 39 constitute a plurality of neurons.

[0029] The inputs (WLx, EGx, CGx, and optionally BLx and SLx) to the VMM array 32 in FIG. 7 can be analog levels, binary levels, or digital bits (in which case the DACs are configured to convert the digital bits to the appropriate input analog levels), and the outputs can be analog levels, binary levels, or digital bits (in which case the output ADCs are configured to convert the output analog levels to digital bits).

[0030] FIG8 is a block diagram illustrating the use of various layers of VMM array 32, here labeled VMM arrays 32a, 32b, 32c, 32d, and 32e. As shown in FIG8 , the input, designated Inputx, is converted from digital to analog by digital-to-analog converter 31 and provided to input VMM array 32a. The converted analog input can be either voltage or current. The first layer of input D / A conversion can be performed using a function or LUT (look-up table) that maps the input Inputx to analog levels suitable for the matrix multipliers of input VMM array 32a. Input conversion can also be performed using an analog-to-analog (A / A) converter to convert external analog inputs into mapped analog inputs for input VMM array 32a.

[0031] The output generated by input VMM array 32a is provided as input to the next VMM array (hidden level 1) 32b, which in turn generates outputs that are provided as input to the next VMM array (hidden level 2) 32c, and so on. The various layers of VMM array 32 serve as different synaptic and neuron layers of a convolutional neural network (CNN). Each VMM array 32a, 32b, 32c, 32d, and 32e can be a separate physical non-volatile memory array, or multiple VMM arrays can utilize different portions of the same physical non-volatile memory array, or multiple VMM arrays can utilize overlapping portions of the same physical non-volatile memory array. The example shown in FIG8 contains five layers (32a, 32b, 32c, 32d, 32e): an input layer (32a), two hidden layers (32b, 32c), and two fully connected layers (32d, 32e). One of ordinary skill in the art will appreciate that this is merely an example and that the system may alternatively include more than two hidden layers and more than two fully connected layers. Vector-Matrix Multiplication (VMM) Array

[0032] FIG9 depicts a neuron VMM array 900, which is particularly suitable for memory cells 310 shown in FIG3 and serves as the synapse-to-neuron portion between the input layer and the next layer. VMM array 900 includes a memory array 901 of non-volatile memory cells and a reference array 902 of non-volatile reference memory cells (at the top of the array). Alternatively, another reference array can be placed at the bottom.

[0033] In VMM array 900, control gate lines, such as control gate line 903, run vertically (thus, reference array 902 in the column direction is orthogonal to control gate line 903), and erase gate lines, such as erase gate line 904, run horizontally. Here, inputs to VMM array 900 are provided on control gate lines (CG0, CG1, CG2, CG3), and outputs from VMM array 900 appear on source lines (SL0, SL1). In one example, only even-numbered columns are used, and in another, only odd-numbered columns are used. The current placed on each source line (SL0, SL1, respectively) performs a summation function on all currents from the memory cells connected to that particular source line.

[0034] As described herein for the neural network, the non-volatile memory cells of the VMM array 900, that is, the memory cells 310 of the VMM array 900, may be configured to operate in the sub-threshold region.

[0035] The non-volatile reference memory cell and the non-volatile memory cells described herein are biased in weak inversion (sub-threshold region): Ids = Io * e (Vg- Vth) / nVt= w * Io * e (Vg) / nVt, where w = e (-Vth) / nVt Where Ids is the drain-to-source current; Vg is the gate voltage on the memory cell; Vth is the threshold voltage of the memory cell; Vt is the thermal voltage = k*T / q, where k is the Boltzmann constant, T is the temperature in kelvins, and q is the electron charge; n is the slope factor = 1 + (Cdep / Cox), where Cdep = the capacitance of the depletion layer and Cox is the capacitance of the gate oxide layer; Io is the memory cell current at a gate voltage equal to the threshold voltage, and Io is proportional to (Wt / L)*u*Cox* (n-1) * Vt², where u is the carrier mobility, and Wt and L are the width and length of the memory cell, respectively.

[0036] For I to V logarithmic converters that use memory cells (such as reference memory cells or peripheral memory cells) or transistors to convert input current to input voltage: Vg= n*Vt*log [Ids / wp*Io] Here, wp is the reference or peripheral memory cell w.

[0037] For a memory array used as a vector matrix multiplier VMM array with current input, the output current is: Iout = wa * Io * e (Vg) / nVt, that is Iout = (wa / wp) * Iin = W * Iin W = e (Vthp-Vtha) / nVt Here, wa = w of each memory cell in the memory array. Vthp is the effective threshold voltage of the peripheral memory cells, and Vtha is the effective threshold voltage of the main (data) memory cells. It should be noted that the transistor threshold voltage is a function of the substrate base bias voltage, and the substrate base bias voltage, denoted as Vsb, can be adjusted to compensate for various conditions at this temperature. The threshold voltage Vth can be expressed as: Vth = Vth0 + γ (SQRT |Vsb-2*φF)-SQRT |2* φF |) Where Vth0 is the threshold voltage with zero substrate bias, φF is the surface potential, and γ is the matrix effect parameter.

[0038] The word line or control gate may be used as the input of the memory cell for the input voltage.

[0039] Alternatively, the flash memory cells of the VMM array described herein may be configured to operate in the linear region: Ids = β* (Vgs-Vth)*Vds;β = u*Cox*Wt / L W = α (Vgs-Vth) This means that the weight W in the linear region is proportional to (Vgs-Vth).

[0040] The word line or control gate or bit line or source line can be used as the input of the memory cell operating in the linear region. The bit line or source line can be used as the output of the memory cell.

[0041] For an I-to-V linear converter, a memory cell (such as a reference memory cell or a peripheral memory cell) or a transistor operating in a linear region may be used to linearly convert an input / output current into an input / output voltage.

[0042] Alternatively, the memory cells of the VMM array described herein may be configured to operate in the saturation region: Ids = 1 / 2 * β* (Vgs-Vth) 2;β = u*Cox*Wt / L Wα (Vgs-Vth) 2, which means that the weight W is proportional to (Vgs-Vth) 2.

[0043] The word line, control gate, or erase gate can be used as the input of a memory cell operating in the saturation region. The bit line or source line can be used as the output of an output neuron.

[0044] Alternatively, the memory cells of the VMM array described herein may be used in all regions or combinations thereof (subthreshold, linear, or saturation regions) of each or multiple layers of a neural network.

[0045] Other examples of the VMM array 32 of Figure 7 are described in U.S. Patent No. 10,748,630, which is incorporated herein by reference. As described in that application, source lines or bit lines can be used as neuron outputs (current summing outputs).

[0046] FIG10 depicts a neuron VMM array 1000, which is particularly well-suited for memory cells 210 shown in FIG2 and serves as a synapse between the input layer and the next layer. VMM array 1000 includes a memory array 1003 of non-volatile memory cells, a reference array 1001 of first non-volatile reference memory cells, and a reference array 1002 of second non-volatile reference memory cells. Reference arrays 1001 and 1002, arranged in the row direction of the array, are used to convert current inputs flowing into terminals BLR0, BLR1, BLR2, and BLR3 into voltage inputs WL0, WL1, WL2, and WL3. In practice, the first and second non-volatile reference memory cells are diode-connected through multiplexers 1014 (partially depicted), into which the current inputs flow. The reference cells are tuned (e.g., programmed) to a target reference level. The target reference level is provided by a reference mini-array matrix (not shown).

[0047] Memory array 1003 serves two purposes. First, it stores weights to be used by VMM array 1000 on its individual memory cells. Second, memory array 1003 efficiently multiplies inputs (i.e., current inputs provided at terminals BLR0, BLR1, BLR2, and BLR3, which are converted by reference arrays 1001 and 1002 into input voltages supplied to word lines WL0, WL1, WL2, and WL3) by the weights stored in memory array 1003. It then sums all the results (memory cell currents) to produce outputs on respective bit lines (BL0-BLN), which serve as inputs to the next layer or the final layer. By performing multiplication and addition functions, memory array 1003 eliminates the need for separate multiplication and addition logic circuits and is also power-efficient. Here, voltage inputs are provided on word lines WL0, WL1, WL2, and WL3, and outputs appear on respective bit lines BL0 through BLN during a read (inference) operation. The current placed on each of the bit lines BL0-BLN performs a summing function of the current from all non-volatile memory cells connected to that particular bit line.

[0048] Table 5 describes the operating voltages and currents for VMM array 1000. The rows in the table indicate the voltages applied to the word line for a selected cell, the word line for unselected cells, the bit line for a selected cell, the bit line for unselected cells, the source line for a selected cell, and the source line for unselected cells. The columns indicate read, erase, and program operations. Table 5: Operations of the VMM array 1000 of FIG. 10 WL WL-Not Selected BL BL-Not selected SL SL-Not Selected Read 1-3.5V -0.5V / 0V 0.6-2V (Ineuron) 0.6V-2V / 0V 0V 0V Erase ~5-13V 0V 0V 0V 0V 0V Programming 1-2V -0.5V / 0V 0.1-3 uA Vinh ~2.5V 4-10V 0-1V / FLT

[0049] FIG11 depicts a neuron VMM array 1100, which is particularly well-suited for memory cells 210 shown in FIG2 and serves as the synaptic connection between the input layer and the next layer. VMM array 1100 includes a memory array 1103 of nonvolatile memory cells, a reference array 1101 of first nonvolatile reference memory cells, and a reference array 1102 of second nonvolatile reference memory cells. Reference arrays 1101 and 1102 extend in the column direction of VMM array 1100. VMM array 1100 is similar to VMM 1000, except that in VMM array 1100, word lines extend vertically. Here, inputs are provided on word lines (WLA0, WLB0, WLA1, WLB1, WLA2, WLB2, WLA3, WLB3), and outputs appear on source lines (SL0, SL1) during a read operation. The current placed on each source line performs a summing function on all currents from the memory cells connected to that particular source line.

[0050] Table 6 describes the operating voltages and currents for VMM array 1100. The rows in the table indicate the voltages applied to the word line for a selected cell, the word line for unselected cells, the bit line for a selected cell, the bit line for unselected cells, the source line for a selected cell, and the source line for unselected cells. The columns indicate read, erase, and program operations. Table 6: Operations of the VMM array 1100 of FIG. 11 WL WL-Not Selected BL BL-Not selected SL SL-Not Selected Read 1-3.5V -0.5V / 0V 0.6-2V 0.6V-2V / 0V ~0.3-1V (Ineuron) 0V Erase ~5-13V 0V 0V 0V 0V SL-Suppression (~4-8V) Programming 1-2V -0.5V / 0V 0.1-3 uA Vinh ~2.5V 4-10V 0-1V / FLT

[0051] FIG12 illustrates a neuron VMM array 1200, which is particularly well-suited for memory cell 310 shown in FIG3 and serves as the synaptic interface between the input layer and the next layer. VMM array 1200 includes a memory array 1203 of nonvolatile memory cells, a reference array 1201 of first nonvolatile reference memory cells, and a reference array 1202 of second nonvolatile reference memory cells. Reference arrays 1201 and 1202 are used to convert current inputs flowing into terminals BLR0, BLR1, BLR2, and BLR3 into voltage inputs CG0, CG1, CG2, and CG3. In practice, the first and second nonvolatile reference memory cells are diode-connected through multiplexers 1212 (only partially illustrated), with current inputs flowing into these multiplexers via BLR0, BLR1, BLR2, and BLR3. Multiplexers 1212 each include a respective multiplexer 1205 and a cascode transistor 1204 to ensure a constant voltage on the bit line (such as BLR0) of each of the first and second non-volatile reference memory cells during a read operation. The reference cells are tuned to a target reference level.

[0052] Memory array 1203 is used for two purposes. First, it stores weights to be used by VMM array 1200. Next, the memory array 1203 effectively multiplies the inputs (current inputs provided to terminals BLR0, BLR1, BLR2, and BLR3, which the reference arrays 1201 and 1202 convert into input voltages to be supplied to the control gates (CG0, CG1, CG2, and CG3) by the weights stored in the memory array, and then adds all the results (cell currents) to produce an output, which is presented on BL0 through BLN and will be the input to the next layer or the input to the final layer. By performing multiplication and addition functions, the memory array eliminates the need for separate multiplication and addition logic circuits and is also power efficient. Here, the inputs are provided on the control gate lines (CG0, CG1, CG2, and CG3), and the outputs appear on the bit lines (BL0-BLN) during a read operation. The current placed on each bit line performs a summing function on all the currents from the memory cells connected to that particular bit line.

[0053] VMM array 1200 implements unidirectional tuning for the non-volatile memory cells in memory array 1203. That is, each non-volatile memory cell is erased and then partially programmed until the desired charge on the floating gate is reached. If excessive charge is placed on the floating gate (causing an incorrect value to be stored in the cell), the sequence of cell erase and partial programming operations begins again. As shown, two columns sharing the same erase gate (such as EG0 or EG1) need to be erased together (this may be called a page erase), and then each cell is partially programmed until the desired charge on the floating gate is reached.

[0054] Table 7 describes the operating voltages and currents for VMM array 1200. The rows in the table indicate the voltages applied to the word line for a selected cell, the word line for an unselected cell, the bit line for a selected cell, the bit line for an unselected cell, the control gate for a selected cell, the control gate for an unselected cell in the same sector as the selected cell, the control gate for an unselected cell in a different sector from the selected cell, the erase gate for a selected cell, the erase gate for an unselected cell, the source line for a selected cell, and the source line for an unselected cell. The columns indicate read, erase, and program operations. Table 7: Operations of the VMM array 1200 of FIG. 12 WL WL-Not Selected BL BL-not Selected CG CG-not Selected phase Same sector CG-not Selected EG EG-Not Selected SL SL-Not Selected Read 1.0-2V -0.5 V / 0 V 0.6-2V (Ineuron) 0V 0-2.6V 0-2.6V 0-2.6V 0-2.6V 0-2.6V 0V 0V Erase 0V 0V 0V 0V 0V 0-2.6V 0-2.6V 5-12V 0-2.6V 0V 0V Programming 0.7-1V -0.5 V / 0 V 0.1-1uA Vinh (1-2V) 4-11V 0-2.6V 0-2.6V 4.5-5V 0-2.6V 4.5-5V 0-1V

[0055] FIG13 depicts a neuron VMM array 1300, which is particularly well-suited for memory cell 310 shown in FIG3 and serves as the synaptic link between the input layer and the next layer. VMM array 1300 includes a memory array 1303 of nonvolatile memory cells, a reference array 1301 of first nonvolatile reference memory cells, and a reference array 1302 of second nonvolatile reference memory cells. EG lines EGR0, EG0, EG1, and EGR1 run vertically, while CG lines CG0, CG1, CG2, and CG3 and SL lines WL0, WL1, WL2, and WL3 run horizontally. VMM array 1300 is similar to VMM array 1400, except that VMM array 1300 implements bidirectional tuning. Due to the use of separate EG lines, individual cells can be fully erased, partially programmed, and partially erased as needed to achieve the desired charge on the floating gate. As shown, reference arrays 1301 and 1302 convert input currents at terminals BLR0, BLR1, BLR2, and BLR3 into control gate voltages CG0, CG1, CG2, and CG3 to be applied to the memory cells in the column direction (via the action of diode-connected reference cells through multiplexer 1314). The current outputs (neurons) are on bit lines BL0-BLN, where each bit line sums all currents from the non-volatile memory cells connected to that bit line.

[0056] Table 8 describes the operating voltages and currents for the VMM array 1300. The rows in the table indicate the voltages applied to the word line for a selected cell, the word line for unselected cells, the bit line for a selected cell, the bit line for an unselected cell, the control gate for a selected cell, the control gate for an unselected cell in the same sector as the selected cell, the control gate for an unselected cell in a different sector than the selected cell, the erase gate for a selected cell, the erase gate for an unselected cell, the source line for a selected cell, and the source line for an unselected cell. The columns indicate read, erase, and program operations. Table 8: Operations of the VMM array 1300 of FIG. 13 WL WL-not Selected BL BL-not Selected CG CG-not Selected phase Same sector CG-not Selected EG EG-Not Selected SL SL-Not Selected Read 1.0-2V -0.5 V / 0 V 0.6-2V (Ineuron) 0V 0-2.6V 0-2.6V 0-2.6V 0-2.6V 0-2.6V 0V 0V Erase 0V 0V 0V 0V 0V 4-9V 0-2.6V 5-12V 0-2.6V 0V 0V Programming 0.7-1V -0.5 V / 0 V 0.1-1uA Vinh (1-2V) 4-11V 0-2.6V 0-2.6V 4.5-5V 0-2.6V 4.5-5V 0-1V

[0057] FIG22 illustrates a neuron VMM array 2200, which is particularly suitable for memory cell 210 shown in FIG2 and serves as the synapse-to-neuron portion between the input layer and the next layer. In VMM array 2200, inputs INPUT 0, ..., INPUT N are received on bit lines BL 0, ..., BL N, respectively, and outputs OUTPUT 1, OUTPUT 2, OUTPUT 3, and OUTPUT 4 are generated on source lines SL 0, SL 1, SL 2, and SL 3, respectively.

[0058] FIG23 illustrates a neuron VMM array 2300, which is particularly suitable for memory cell 210 shown in FIG2 and serves as part of the synapse between the input layer and the next layer to the neuron. In this example, inputs INPUT 0, INPUT 1, INPUT 2, and INPUT 3 are received on source lines SL0, SL1, SL2, and SL3, respectively, and outputs OUTPUT 0, ..., OUTPUT N are generated on bit lines BL0, ..., BLN.

[0059] FIG24 illustrates a neuron VMM array 2400, which is particularly suitable for the memory cell 210 shown in FIG2 and serves as the synapse-to-neuron portion between the input layer and the next layer. In this example, inputs INPUT 0, ..., INPUT M are received on word lines WL0, ..., WL M, respectively, and outputs OUTPUT 0, ..., OUTPUT N are generated on bit lines BL0, ..., BL N, respectively.

[0060] FIG25 illustrates a neuron VMM array 2500, which is particularly suitable for memory cell 310 shown in FIG3 and serves as the synapse-to-neuron portion between the input layer and the next layer. In this example, inputs INPUT 0, ..., INPUT M are received on word lines WL0, ..., WL0 M, respectively, and outputs OUTPUT 0, ..., OUTPUT N are generated on bit lines BL0, ..., BL0 N, respectively.

[0061] FIG26 illustrates a neuron VMM array 2600, which is particularly suitable for the memory cell 410 shown in FIG4 and serves as the synapse-to-neuron portion between the input layer and the next layer. In this example, inputs INPUT 0, ..., INPUT n are received on vertical control gate lines CG 0, ..., CG N, respectively, and outputs OUTPUT 1 and OUTPUT 2 are generated on source lines SL 0 and SL 1.

[0062] FIG27 depicts a neuron VMM array 2700, which is particularly suitable for memory cell 410 shown in FIG4 and serves as the synapse-to-neuron portion between the input layer and the next layer. In this example, inputs INPUT 0, ..., INPUT N are received at the gates of bitline control gates 2701-1, 2701-2, ..., 2701-(N-1), and 2701-N, respectively, which are coupled to bitlines BL0, ..., BLN, respectively. Exemplary outputs OUTPUT 1 and OUTPUT 2 are generated on source lines SL0 and SL1.

[0063] FIG28 depicts a neuron VMM array 2800, which is particularly suitable for memory cell 310 shown in FIG3 , memory cell 510 shown in FIG5 , and memory cell 710 shown in FIG7 , and serves as the synapse-to-neuron portion between the input layer and the next layer. In this example, inputs INPUT 0, ..., INPUT M are received on word lines WL0, ..., WL1 M, and outputs OUTPUT 0, ..., OUTPUT N are generated on bit lines BL0, ..., BL1 N, respectively.

[0064] FIG29 depicts a neuron VMM array 2900, which is particularly suitable for memory cell 310 shown in FIG3 , memory cell 510 shown in FIG5 , and memory cell 710 shown in FIG7 , and serves as the synapse-to-neuron portion between the input layer and the next layer. In this example, inputs INPUT 0, ..., INPUT M are received on control gate lines CG 0, ..., CG M. Outputs OUTPUT 0, ..., OUTPUT N are generated on vertical source lines SL 0, ..., SL N, respectively, where each source line SL i is coupled to the source lines of all memory cells in row i.

[0065] FIG30 illustrates a neuron VMM array 3000, which is particularly suitable for memory cell 310 shown in FIG3 , memory cell 510 shown in FIG5 , and memory cell 710 shown in FIG7 , and serves as the synapse-to-neuron portion between the input layer and the next layer. In this example, inputs INPUT 0, ..., INPUT M are received on control gate lines CG 0, ..., CG M. Outputs OUTPUT 0, ..., OUTPUT N are generated on vertical bit lines BL 0, ..., BL N, respectively, where each bit line BL i is coupled to the bit lines of all memory cells in row i. Long and short-term memory

[0066] Prior art includes a concept called long short-term memory (LSTM). LSTM cells are often used in neural networks. LSTM allows neural networks to remember information for a predetermined, arbitrary time interval and use that information in subsequent operations. Conventional LSTM cells consist of a cell, an input gate, an output gate, and a forget gate. These three gates regulate the flow of information into and out of the cell and the time interval over which information is remembered within the LSTM. VMMs are particularly well-suited for LSTM cells.

[0067] FIG14 illustrates an exemplary LSTM 1400. LSTM 1400 in this example includes cells 1401, 1402, 1403, and 1404. Cell 1401 receives an input vector x0 and generates an output vector h0 and a cell state vector c0. Cell 1402 receives an input vector x1, the output vector (hidden state) h0 from cell 1401, and the cell state c0 from cell 1401, and generates an output vector h1 and a cell state vector c1. Cell 1403 receives an input vector x2, the output vector (hidden state) h1 from cell 1402, and the cell state c1 from cell 1402, and generates an output vector h2 and a cell state vector c2. Cell 1404 receives input vector x3, output vector (hidden state) h2 from cell 1403, and cell state c2 from cell 1403, and produces output vector h3. Additional cells may be used, and an LSTM with four cells is merely an example.

[0068] FIG15 depicts an exemplary implementation of an LSTM cell 1500, which may be used for cells 1401, 1402, 1403, and 1404 in FIG14. LSTM cell 1500 receives an input vector x(t), a cell state vector c(t-1) from the previous cell, and an output vector h(t-1) from the previous cell, and generates a cell state vector c(t) and an output vector h(t).

[0069] LSTM cell 1500 includes sigmoid function components 1501, 1502, and 1503, each of which uses a number between 0 and 1 to control the amount of each component in the input vector that is allowed to pass through the output vector. LSTM cell 1500 also includes hyperbolic tangent components 1504 and 1505 for applying the hyperbolic tangent function to the input vector, multiplier components 1506, 1507, and 1508 for multiplying two vectors together, and addition component 1509 for adding two vectors together. The output vector h(t) can be provided to the next LSTM cell in the system, or it can be accessed for other purposes.

[0070] FIG16 depicts LSTM cell 1600, which is an example of an implementation of LSTM cell 1500. For the convenience of the reader, the same numbering as in LSTM cell 1500 is used in LSTM cell 1600. Sigmoid function components 1501, 1502, and 1503, as well as hyperbolic tangent component 1504, each include multiple VMM arrays 1601 and an activation function block 1602. Therefore, it can be seen that VMM arrays are particularly suitable for LSTM cells used in certain neural network systems. Multiplier components 1506, 1507, and 1508, as well as adder component 1509, are implemented digitally or analogically. Activation function block 1602 can be implemented digitally or analogically.

[0071] An alternative to LSTM cell 1600 (and another example of an implementation of LSTM cell 1500) is shown in Figure 17. In Figure 17, sigmoid function components 1501, 1502, and 1503 and hyperbolic tangent component 1504 share the same physical hardware (VMM array 1701 and activation function block 1702) in a time-multiplexed manner. LSTM cell 1700 also includes a multiplier component 1703 for multiplying two vectors together, an addition component 1708 for adding two vectors together, a hyperbolic tangent component 1505 (which includes an activation function block 1702), a register 1707 for storing the value i(t) when it is output from the sigmoid function block 1702, a register 1704 for storing the value f(t)*c(t-1) when it is output from the multiplier component 1703 through a multiplexer 1710, a register 1705 for storing the value i(t)*u(t) when it is output from the multiplier component 1703 through the multiplexer 1710, and a register 1706 for storing the value o(t)*c(t) when it is output from the multiplier component 1703 through the multiplexer 1710 and the multiplexer 1709.

[0072] LSTM cell 1600 contains multiple sets of VMM arrays 1601 and respective activation function blocks 1602, while LSTM cell 1700 contains one set of VMM arrays 1701 and activation function blocks 1702, which are used to represent multiple layers in an instance of LSTM cell 1700. LSTM cell 1700 requires less space than LSTM 1600 because LSTM cell 1700 requires ¼ of the space for VMM and activation function blocks compared to LSTM cell 1600.

[0073] It will be further understood that an LSTM cell will typically include multiple VMM arrays, each of which utilizes functionality provided by certain circuit blocks external to the VMM arrays, such as the summer and excitation function blocks, as well as the high-voltage generation block. Providing a separate circuit block for each VMM array would require a significant amount of space within the semiconductor device and would be somewhat inefficient. Therefore, the example described below reduces the circuitry provided external to the VMM arrays themselves. Gated recursive unit

[0074] Analogous VMM implementations can be used in Gated Recurrent Unit (GRU) systems. A GRU is a gated mechanism in recurrent neural networks. A GRU is similar to an LSTM, except that a GRU cell typically contains fewer components than an LSTM cell.

[0075] FIG18 depicts an exemplary GRU 1800. GRU 1800 in this example includes cells 1801, 1802, 1803, and 1804. Cell 1801 receives an input vector x0 and produces an output vector h0. Cell 1802 receives an input vector x1, the output vector h0 from cell 1801, and produces an output vector h1. Cell 1803 receives an input vector x2 and the output vector (hidden state) h1 from cell 1802, and produces an output vector h2. Cell 1804 receives an input vector x3 and the output vector (hidden state) h2 from cell 1803, and produces an output vector h3. Additional cells may be used, and a GRU with four cells is merely an example.

[0076] FIG19 depicts an exemplary implementation of a GRU cell 1900, which may be used in cells 1801, 1802, 1803, and 1804 of FIG18. GRU cell 1900 receives an input vector x(t) and an output vector h(t-1) from the previous GRU cell and generates an output vector h(t). GRU cell 1900 includes sigmoid function components 1901 and 1902, each of which applies a number between 0 and 1 to the components of the output vector h(t-1) and the input vector x(t). GRU cell 1900 also includes a hyperbolic tangent component 1903 for applying a hyperbolic tangent function to the input vector, a plurality of multiplier components 1904, 1905, and 1906 for multiplying two vectors together, an addition component 1907 for adding two vectors together, and a complement component 1908 for subtracting the input from 1 to generate the output.

[0077] FIG20 depicts GRU cell 2000, which is an example of an implementation of GRU cell 1900. For the convenience of the reader, the same numbering as in GRU cell 1900 is used in GRU cell 2000. As shown in FIG20, sigmoid function components 1901 and 1902 and hyperbolic tangent component 1903 each include multiple VMM arrays 2001 and an activation function block 2002. Thus, it can be seen that VMM arrays are particularly useful for GRU cells used in certain neural network systems. Multiplier components 1904, 1905, and 1906, addition component 1907, and complement component 1908 are implemented digitally or analogically. Activation function block 2002 can be implemented digitally or analogically.

[0078] An alternative to GRU cell 2000 (and another example of an implementation of GRU cell 1900) is shown in FIG21. In FIG21, GRU cell 2100 utilizes a VMM array 2101 and an activation function block 2102, which, when configured as a sigmoid function, applies a number between 0 and 1 to control the amount of each component in the input vector that is allowed to pass through to the output vector. In FIG21, sigmoid function components 1901 and 1902 and hyperbolic tangent component 1903 share the same physical hardware (VMM array 2101 and activation function block 2102) in a time-multiplexed manner. The GRU cell 2100 also includes a multiplier component 2103 for multiplying two vectors together, an addition component 2105 for adding two vectors together, a complement component 2109 for subtracting an input from 1 to produce an output, a multiplexer 2104, a register 2106 for storing the value h(t-1)*r(t) when it is output from the multiplier component 2103 through the multiplexer 2104, a register 2107 for storing the value h(t-1)*z(t) when it is output from the multiplier component 2103 through the multiplexer 2104, and a register 2108 for storing the value h^(t)*(1-z(t)) when it is output from the multiplier component 2103 through the multiplexer 2104.

[0079] GRU cell 2000 contains multiple sets of VMM arrays 2001 and excitation function blocks 2002, while GRU cell 2100 contains one set of VMM arrays 2101 and excitation function blocks 2102, which are used to represent multiple layers in an instance of GRU cell 2100. GRU cell 2100 requires less space than GRU cell 2000 because GRU cell 2100 requires 1 / 3 of the space for VMM and excitation function blocks compared to GRU cell 2000.

[0080] It will be further understood that a GRU system will typically include multiple VMM arrays, each of which utilizes functionality provided by certain circuit blocks external to the VMM arrays, such as the summer and excitation function blocks, as well as the high-voltage generation block. Providing a separate circuit block for each VMM array would require a significant amount of space within the semiconductor components and would be somewhat inefficient. Therefore, the example described below reduces the circuitry provided external to the VMM arrays themselves. Input and output

[0081] The inputs to the VMM array can be analog levels, binary levels, pulses, time-modulated pulses, or digital bits (in which case a DAC is used to convert the digital bits to the appropriate input analog levels), and the outputs can be analog levels, binary levels, timed pulses, pulses, or digital bits (in which case an output ADC is used to convert the output analog levels to digital bits). Differential Cell

[0082] Generally speaking, for each memory cell in the VMM array, each weight W can be implemented by a single memory cell, a differential cell, or two hybrid memory cells (the average of the two cells). In the differential cell case, two memory cells are used to implement the weight W as a differential weight (W = W+ - W-). In the two hybrid memory cells case, two memory cells are used to implement the weight W as the average of the two cells.

[0083] FIG31 depicts a VMM system 3100. In some examples, the weights W stored in the VMM array are stored as a differential pair of W+ (positive weight) and W− (negative weight), where W = (W+) - (W−). In VMM system 3100, half the bit lines are designated as W+ lines, i.e., bit lines connected to memory cells that will store positive weights W+, and the other half are designated as W− lines, i.e., bit lines connected to memory cells that implement negative weights W−. The W− lines are interspersed with the W+ lines in an alternating pattern. Subtraction operations are performed by summing circuits, such as summing circuits 3101 and 3102, that receive current from the W+ and W− lines. The outputs of the W+ and W− lines are combined, effectively yielding W = W+ - W− for each pair of (W+, W−) cells for all (W+, W−) line pairs. Although described above with respect to W- lines being interspersed among W+ lines in an alternating manner, in other examples, the W+ lines and W- lines may be arbitrarily located anywhere in the array.

[0084] 32 depicts another example. In a VMM system 3210, positive weights W+ are implemented in a first array 3211 and negative weights W- are implemented in a second array 3212, which is separate from the first array, and the resulting weights are appropriately combined by a summing circuit 3213.

[0085] FIG33 depicts a VMM system 3300. Weights W stored in the VMM array are stored as differential pairs W+ (positive weights) and W− (negative weights), where W = (W+) - (W−). VMM system 3300 includes arrays 3301 and 3302. Half the bit lines in each of arrays 3301 and 3302 are designated as W+ lines, i.e., bit lines connected to memory cells that will store positive weights W+. The other half of the bit lines in each of arrays 3301 and 3302 are designated as W− lines, i.e., bit lines connected to memory cells that implement negative weights W−. The W− lines are interspersed alternately with the W+ lines. Subtraction operations are performed by summing circuits, such as summing circuits 3303, 3304, 3305, and 3306, that receive current from the W+ and W− lines. The outputs of the W+ and W- lines from each array 3301 and 3302, respectively, are combined, effectively yielding W = W+ - W- for each pair of (W+, W-) cells for all pairs of (W+, W-) lines. Furthermore, the W values ​​from each array 3301 and 3302 can be further combined via summing circuits 3307 and 3308, such that each W value is the result of subtracting the W value from array 3302 from the W value from array 3301. This means that the final result from summing circuits 3307 and 3308 is the difference of two difference values. The number of weights per layer

[0086] Each non-volatile memory cell used in an analog neural memory system can be erased and programmed to maintain a specific and precise charge level, or number of electrons, in its floating gate. Each floating gate can hold one of N different values, where N is the number of different weights that can be represented by each cell. Examples of N include 16, 32, 64, 128, and 256.

[0087] Referring again to Figures 6 and 8, a neural network contains multiple layers. Some layers do not require the use of an analog neural memory system. For example, some layers, such as the first layer of a residual neural network (RESNET), only require cells capable of storing two weights, "0" or "1," meaning N = 2, which are binary weights. These layers do not require the use of an analog neural memory system because a simpler digital system would suffice. Using an analog neural memory system for these layers is inefficient because it provides a level of precision not required by layers requiring N = 2 (e.g., N = 256). Summary of the Invention

[0088] Various embodiments of a neural network comprising one or more analog in-memory computation engines and one or more digital in-memory computation engines are disclosed. This allows the digital CIM engine to be used in situations where layers contain binary weights but not analog weights. Simple diagram description

[0089] FIG1 is a diagram illustrating an artificial neural network. FIG. 2 depicts a prior art split-gate flash memory cell. FIG. 3 illustrates another prior art split-gate flash memory cell. FIG. 4 illustrates another prior art split-gate flash memory cell. FIG. 5 illustrates another prior art split-gate flash memory cell. FIG6 is a diagram illustrating different levels of an exemplary artificial neural network utilizing one or more non-volatile memory arrays. FIG7 is a block diagram illustrating a VMM system. FIG8 is a block diagram illustrating an exemplary artificial neural network utilizing one or more VMM systems. FIG9 illustrates another example of a VMM system. FIG. 10 illustrates another example of a VMM system. FIG. 11 illustrates another example of a VMM system. FIG. 12 illustrates another example of a VMM system. FIG. 13 illustrates another example of a VMM system. FIG14 depicts a prior art long short-term memory system. FIG. 15 depicts an exemplary cell for use in a long short-term memory system. FIG. 16 depicts an exemplary implementation of the cell of FIG. 15 . FIG. 17 depicts another exemplary embodiment of the cell of FIG. 15 . FIG. 18 depicts a prior art gated recursive cell system. FIG. 19 depicts an exemplary cell for use in a gated recursive cell system. FIG. 20 depicts an exemplary implementation of the cell of FIG. 19 . FIG. 21 depicts another exemplary embodiment of the cell of FIG. 19 . FIG22 illustrates another example of a VMM system. FIG23 illustrates another example of a VMM system. FIG. 24 illustrates another example of a VMM system. FIG25 illustrates another example of a VMM system. FIG. 26 illustrates another example of a VMM system. FIG. 27 illustrates another example of a VMM system. FIG28 illustrates another example of a VMM system. FIG29 illustrates another example of a VMM system. FIG30 illustrates another example of a VMM system. FIG31 illustrates another example of a VMM system. FIG32 illustrates another example of a VMM system. FIG33 illustrates another example of a VMM system. FIG34 depicts an analog in-memory computing system. FIG35 depicts a digital memory computing system. FIG. 36 depicts a hybrid system comprising an analog in-memory computing system and a digital in-memory computing system. FIG37 depicts an analog in-memory computation engine. FIG38 depicts an exemplary digital in-memory computation engine. FIG39 depicts another exemplary digital in-memory computation engine. FIG40 depicts another exemplary digital in-memory computation engine. FIG41A depicts a multiplication logic cell. FIG41B depicts a 2-bit adder logic cell. FIG41C depicts a digital CIM cell. FIG42 depicts another exemplary digital in-memory computation engine. Figure 43 describes the shift and adder tree. Figure 44 depicts another shift and adder tree. FIG45 depicts an exemplary dynamic weight engine. FIG46 depicts another exemplary dynamic weight engine. FIG47 depicts an integrating analog-to-digital converter. FIG48 depicts an integrating analog-to-digital converter. 49A and 49B illustrate an output circuit including a current-to-voltage converter and an analog-to-digital converter. FIG50 describes a method performed by the hybrid system of FIG36. Implementation Method

[0090] Analog memory computing engine

[0091] 34 illustrates a block diagram of an analog compute-in-memory (CIM) engine 3400. Analog CIM engine 3400 includes a VMM array 3401 (which may also be referred to as a neural network array), a row decoder 3402, a high-voltage decoder 3403, a column decoder 3404, a bitline driver 3405 (such as bitline control circuitry for programming), input circuitry 3406, output circuitry 3407, control logic 3408, and a bias generator 3409. Analog CIM engine 3400 further includes a high-voltage generation block 3410, which includes a charge pump 3411, a charge pump regulator 3412, and a high-voltage level generator 3413. Analog CIM engine 3400 further includes a (programming / erasing, or weight tuning) algorithm controller 3414, analog circuitry 3415, a control engine 3416 (which may include, but is not limited to, functions such as arithmetic functions, activation functions, and embedded microcontroller logic), test control logic 3417, and a static random access memory (SRAM) block 3418 for storing data such as intermediate data for input circuits (e.g., activation data) or intermediate data for output circuits (neuron output data, partial and output neuron data), or data input for programming (e.g., data input for an entire column or multiple columns). VMM array 3401 includes an array of non-volatile memory cells arranged in rows and columns, wherein the non-volatile memory cells are of the type shown as memory cells 210, 310, 410, or 510 in FIG. 2 , FIG. 3 , FIG. 4 , or FIG. 5 , respectively, or of other types known to those skilled in the art. In one example, the non-volatile memory cell is a split-gate flash memory cell as shown in Figure 2, Figure 3, or Figure 4. In another example, the non-volatile memory cell is a stacked-gate flash memory cell as shown in Figure 5.

[0092] Input circuit 3406 may include circuits such as a DAC (digital-to-analog converter), a DPC (digital-to-pulse converter or digital-to-time-modulated-pulse converter), an AAC (analog-to-analog converter, such as a current-to-voltage converter or a logarithmic converter), a PAC (pulse-to-analog level converter), or any other type of converter. Input circuit 3406 may implement one or more of normalization, linear or nonlinear upscaling / downscaling functions, or arithmetic functions. Input circuit 3406 may implement a temperature compensation function for the input level. Input circuit 3406 may implement an excitation function such as a rectified linear excitation function (ReLU) or a sigmoid. Input circuit 3406 may store digital excitation data to be applied as an input signal or combined with an input signal during programming or read operations. The digital excitation data may be stored in a register. Input circuit 3406 may include circuitry for driving array terminals such as the CG, WL, EG, and SL lines, which may include sample-and-hold circuits and buffers. A DAC can be used to convert digital excitation data into an analog input voltage to be applied to the array.

[0093] Output circuit 3407 may include circuits such as an ITV (current-to-voltage circuit), an ADC (analog-to-digital converter, which converts the neuron analog output into digital bits), an AAC (analog-to-analog converter, such as a current-to-voltage converter or a logarithmic converter), an APC (analog-to-pulse converter or an analog-to-time-modulated pulse converter), or any other type of converter. Output circuit 3407 may convert the array output into excitation data. Output circuit 3407 may implement an excitation function, such as a ReLU or a sigmoid. Output circuit 3407 may also implement one or more of statistical normalization, regularization, upscaling / downscaling / gain functions, statistical rounding, or arithmetic functions (e.g., addition, subtraction, division, multiplication, shift, logarithm) for the neuron output. Output circuit 3407 can implement a temperature compensation function for neuron outputs or array outputs (such as bit line outputs) to keep the array's power consumption approximately constant over a temperature range or to improve the accuracy of the array (neuron) output, such as by keeping the IV slope approximately the same over a temperature range. Output circuit 3407 can include a register for storing output data. Computing in digital memory

[0094] 35 depicts a block diagram of a digital compute-in-memory (CIM) engine 3500. The digital CIM engine 3500 contains many, but not all, of the components contained in the analog CIM engine 3400.

[0095] Digital CIM engine 3500 includes array 3501, row decoder 3502, column decoder 3504, bitline driver 3505 (such as bitline control circuitry for programming), input circuitry 3506, output circuitry 3507, control logic 3508, and bias generator 3509. Digital CIM engine 3500 further includes algorithm controller 3514, analog circuitry 3515, control engine 3516, and test control logic 3517. Array 3501 includes an array of non-volatile memory cells arranged in rows and columns, where the non-volatile memory cells are of the type shown as memory cells 210, 310, 410, or 510 in FIG. 2 , FIG. 3 , FIG. 4 , or FIG. 5 , respectively, or other types known to those skilled in the art. In one example, the non-volatile memory cells are split-gate flash memory cells as shown in FIG. 2 , FIG. 3 , or FIG. 4 . In another example, the non-volatile memory cells are stacked-gate flash memory cells as shown in Figure 5. Array 3501 operates in the digital domain, where the values ​​stored in each non-volatile memory cell are binary values ​​rather than analog values ​​as in analog CIM engine 3400.

[0096] Input circuit 3506 may include circuitry such as a DAC, DPC, AAC, PAC, or any other type of converter. Input circuit 3506 may implement one or more of normalization, linear or nonlinear upscaling / downscaling functions, or arithmetic functions. Input circuit 3506 may implement a temperature compensation function for the input level. Input circuit 3506 may implement an excitation function such as a ReLU or a sigmoid. Input circuit 3506 may store digital excitation data to be applied as an input signal or combined with an input signal during programming or read operations. The digital excitation data may be stored in a register. Input circuit 3506 may include circuitry for driving array terminals such as the CG, WL, EG, and SL lines, which may include sample-and-hold circuitry and buffers. A DAC may be used to convert the digital excitation data into an analog input voltage to be applied to the array. Output circuit 3507 may include circuitry such as an ITV, ADC, AAC, APC, or any other type of converter. Output circuit 3507 may convert the array output into excitation data. Output circuit 3507 may implement an activation function such as a ReLU or a sigmoid. Output circuit 3507 may implement one or more of statistical normalization, regularization, upscaling / downscaling / gain functions, statistical rounding, or arithmetic functions (e.g., addition, subtraction, division, multiplication, shift, logarithm) for neuron outputs. Output circuit 3507 may implement a temperature compensation function for neuron outputs or array outputs (such as bit line outputs) to maintain approximately constant array power consumption over a temperature range or to improve the accuracy of array (neuron) outputs, such as by maintaining approximately the same IV slope over a temperature range. Output circuit 3507 may include a register for storing output data.

[0097] In an alternative example, the digital CIM engine 3500 does not utilize any analog circuitry and operates solely in the digital domain. For example, the input circuit 3506 and the output circuit 3507 can be formed from digital logic circuits operating in the digital domain, and the analog circuitry 3515 can be removed. A hybrid system that includes an analog in-memory computing engine, a digital in-memory computing engine, and a dynamic weighting engine

[0098] 36 illustrates a hybrid system 3600. The hybrid system includes analog CIM engines 3601, 3602, and 3604; digital CIM engines 3605 and 3607; digital computation engines 3606 and 3608; a dynamic weighting engine 3603; and a system bus 3609.

[0099] Analog CIM engines 3601, 3602, and 3604 store analog weights (e.g., N = 256) in non-volatile memory cells of respective VMM arrays (e.g., VMM array 3401) and perform VMM operations on those analog weights. The VMM systems depicted in Figures 9-13 and 22-33 are examples of analog CIM engines.

[0100] Digital compute-in-memory (CIM) engines 3605 and 3607 store digital weights (i.e., N=2) in non-volatile memory cells of respective VMM arrays (e.g., VMM array 3501) and perform VMM operations on those digital weights. Alternatively, digital CIM engines 3605 and 3607 may store the digital weights in a non-volatile macro or on-chip. Unlike analog CIM engines, digital CIM engines can utilize SRAM or DRAM cell arrays and non-volatile memory cells with floating gates, such as those shown in Figures 2 through 5.

[0101] Digital computing engines 3606 and 3608 are microprocessors, digital signal processors, or other digital computing engines, such as graphics processing units (GPUs), tensor processing units (TPUs), dedicated multiply-add-accumulate (MAC) units, dedicated single instruction multiple data (SIMD) processors, vector extensions, etc. Digital computing engines 3606 and 3608 can perform integer or floating-point calculations. They can use weights from the VMM array in the analog CIM engine or from a digital non-volatile memory macro or chip.

[0102] Dynamic weight engine 3603 is a component whose stored weights can be modified by changing bias voltage or bias current without performing a separate erase or program operation. Because the weights can be modified without performing a separate erase or program operation, they are considered dynamic weights. Weights can be transferred from the VMM array in an analog CIM engine or from a digital non-volatile memory macro or chip. An example of dynamic weight engine 3603 is shown in Figures 45 and 46, where the bias voltage is transferred to the gates of transistors in the unit CIM array cell. Dynamic weight control and bias circuitry are not shown.

[0103] The dynamic weight engine 3603, analog CIM engines 3601, 3602 and 3604, digital CIM engines 3605 and 3607, and digital computation engines 3606 and 3608 are coupled to a system bus 3609, which enables all coupled components to communicate with each other.

[0104] The microcontroller 3610, SRAM 3611, vector register 3612, and peripheral control and interface logic 3613 assist in certain functions and operations in the hybrid system 3600 (such as, but not limited to, controlling access to the SRAM 3611, controlling data flow to various computing engines, controlling communication between the internal system bus 3609 and external buses, executing activation functions, performing read, erase, and program operations on non-volatile memory (NVM) or volatile memory (VM), or performing weight transfer between computing engines).

[0105] FIG50 illustrates an exemplary method that may be performed by hybrid system 3600. First, the system performs a vector-matrix multiplication operation in a first layer of a neural network using digital in-memory compute engine 3605 or 3607 (5001). Next, the system performs a vector-matrix multiplication operation in a second layer of the neural network, different from the first layer, using analog in-memory compute engine 3601, 3602, or 3604 (5002). Optionally, the system transfers dynamic weights from dynamic weight engine 3603 to one or more of analog in-memory compute engines 3601, 3602, or 3604 and / or digital in-memory compute engines 3605 or 3607 (5003). Optionally, the system stores the dynamic weights in analog in-memory compute engine 3601, 3602, or 3604.

[0106] FIG37 illustrates a portion of an example of an analog CIM engine 3700. Here, two columns and four rows of nonvolatile memory cells are shown. The memory cells operate in the subthreshold region and can store analog weights. Stimulus inputs are provided to each selected column on a wordline or control gate line. The memory cells in that column multiply their stimulus inputs by the stored analog weights and output a current representing the product in the bitline coupled to the row of cells. The currents are then summed along the bitlines (output lines). The array output currents are then converted by ADC circuitry (not shown) to produce digital output bits. The stimulus input can be provided as a pulse width (PW), the output of a fixed-offset 1-bit digital-to-analog converter, or the output of an n-bit DAC. Other examples of a portion of an analog CIM engine are shown in FIG9 through FIG13 and FIG22 through FIG33.

[0107] Portions of the analog CIM engine 3700 can also be used as analog memory weight storage. In that case, a word of memory cells (e.g., 64 or 128 cells from a selected column) is selected, and the output digital output bits from the ADC represent the weight values ​​stored by the portion of the analog CIM engine 3700.

[0108] FIG38 depicts a portion of an exemplary digital CIM engine 3800. The portion of digital CIM engine 3800 includes an array of current-based CIM SRAM memory cells 3801, 3802, 3803, and 3804 arranged in rows and columns. CIM SRAM memory cell 3801 will be described in detail, with the understanding that the other CIM SRAM memory cells 3801 have the same design. In CIM SRAM memory cell 3801, if the weight stored in the SRAM is "1," NMOS transistor 3816 turns on, and current NC flows from transistor 3815 to transistor 3817 toward bit line BL0. If the weight stored in the SRAM is "0," NMOS transistor 3816 turns off, and current NC is prevented from flowing from transistor 3815 to transistor 3817 toward bit line BL0. Current NC represents the effective value of the weight stored in the CIM SRAM cell.

[0109] During operation of the digital CIM engine, input is applied to input line IN0. SRAM memory cell 3801 performs a multiplication on the input on IN0 and the stored weight. If IN0 is "1," NMOS transistor 3817 turns on, passing current NC (representing the stored weight value) to bit line BL0. If IN0 is "0," NMOS transistor 3817 turns off, and no current NC is passed to bit line BL0.

[0110] If the stored weight is "1" and if the input on the input line is "1," bit line BL0 will receive current NC because NMOS transistors 3815, 3816, and 3817 are all on. If the stored weight is "0" or if the input on the input line is "0," bit line BL0 will be disconnected from current NC and will not contain any current attributable to SRAM memory cell 3801.

[0111] CIM SRAM memory cells 3802 , 3803 , and 3804 and other CIM SRAM memory cells in the digital CIM engine operate in the same manner as CIM SRAM memory cell 3801 .

[0112] The bit lines of the digital CIM engine, such as BL0 and BL1, will sum the currents of all CIM SRAM memory cells connected to them. This means that BL0 will contain the sum of all currents of all CIM SRAM memory cells in row 0 (all NC currents with weight = "1"), BL1 will contain the sum of all currents of all CIM SRAM memory cells in row 1, and so on. This current can be converted into digital output bits by ADC circuitry or multi-bit sense amplifier circuitry.

[0113] 39 depicts a portion of another exemplary digital CIM engine 3900. The portion of the digital CIM engine 3900 includes an array of charge-based CIM SRAM memory cells 3901, 3902, 3903, and 3904 arranged in rows and columns.

[0114] CIM SRAM memory cell 3901 represents CIM SRAM memory cells 3902, 3903, and 3904, as well as other CIM SRAM memory cells in the digital CIM engine, and its operation will now be described. SRAM memory cell 3901 includes inverters 3911 and 3912, which form a latch; NMOS transistors 3913, 3914, 3916, 3917, and 3918; and capacitor 3915. A charge (representing a weight value) Q, equal to C (the capacitor's capacitance) * Vdds (the SRAM memory cell's supply voltage), is stored on capacitor 3915 in the SRAM cell. The latch formed by inverters 3911 and 3912 can store a value of "0" or "1" (the output of inverter 3912), where "0" represents zero charge and "1" represents a charge of Q, which is the effective weight of SRAM memory cell 3901. When the output of inverter 3912 is "0," meaning ground (0 V), capacitor 3915 is charged to ground through transistor 3916, resulting in Q = 0 (indicating a "0" weight). When the output of inverter 3912 is "1," for example, Vdds, capacitor 3915 is charged to Vdds through transistor 3916, resulting in Q = C*Vdds (indicating a "1" weight). When transistor 3916 is off (ENSB = "0"), transistor 3917 is on (input IN0 = "1"), and transistor 3918 is on (ENS = "1"), the stored charge is transferred to bit line BL0. For a weight of "1," if input IN0 = "1" (meaning transistor 3917 is on), a charge Q (= C*Vdss) is transferred to bit line BL0.

[0115] The weight is stored in SRAM memory cell 3901 through a write operation. During a write operation, word line WL0 is asserted, which turns on NMOS transistors 3913 and 3914. A "1" is stored in the latch by driving BLS0B high and BLS0 low, and a "0" is stored in the latch by driving BLS0 high and BLS0B low.

[0116] During an inference or read operation of the digital CIM engine, input is applied to input line IN0, and ENSB is asserted. SRAM memory cell 3901 performs a multiplication on the input on IN0 and the stored weight. If IN0 is "1," NMOS transistor 3917 turns on, and if WLN0 is "0," NMOS transistor 3917 turns off.

[0117] If the stored weight is "1" and if the input on input line INx is "1," bit line BL0 will receive the charge of capacitor 3915 due to NMOS transistor 3916 being off and transistors 3917 and 3918 being on. If the stored weight is "0" or if the input on input line INx0 is "0," bit line BL0 will receive zero charge and will not contain any current component attributable to SRAM memory cell 3801.

[0118] SRAM memory cells 3902 , 3903 , and 3904 , as well as other SRAM memory cells in the digital CIM engine, operate in the same manner as SRAM memory cell 3901 .

[0119] The bit lines of the digital CIM engine, such as BL0 and BL1, will sum the charge of all the SRAM memory cells connected to them. This means that BL0 will contain the sum of all the charges output by all the SRAM memory cells in row 0, BL1 will contain the sum of all the charges output by all the SRAM memory cells in row 1, etc. This summed charge can be converted into digital output bits by an ADC circuit, such as the ADC circuits shown in Figures 44 and 46 below.

[0120] FIG40 illustrates a portion of another exemplary digital CIM engine 4000. This portion of digital CIM engine 4000 includes a subarray of CIM digital cells, each of which stores a 1-bit weight w. In this example, subarrays 4010-11 and 4010-12 are shown in detail. Horizontally, digital CIM engine 4000 includes m subarrays (such as subarrays 4010-11, 4010-12, ..., 4010-1m in row 1). Vertically, digital CIM engine 4000 includes n subarrays (such as subarrays 4010-11, ..., 4010-n1 in row 1).

[0121] Subarray 4010-11 will now be described as an example. The discussion regarding subarray 4010-11 also applies to other subarrays. Subarray 4010-11 includes an array of CIM digital cells, such as CIM digital cells 4001, arranged in rows and columns. In this example, subarray 4010-11 includes four columns and four rows of CIM digital cells, but it should be understood that subarray 4010-11 and other subarrays can each alternatively include any number of columns and rows, such as eight columns and eight rows, to accommodate an 8-bit input and an 8-bit weight (meaning that each of the eight CIM digital cells stores one bit, for a total of eight bits of weight), resulting in a 16-bit output (because an 8-bit input multiplied by an 8-bit weight yields a 16-bit output). In this example, subarray 4010-11 implements a 4-bit input multiplied by a 4-bit weight, resulting in an output DOUT[7:0] = IN[3:0] * W[3:0], where each row of four CIM digital cells stores the 4-bit weight W[3:0], and all four rows store the same weight W[3:0]. A MUX block is used to route the output from each subarray 4010x onto the main output bus DOUTx.

[0122] The digital CIM engine 4000 receives inputs for multiple rows. In this example, subarray 4010-11 receives inputs IN[0], IN[1], IN[2], and IN[3] on its four rows. The 4-bit inputs are multiplied by the 4-bit weights W[3:0] stored in each row. Each digital cell, such as digital cell 4001, receives inputs on its port I1, multiplies the inputs by the 1-bit weight value (W) stored in the digital cell, and adds the resulting product to any data received on its port I2 from the digital cell in the row above it, along with any carryover bits received on its port C1 from the adjacent digital cell to its right in the same row. In the example shown, the leftmost row (Col3) represents the most significant bit of the weight, the row to the right (Col2) represents the next most significant bit of the weight, the row to the right (Col1) represents the next most significant bit of the weight, and the last row (Col0) represents the least significant bit of the weight.

[0123] Each CIM digital cell, such as CIM digital cell 4001, includes a memory component, such as an SRAM cell, a latch, or a register for storing weights W. Each CIM digital cell, such as CIM digital cell 4001, also includes digital multiplication and addition logic. Inputs are received digitally by the digital components in each row of port I1 of each CIM digital cell. The inputs received on port I1 are multiplied by the weights stored in the SRAM of the CIM digital cell, and the results are then summed and output row by row. The output (O) of a particular CIM digital cell is provided as input (I2) to the CIM digital cell below it in the same row, and the carry-out (CO) from the particular CIM digital cell is provided as the carry-in (CI) to the CIM digital cell to its left in the same column because that cell represents a higher-order bit than the particular CIM digital cell.

[0124] The multiplexer block 4020-11 is used to respond to the enable signal EN0 to enable the output from the sub-array 4010-11 to enter the main output bus DOUTx. Other sub-arrays have similar multiplexer blocks.

[0125] 41A , 41B , and 41C depict exemplary components that may be used in various CIM digital cells of FIG. 40 , such as CIM digital cell 4001 .

[0126] 41A , a multiplication logic cell 4101 receives two inputs I1 and I2 and outputs their product O according to the truth table shown in Table 9: Table 9: Truth table for multiplication logic cell 4101 I1 I2 O 0 0 0 0 1 0 1 0 0 1 1 1

[0127] 41B , a 2-bit adder logic cell 4102 receives inputs I1 and I2 and a carry input CI, and generates an output O and a carry output CO according to the truth table shown in Table 10: Table 10: Truth table for 2-bit adder logic cell 4102 I1 I2 C1 O CO 0 0 0 0 0 0 1 0 1 0 1 0 0 1 0 1 1 0 0 1 0 0 1 1 0 0 1 1 0 1 1 0 1 0 1 1 1 1 1 1

[0128] 41C , the digital CIM cell 4103 stores the weight W in the SRAM cell 4104. It multiplies W by the received input I2 using the multiplication logic cell 4101 and adds the resulting product P to the received inputs I1 and CI using the 2-bit adder logic cell 4102 to generate outputs O and CO. In this way, each digital CIM cell 4103 performs a bit-by-bit multiplication operation (W*I2) and adds the result to the received inputs (I1 and CI) to perform a vector-matrix multiplication operation using the stored digital value W.

[0129] FIG42 illustrates a portion of another exemplary digital CIM engine 4200. This portion of digital CIM engine 4200 includes an array of blocks, each of which includes a multiplier array and a shift and adder tree. In this example, blocks 4210-11, 4210-12, 4210-21, and 4210-22 are described in detail. Horizontally, digital CIM engine 4200 includes m blocks (such as blocks 4210-11, 4210-12, ..., 4210-1m in row 1). Vertically, digital CIM engine 4200 includes n blocks (such as blocks 4210-11, 4210-21, ..., 4210-n1 in row 1).

[0130] Block 4210-11 will now be described as an example. The discussion regarding block 4210-11 also applies to the other blocks. Block 4210-11 includes an array of multipliers arranged in columns and rows, such as multiplier 4201. In this example, block 4210-11 includes four columns and four rows of multipliers, but it should be understood that block 4210-11 and all other blocks can substitute any number of columns and rows, such as eight columns and eight rows to accommodate an 8-bit input and an 8-bit weight, resulting in a 16-bit output, based on the same principles discussed herein.

[0131] The digital CIM engine 4200 receives inputs for complex columns. In this example, block 4210-11 receives inputs IN[0], IN[1], IN[2], and IN[3] on its four columns. Each multiplier, such as multiplier 4201, receives the input and multiplies the input by a 1-bit weight (W) stored in the multiplier, and outputs the product. The product is provided to the shift and adder tree 4202, which adds the products and performs shift operations as needed to reflect the fact that each column (IN[0], IN[1], IN[2], and IN[3]) represents a different digit in the binary input (IN[3:0]). That is, IN[0] represents 20, IN[1] represents 21, IN[2] represents 22, and IN[3] represents 23. The multiplexer block 4203 is used to respond to the enable signal EN0 to enable the output from each block, such as block 4210-11, to enter the main output bus DOUTx. Other blocks have similar multiplexer blocks.

[0132] FIG43 depicts a shift and adder tree 4300, which is an example of an implementation of the shift and adder tree 4202 used in FIG42. Each multiplier, such as multiplier 4201 in block 4210-11, receives a 1-bit input, multiplies the input by a 1-bit weight, and outputs a 1-bit value provided to the shift and adder tree 4202. For each column, the shift and adder tree 4300 receives four 1-bit values, which are treated as a single 4-bit value arranged in an order corresponding to rows D3, D2, D1, and D0, respectively. The 4-bit values ​​are X4-0 for column 0 (which receives input IN[0]), X4-1 for column 1 (which receives input IN[1]), X4-2 for column 2 (which receives input IN[2]), and X4-3 for column 3 (which receives input IN[3]).

[0133] Shift and adder 4301 first adds X4-0 to X4-1 (where the digits of X4-1 are shifted left by one digit due to the fact that column 1 represents the value of the input digit to the left of column 0; for example, if X4-1 is 1010, the shifted version will be 10100), resulting in a 6-bit value. Shift and adder 4302 adds that 6-bit value to a shifted version of X4-2 (where X4-2 is shifted by two digits), resulting in a 7-bit value. Shift and adder 4303 then adds that 7-bit value to X4-3 (where X4-3 is shifted by three digits), resulting in an 8-bit value representing the sum of the products received from the block's multipliers.

[0134] FIG44 depicts a shift and adder tree 4400, which is an example of an implementation of the shift and adder tree 4202 used in FIG42. Each multiplier, such as multiplier 4201 in block 4210-11, receives a 1-bit input, multiplies the input by a 1-bit weight, and outputs a 1-bit value provided to the shift and adder tree 4202. For each column, the shift and adder tree 4400 receives four 1-bit values, which are treated as a single 4-bit value arranged in an order corresponding to rows D3, D2, D1, and D0, respectively. The 4-bit values ​​are X4-0 for column 0 (which receives input IN[0]), X4-1 for column 1 (which receives input IN[1]), X4-2 for column 2 (which receives input IN[2]), and X4-3 for column 3 (which receives input IN[3]). Shift and adder 4401 first adds X4-0 and X4-1 (where the digits of X4-1 are shifted left by one digit due to the fact that column 1 represents the value of the input digit to the left of column 0; for example, if X4-1 is 1010, the shifted version will be 10100), resulting in a 6-bit value X6. Shift and adder 4302 then adds a shifted version of X4-2 (where X4-2 is shifted by two digits) and a shifted version of X4-3 (where X4-3 is shifted by three digits), resulting in a 7-bit value X7. Shift and adder 4403 then adds X6 and X7 together (without shifting), resulting in an 8-bit value X8, which represents the sum of the products received from the block's multipliers.

[0135] Another use of digital CIM arrays is for floating point numbers, where a first array is used for the exponent part and a second array is used for the mantissa part (fraction part), where appropriate realignment and recombination of the array outputs is performed to reconstruct the final floating point number.

[0136] Figure 45 depicts a portion of a dynamic weight engine 4500. The dynamic weight engine 4500 includes CIM dynamic weight cells 4501, 4502, 4503, and 4504 arranged in rows and columns.

[0137] CIM dynamic weight cell 4501 represents CIM dynamic weight cells 4502, 4503, and 4504, as well as other CIM memory cells in the dynamic weight engine. Its operation will now be described. CIM dynamic weight cell 4501 includes NMOS transistors 4511, 4512, 4514, and 4515, and capacitor 4513. CIM dynamic weight cell 4501 can store weights that can be dynamically changed by modifying its inputs. When WL0 and WBL0 are asserted, CIM dynamic weight cell 4501 is selected. When WL0 is asserted, NMOS transistors 4511 and 4514 are turned on. When WBL0 is asserted, NMOS transistor 4512 is turned on. A voltage W representing the weight applied by CIM dynamic weight cell 4501 is applied to WWL0 and passed to the gate of NMOS transistor 4515 and stored by capacitor 4513 coupled between the gate of NMOS transistor 4515 and source line SL0 for column 0. NMOS transistor 4515 draws current from bit line BL0, where the drawn current varies with voltage W.

[0138] CIM dynamic weight cells 4502 , 4503 , 4504 and other CIM dynamic weight cells in the dynamic weight engine operate in the same manner as CIM dynamic weight cell 4501 .

[0139] The bit lines of a CIM dynamic weight engine, such as BL0 and BL1, will sum the currents drawn by all CIM dynamic weight cells connected to them. This means that BL0 will contain the sum of all currents drawn by all CIM dynamic weight cells in row 0, BL1 will contain the sum of all currents drawn by all CIM dynamic weight cells in row 1, and so on. This current can be converted into digital output bits by an ADC circuit, such as the ADC circuits shown in Figures 47 and 49 below.

[0140] Figure 46 depicts a portion of a dynamic weight engine 4600. The dynamic weight engine 4600 includes CIM dynamic weight cells 4601, 4602, 4603, and 4604 arranged in columns and rows.

[0141] CIM dynamic weight cell 4601 represents CIM dynamic weight cells 4602, 4603, and 4604, as well as other CIM dynamic weight cells in the dynamic weight engine, and its operation will now be described. CIM dynamic weight cell 4601 includes NMOS transistors 4611, 4612, 4613, 4615, and 4616, and capacitor 4614. CIM dynamic weight cell 4601 can store weights that can be dynamically changed by modifying its inputs. When WL0 and WBL are asserted, CIM dynamic weight cell 4601 is selected. When WL0 is asserted, NMOS transistors 4611 and 4615 are turned on. When WBL is asserted, NMOS transistor 4613 is turned on. Voltage W, representing the weight applied by CIM dynamic weight cell 4601, is applied to WWL0 and passed to the gate of NMOS transistor 4616 and stored by capacitor 4614 coupled between the gate of NMOS transistor 4616 and source line SL0 for column 0. NMOS transistor 4616 draws current from bit line BL0, where the drawn current varies with voltage W.

[0142] CIM dynamic weight cells 4602 , 4603 , 4604 and other CIM dynamic weight cells in the dynamic weight engine operate in the same manner as CIM dynamic weight cell 4601 .

[0143] The bit lines of a CIM dynamic weight engine, such as BL0 and BL1, will sum the currents drawn by all CIM dynamic weight cells connected to them. This means that BL0 will contain the sum of all currents drawn by all CIM dynamic weight cells in row 0, BL1 will contain the sum of all currents drawn by all CIM dynamic weight cells in row 1, and so on. This current can be converted into digital output bits by an ADC circuit, such as the ADC circuits shown in Figures 47 and 49 below.

[0144] FIG47 illustrates an example of an integrating ADC 4700, which is used to convert the charge stored in cells that are part of the digital CIM engine 3900 in FIG39 into digital pulses or digital output bits. Integrating ADC 4700 converts the analog output charge (charge CINT stored in capacitor 4702) in the neuron output block into a digital pulse whose width varies proportionally to the magnitude of the analog output current in the neuron output block. Capacitor 4702 represents the sum of all capacitors from the selected CIM volatile SRAM memory cell in FIG39 (connected in parallel via a pass transistor). ADC 4700 includes a current reference 4707, an operational amplifier 4701, a comparator 4704, an AND gate 4740, and a counter 4720. The charges of the individual capacitors in the CIM volatile SRAM memory cell are summed and then reconfigured as integrating capacitor 4702 on operational amplifier 4701. Reference current 4707 discharges all stored charge in integrating capacitor 4702, and counter 4720 generates pulses until the discharge is complete. The pulses represent digital output bits, as shown in FIG48.

[0145] 48 depicts an integrating ADC 4800 for converting current received from one or more rows of the analog CIM engine in FIG. 37 , the digital CIM engine 3800 in FIG. 38 , the dynamic weight engine 4500 in FIG. 45 , and the dynamic weight engine 4600 in FIG. 46 into digital pulses or digital output bits.

[0146] ADC 4800 converts current I NEU into digital pulses EC whose width varies in proportion to the magnitude of the current. An integrator comprising an integrating operational amplifier 4801 and an integrating capacitor 4802 integrates I NEU 4806 with a reference current I REF provided by a current source 4807.

[0147] Optionally, current source 4807 may comprise a bandgap filter having a temperature coefficient of zero or a temperature coefficient that tracks the neuron current INEU. The temperature coefficient that tracks INEU may be obtained from a lookup table (not shown) containing values ​​determined during the test phase.

[0148] During the initialization phase, switch 4808 is closed. Vout 4803 and the input to the negative terminal of operational amplifier 4801 will then become equal to VREF. Thereafter, switch 4808 is opened, and switch S2 is closed, while switch S1 remains open, and constant reference current IREF 4807 is integrated up during a fixed period tref. During the fixed period tref, Vout rises, and its slope reflects the value of constant reference current IREF 4807. Thereafter, switch S2 is opened, and switch S1 is closed, and during a period tmeas, neuron current INEU 4806 is integrated down for a period tmeas (during which Vout falls), where tmeas is the time it takes to integrate Vout down to VREF, as indicated by the change in the output of comparator 4804.

[0149] Output EC 4805 will be high when VOUT > VREFV and low otherwise. EC 4805 thus generates a pulse whose width reflects the period tmeas, which is proportional to current I NEU 4806. Thus, output neuron current I NEU 4806 is converted into digital pulse EC 4805, where the width of digital pulse EC 4805 varies in proportion to the magnitude of output neuron current I NEU 4806.

[0150] Current I NEU4806 = tmeas / tref * IREF. For example, for a desired output bit resolution of 10 bits, tref is equal to a period of 1024 clock cycles. The period tmeas depends on the value of I NEU4806 and the change in the value of Iref from 0 to 1024 clock cycles. The neuron current I NEU4806 affects the rate and slope of charging.

[0151] Optionally, output pulse EC 4805 can be converted into a series of pulses of uniform duration for transmission to a subsequent block of the circuitry, such as the input block of a CIM engine. At the beginning of period tmeas, output EC 4805 is input into AND gate 4840 along with reference clock 4841. The output will be pulse train 4842 during the period when VOUT > VREF (where the frequency of the pulses in pulse train 4842 is the same as the frequency of clock 4841). The number of pulses is proportional to period tmeas, which is proportional to current INEU 4806.

[0152] Optionally, pulse train 4843 can be input to counter 4820, which counts the number of pulses in pulse train 4842 and generates count value 4821, which is a digital count of the number of pulses in pulse train 4842, which is proportional to neuronal current I NEU 4806. Count value 4821 comprises a set of digital bits. In another example, integrating ADC 4800 can convert neuronal current I NEU 4806 into pulses, where the pulse width is inversely proportional to the magnitude of neuronal current I NEU 4806. This inversion can be performed digitally or analogously and converted into a series of pulses or digital bits for output to subsequent circuitry.

[0153] FIG49A illustrates an exemplary output circuit 4900 that can convert array current into digital output bits. Output circuit 4900 includes a current-to-voltage converter (ITV) 4901 and an analog-to-digital converter (ADC) 4902. ITV 4901 converts the array current into a voltage that is then digitized by ADC 4902. ADC 4902 can be a successive approximation register (SAR) ADC.

[0154] 49B depicts an exemplary output circuit 4950 including an ITV 4951 that receives a differential input and produces a differential output, and an ADC 4952 that receives a differential input and produces a single-ended digital output from the differential outputs of the ITV 4951. Examples of implementations of current-to-voltage converters and analog-to-digital converters are described in U.S. patent application Ser. No. 17 / 521,772, which is incorporated herein by reference.

[0155] Other examples for analog and digital CIM engines may utilize, but are not limited to, NAND memory, resistive RAM (ReRAM), magnetic RAM (MRAM), or dynamic RAM (DRAM).

[0156] It should be noted that as used herein, the terms "over" and "on" both include "directly over" (without intervening materials, elements, or spaces disposed therebetween) and "indirectly on" (with intervening materials, elements, or spaces disposed therebetween). Similarly, the term "adjacent" includes "directly adjacent" (without intervening materials, elements, or spaces disposed therebetween) and "indirectly adjacent" (with intervening materials, elements, or spaces disposed therebetween), "mounted to" includes "directly mounted to" (without intervening materials, elements, or spaces disposed therebetween) and "indirectly mounted to" (with intervening materials, elements, or spaces disposed therebetween), and "electrically coupled to" includes "directly electrically coupled to" (without intervening materials or elements electrically connecting the elements together) and "indirectly electrically coupled to" (with intervening materials or elements electrically connecting the elements together). For example, forming a component "over a substrate" can include forming the component directly on the substrate without intervening materials / elements, as well as forming the component indirectly on the substrate with one or more intervening materials / elements disposed therebetween.

[0157] 12:Semiconductor substrate 14: Source region 16: Drain area 18: Channel Area 20: Floating Gate 22: Word line terminal / select gate 24: Bit line 28: Control gate 30: Erase Gate 31: Digital to Analog Converter 32,1601,1701,2001,3401: vector-matrix multiplication array 32a: Vector-matrix multiplication array / input vector-matrix multiplication array / input layer 32b, 32c: vector-matrix multiplication array / hidden layer 32d, 32e: vector-matrix multiplication array / fully connected layer 33: Non-volatile memory cell array 34: Erase gate and word line gate decoder 35: Control gate decoder 36: Bit Line Decoder 37: Source Line Decoder 38: Differential Summer / Summing Operational Amplifier 39,1602,1702,2002: incentive function block 210,710: memory cells 310: Four-gate memory cell 410: Tri-Gate Memory Cell 510: Stacked Gate Memory Cell / Flash Memory Cell 900,1000,1100,1200,1300,2200,2300,2400,2500,2600,2700, 2800, 2900, 3000: Neuron vector matrix multiplication array / vector matrix multiplication array 901, 1003, 1103, 1203, 1303: Memory array of non-volatile memory cells 902: Reference array of non-volatile reference memory cells 903: Control gate line 904: Erase gate line 1001, 1101, 1201, 1301: Reference array of the first non-volatile reference memory cell 1002, 1102, 1202, 1302: Reference array of the second non-volatile reference memory cell 1014,1205,1212: Diode-connected through-multiplexer 1204: Cascade transistor 1314: Diode-connected reference cell through multiplexer 1400: Long and short-term memory 1401, 1402, 1403, 1404, 1801, 1802, 1803, 1804: Cell 1500,1600: Long and short-term memory cells 1501, 1502, 1503, 1901, 1902: S-type function components 1504,1505,1903: Hyperbolic tangent component 1506, 1507, 1508, 1703, 1904, 1905, 1906, 2103: Multiplier components 1509, 1708, 1907, 2105: Additive components 1704, 1705, 1706, 1707, 2106, 2107, 2108: registers 1709, 1710, 2104: Multiplexer 1800:Gate-controlled recursive unit 1900, 2000: gated recursive unit cell 1908,2109: complementary components 2701-1, 2701-2…2701-(N-1), 2701-N: Bit line control gate 3100,3210,3300: Vector-matrix multiplication system 3101, 3102, 3213, 3303, 3304, 3305, 3306, 3307, 3308: Summation circuit 3211: First Array 3212: Second Array 3301,3302,3501: array 3400,3601,3602,3604,3700: Computing engine in analog memory 3402,3502: Column decoder 3403: High Voltage Decoder 3404,3504: row decoder 3405,3505:Bit line driver 3406,3506: Input circuit 3407,3507,4900,4950: Output circuit 3408,3508: Control logic 3409,3509: Bias generator 3410: High voltage generation block 3411: Charge Pump 3412: Charge Pump Regulator 3413: High Voltage Level Generator 3414,3514: Algorithm controller 3415,3515:Analog circuit system 3416,3516: Control Engine 3417,3517: Test control logic 3418: Static Random Access Memory Block 3606,3608: Digital computing engine 3911,3912: Reverser 3915,4513,4614,4702:Capacitors 3500, 3605, 3607, 3800, 3900, 4000, 4200: Computing engine in digital memory 3801,3802,3803,3804,3901,3902,3903,3904: Compute static random access memory cells in memory 3815,3816,3817,3913,3914,3916,3917,3918,4511,4512,4514,4515,4611,4612,4613,4615,4616: NMOS transistors 3600: Hybrid System 3603,4500,4600: Dynamic Weight Engine 3609: System bus 3610:Microcontroller 3612: Static Random Access Memory 3613: Peripheral Control and Interface Logic 4001: Computing digital cells in memory 4010-11, 4010-12…4010-1m, 4010-n1, 4010x: Subarray 4020-11: Multiplexer Block 4101:Multiplication logic cell 4102: 2-bit adder logic cell 4103: Computing cells in digital memory 4104: Static Random Access Memory Cell 4201:Multiplier 4202,4300,4400: shift and adder tree 4203:Multi-tasking block 4210-11, 4210-12, 4210-21, 4210-22…4210-n1: Block 4301, 4401, 4403: Shift and adder 4501,4502,4503,4504,4601,4602,4603,4604: Calculate dynamic weight cells in memory 4700,4800: Integral Analog-to-Digital Converter 4701: Operational Amplifier 4704,4804: Comparator 4707: Reference current 4720,4820:Counter 4740:AND gate 4801: Integrating Operational Amplifier 4802: Integrating capacitor 4805: Output EC 4806:Neural Current 4807: Current Source / Constant Reference Current 4808,S1,S2: switch 4821:Count value 4841: Reference clock 4842,4843: Pulse train 4901,4951: Current to Voltage Converter 4902,4952:Analog-to-digital converter

Claims

1. A hybrid system comprising: an analog-memory computation engine for performing operations on analog weights in a first layer of a neural network; a digital-memory computation engine for performing operations on digital weights in a second layer of the neural network; and a dynamic weight engine for performing operations on dynamic weights in a third layer of the neural network, the third layer including an array having a plurality of non-volatile memory cells, each of the non-volatile memory cells including a floating gate, the dynamic weights being provided by a bias voltage received from outside the dynamic weight engine, at least one of the operations including modifying the dynamic weights by changing the bias voltage and applying the bias voltage to the non-volatile memory cells without modifying the charge stored in the floating gate of the non-volatile memory cells; wherein, The first layer, the second layer, and the third layer are different layers in the neural network.

2. The system of request item 1 includes: a system bus coupled to the computation engine in the analog memory, the computation engine in the digital memory, and the dynamic weight engine.

3. As in request item 1, where, The computing engine in this analog memory contains multiple non-volatile memory cells configured in columns and rows.

4. As in request item 3, where, These non-volatile memory cells are stacked gate flash memory cells.

5. The system as described in request item 3, wherein, These non-volatile memory cells are discrete gate flash memory cells.

6. As in request item 1, where, The computing engine in this digital memory contains multiple static random access memory (SRAM) cells configured in columns and rows.

7. As in request item 1, where, The computing engine in this digital memory contains a plurality of CIM digital cells.

8. The system as described in request item 7, wherein, Each of the multiple CIM digital cells contains a multiplication logic cell and a 2-bit adder logic cell.

9. The system as described in request item 7, wherein, The computing engine in the digital memory contains a shifter and adder tree coupled to one of the plurality of CIM digital cells.

10. A method of operating a hybrid system, comprising: performing vector-matrix multiplication on digital weights in a first layer of a neural network using a computing engine in digital memory; performing vector-matrix multiplication on analog weights in a second layer of the neural network using a computing engine in analog memory; and performing vector-matrix multiplication on dynamic weights in a third layer of the neural network using a dynamic weight engine, the third layer including an array having a plurality of non-volatile memory cells, each of the non-volatile memory cells including a floating gate, the dynamic weights being provided by a bias voltage received from outside the dynamic weight engine, at least one of the operations comprising modifying the dynamic weights by changing the bias voltage and applying the bias voltage to the non-volatile memory cells without modifying the charge stored in the floating gate of the non-volatile memory cells; wherein, The first layer, the second layer, and the third layer are different layers in the neural network.

11. As in request item 10, wherein, The computing engine in this analog memory contains multiple non-volatile memory cells configured in columns and rows.

12. A method of operating a hybrid system, comprising: providing numerical values ​​derived from analog weights as dynamic weights in a dynamic weighting engine, the analog weights being received from a computation engine in an analog memory, the dynamic weights being provided by a bias voltage received from outside the dynamic weighting engine, and modifying the dynamic weights by changing the bias voltage and applying the bias voltage to a plurality of non-volatile memory cells, each of the non-volatile memory cells including a floating gate, but without modifying the charge stored in the floating gate of the non-volatile memory cell.

Images