In-memory computing devices and methods for performing mac operations

Abstract

The present invention relates to apparatuses and methods for performing MAC operations using a TCAM array as a compute-in-memory (CIM) device, which can achieve higher computation throughput, higher performance, and lower energy consumption compared to computation using a processor located outside of a memory array. In some embodiments, weights in a weight matrix can be programmed in multiple SRAMs of a TCAM bitcell array. Each SRAM can operate as a multiplier that performs multiplication between the stored weight and an input activation value applied on a search line in the TCAM bitcell array. Two SRAMs within a TCAM bitcell can independently operate to independently receive two input activation values on their respective select lines and perform multiplication operations with the weights stored in each respective SRAM.

Description

TECHNICAL FIELD

[0001] The present application relates generally to compute-in-memory (CIM) devices, and more particularly to CIM devices that can perform multiply and accumulate (MAC) operations. BACKGROUND

[0002] Deep learning, machine learning, neural networks, and other matrix-based differentiable programs are used to solve a variety of problems, including natural language processing and object recognition in images. Solving these problems often involves performing computations based on matrix vector multiplication. For example, in a neural network having a plurality of neurons, the activation values input to a neuron can be viewed as an input activation vector, and the output activation values from a neuron can be viewed as an output activation vector. The computation of the neural network often involves matrix vector multiplication of the input activation vector with a weight matrix to compute the output activation vector. Typically, the weight matrix can be rectangular, and the length of the output activation vector is not necessarily the same as the length of the input activation vector. The computation of the neural network often involves computation of multiply and accumulate (MAC) operations performed on data values, including input / output activation vectors and weights. A MAC operation refers to a multiplication operation between two values, and, subsequently, accumulation of a series of multiplication results to provide an output MAC value.

[0003] The computation of large deep neural networks involves large-scale parallel computation of many different data values. Computation operations (e.g., arithmetic or logical operations) are typically performed by a processor based on data transfers between the processor and a storage device (e.g., a memory array). As the data throughput requirements of large-scale parallel neural networks increase, the slow data transfers of fast processors and memories can sometimes become a bottleneck for machine learning applications.

[0004] Content-addressable memory (CAM) is a type of computer memory designed for search-intensive applications. Due to its parallel nature, CAMs are much faster at searching than random access memory (RAM) architectures. CAMs are commonly used in inter-network routers and switches, which can improve the speed of route lookups, packet classification, and datagram forwarding. Ternary CAMs or TCAMs are designed to store and query data using three different inputs (0, 1, and X). The “X” input (often referred to as “don’t care” or “wildcard” state) enables TCAMs to perform a broader search based on pattern matching, as opposed to ternary CAMs, which only use 0s and 1s to perform an exact match search. SUMMARY

[0005] The present disclosure proposes an in-memory computing device and a method for performing MAC operations to achieve higher computational throughput and higher performance.

[0006] In a first aspect, the present disclosure provides an in-memory computing device comprising: a memory array comprising a plurality of ternary content addressable memory (TCAM) bitcells arranged into a plurality of rows, wherein each TCAM bitcell comprises two static random access memories (SRAMs), each SRAM configured to apply a weight stored therein to an input activation to produce an output value; and a plurality of activation lines, each activation line interconnected to the TCAM bitcells of a respective row of the plurality of rows and configured to receive the output values from the two SRAMs in each TCAM bitcell located within the respective row.

[0007] In some embodiments, the in-memory computing device further comprises: a summer having a plurality of summer inputs, each summer input coupled to a respective activation line of the plurality of activation lines, the summer configured to produce a multiply-accumulate (MAC) value based on the output values of the two SRAMs.

[0008] In some embodiments, each TCAM bitcell of the plurality of TCAM bitcells comprises: a first SRAM configured to receive an input activation from a first search line and a second SRAM configured to receive an input activation from a second search line.

[0009] In some embodiments, the in-memory computing device further comprises a plurality of inverters, wherein each activation line is coupled to a respective summer input via a respective inverter of the plurality of inverters.

[0010] In some embodiments, the TCAM bitcells within the memory array are 16-transistor (16-T) TCAM bitcells.

[0011] In some embodiments, the memory array is configured to receive an input activation vector and a weight matrix.

[0012] In some embodiments, the input activations of the input activation vector are provided to the TCAM bitcells of two or more different rows.

[0013] In some embodiments, the input activation vector is received in a time-division multiplexed manner.

[0014] In some embodiments, within each SRAM, the output value is produced based on a multiplication of the weight and the input activation.

[0015] In some embodiments, the memory array is located in a first layer, the first search line is located in a second layer parallel to the first layer, and the plurality of activation lines include conductors located in a third layer parallel to and offset from the second layer.

[0016] In some embodiments, the plurality of TCAM bit cells are arranged in a plurality of columns, and the in-memory computing device further includes: routing circuitry located next to one of the plurality of columns and configured to route input activation to some or all of the plurality of TCAM bit cells.

[0017] In some embodiments, the in-memory computing device further includes an SRAM edge cell located next to a row of the plurality of TCAM bit cells.

[0018] In some embodiments, the memory array includes a first subarray and a second subarray located below the first subarray along the column direction, with the SRAM edge cell located between the first subarray and the second subarray.

[0019] In some embodiments, the in-memory computing device further includes: routing circuitry located next to one of the plurality of rows and configured to route input activation to some or all of the plurality of TCAM bit cells.

[0020] In some embodiments, the memory array includes a first subarray and a second subarray located below the first subarray along the column direction, and there are no SRAM edge cells between the first subarray and the second subarray.

[0021] In a second aspect, the present invention provides a method for performing a multiplication and accumulation MAC operation, characterized in that the method performs the MAC operation on an input activation vector and a weight matrix by operating a tri-state content-addressable memory (TCAM) cell array, wherein a plurality of TCAM bit cells in the TCAM cell array are arranged in a plurality of rows and each TCAM bit cell includes two static random access memories (SRAMs), and the method includes: storing a plurality of weights in a plurality of SRAMs within the TCAM cell array; multiplying the input activation vector with the plurality of weights using the plurality of SRAMs to generate a plurality of output values; collecting the output values ​​from the two SRAMs in each TCAM bit cell of the respective row using activation lines; and summing the output values ​​from the activation lines to generate a MAC value.

[0022] In some embodiments, summing the output values ​​from the activation line includes coupling the activation line to an adder tree.

[0023] In some embodiments, the multiplication operation is performed in a time-division multiplexing manner.

[0024] In some embodiments, the act of multiplying includes providing a first input activation of the input activation vector from a first search line of the array of TCAM cells to a first SRAM in the TCAM bitcell and providing a second input activation of the input activation vector from a second search line of the array of TCAM cells to a second SRAM in the TCAM bitcell.

[0025] In some embodiments, the act of multiplying further includes providing the first input activation to the SRAMs in the TCAM bitcells of two or more different rows of the plurality of rows.

[0026] These and other objects of the present application will no doubt become obvious to those of ordinary skill in the art after reading the following detailed description of the preferred embodiment, which is presented in connection with the following drawings. BRIEF DESCRIPTION OF DRAWINGS

[0027] The present application can be more fully understood from the following detailed description taken in connection with the accompanying drawings, in which:

[0028] Figure 1 is a schematic diagram illustrating an exemplary neural network representation of a MAC operation that is performed using the CIM apparatus disclosed herein.

[0029] Figure 2 is a schematic diagram of an exemplary 16-transistor (16-T) TCAM bitcell used in a CIM apparatus to perform a MAC operation, in accordance with some embodiments of the application.

[0030] Figure 3A is a schematic diagram of a CIM apparatus architecture using an array of TCAM bitcells, in accordance with a first embodiment of the application.

[0031] Figure 3B is a schematic diagram illustrating Figure 3A is a schematic circuit diagram of the same CIM apparatus shown, and detailing the SRAM circuitry and input activation bus architecture within each TCAM bitcell.

[0032] Figure 4A is a schematic diagram of a CIM apparatus architecture in which addition is performed in a time-division multiplexed manner, in accordance with a second embodiment of the application.

[0033] Figure 4B is a schematic diagram of a CIM apparatus architecture in which addition is performed in a time-division multiplexed manner, in accordance with a second embodiment of the application.Figure 4A An exemplary timing diagram of the CIM device 200 is shown.

[0034] Figure 5 is a schematic diagram of a CIM device having multiple tiles according to a third embodiment of the present application.

[0035] Figure 6 is a schematic diagram of a CIM device having multiple tiles with a transposed input activation routing scheme according to a fourth embodiment of the present application.

[0036] Figure 7 is a schematic diagram of another variant of a CIM device having a contiguous tile architecture according to a fifth embodiment of the present application.

[0037] Figure 8 is a block diagram schematic of an illustrative computing device 1000.

[0038] In the following detailed description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the application. It will be apparent, however, that one or more embodiments can be practiced without these specific details, in different ways, and that different embodiments can be practiced with different combinations of elements. It should be noted that examples can be described in terms of sequential blocks or steps. Unless otherwise specified, these blocks or steps need not be performed in the order in which they are described. Non-sequential or parallel execution can be appropriate in some examples. In addition, unless otherwise specified, the examples are not limited to any particular hardware or software architecture. DETAILED DESCRIPTION

[0039] The following description is provided for the purpose of illustrating the embodiments of the application, and is not intended to limit the scope of the application. In the description, certain terminology is used to describe particular elements. Those of ordinary skill in the art will recognize that the terminology is used for the purpose of description and should not be construed to limit the elements described. The terms "element," "system" and "device" as used throughout this description and in the claims, can represent a physical entity, hardware, software, or a combination of hardware and software. The terms "comprising," "including," and "having," as used throughout this description and in the claims, are used in their open-ended, non-limiting sense. The term "coupled" as used throughout this description and in the claims means an indirect or direct electrical connection. Thus, if a device is coupled to another device, that device can be electrically connected to the other device directly, or through an intervening device or connection.

[0040] Wherever possible, similar reference numbers have been used in the drawings to indicate similar elements. The drawings are not necessarily to scale and the emphasis is generally placed upon illustrating the principles of the embodiments rather than to the details thereof.

[0041] The term "substantially" or "approximately" as used herein refers to an acceptable range in which a person skilled in the art can solve the technical problems to be solved and substantially achieve the technical effects to be achieved. For example, "substantially equal" refers to a certain error from "exactly equal" in a manner acceptable to a person skilled in the art without affecting the correctness of the results.

[0042] Disclosed are apparatuses and methods for performing MAC operations that utilize a TCAM array as a compute-in-memory (CIM) device. CIM (compute-in-memory) or in-memory computation (or interchangeably referred to as "memory-in-computing") is a technique that performs computations on data using internal circuitry located within a memory array without sending such data to a processor, and can achieve higher computation throughput and higher performance compared to computations utilizing a processor located outside the memory array. CIM or in-memory computation can also reduce energy consumption due to reduced data movement between an external processor and a memory. In embodiments of the present disclosure, the terms compute-in-memory device and in-memory computation device are used interchangeably.

[0043] Some embodiments relate to a CIM device that includes an array of TCAM bit cells (also interchangeably as an array of TCAM bit cells or an array of TCAM cells) arranged in rows and columns. Each TCAM bit cell has two static random-access memory (SRAM) as shown in 6-T SRAM + 2 transistors in Figure 2 Figure 2 Each SRAM as shown in Figure 2 ​The "6T SRAM" and the corresponding two transistors noted in the figure can operate as a multiplier that performs multiplication between the stored weight and an input activation value applied to the search line in the TCAM bitcell array. The two SRAMs within the TCAM bitcell can operate independently to receive two input activation values independently on their respective select lines and perform multiplication operations with the weights stored in each respective SRAM. It is understood that the structure of a 16-T TCAM bitcell is well known to those of ordinary skill in the art and the specific structure and function of the TCAM bitcell will not be described in detail herein.

[0044] In some embodiments, multiple TCAM bitcells in the same row are interconnected with an activation line to receive an output value that is the result of multiplication performed by the SRAMs in the row of TCAM bitcells. Summing for the MAC operation is provided by an adder tree coupled to the activation line. In some embodiments, each row of TCAM bitcells is provided with a respective activation line to receive output multiplication values from the TCAM bitcells in the row. The activation line is coupled to the adder tree for the MAC operation.

[0045] According to an aspect of the present application, since the activation line for a row of TCAM cells intersects with multiple select lines (which extend in the column direction) for multiple TCAM bitcells in the row, in order to avoid shorting between the activation line and the select lines, the activation line can be implemented by a conductor that is disposed in a different layer / plane than the layers / planes in which the select lines are disposed within the substrate for the TCAM bitcell array, e.g., in a higher metal layer. One or more vertical via structures can be provided to couple the activation line to the respective TCAM bitcells to receive the output values.

[0046] Some aspects relate to operations of a CIM device performing MAC computations. One aspect relates to programming weights in a weight matrix into SRAM of a TCAM bitcell array based on a MAC operation to be performed. In some embodiments, input activation values in an input activation vector are provided to different rows of TCAM bitcells. Since each TCAM bitcell provides two SRAM to store two weights, a multiplication operation with two weights can be performed in a time-multiplexed manner. In one example, a controller can cause the two multiplication operations to be performed in different non-overlapping phases within one clock cycle to produce a time-multiplexed output value. In such an example, an addition of the time-multiplexed output value can be performed in a clock cycle after the multiplication is completed.

[0047] In some embodiments, a CIM device includes multiple sub-arrays or tiles of a TCAM bitcell array. In a first embodiment, a first block is located above a second block in a column direction, the CIM device includes an SRAM edge cell separating the two blocks such that search lines between the adjacent first and second blocks can be used to independently provide different input activation vectors. In such an embodiment, an input activation vector can be routed horizontally along a side of a block through routing circuitry along a column of TCAM bitcells. In another embodiment, routing of an input activation vector can be provided across columns of different blocks. In such an embodiment, since search lines can be shared across adjacent blocks, SRAM edge cells are not needed between adjacent blocks and higher array efficiency can be achieved.

[0048] Some aspects can utilize existing foundry designs for TCAM bitcells without or with minimal hardware changes, which can reduce design and manufacturing costs of the disclosed CIM devices.

[0049] Embodiments disclosed herein can be used for computations of neural networks in artificial intelligence (AI) applications, some examples of which are described in detail below. It should be understood that embodiments of the present disclosure can also be used for other applications, such as but not limited to image processing, image classification, and face recognition using images captured by a camera. In some embodiments, a CIM device is part of a mobile device.

[0050] The above aspects and embodiments, as well as additional aspects and embodiments, are described further below. These aspects and / or embodiments can be used individually, together, or in any arbitrary combination of two or more, without limitation.

[0051] Figure 1 is a schematic diagram of an exemplary neural network representation showing a MAC operation performed with a CIM device disclosed herein. Figure 1 An input activation vector (a0, a1, a2, a3) is shown, which can represent four input activation values from a previous layer. The neural network applies a weight matrix where each weight w i,j represents a weight for the respective input activation and neuron, respectively. The neural network produces an output activation vector (N0, N1, N2, N3) based on a vector-matrix multiplication of the input activation vector and the weight matrix, which has four output values N0...N3 for four respective neurons of a next layer. The vector-matrix multiplication can be performed with MAC operations. It should be understood that while Figure 1 A 4x4 square weight matrix is shown, but this is for illustrative purposes only and the present invention is not limited to this example. It can be appreciated that the number of neurons of a previous layer can differ from the number of neurons of a next layer, and that the input activation vector and the output activation vector can have any suitable size and / or dimension.

[0052] Figure 2 is a schematic diagram of an exemplary 16-transistor (16-T) TCAM bitcell that can be used in a CIM device for performing MAC operations, shown in accordance with some embodiments. Figure 2 A TCAM bitcell 110 is shown that includes a first SRAM 101 and a second SRAM 102. The TCAM bitcell 110 is a 16-T TCAM bitcell, where each of the SRAMs 101, 102 can be implemented as a 6-T SRAM plus 2 respective transistors. It should be noted that any generation of TCAM bitcell now known or developed in the future can be used for the TCAM bitcell 110, and the present invention is not limited in this regard.

[0053] As Figure 2As shown, any known technique for programming TCAM bit cells 110 can be used to store charges Q0 and Q1 on SRAMs 101 and 102, where the bit values ​​of Q0 and Q1 can represent two weight values ​​of a weight matrix. The first SRAM 101 receives a first input value from a first search line SL, while the SRAM 102 receives a second input value b from a second search line SLB. In this embodiment, each of the SRAMs 101 and 102 can operate as an independent multiplier of the input value and the stored charge, where the output value is provided at a common output node 103 shared by the SRAMs 101 and 102. For example, an inverting output node MULT is coupled to the common output node 103 via an inverter 130. Those skilled in the art will understand that TCAM bit cell 110 is equivalent to multiplying the stored weight Q0 with the first input value received from the first search line SL and then inverting it. Therefore, the MULT of the output value of TCAM bit cell 110 after inverter 130 is equivalent to the product of Q0 and the first input value. In other words, TCAM bit cell 110 performs a multiplication and inversion operation, which those skilled in the art will understand is also a multiplier operation. Figure 2 As shown in truth table 1011, the bit value of MULT represents the product of the first input value a and the weight Q0 of the first SRAM. For example, MULT is 1 only when both the first input value a and the weight Q0 are 1. Figure 2 As shown in truth table 1021, the bit value of MULT represents the product of the second input value b and the weight Q1 of the second SRAM. Although Figure 2 The truth value represents a multiplication with single-bit weights Q0 and Q1, but multi-bit multiplication is also applicable, for example, by storing multiple bit values ​​in capacitors in SRAM.

[0054] According to one aspect, when TCAM 110 is used in a CIM device, input values ​​a and b are applied independently to SL and SLB, allowing SRAMs 101 and 102 to perform independent multiplications. This operation differs from some non-CIM operations using TCAM, such as storage or retrieval, where SL and SLB are not independent operations. Any suitable technique for independently controlling the routing of different input values ​​to SL and SLB can be used. In one example, the input values ​​can be provided sequentially or in a time-multiplexed manner, such that only one multiplication operation is performed within TCAM 110 at a time interval.

[0055] Figure 2An activation line 120 is also shown extending across two search lines SL and SLB and horizontally to interconnect the common output node 103 to the input of the inverter 130. In some embodiments, to avoid shorting the activation line 120 with the search lines SL, SLB, the activation line 120 is implemented as a conductor ML extending in a metal layer, where the metal layer is in a different layer / plane (in this disclosure, “layer” and “plane” are used interchangeably) offset from the layer / plane where the search lines SL and SLB are located, and parallel to the surface of the semiconductor substrate of the TCAM 110. In a non-limiting example, the activation line 120 is disposed in the M3 metal layer. In other words, in some embodiments, the memory array, the search lines, and the activation line are in different metal layers, respectively. In embodiments where the TCAM 110 uses an existing foundry TCAM design, the activation line 120 is added without significantly increasing the cost or complexity of the overall memory array circuit design.

[0056] Figure 3A is a schematic diagram of a CIM device architecture utilizing a TCAM bitcell array according to a first embodiment of the present disclosure. Figure 3A A CIM device 100 is shown, which includes a TCAM bitcell array 110 arranged in four rows and four columns. Within the first row 112-0, the output nodes of the four TCAM bitcells are interconnected together to an activation line 120-0, which extends in the row direction. The activation line 120-0 is coupled through an inverter to an adder tree 140, and provides an inverted output MULT[0] to the adder tree 140.

[0057] Figure 3A An example of loading weights of a weight matrix to different SRAM locations of the TCAM bitcell array and routing of input activation values is illustrated to perform vector matrix multiplication between an 8-bit input activation vector (a0, a1, … a7) and an 8x4 weight matrix Figure 3A ​The array of TCAM bitcells in is configured to take 8 different input activation values a0, a1,... a7, which are routed along the leftmost column. For example, with the first row 112-0, the first input activation value a0 is provided at the search line 101a of the first SRAM 101, which multiplies a0 with the weight w(0,0). The second input activation value a1 is provided at the search line 102b of the second SRAM 102, which multiplies a1 with the weight w(1,0). The results from the multiplications of the two SRAMs 101, 102 are provided as output values to the activation line 120-0 as MULT[0]. The adder tree 140 is configured to receive the products a0-w(0,0) and a1-w(1,0) from MULT[0] and perform the accumulation operation accordingly.

[0058] Still referring to Figure 3A the first row 112-0 in, which illustrates how each of the 8 SRAMs within the first row is respectively coupled to the activation bus architecture of one of the 8 input activation values a0, a1,... a7. During operation, 8 input activation values are provided each time. For example, the routing circuitry can activate the search lines corresponding to one of the input activation values only, while disabling the search lines of the remaining 7 input activation values. The 8 products of the SRAMs in the four TCAM bitcells in the first row 112-0 are represented in MULT[0]. The adder tree 140 accumulates the products in MULT[0] based on a0-w(0,0) + a1-w(1,0) + a2-w(2,0) + a3-w(3,0) + a4-w(4,0) + a5-w(5,0) + a6-w(6,0) + a7-w(7,0) to provide the MAC value N0.

[0059] Still referring to Figure 3A which shows that each of the remaining three rows 112-2, 112-2, 112-3 of TCAM bitcells is coupled to the adder tree 140 via a corresponding activation line 120-1, 120-2, 120-3. Notably, each input activation value is provided to only one of the SRAMs within each row, and, the input activation values are provided to each of the four rows. Thus, four multiplication operations can be performed simultaneously in the four rows to yield four MULT values, and the four MULT values can be output in parallel to the adder tree 140.

[0060] Figure 3B is a schematic circuit diagram showing Figure 3A the same CIM device shown in, and details the SRAM circuitry and input activation bus architecture within each TCAM bitcell. For example, Figure 3BThe first SRAM of the top left TCAM bitcell is shown coupled to a first search line 101a, while the second SRAM of the same TCAM bitcell is coupled to a second search line 102b.

[0061] The TCAM-based CIM device 100 can provide several advantages. The CIM array architecture has a high degree of scalability to provide flexibility to applications with different matrix and vector size requirements across different technology generations. For example, while Figure 3A A MAC operation with an 8x4 weight matrix is shown, but the CIM device 100 can be scaled to accommodate any size weight matrix. For example, an 8x8 weight matrix can be used by adding more 4-row TCAM bitcells. Similarly, a 16x4 weight matrix can be computed by scaling each of the four rows to have 8 columns. The length of the activation line 120-0 can be increased along the row direction to accommodate the additional columns. A large number of multiplier cells can be provided based on any suitable SRAM / TCAM read / write circuit architecture now known or developed in the future to benefit from the high storage density of the TCAM bitcell array. For example, scaling can be provided by using more bitcell columns and rows within the array, and / or by using a set of subarrays as described below.

[0062] Referring back to Figure 3A An SRAM edge cell 130 is provided below the row 112-3. The edge cell 130 can provide electrical isolation between the four rows 112-0, 112-1 (second row), 112-3 (third row), 112-3 (fourth row) and additional circuitry (e.g., located below the Figure 3A The edge cell 130 shown below the additional row TCAM bitcells in the different block.

[0063] Figure 4A is a schematic diagram illustrating the execution of an addition operation in a CIM device architecture in a time-division multiplexed manner according to a second embodiment of the present invention. Figure 4A A CIM device 200 is shown with two rows 212-0, 212-1, each having 4 TCAM bitcells. In the first row 212-0, an activation line 220-0 is coupled to each TCAM bitcell to provide an output value MULT[0] to an adder tree 240.

[0064] The CIM device 200 allows one input activation value (e.g., a0) to be time-division multiplexed with different weights. As shown, the product with a0is configured to be performed in two stages. For example, a 0_p1 is provided to w(0,0) in the SRAM 201, while a 0_p1w(0,2) is provided to SRAM 202 in the same row 212-0 as SRAM 201. Multiplications a0-w(0,0) and a0-w(0,2) can be performed at different times, e.g. in two non-overlapping phases. This is illustrated in Figure 4B , Figure 4B a schematic timing diagram for operating the CIM device 200 according to some embodiments. Figure 4A

[0065] Figure 4B It is shown that the products a0-w(0,0) and a0-w(0,1) can be performed in parallel during a first phase 264 of a clock cycle 262, in which a 0_p1 is active and a 0_p2 is disabled. The products a0-w(0,2) and a0-w(0,3) can be performed during a second phase 266 of the clock cycle 262, in which a 0_p2 is active and a 0_p1 is disabled. The clock cycle 262 can be a clock signal CLK provided by circuitry within the CIM device 200, but the invention is not limited thereto, i.e. any suitable timing signal can be used.

[0066] The weight matrix mapping in the second embodiment shown in Figure 4A is also different from the weight matrix mapping in the first embodiment shown in Figure 3A . Thus, unlike the CIM device 100, in the CIM device 200 one input activation value a0 is interconnected with the multiplication output values of different weights by a single activation line, instead of multiple activation lines in the CIM device 100. For example, in Figure 4A both a0-w(0,0) and a0-w(0,2) are interconnected to MULT[0] by the activation line 220-0 in the first row, whereas this same multiplication requires two rows MULT[0] and MULT[2] in Figure 3A . Thus, the second embodiment can provide some advantages over the first embodiment. Since less rows are required, the adder height along the column direction of the semiconductor substrate can be made more compact, thereby improving the area utilization on the semiconductor substrate. As another advantage, since the SRAMs for w(0,0) and w(0,2) are switched at different time phases, there are fewer bit cells switched at a given time. Thus, the switching current can be more evenly distributed over time, thereby reducing the average switching current and reducing local resistive IR drops. At the same time, since the second embodiment performs less computations in parallel, the computation is slower compared to the first embodiment. ​

[0067] Figure 5 This is a schematic diagram of a CIM device having multiple groups (tiles, or interchangeably referred to as "blocks") according to a third embodiment of the present invention. Figure 5 As shown, the CIM device 300 includes four blocks (or subarrays) 301, 302, 303, and 304 of TCAM bit cells. Each block is similar in many ways to Figure 3A The TCAM bit cell array is shown. The activation input value a0 provided to block 301 is routed via routing circuitry 351 located at the left edge of block 301 and parallel to the column direction. The activation input value a1 provided to block 302 is routed via routing circuitry 352 located at the left edge of block 302 and parallel to the column direction. Block 302 is located below block 301 and separated from block 301 by one or more SRAM edge cells 130 to cut off the selection line in one block from extending into different blocks, thereby providing electrical isolation between different activation inputs. This block can be utilized... Figure 3A The first embodiment shown, Figure 4A The second embodiment shown, a combination of the two embodiments, or any other suitable arrangement of the TCAM bit cell array is implemented.

[0068] exist Figure 5 In this circuit, routing circuit 351 can be a demultiplexer used to direct a0 to one of eight different inputs for inputting an activation value, based on, for example, a 3-bit control input signal transmitted from a controller (not shown) to the demultiplexer. It should be understood that any suitable decoding circuitry can be used to route signals to multiple inputs of another circuit. In one example, routing circuit 351 may include a word-line decoder.

[0069] Figure 6 This is a schematic diagram of a CIM device comprising multiple blocks, illustrating a scheme for activating circuitry using a transposed input according to a fourth embodiment of the present invention. Figure 6 As shown, CIM device 400 and Figure 5 The CIM device 300 differs from the one described above in that, using routing circuitry 451 located next to the bottom row of array 401, the input activation values ​​are routed from below array 401. Due to this vertical routing of the input lines, the array 401 of TCAM bit cells can be extended vertically into multiple rows (e.g., as shown in the image). Figure 6The 64 SRAM rows shown are not broken up into 8-bit or 4-bit subarrays (which are separated by SRAM edge cells for isolation). Because there is no SRAM edge cell footprint between rows, this embodiment can greatly improve area utilization efficiency.

[0070] The configuration of the weight matrix array 401 can be programmed with respect to the routing of the input activation values, similar to Figure 3A the first embodiment shown, Figure 4A the second embodiment shown, a combination of the two embodiments, or any other suitable arrangement of TCAM bitcell arrays to provide the MAC operation of the input activation vector with the weight matrix.

[0071] In Figure 6 each row of TCAM bitcells, the output values are interconnected by horizontal activation lines in the row direction to 4-bit adders 440, which in turn are connected to an adder tree 442 configured to accumulate the values received by the adder tree. As will be appreciated, the role of the adder is to sum the results of the operation of two adjacent activation values (4-bit), while the adder tree is to accumulate the outputs of the adders. Both have the same function, but the structure is different, the latter optimizes the logic, reduces the area, delay and power consumption in the process of multi-stage accumulation, and those skilled in the art will understand the structure of the adder and the adder tree, therefore, it will not be described in detail herein.

[0072] Figure 7 is a schematic diagram of another variant of a CIM device with a contiguous block architecture according to a fifth embodiment of the application. In Figure 7 the CIM device 500 includes contiguous TCAM bitcell arrays 501, 502 extending in row and column directions. The routing of the input activation vector is provided by routing circuitry 551 located below the array 501 and alongside the bottom row of the array 501. Similar to Figure 6 the embodiment in Figure 5 , this vertical wiring of the input data allows the memory blocks to be stacked in the column direction to provide 32, 64, 128, 256 rows or more for each bitline. With respect to the output, an array of 4-bit adders 540 can be provided and arranged in the column direction, and the accumulation operation is further provided by adder / accumulators 542. Because the use of SRAM edge cells is minimized, the contiguous block architecture of Figure 7 can achieve significant area utilization efficiency compared to the third embodiment in

[0073] Figure 8 is a block diagram schematic of an illustrative computing device 1000 in which some embodiments can be practiced. In Figure 8In particular embodiments, computing device 1000 can be a desktop computer, a server, or a high-performance computing system such as a machine learning accelerator. Computing device 1000 can also be a portable, handheld, or wearable electronic device. In some embodiments, computing device 1000 can be a smartphone, a personal digital assistance (PDA), a tablet computer, a smartwatch. Computing device 1000 can be powered by a battery such as a rechargeable battery. Computing device 1000 can also be a general-purpose computer, as aspects of the present application are not limited to portable or battery-powered devices. Computing device 1000 can be a CIM device capable of performing MAC operations according to any of the embodiments disclosed herein.

[0074] Computing device 1000 includes a central processing unit (CPU) 12 having one or more processors, a graphics processing unit (GPU) 14 having one or more graphics processors, and a memory (e.g., a non-transitory computer-readable storage medium) 16, e.g., including volatile and / or non-volatile memory. Memory 16 can store one or more instructions to program CPU 12 and / or GPU 14 to perform any of the functions described herein. Memory 16 can include one or more TCAM bitcell arrays that can perform MAC operations according to the disclosure described herein. Memory 16 can also include routing circuitry and one or more memory controllers configured to route input activation values and adder tree programming operations.

[0075] Computing device 1000 can have one or more input devices and / or output devices, such as user input interface 18 and output interface 17, as shown in FIG. 1. Figure 8 These devices can be used to present a user interface. Examples of output interfaces that can be used to provide a user interface include printers or display screens for visual presentation of output and speakers or other sound generating devices for audible presentation of output. Examples of input interfaces that can be used for a user interface include keyboards, including virtual keyboards, and pointing devices, such as mice, touch pads, and digitizers for pen, touch, or finger input. As another example, input interface 18 can include one or more microphones for capturing audio signals, one or more cameras and light sensors for capturing visual signals, and output interface 17 can include a display screen for visual presentation and / or a speaker for audible presentation of images or text to a user 30 of computing device 1000.

[0076] AsFigure 8 As shown, computing device 1000 can include one or more network interfaces 19 to enable communication via various networks, such as communication network 20. Examples of such networks include local or wide area networks, such as an enterprise network or the Internet. Such networks can be based on any suitable technology and can operate according to any suitable protocol and can include wireless networks, wired networks or fiber optic networks. Examples of network interfaces include Wi-Fi, WiMAX, 3G, 4G, 5G NR, white space, 802.1 lx, satellite, Bluetooth, near field communication (NFC), LTE, GSM / WCDMA / HSPA, CDMA lx / EVDO, DSRC, GPS, etc. Although not shown, computing device 1000 can also include one or more high speed data buses (interconnecting the components shown in the Figure 8 FIG. 1) and a power subsystem (to provide power to the components).

[0077] The aspects and embodiments described above can be used alone or in combination with each other, or in any suitable combination of two or more, as the application is not limited in this respect.

[0078] Having thus described some aspects of at least one embodiment of the application, it is to be appreciated that various alterations, modifications, and improvements will readily occur to those skilled in the art. Such alterations, modifications, and improvements are intended to be part of this disclosure, and are intended to be within the spirit and scope of the application. Further, although advantages of the present application are indicated, it should be appreciated that not every embodiment of the technology described herein will include every described advantage. Some embodiments can not implement any advantages or only a subset of the advantages.

[0079] Further, the application can be embodied as a method, of which an example has been provided. The acts performed as part of the method can be ordered in any suitable way. Accordingly, embodiments can be constructed in which acts are performed in an order different than illustrated, which can include performing some acts simultaneously, even though shown as being performed sequentially in illustrative embodiments.

[0080] In some embodiments, the terms “about,” “substantially,” and “approximately” can be used to indicate that a value is within ±20% of a target value, in some embodiments, within ±10% of a target value; in some embodiments, within ±5% of a target value. In some embodiments, within ±2% of a target value. The terms “about” and “approximately” can include the target value.

[0081] The use of ordinal terms such as "first", "second", "third", etc. in the claims to modify a claim element does not by itself connote any priority, precedence or order of one claim element over another, or of executing the method actions in a temporal sequence, but is merely used to distinguish one claim element from another.

[0082] While the application has been described by way of example and in terms of the preferred embodiment, it is to be understood that the application is not limited to the disclosed embodiment. To the contrary, it is intended to cover various modifications and equivalent structures (as would be apparent to one skilled in the art) and does so expressly, as for example, the combinations or sub-combinations of different features disclosed in the above-described embodiments. Therefore, the scope of the claims should be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures.

Claims

1. An in-memory computing device, comprising: comprises: a memory array comprising a plurality of ternary content addressable memory (TCAM) bitcells arranged in a plurality of rows, wherein each TCAM bitcell comprises two static random access memories (SRAMs), two independent input activations are applied to two search lines of a single TCAM bitcell respectively, each SRAM is configured to multiply a respective weight stored therein with an input activation of a respective search line to generate an output value, wherein the two multiplication operations in a single TCAM bitcell are performed in a time-division multiplexed manner; and a plurality of activation lines, each activation line is interconnected to the TCAM bitcells of a respective row of the plurality of rows and is configured to receive the output values from the two SRAMs in each TCAM bitcell located within the respective row.

2. The in-memory computing device of claim 1, wherein, The in-memory computing device further comprises: a summer having a plurality of summer inputs, each summer input is coupled to a respective activation line of the plurality of activation lines, the summer is configured to generate a multiply-accumulate (MAC) value based on the output values of the two SRAMs.

3. The in-memory computing device of claim 1, wherein, Each TCAM bitcell of the plurality of TCAM bitcells comprises: a first SRAM configured to receive an input activation from a first search line and a second SRAM configured to receive another input activation from a second search line.

4. The in-memory computing device of claim 1, wherein, The in-memory computing device further comprises a plurality of inverters, wherein each activation line is coupled to a respective summer input via a respective inverter of the plurality of inverters.

5. The in-memory computing device of claim 1, wherein, The TCAM bitcells within the memory array are 16-transistor (16-T) TCAM bitcells.

6. The in-memory computing device of claim 1, wherein, The memory array is configured to receive an input activation vector and a weight matrix.

7. The in-memory computing device of claim 6, wherein, The input activations of the input activation vector are provided to the TCAM bitcells of two or more different rows.

8. The in-memory computing device of claim 6, wherein, The input activation vector is received in a time-division multiplexed manner.

9. The in-memory computing device of claim 1, wherein, Within each SRAM, the output value is generated based on a multiplication of the weight and the input activation.

10. The in-memory computing device of claim 3, wherein, The memory array is located in a first layer, the first search line is located in a second layer parallel to the first layer, and the plurality of activation lines comprises conductors located in a third layer parallel to and offset from the second layer.

11. The in-memory computing device of claim 3, wherein, The plurality of TCAM bitcells are arranged in a plurality of columns, and the in-memory computing device further comprises: routing circuitry located alongside one of the plurality of columns and configured to route input activations to some or all of the plurality of TCAM bitcells.

12. The in-memory computing device of claim 11, wherein, The in-memory computing device further comprises: an SRAM edge cell located alongside a row of the plurality of TCAM bitcells.

13. The in-memory computing device of claim 12, wherein, The memory array comprises: a first sub-array and a second sub-array located below the first sub-array in a column direction, the SRAM edge cell is located between the first sub-array and the second sub-array.

14. The in-memory computing device of claim 3, wherein, The in-memory computing device further comprises: routing circuitry located alongside one of the plurality of rows and configured to route input activations to some or all of the plurality of TCAM bitcells.

15. The in-memory computing device of claim 14, wherein, The memory array comprises: a first sub-array and a second sub-array located below the first sub-array in a column direction, and there is no SRAM edge cell between the first sub-array and the second sub-array.

16. A method for performing a multiply and accumulate, MAC, operation, the method comprising: The method performs the MAC operation on an input activation vector and a weight matrix by manipulating an array of ternary content addressable memory (TCAM) cells, a plurality of TCAM bit cells in the array of TCAM cells are arranged into a plurality of rows and each TCAM bit cell includes two static random access memories (SRAMs), wherein two independent input activations are applied to two search lines of a single TCAM bit cell, and the method includes: storing a plurality of weights in a plurality of SRAMs within the array of TCAM cells; multiplying the input activation vector with the plurality of weights using the plurality of SRAMs to produce a plurality of output values, wherein each SRAM is configured to multiply a respective weight stored therein with an input activation of a respective search line to produce an output value, wherein the two multiplication operations in a single TCAM bit cell are performed in a time-division multiplexed manner; collecting the output values from the two SRAMs in each TCAM bit cell in a respective row of the plurality of rows using an activation line; and summing the output values from the activation line to produce a MAC value.

17. The method of claim 16, wherein, Summing the output values from the activation line includes coupling the activation line to a tree of adders.

18. The method of claim 16, wherein, The act of multiplying is performed in a time-division multiplexed manner.

19. The method of claim 16, wherein, The act of multiplying includes: providing a first input activation of the input activation vector to a first SRAM in a TCAM bit cell from a first search line of the array of TCAM cells; and providing a second input activation of the input activation vector to a second SRAM in the TCAM bit cell from a second search line of the array of TCAM cells.

20. The method of claim 19, wherein, The act of multiplying further includes: providing the first input activation to the SRAMs in the TCAM bit cells of two or more different rows of the plurality of rows.

Images