Configurable in-memory computing engines, platforms, bit cells, and layouts therefor

The bit-parallel/bit-serial method with ADCs and data reshaping buffers in in-memory computing systems addresses the limitations of 1-b representations, enhancing efficiency and scalability for multi-bit operations in machine learning and artificial intelligence applications.

JP7587823B2Active Publication Date: 2024-11-21THE TRUSTEES OF PRINCETON UNIV

Patent Information

Application Number
JP2020570472
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2018-11-07
Filing Date
2019-06-18
Publication Date
2024-11-21
Estimated Expiration
2039-06-18

AI Technical Summary

Technical Problem

Existing in-memory computing systems are limited by their inability to efficiently perform multi-bit operations and are constrained by 1-b representations, leading to energy inefficiencies and scalability issues in matrix-vector products, particularly in machine learning and artificial intelligence applications.

Method used

The implementation of a bit-parallel/bit-serial (BP/BS) method for in-memory computing, incorporating analog-to-digital converters (ADCs) and a configurable near-memory computing path to support multi-bit operations, along with data reshaping buffers for interfacing with microprocessors, enabling programmability and energy-efficient matrix-vector multiplication.

Benefits of technology

This approach enhances the efficiency and scalability of in-memory computing by allowing multi-bit operations, reducing energy consumption and maintaining signal-to-quantization noise ratio (SQNR), thus improving performance in machine learning and artificial intelligence applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007587823000004
    Figure 0007587823000004
  • Figure 0007587823000005
    Figure 0007587823000005
  • Figure 0007587823000006
    Figure 0007587823000006
Patent Text Reader

Abstract

Various embodiments include a system, method, architecture, mechanism, or apparatus for providing programmable or pre-programmed in-memory computing operations, including a massively parallel bit-wise input signal, a reshaping buffer configured to reshape a sequence of received data words to form a plurality of CIM channel output signals, thereby providing a sequence of multi-bit output words, control circuitry configured to cause the CIM array to perform multi-bit computing operations on the input signal and the stored signal using single-bit internal circuitry and signals, and a near-memory computing path configured to provide a sequence of multi-bit output words as a computing result.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 686,296, filed June 18, 2018, U.S. Provisional Patent Application No. 62 / 702,629, filed July 24, 2018, U.S. Provisional Patent Application No. 62 / 754,805, filed November 2, 2018, and U.S. Provisional Patent Application No. 62 / 756,951, filed November 7, 2018, which are incorporated by reference in their entireties herein.

[0002] The present invention relates to the field of in-memory computing and matrix-vector multiplication. [Background technology]

[0003] In recent years, charge-domain in-memory computing has emerged as a robust and scalable way of performing in-memory computing. Here, an arithmetic operation within a memory bit cell delivers its result as a charge, typically using voltage-to-charge conversion through a capacitor. Thus, the bit cell circuitry involves the appropriate switching of a local capacitor in a given bit cell, which is also appropriately coupled with other bit cell capacitors to produce an aggregate arithmetic result across the combined bit cells. In-memory computing is well suited to implement matrix-vector multiplication, where matrix elements are stored in a memory array and vector elements are broadcast in parallel across the memory array. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Provisional Patent Application No. 62 / 555,959, "Analog Switched-Capacitor Neural Network," filed September 8, 2017 Summary of the Invention [Problem to be solved by the invention]

[0005] Various deficiencies in the prior art are addressed by a system, method, architecture, mechanism, or apparatus that provides programmable or pre-programmed in-memory computing operations. [Means for solving the problem]

[0006] One embodiment provides an in-memory computing architecture comprising: a reshaping buffer configured to reshape a sequence of received data words to form a massively parallel bit-wise input signal; a compute-in-memory (CIM) array of bit cells configured to receive the massively parallel bit-wise input signal via a first CIM array dimension and one or more stored signals via a second CIM array dimension, where each of a plurality of bit cells associated with a common stored signal forms a respective CIM channel configured to provide a respective output signal; an analog-to-digital converter (ADC) circuit configured to provide a sequence of multi-bit output words by processing the plurality of CIM channel output signals; a control circuit configured to cause the CIM array to perform multi-bit computing operations on the input and stored signals using single-bit internal circuitry and signals; and a near-memory computing path configured to provide a sequence of multi-bit output words as a computing result.

[0007] Additional objects, advantages and novel features of the invention will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following description, or may be learned by the practice of the invention. The objects and advantages of the invention may be realized and attained by means of the instrumentalities and combinations particularly pointed out in the appended claims.

[0008] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the general description of the invention given above, and the detailed description of the embodiments given below, serve to explain the principles of the invention. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates a typical structure of an in-memory computing architecture. [Diagram 2] FIG. 2 is a high-level block diagram of an example architecture according to one embodiment. [Diagram 3] FIG. 3 is a high-level block diagram of an exemplary compute-in-memory unit (CIMU) suitable for use in the architecture of FIG. [Figure 4] 3 illustrates a high-level block diagram of an Input-Activation Vector Reshaping Buffer (IA BUFF) suitable for use in the architecture of FIG. 2, according to one embodiment. [Diagram 5] FIG. 3 is a high-level block diagram of a CIMA read / write buffer suitable for use in the architecture of FIG. 2, according to one embodiment. [Figure 6] 3 illustrates a high-level block diagram of a Near-Memory Datapath (NMD) module suitable for use in the architecture of FIG. 2, according to one embodiment. [Figure 7] 3 illustrates a high-level block diagram of a direct memory access (DMA) module suitable for use in the architecture of FIG. 2, according to one embodiment. [Figure 8A] FIG. 3 is a high level block diagram of different embodiments of CIMA channel digitization / weighting suitable for use in the architecture of FIG. 2. [Figure 8B] FIG. 3 is a high level block diagram of different embodiments of CIMA channel digitization / weighting suitable for use in the architecture of FIG. 2. [Figure 9] FIG. 2 is a flow diagram of a method according to one embodiment. [Figure 10] FIG. 1 is a circuit diagram of a multiplying bit-cell. [Figure 11] FIG. 1 is a circuit diagram of three M-BCs configured to perform an XNOR function. [Figure 12] FIG. 2 is a circuit diagram of an M-BC according to one embodiment. [Figure 13] FIG. 14 illustrates an example IC of the M-BC of FIG. [Figure 14A] FIG. 1 is a block diagram of a bit cell having a switched coupling structure. [Figure 14B] FIG. 1 is a block diagram of a bit cell having a non-switched coupling structure. [Figure 15] FIG. 2 is a circuit diagram of a bit cell circuit having a non-switched coupling structure according to one embodiment. [Figure 16] FIG. 1 is a circuit diagram of two-way interleaving of bit cell layouts according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] It should be understood that the accompanying drawings are not necessarily drawn to scale, but rather show some simplified representations of various features illustrating the basic principles of the invention. For example, the specific design features of the sequence of operations as disclosed herein, including the specific dimensions, orientations, arrangements, and shapes of the various illustrated components, will be determined in part by the specific intended application and environment of use. Certain features of the described embodiments have been enlarged or distorted relative to other features for ease of visualization and clear understanding. In particular, thin features may be shown darker for clarity and explanation.

[0011] Before describing the present invention in more detail, it is to be understood that this invention is not limited to particular embodiments described, as such may, of course, vary. Also, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting, since the scope of the present invention will be limited only by the appended claims.

[0012] Where a range of values ​​is given, it is understood that each intervening value between the upper and lower limits of that range, to the tenth of the minimum division unless the context clearly dictates otherwise, and any other stated or intervening value in the stated range, is encompassed within the invention. The upper and lower limits of these smaller ranges may be independently included in the smaller ranges, and are also encompassed within the invention, subject to any specifically excluded limit in the stated range. Where the stated range includes one or both of the stated limits, ranges excluding either or both of those included limits are also included in the invention.

[0013] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this invention belongs. Although any methods and materials similar or equivalent to those described herein can also be used in the practice or testing of the present invention, a limited number of exemplary methods and materials are described herein. As used herein and in the appended claims, the singular forms "a", "an" and "the" include plural references unless the context clearly dictates otherwise.

[0014] Disclosed embodiments include systems and methods that enable programmability of a highly efficient linear algebra computing architecture for modern VLSI implementation. The computing architecture disclosed herein is referred to as "in-memory computing" and the programmability of this architecture allows it to be widely used across a range of applications, particularly for machine learning and artificial intelligence. The disclosed approach incorporates a range of configurability features for in-memory computing arrays, which in turn now can be integrated in a programmable platform to address energy and throughput for a wide range of machine learning and artificial intelligence applications.

[0015] Linear algebra operations, especially matrix-vector multiplication, have become very prominent in new workloads, especially from machine learning (ML) and artificial intelligence (AI). The dimensions of the matrices and vectors can often be very large (typically >100). Due to the way elements are reused in the above operations, they are usually stored in embedded or off-chip memory (depending on the number of elements that have to be stored). Implementations in modern VLSI technologies have shown that, in practice, the energy and latency of accessing data from such memory substantially exceeds the actual operations on the data. This has limited the energy / latency reductions achievable by traditional accelerators, thereby decoupling memory and operations and motivating the in-memory computing paradigm. Within the context of an in-memory computing system, the raw data is not accessed from memory, but the results of operations on many bits of the raw data are accessed, thereby amortizing the energy and latency of the access.

[0016] The disclosed systems, methods, and portions thereof, enable configurability and programmability of highly efficient linear algebra computing architectures for modern VLSI implementations and the like, and even leading to integrated circuit implementations. The computing architectures disclosed herein may be broadly referred to as "in-memory computing," and the programmability of the architectures enables a wide range of uses across a range of applications, including matrix-vector operations such as those used in machine learning, artificial intelligence, and other applications. In various embodiments, the disclosed systems, methods, and portions thereof enable configurability and programmability of in-memory computing architectures using hybrid analog / digital computing methods that employ parallel and serial operations.

[0017] While the disclosed approach may introduce certain forms of quantization noise during computation, this is controlled by a number of architectural features disclosed herein, allowing in some cases operation to exhibit quantization noise similar to that found in standard fixed-point precision computing.

[0018] The disclosed system, or portions thereof, may be implemented as an integrated circuit and various features thereof may be implemented as a Field Programmable Gate Array (FPGA) as well as simulated using Verilog / transistor level simulation.

[0019] The disclosed approach may be applied to a range of in-memory computing architectures, including those disclosed in Provisional Patent Application No. 62 / 555,959, filed September 8, 2017, entitled "Analog Switched-Capacitor Neural Network," which is incorporated by reference in its entirety as if fully set forth herein.

[0020] The disclosed approach enables the configurability and programmability of in-memory computing, which is purported to be an energy-efficient approach to linear algebra operations, enabling the use of the disclosed architecture in a broad range of applications.

[0021] Further disclosed embodiments provide a bit-parallel / bit-serial approach to multi-bit in-memory computing. In particular, disclosed herein is an in-memory computing architecture in which memory bit cells perform operations on 1-b operands, and the hardware is extended to enable operations on multi-bit operands using a bit-parallel / bit-serial (BP / BS) method in which multiple bits of one operand are mapped to parallel bit cells, and multiple bits of the other operand are input consecutively. The disclosed approach considers in-memory computing where the operation output bit cells are digitized and then fed to further operations, possible across different bit cell output operations in time and space, in-memory computing where multi-bit operand operations are performed on the digitized output in a bit-parallel / bit-serial (BP / BS) manner, in-memory computing where the BP / BS approach uses a two's complement representation, for example by using a bitwise AND operation by the bit cells, and in-memory computing where a different number representation is used, for example where 1 / 0 bits are mathematically taken to have values ​​of +1 / -1, so that the BP / BS approach allows for the use of an XNOR operation by the bit cells.

[0022] Configurable in-memory computing engine and platform Various embodiments relate to integrating configurability and hardware support for in-memory computing accelerators to enable the programmability and virtualization required for scaling to viable applications. Typically, in-memory computing implements matrix-vector multiplication, where matrix elements are stored in a memory array and vector elements are broadcast in parallel across the memory array. Some aspects of the embodiments relate to enabling programmability and configurability of such architectures.

[0023] In-memory computing typically involves 1-b representations for matrix elements, vector elements, or both. This is because the memory stores data in independent bit cells that are broadcast in a parallel and homogeneous manner, without providing the different binary weighted combinations between bits that are required for multi-bit operations. In this invention, the extension to multi-bit matrix and vector elements is achieved with a bit-parallel / bit-serial (BPBS) method.

[0024] A highly configurable / programmable near-memory computing data path is included to enable common operations often involved in matrix-vector multiplication. Both of these enable the operations required to extend from the bitwise operations of in-memory computing to multi-bit operations, and for versatility, it supports multi-bit operations and is no longer constrained by the 1-b representation specific to in-memory computing. Since programmable / configurable multi-bit computing is more efficient in the digital domain, in the present invention, analog-to-digital conversion is performed after in-memory computing, and in certain embodiments, the configurable data path is multiplexed between 8 ADC / in-memory computing channels, although other multiplexing ratios are also possible. This is also in line with the BPBS method used for multi-bit matrix element support in this embodiment, where support for up to 8-b operands is provided.

[0025] Since sparsity of input vectors is common in many linear algebra applications, the present invention incorporates support to allow sparsity control proportional to energy. This is achieved by masking the broadcast of bits from the input vector that correspond to zero-valued elements (such masking is done for all bits in a bit-serial process). This saves broadcast energy as well as computational energy in the memory array.

[0026] Assuming an internal bit-wise arithmetic architecture and an external digital word architecture of a typical microprocessor for in-memory computing, data reshaping hardware is used both at the arithmetic interface through which input vectors are supplied, and at the memory interface through which matrix elements are written and read.

[0027] FIG. 1 illustrates a typical structure of an in-memory computing architecture. Given a memory array (which may be based on standard or modified bit cells), the in-memory computing includes two additional "orthogonal" sets of signals: (1) input lines and (2) storage lines. With reference to FIG. 1, a two-dimensional array of bit cells is illustrated, with each of a number of in-memory computing channels 110 including a respective column of bit cells, with each of the bit cells per channel associated with a common storage line and bit line (column) and a respective input line and word line (row). It is noted that the columns and rows of signals are described herein as "orthogonal" with respect to each other simply to indicate the row / column relationship within the context of an array of bit cells, such as the two-dimensional array of bit cells illustrated in FIG. 1. The term "orthogonal" as used herein is not intended to convey any particular geometric relationship.

[0028] The input / bit and storage / bit sets of signals may be physically coupled to existing signals in the memory (e.g., word lines, bit lines) or may be separate. To implement a matrix-vector multiplication, the matrix elements are first loaded into the memory cells. Then, multiple input vector elements (possibly all of them) are applied at once via the input lines. This causes a local arithmetic operation, typically some form of multiplication, to occur in each of the memory bit cells. The result of this arithmetic operation is then driven onto a shared storage line. In this way, the storage line represents the result of the operation on multiple bit cells activated by the input vector elements. This is in contrast to standard memory access, where bit cells are accessed one at a time via bit lines and activated by a single word line.

[0029] In-memory computing as described above has a number of important attributes. First, the computations are typically analog. This is because the constrained structure of memories and bit cells requires a richer computational model than can be realized by a simple digital switched abstraction. Second, local operations on bit cells typically involve operations on the 1-b representations stored in the bit cells. This is because bit cells in standard memory arrays are not coupled to each other by any binary weighting method, and such coupling must be realized by a method of accessing / reading the bit cells from the edge. In the following, we describe the extensions to in-memory computing proposed in this invention.

[0030] Extension to near memory and multi-bit operations While in-memory computing has the potential to handle matrix-vector multiplication in ways that traditional digital acceleration cannot, a typical computation pipeline may include a range of other operations involving matrix-vector multiplication. While such operations are usually well handled by traditional digital acceleration, it may nevertheless be of high value to place such acceleration hardware close to the in-memory computing hardware in an appropriate architecture to handle the parallelism, high throughput (hence the need for high communication bandwidth), and common computation patterns associated with in-memory computing. Since it is preferable to perform the majority of the operations involved in the digital domain, analog-to-digital conversion by an ADC is included after each of the in-memory computing accumulation lines, hence referred to as an in-memory computing channel. The primary challenge is to incorporate the ADC hardware into the pitch of each in-memory computing channel, but the appropriate layout approach taken in the present invention makes this possible.

[0031] The incorporation of an ADC after each computation channel allows an efficient way of extending in-memory computing to support multi-bit matrix and vector elements via bit-parallel / bit-serial (BPBS) computation, respectively. Bit-parallel computation involves loading different matrix element bits in different in-memory computing columns. The ADC outputs from the different columns are then appropriately bit-shifted to represent the corresponding bit weightings, and a digital accumulation in all of the columns is performed to obtain the multi-bit matrix element computation result. Bit-serial computation, on the other hand, involves applying each bit of the vector element one at a time, storing the ADC output each time, and appropriately bit-shifting the stored output, before digital accumulation with the next output corresponding to the subsequent input vector bit. Such a BPBS approach that allows mixing of analog and digital computation is highly efficient because it leverages analog (1-b) highly efficient low precision schemes along with digital (multi-bit) highly efficient high precision schemes while overcoming the access costs associated with traditional memory operations.

[0032] While a range of near-memory computing hardware is possible, the details of the hardware incorporated in this embodiment of the invention are described below. To facilitate the physical layout of the multi-bit digital hardware described above, eight in-memory computing channels are multiplexed into each near-memory computing channel. We note that this allows the highly parallel operation of in-memory computing to match throughput with the high frequency operation of digital near-memory computing (highly parallel analog in-memory computing operates at a lower clock frequency than digital near-memory computing). Thus, each near-memory computing channel includes a look-up table (LUT), a fixed nonlinear function implementation, along with digital barrel shifters, multipliers, and accumulators. Additionally, a configurable finite-state machine (FSM) associated with the near-memory computing hardware is incorporated to control the operation of the entire hardware.

[0033] Input interfacing and bit scalability control To integrate in-memory computing with programmable microprocessors, the internal bit-wise operations and representations need to be properly interfaced with the external multi-bit representations used in typical microprocessor architectures. Therefore, a data reshaping buffer is included in both the input vector interface and the memory read / write interface through which the matrix elements are stored in the memory array. The design details used for the embodiment of the present invention are described below. The data reshaping buffer allows bit-width scalability of the input vector elements while maintaining maximum bandwidth for data transfer between the in-memory computing hardware and external memory, as well as to other architecture blocks. This data reshaping buffer consists of a register file that acts as a line buffer that receives incoming parallel multi-bit data for the input vectors, element by element, and provides outgoing parallel single-bit data for all vector elements.

[0034] In addition to word-wise / bit-wise interfacing, hardware support for convolution operations applied to input vectors is also included. Such operations are prominent in convolutional-neural networks (CNNs). In this case, matrix-vector products are performed with only a subset of new vector elements that need to be fed in (other input vector elements are buffered and simply shifted appropriately). This relieves the bandwidth constraints of getting data to high-throughput in-memory computing hardware. In an embodiment of the present invention, the convolution support hardware, which must perform the proper bit-serial ordering of multi-bit input vector elements, is implemented in a specialized buffer whose appropriately read-out output shifts the data for a configurable convolution stride.

[0035] Dimension and Sparsity Control For programmability, two additional considerations must be accommodated by the hardware: (1) matrix / vector dimensions may vary from application to application, and (2) in many applications vectors will be sparse.

[0036] In-memory computing hardware often incorporates controls to enable / disable tile parts of the array, so that energy is consumed only for the dimensional levels desired in the application. However, in the BPBS approach used, the input vector dimension has important implications on the computational energy and SNR. For SNR, if we perform bitwise operations in each in-memory computing channel, assuming that the operation between each input (fed on the input line) and the data stored in the bit cell produces a 1-bit output, the number of possible distinct levels on the storage line is equal to N+1, where N is the input vector dimension. This implies that a log2(N+1)-bit ADC is required. However, ADCs have an energy cost that scales strongly with the number of bits. It is therefore advantageous to support N that is very large but smaller than log2(N+1) bits in the ADC, to reduce the relative contribution of the ADC energy. The consequence of doing this is that the signal-to-quantization-noise ratio (SQNR) of the computational operation is reduced with the number of ADC bits, unlike standard fixed-precision operations. Therefore, hardware support for configurable input vector dimensions, along with the corresponding energy consumption, is essential to support varying application level dimensionality and SQNR requirements. For example, if a reduced SQNR is acceptable, then large dimensional input vector segments must be supported. On the other hand, if a high SQNR must be maintained, then lower dimensional input vector segments must be supported, with the dot product result being from multiple input vector segments that can be combined from different in-memory computing banks (thus, in particular, the input vector dimensions can be reduced to a level set by the number of ADC bits to ensure an operation ideally consistent with standard fixed-precision operations). The mixed analog / digital approach taken in the present invention makes this possible.That is, input vector elements can be masked to filter broadcasts only along the desired dimension, saving broadcast energy and bit-cell operation energy proportional to the input vector dimension.

[0037] For sparsity, a similar masking approach can be applied across bit-serial operations to prevent broadcasting of all input vector element bits that correspond to zero-valued elements. We note that the BPBS approach used is particularly helpful in achieving this, since the expected number of non-zero elements is often known in sparse linear algebra applications, while the input vector dimension can be large. Thus, the BPBS approach allows us to increase the input vector dimension and still ensure that the number of levels that need to be supported on the accumulation line is within the ADC resolution, thereby ensuring a high operational SQNR. While the expected number of non-zero elements is known, it is still essential to support a variable number of actual non-zero elements, which can vary from input vector to input vector. This is easily achieved in a mixed analog / digital approach, since the masking hardware simply counts the number of zero-valued elements for a given vector and then applies a corresponding offset to the final dot product result in the digital domain after the BPBS operation.

[0038] Exemplary Integrated Circuit Architecture 2 illustrates a high-level block diagram of an example architecture according to one embodiment. In particular, the example architecture of FIG. 2 has been implemented as an integrated circuit using VLSI fabrication techniques using specific components and functional elements for testing various embodiments herein. It is understood that alternative embodiments having different components (e.g., larger or more powerful CPUs, memory elements, processing elements, etc.) are contemplated by the inventors as being within the scope of this disclosure.

[0039] As shown in FIG. 2, the architecture 200 includes a central processing unit (CPU) 210 (e.g., a 32-bit RISC-V CPU), a program memory (PMEM) 220 (e.g., 128 KB program memory), a data memory (DMEM) 230 (e.g., 128 KB data memory), an external memory interface 235 (e.g., illustratively configured to expand accessible memory by accessing one or more 32-bit external memory devices (not shown)), a boot loader module 240 (e.g., configured to access an 8 KB off-chip EEPROM (not shown)), a compute-in-memory unit (CIMU) 300 including various configuration registers 255 and configured to perform in-memory computing and various other functions in accordance with embodiments described herein, a direct memory access (DMA) module 260 including various configuration registers 265, and a universal asynchronous receiver (UART) 270 for transmitting and receiving data. 2. The architecture 200 of FIG. 2 also includes various support / peripheral modules, such as a SoC configuration module (not shown), a general purpose input / output (GPIO) module 273, and various timers 274. Other elements not shown here may also be included in the architecture 200 of FIG. 2, such as an SoC configuration module (not shown).

[0040] While CIMU 300 is very well suited for matrix vector products and the like, other types of operations / computations may be better performed by non-CIMU computing devices. Thus, in various embodiments, a close coupling between CIMU 300 and near memory is provided such that the selection of the computing device to which a particular operation and / or function is assigned can be controlled to provide a more efficient computing function.

[0041] Figure 3 illustrates a high-level block diagram of an example compute-in-memory unit (CIMU) 300 suitable for use in the architecture of Figure 2. The following description relates to the architecture 200 of Figure 2 as well as an example CIMU 300 suitable for use within the context of that architecture 200.

[0042] Generally speaking, CIMU 300 provides programmable in-memory computation functions, such as matrix vector multiplication, by including various structural elements including, for example, a computation-in-memory array (CIMA) of bit cells that are configured via various configuration registers. In particular, the exemplary CIMU 300 is configured as a 590kb, 16-bank CIMU assigned to multiply an input matrix X by an input vector A to obtain an output matrix Y.

[0043] Referring to FIG. 3, the CIMU 300 is illustrated as including a computation-in-memory array (CIMA) 310, an input activation vector reshaping buffer (IA BUFF) 320, a sparsity / AND logic controller 330, a memory read / write interface 340, a row decoder / WL driver 350, a number of A / D converters 360, and a near memory computing multiply-shift-accumulate data path (NMD) 370.

[0044] The illustrated computation-in-memory array (CIMA) 310 includes a 256×(3×3×256) computation-in-memory array arranged as a 4×4 clock-gated 64×(3×3×64) in-memory computing array, thus having a total of 256 in-memory computing channels (e.g., memory columns) and including 256 ADCs 360 to support the in-memory computing channels.

[0045] The IA BUFF 320 operates, for example, to receive a sequence of 32-bit data words and reshapes the 32-bit data words into a sequence of higher dimensional vectors suitable for processing by the CIMA 310. It should be noted that 32-bit, 64-bit, or any other width data words may be reshaped to match the available or selected size of the compute-in-memory array 310, where the compute-in-memory array 310 itself is configured to operate on higher dimensional vectors and includes elements that may have 2-8 bits, 1-8 bits, or other sizes and apply them in parallel across the array. Also, while the matrix-vector multiplication operations described herein are illustrated as utilizing the entire CIMA 310, in various embodiments, only a portion of the CIMA 310 is used. Furthermore, in various other embodiments, the CIMA 310 and associated logic circuitry are adapted to implement an interleaved matrix-vector multiplication operation, where parallel portions of a matrix are processed simultaneously by respective portions of the CIMA 310.

[0046] In particular, the IA BUFF 320 reshapes sequences of 32-bit data words into highly parallel data structures that may be added to the CIMA 310 all at once (or at least in large chunks) and properly ordered in a bit-serial manner. For example, a 4-bit operation with 8 vector elements may be associated with a high-dimensional vector of 2000n-bit data elements or more. The IA BUFF 320 shapes this data structure.

[0047] As shown, IA BUFF 320 is configured to receive an input matrix X, e.g., as a sequence of 32-bit data words, and resize / rearrange the received sequence of data words according to the size of CIMA 310 to provide a data structure including, e.g., 2303n-bit data elements. Each of those 2303n-bit data elements, along with a respective masking bit, is transmitted from IA BUFF 320 to sparsity / AND logic controller 330.

[0048] The sparsity / AND logic controller 330 is configured to receive, for example, 2303n-bit data elements and respective masking bits and invoke the sparsity function accordingly, where zero-value data elements (such as those indicated by the respective masking bits) are not propagated to the CIMA 310 for processing, thus saving energy specifically required for the processing of such bits by the CIMA 310.

[0049] In operation, the CPU 210 reads the PMEM 220 and the boot loader 240 via direct data paths implemented in a standard manner. The CPU 210 may access the DMEM 230, the IA BUFF 320, and the memory read / write buffer 340 via direct data paths implemented in a standard manner. All of these memory modules / buffers, the CPU 210, and the DMA module 260 are connected by the AXI bus 281. The chip configuration modules and other peripheral modules are grouped by the APB bus 282 and attached to the AXI bus 281 as slaves. The CPU 210 is configured to write to the PMEM 220 via the AXI bus 281. The DMA module 260 is configured to access the DMEM 230, the IA BUFF 320, the memory read / write buffer 340, and the NMD 370 via dedicated data paths, and to access all of the other accessible memory space via the AXI / APB bus, for example according to the DMA controller 265. CIMU 300 performs the BPBS matrix-vector multiplication described above. Further details of these and other embodiments are provided below.

[0050] Thus, in various embodiments, the CIMA operates in a bit-serial-bit-parallel (BPBS) manner to receive vector information, perform a matrix-vector product, and provide a digitized output signal (i.e., Y=AX) that may be further processed by another arithmetic function as necessary to provide a composite matrix-vector product function. Generally speaking, embodiments described herein provide an in-memory computing architecture including: a reshaping buffer configured to reshape a sequence of received data words to form a massively parallel bit-wise input signal; a compute-in-memory (CIM) array of bit cells configured to receive said massively parallel bit-wise input signal via a first CIM array dimension and one or more stored signals via a second CIM array dimension, wherein each of a plurality of bit cells associated with a common stored signal forms a respective CIM channel configured to provide a respective output signal; an analog-to-digital converter (ADC) circuit configured to provide a sequence of multi-bit output words by processing the plurality of CIM channel output signals; a control circuit configured to cause the CIM array to perform multi-bit computing operations on said input and stored signals using single-bit internal circuitry and signals; and a near memory computing path configured to provide a sequence of multi-bit output words as a computing result.

[0051] Memory Map and Programming Model Because the CPU 210 is configured to directly access the IA BUFF 320 and the memory read / write buffer 340, these two memory spaces appear similar to the DMEM 230 in terms of latency and energy from the perspective of a user program, especially for structured data such as array / matrix data. In various embodiments, when the in-memory computing feature is not activated or is partially activated, the memory read / write buffer 340 and the CIMA 310 may be used as normal data memory.

[0052] FIG. 4 illustrates a high-level block diagram of an input activation vector reshaping buffer (IA BUFF) 320 suitable for use in the architecture of FIG. 2, according to one embodiment. The illustrated IA BUFF 320 supports input activation vectors having element precision from 1 bit to 8 bits, with other precisions being considered in various embodiments. According to the bit-serial flow mechanism described herein, specific bits of all elements in the input activation vector are broadcast at once to the CIMA 310 for a matrix-vector multiplication operation. However, the high parallelism of this operation requires that elements of the high-dimensional input activation vector be provided with maximum bandwidth and minimum energy, otherwise the throughput and energy efficiency advantages of in-memory computing would not be utilized. To achieve this, the input activation reshaping buffer (IA BUFF) 320 may be configured such that in-memory computing can be incorporated into the 32-bit (or other bit width) architecture of a microprocessor, whereby the corresponding hardware for 32-bit data transfer is fully utilized for the highly parallel internal organization of in-memory computing.

[0053] Referring to FIG. 4, the IA BUFF 320 receives a 32-bit input signal that may contain input vector elements with a bit precision of 1 to 8 bits. The 32-bit input signal is first stored in 4×8-b registers 410, totaling 24 (here shown as registers 410-0 to 410-23). ​​These registers 410 feed their contents to 8 register files (shown as register files 420-0 to 420-8) each with 96 columns, so that an input vector with a dimension of up to 3×3×256=2304 is arranged with its elements in parallel columns. This is done in the case of 8-b input elements, where the 24 4×8-b registers 410 provide 96 parallel outputs to one of the register files 420, and in the case of 1-b input elements, the 24 4×8-b registers 410 provide 1536 parallel outputs to all eight register files 420 (or intermediate configurations for other bit precisions). The height of each register file column is 2x4x8-b, allowing each input vector (up to 8-bit element precision) to be stored in 4 segments, enabling double buffering when all input vector elements are loaded. On the other hand, when only about 1 / 3 of the input vector elements are loaded (i.e., CNN with a stride of 1), one of each 4 register file columns acts as a buffer, allowing data from the 3 columns to be forward propagated to the CIMU for operation.

[0054] Thereby, of the 96 columns output by each register file 420, only 72 are selected by the respective circular barrel shift interface 430, giving a total of 576 outputs for eight register files 420 at a time. These outputs correspond to one of the four input vector segments stored in that register file. Thus, four cycles are required to load all of the input vector elements into the sparsity / AND logic controller 330 in 1-b registers.

[0055] To exploit sparsity in the input activation vector, a mask bit is generated for each data element while the CPU 210 or DMA 260 writes to the reshaping buffer 320. The masked input activations prevent charge-based computation operations in the CIMA 310, thereby saving computational energy. This mask vector is also stored in an SRAM block and is organized similarly to the input activation vector, but with a 1-bit representation.

[0056] The 4-to-3 barrel shifter 430 is used to support VGG-style (3x3 filter) CNN operations: only one out of three of the input activation vectors need to be updated when moving to the next filtering operation (convolutional reuse), thereby saving energy and improving throughput.

[0057] Figure 5 illustrates a high-level block diagram of a CIMA read / write buffer 340 suitable for use in the architecture of Figure 2, according to one embodiment. The illustrated CIMA read / write buffer 340 is organized as, for example, a 768-bit wide static random access memory (SRAM) block 510, while the illustrated CPU's word width is 32 bits in this example, and the read / write buffer 340 is used to interface therebetween.

[0058] The read / write buffer 340 as shown includes a 768-bit write register 511 and a 768-bit read register 512. The read / write buffer 340 generally acts like a cache for wide SRAM blocks in the CIMA 310, but with some differences. For example, the read / write buffer 340 writes back to the CIMA 310 only if the CPU 210 writes to a different row, but a read of a different row does not trigger a writeback. If the read address matches the tag of the write register, then instead of reading from the CIMA 310, modified bytes (shown as dirty bits) in the write register 511 are bypassed to the read register 512.

[0059] Accumulation line analog-to-digital converters (ADCs). Each accumulation line from the CIMA310 has an 8-bit SAR ADC, matching the pitch of the in-memory computing channels. To save area, the finite-state machine (FSM) that controls the bit cycling of the SAR ADCs is shared among the 64 ADCs required in each in-memory computing tile. The FSM control logic consists of 8+2 shift registers that generate pulses to reset, sample, and cycle through 8-bit decision stages. The shift register pulses are broadcast to the 64 ADCs, where they are buffered locally and used to trigger local comparator decisions, store the corresponding bit decisions in the local ADC code registers, and then trigger the next capacitor-DAC configuration. Precision metal-oxide-metal (MOM) capacitance may be used to allow for a small capacitor array for each ADC.

[0060] Figure 6 illustrates a high-level block diagram of a near memory data path (NMD) module 600 suitable for use in the architecture of Figure 2, according to one embodiment, although digital near memory computing having other features may be used. The NMD module 600 illustrated in Figure 6 illustrates a digital computation data path after the ADC output that supports multi-bit matrix multiplication with the BPBS method.

[0061] In a specific embodiment, the 256 ADC outputs are organized into groups of 8 for digital computation flow, allowing support for matrix element configurations up to 8 bits. Thus, the NMD module 600 includes 32 identical NMD units. Each NMD unit consists of eight ADC outputs 610 and multiplexers 610 / 620 to select among the corresponding biases 621, multiplicands 622 / 623, shift counts 624 and accumulation registers; an adder 631 with 8-bit unsigned input and 9-bit signed input to subtract the global bias and mask count; a signed adder 632 to compute local biases for neural network tasks; a fixed-point multiplexer 633 to perform scaling; a barrel shifter 634 to compute the exponent of the multiplicand and perform shifts for different bits in the weight elements; a 32-bit signed adder 635 to perform accumulation; eight 32-bit accumulation registers 640 to support weights with 1, 2, 4, and 8 bit configurations; and a ReLU unit 650 for neural network applications.

[0062] Figure 7 illustrates a high-level block diagram of a direct memory access (DMA) module 700 suitable for use in the architecture of Figure 2, according to one embodiment. The illustrated DMA module 700 includes, for example, two channels supporting simultaneous data transfers with different hardware resources, and five independent data paths with each of the DMEM, IA BUFF, CIMU R / W BUFF, NMD result, and AXI4 buses.

[0063] Bit-parallel / bit-serial (BPBS) matrix vector multiplication Multi-bit MVM JPEG0007587823000001.jpg1126 The BPBS method for is shown in Figure 8. A is the matrix element a m,n corresponds to the number of bits used for x is the input vector element x nwhere N corresponds to the number of bits used for n are the mask bits used for sparsity and dimensionality control). m,n The bits of x are mapped to parallel CIMA columns. n Multiple bits of are input in series. Thus, multi-bit multiplication and accumulation can be achieved via in-memory computing by either bitwise XNOR or bitwise AND, both of which are supported by the multiplication bit cell (M-BC) of the present embodiment. Notably, bitwise AND differs from bitwise XNOR in that its output must remain low when the input vector element bit is low. The M-BC of the present embodiment involves inputting the input vector element bits (one at a time) as differential signals. The M-BC implements XNOR, except that each logic "1" output in the truth table is calculated by multiplying V by the true and complement signals of the input vector element bit, respectively. DD Therefore, AND is easily achieved by simply masking the complement signal so that the output remains low, producing a truth table corresponding to AND.

[0064] Bitwise AND can support multi-bit matrix and standard two's complement representation for input vector elements, including appropriately applying a negative sign to the column operation that corresponds to the most significant bit (MSB) in the digital domain after the ADC before adding the digitized output to the output of the other column operation.

[0065] Bitwise XNOR requires a slight modification of number representation: the constituent bits map to +1 / -1 instead of 1 / 0, and two bits with equal LSB weighting are required to properly represent zero. This is done as follows: First, each B-bit operand (in standard two's complement representation) is decomposed into a B+1 bit signed integer. For example, y is decomposed into B+1 positive / negative 1 bit JPEG0007587823000002.jpg1150 Break it down into JPEG0007587823000003.jpg1151 get.

[0066] By mapping 1 / 0 value bits to +1 / -1 mathematical values, bitwise in-memory computing multiplication may be realized via a logical XNOR operation. Thus, an M-BC that performs a logical XNOR with the difference signals for the input vector elements can enable signed multi-bit multiplication by bit-weighting and adding the digitized outputs from the column operations.

[0067] While AND-based M-BC multiplication and XNOR-based M-BC multiplication represent two options, other options are possible by using appropriate number representations with the logical operations possible in M-BC. Such alternatives are beneficial. For example, XNOR-based M-BC multiplication is suitable for binarized (1-b) operations, while AND-based M-BC multiplication allows more standard number representations for easier integration within digital architectures. Furthermore, the two approaches produce slightly different signal-to-quantization-noise ratios (SQNR), and can therefore be selected based on the needs of the application.

[0068] Heterogeneous Computing Architectures and Interfaces Various embodiments described herein contemplate various aspects of charge domain in-memory computing where a bit cell (or multiplication bit cell: M-BC) drives an output voltage corresponding to the result of an operation onto a local capacitor. The capacitors from the in-memory computing channel (column) are then coupled to achieve accumulation via charge redistribution. As mentioned above, such capacitors may be formed using specific geometries that are very easy to replicate in VLSI or other processing, such as through wires that are coupled via an electric field simply because of their close proximity to each other. Thereby, the local bit cell formed as a capacitor stores a charge representing a one or zero, while locally adding up all of the charges of multiple capacitors or bit cells allows the implementation of the multiplication and accumulation / addition functions that are the underlying operations in matrix vector multiplication.

[0069] Advantageously, the various embodiments described above provide improved bit cell based architectures, computing engines, and platforms. Matrix vector multiplication is one operation that is not efficiently performed by standard digital processing or digital acceleration. Thus, performing this one type of computation-in-memory operation provides a significant advantage over existing digital designs. However, many other types of operations are performed using digital designs.

[0070] Various embodiments contemplate mechanisms for connecting / interfacing the above-described bit cell based architectures, computing engines, platforms, etc., to traditional digital computing architectures and platforms, e.g., to form heterogeneous computing architectures. In this manner, computational operations well suited to bit cell architecture processing (e.g., matrix vector processing) are processed as described above, while other computational operations well suited to traditional computer processing are processed via traditional computer architectures. That is, various embodiments provide a computing architecture that includes the highly parallel processing mechanisms described herein, which can be externally coupled to more traditional digital computing architectures by connecting to multiple interfaces. This allows the digital computing architecture to be directly and efficiently aligned with an in-memory computing architecture, allowing the two to be located in close proximity to minimize data transfer overhead between the two. For example, machine learning applications may include 80% to 90% matrix vector operations, while still performing 10% to 20% of other types of computations / operations. By combining the in-memory computing described herein with near-memory computing that is more traditional in architecture, the resulting system provides much greater configurability for performing many types of processing. Thus, various embodiments contemplate near-memory digital computing combined with the in-memory computing described herein.

[0071] The in-memory operations described herein are massively parallel, but single-bit operations. For example, often only one bit is stored in a bit cell; either a 1 or a 0. The signal driven into the bit cell is typically an input vector (i.e., each matrix element is multiplied by each vector element in a 2D vector multiplication operation). This vector element is placed into a signal that is also digital, and is only one bit so that the vector elements are also one bit.

[0072] Various embodiments expand matrices / vectors from 1-bit elements to multi-bit elements using a bit-parallel / bit-serial approach.

[0073] Figures 8A and 8B illustrate high level block diagrams of various embodiments of CIMA channel digitization / weighting suitable for use in the architecture of Figure 2. In particular, Figure 8A illustrates a digital binary weighting and summing embodiment similar to that described above with respect to various other figures, and Figure 8B illustrates an analog binary weighting and summing embodiment with modifications to various circuit elements to enable the use of a smaller number of analog to digital converters than the embodiment of Figure 8A and / or other embodiments described herein.

[0074] As discussed above, various embodiments contemplate a compute-in-memory (CIM) array of bit cells configured to receive massively parallel bit-wise input signals via a first CIM array dimension (e.g., a row of a 2D CIM array) and one or more stored signals via a second CIM array dimension (e.g., a column of a 2D CIM array), where each of a plurality of bit cells associated with a common stored signal (e.g., depicted as a column of bit cells) forms a respective CIM channel configured to provide a respective output signal. An analog-to-digital converter (ADC) circuit is configured to provide a sequence of multi-bit output words by processing the plurality of CIM channel output signals. A control circuit is configured to cause the CIM array to perform multi-bit computing operations on the input and stored signals using single-bit internal circuits and signals, such that an operatively engaged near memory computing path may be configured to provide a sequence of multi-bit output words as a result of the operation.

[0075] 8A, a digital binary weighting and summing embodiment for performing ADC circuit functions is illustrated. In particular, a two-dimensional CIMA 810A receives matrix input values ​​in a first (row) dimension (i.e., via a number of buffers 805) and vector input values ​​in a second (column) dimension, and the CIMA 810A operates in accordance with control circuitry and the like (not shown) to provide various channel output signals CH-OUT.

[0076] The ADC circuit of FIG. 8A provides, for each CIM channel, a respective ADC 760 configured to digitize the CIM channel output signal CH-OUT, and a respective shift register 865 configured to apply a respective binary weighting to the digitized CIM channel output signal CH-OUT to form a respective portion of a multi-bit output word 870.

[0077] 8B, an analog binary weighting and summing embodiment for performing ADC circuit functions is shown, in particular a two-dimensional CIMA 810B receives matrix input values ​​in a first (row) dimension (i.e., via a number of buffers 805) and vector input values ​​in a second (column) dimension, the CIMA 810B operating in accordance with control circuitry and the like (not shown) to provide various channel output signals CH-OUT.

[0078] The ADC circuit of FIG. 8B provides four controllable (or pre-configured) banks of switches 815-1, 815-2, etc. within the CIM ADC 810B which operate to couple and / or isolate capacitors formed therein to implement an analog binary weighting scheme for each of one or more subgroups of channels, each of which provides a single output signal such that only one ADC 860B is required to form each portion of a multi-bit output word by digitizing a weighted analog sum of the CIM channel output signals of a respective subset of the CIM channels.

[0079] Figure 9 illustrates a flow diagram of a method according to one embodiment. In particular, the method 900 of Figure 9 relates to various processing operations implemented by the architectures, systems, etc. described herein, where the input matrix / vector is expanded to be operated on in a bit-parallel / bit-serial approach.

[0080] In step 910, the matrix and vector data are loaded into the appropriate memory locations.

[0081] In step 920, each of the vector bits (MSB to LSB) is processed sequentially. In particular, the MSB of the vector is multiplied by the MSB of the matrix, the MSB of the vector is multiplied by the MSB-1 of the matrix, the MSB of the vector is multiplied by the MSB-2 of the matrix, and so on, until the MSB of the vector is multiplied by the LSB of the matrix. The resulting analog charge results are then digitized for each MSB to LSB vector product, and the resulting results are latched. This process is repeated from vector MSB-1, vector MSB-2, etc. through vector LSB until each of the vector MSB-LSB has been multiplied by each of the MSB-LSB elements of the matrix.

[0082] In step 930, the bits are shifted to apply the appropriate weightings and the results are added together. Note that in some embodiments where analog weighting is used, the shifting operation of step 930 is not necessary.

[0083] Various embodiments enable highly stable and robust operations to be performed in circuits used to store data in high density memories. Moreover, various embodiments advance the computing engines and platforms described herein by enabling higher densities for memory bit cell circuits. Density can be increased due to a more compact layout and also because of improved compatibility of the layout with the highly aggressive design rules (i.e., push rules) used for memory circuits. Various embodiments substantially improve the performance of processors for machine learning and other linear algebra.

[0084] Bit cell circuits usable within in-memory computing architectures are disclosed. The disclosed approach allows for highly stable / robust operations to be performed within circuits used to store data in high density memories. The disclosed approach for robust in-memory computing allows for higher density for memory bit cell circuits compared to known approaches. Density is increased due to a more compact layout and also because of improved compatibility of the layout with highly aggressive design rules (i.e., push rules) used for memory circuits. The disclosed devices can be fabricated using standard CMOS integrated circuit processing.

[0085] Memory accesses account for a major portion of the energy and latency in many computing workloads. In standard memories, memory energy and latency arise because raw data is accessed row by row, incurring communication costs to move the data from its storage point to a computation point outside the memory array. In-memory computing architectures, on the other hand, access many rows at once and amortize the communication costs by accessing computation results on many bits of data stored across rows.

[0086] While such amortization reduces energy and delay by a multiplier (i.e., roughly the number of rows accessed simultaneously), the most important issue is that the signal-to-noise ratio (SNR) of the computation is also reduced by a corresponding factor. This is because computations on a large number of bits typically increase the required dynamic range, but with it, the SNR is constrained within the limited swing of the existing bit lines of the memory. A large part of the computational noise, especially in in-memory architectures, is due to the deformation and nonlinearity of the computational operations performed by the bit cells. In standard memories, the bit cells provide an output current. This has made current-domain computation a natural choice for in-memory computing architectures, with the goal of minimizing changes to the high density of bit cells typically used.

[0087] However, the bit cell current is subject to high levels of variation and nonlinearity that affect the bit cell transistor. This limits the SNR of in-memory computing and therefore the scalability. Improvements are obtained according to various embodiments that use charge domain computation. Here, the computation output from the bit cell is stored as a charge on a capacitor. For example, various embodiments contemplate the use of metal finger capacitors on top of the bit cell transistor, such capacitors do not incur additional area and therefore allow the bit cell to maintain a dense structure. Important for the SNR of the computation, such capacitors also exhibit very good linearity and high stability in the presence of process and temperature variations. This has substantially increased the scalability of in-memory computing.

[0088] While metal finger capacitors can be placed on the bit cells, some circuit changes are required within the bit cells for switched capacitor charge domain operation. This invention focuses on circuits and layouts to enable high density bit cell operation of charge domain in static random access memories (SRAMs). In particular, circuits and layouts are described that improve compatibility with aggressive push design rules used for SRAM bit cells.

[0089] FIG. 10 illustrates a circuit diagram of a multiplication bit cell. The bit cell 900 of FIG. 9 performs operations such as storing, writing, and reading 1-b data, and also allows multiplication between the stored 1-b data and the 1-b IA / IAb signal (difference). Therefore, this structure is called a multiplication bit cell (M-BC). Note that the 1-b multiplication corresponds to a logical XNOR operation. To implement this, a PMOS transistor is added, coupled to the bit cell storage node and driven by the IA / IAb signal, as illustrated in FIG. 10.

[0090] FIG. 11 illustrates a circuit diagram of three M-BCs configured to perform an XNOR function. In particular, the operation of the circuit of FIG. 10 includes, for the three M-BCs, (1) unconditionally discharging the M-BC capacitor by asserting the capacitor shorting switch (TSHORT / TSHORTb) and the discharge NMOS transistor (PRE), (2) deasserting TSHORT / TSHORTb and driving IA / IAb to store the M-BC XNOR output on the local M-BC capacitor, and (3) deasserting IA / IAb and asserting TSHORT to accumulate the charge from all the XNOR results to provide a multiply-accumulate operation. Note that in addition to the eight M-BC transistors, two additional NMOS / PMOS transistors are required per M-BC for TSHORT / TSHORTb implementation. In previous designs, with three M-BCs, the above NMOS / PMOS transistors were placed outside each M-BC to allow efficient sharing of nodes.

[0091] 11A and 11B illustrate an example integrated circuit (IC) layout on an 8-transistor M-BC (FIG. 11A) next to an 8-transistor M-BC of a standard SRAM bit cell (FIG. 11B). The difference in PCB size and complexity between the two types of bit cells can be seen by inspection.

[0092] Figure 12 illustrates an example IC layout of a group of three M-BCs (with metal finger capacitors placed on them) and a TSHORT NMOS / PMOS. Note that the use of PMOS transistors for bit cell operation leads to a balanced use of NMOS and PMOS transistors in the M-BCs, which results in a significantly different IC layout than that of a standard six-transistor (6T) SRAM bit cell. This impacts the likelihood of push rule compliance.

[0093] Various embodiments contemplate a novel M-BC circuit that adds NMOS transistors for bit cell operations, thereby providing an IC layout that achieves both higher density and closer proximity as well as improved push rule compliance compared to standard 6T bit cells.

[0094] Figure 13 illustrates a circuit diagram of an M-BC according to one embodiment. In particular, the M-BC 1300 of Figure 13 implements a 1-b charge domain multiplication (XNOR) using NMOS transistors, where the NMOS inputs IA / IAb are low during unconditional discharge of the local capacitors and then driven differentially for operation.

[0095] FIG. 14 illustrates an example layout of the M-BC of FIG. 13. In particular, the layout of FIG. 14 contemplates compactly including NMOS / PMOS TSHORT switches within a single M-BC (metal finger capacitors are placed on the bit cells). In the layout 1400 of FIG. 14, signals WL, IA / IAb run horizontally while signals BL, Blab, PA, VDD, GND run vertically. This layout has roughly twice the area of ​​a standard 6T cell and takes advantage of the opportunity to share some nodes with surrounding M-BCs.

[0096] While the disclosed approach uses more area than standard memory bit cell circuits, the majority of the bit cells are demonstrated by the push rule, and extensions to this are similar to other structures that have successfully used the push rule. The disclosed approach substantially improves the performance of processors for machine learning and other linear algebra. Such improvements have been experimentally proven over conventional architectures, and the disclosed approach represents a substantial advancement over those architectures.

[0097] As mentioned above, arithmetic operations within a memory bit cell typically use voltage-to-charge conversion across a capacitor to provide a charge for its result. Thus, the bit cell circuitry includes appropriate switching of a local capacitor for a given bit cell, where the local capacitor is also appropriately coupled to other bit cell capacitors to produce an aggregate arithmetic result across the coupled bit cells.

[0098] Disclosed herein are charge injection robust bit cells and bit cell layouts for reconfigurable charge domain in-memory computing. The disclosed apparatus, particularly the bit cell circuit, can be used within in-memory computing architectures. The disclosed approach allows highly stable / robust operations as well as reconfigurable operations within circuits used to store data in high density memories. The disclosed approach allows for greater robustness and reconfigurability for in-memory computing than previous approaches. The disclosed apparatus may be fabricated using standard CMOS integrated circuit processing. The disclosed approach is believed to have significant utility for the semiconductor industry, as it can substantially improve the performance of processors for machine learning and other linear algebra.

[0099] The approach disclosed herein involves two novel aspects of the bit cell circuitry: (1) a configuration in which coupling between bit cell capacitors is achievable without the need for explicit switches (non-switched coupling structure) and (2) a physical layout in which coupled bit cells are interleaved with other coupled bit cells (interleaved layout).

[0100] A non-switched coupling structure refers to a bit cell that provides a result on one of the capacitor plates, where the coupling between the capacitors is achieved through the other capacitor plate. This is in contrast to a switched coupling structure, where the bit cell circuitry provides a result on the same capacitor plate that is ultimately coupled to the other capacitor, usually through a switch.

[0101] FIG. 15A illustrates a block diagram of a bit cell with a switched coupling structure, while FIG. 15B illustrates a block diagram of a bit cell with a non-switched coupling structure. In both cases, the coupling capacitor must first be reset (removing the charge on the capacitor), such as by shorting the output node to which the capacitor is coupled. An arithmetic operation f(.) is then performed locally in the bit cell. This is illustrated for two operands a and b, one operand stored in the bit cell and the other supplied externally from the bit cell periphery. In general, however, structures with more operands are possible. The arithmetic operation then drives a plate of a local capacitor that is coupled to the other capacitor, either through a switch on the sample plate (switched coupling structure) or without a switch on the other plate (non-switched coupling structure). Advantageously, the non-switched coupling structure of FIG. 15B avoids the need for coupling switches in the bit cells and, if implemented with MOSFETs, has the potential to reduce the effects of charge injection errors, such as allowing the switches to absorb / release variable amounts of charge (depending on voltage levels) by the MOSFETs, thereby slightly impairing the operation of the charge domain.

[0102] FIG. 16 illustrates a circuit diagram of a bit cell circuit having a non-switched coupling structure according to one embodiment. It should be noted that other variations of this circuit are possible within the context of the embodiments of the present disclosure. The bit cell 1600 of FIG. 16 allows for the implementation of either an XNOR or AND operation between the stored data W / Wb (in the six-transistor cross-coupled circuit formed by MN1-3 / MP1-2) and the input data IA / IAb. For example, for an XNOR operation, after reset, IA / IAb can be driven complementarily, resulting in the lower plate of the local capacitor being pulled up / down according to IA XNOR W. Meanwhile, for an AND operation, after reset, only IA is driven (IAb remains low), resulting in the lower plate of the local capacitor being pulled up / down according to IA AND W. Advantageously, this structure reduces the total capacitor switching energy due to the resulting series pull-up / pull-down charging structure between all of the coupled capacitors, and reduces the effects of switch charge injection errors due to the elimination of the coupled switches at the output nodes.

[0103] FIG. 17 illustrates a circuit diagram of a two-way interleaving of a bit cell layout according to one embodiment. In particular, the bit cell interleaved layout relates to a layout in which capacitors may be coupled together in two or more sets. Although FIG. 17 illustrates an example of a two-way interleaving case, higher interleaving is also contemplated in various embodiments. Furthermore, the capacitors may be placed on the side of the columns, may actually be placed on the bit cell transistors, and / or may be placed elsewhere adjacent to the bit cell transistors. An advantage of this structure is, among other things, improved configurability. That is, because the outputs are provided on two different nodes, the couplings A and B can be used to implement separate computations. Or, the couplings A and B can be used to implement a joint computation by appropriately combining the outputs on the different nodes, for example via appropriate peripheral circuitry.

[0104] It should be noted that the functions shown and described herein may be implemented by hardware or a combination of software and hardware, using, for example, a general purpose computer, one or more application specific integrated circuits (ASICs), or any other equivalent hardware. It is contemplated that some of the steps described herein may be implemented in hardware, for example as circuitry that cooperates with a processor to perform various method steps. Some of the functions / elements described herein may be implemented as computer program products, where computer instructions, when processed by a computing device, adapt the operation of the computing device such that the methods or techniques described herein are invoked or otherwise provided. Instructions for invoking the methods of the present invention may be stored in a tangible and non-transitory computer readable medium, such as a fixed or removable medium or memory, or may be stored in a memory within a computing device that operates according to the instructions.

[0105] Various modifications may be made to the systems, methods, apparatus, mechanisms, techniques, and portions thereof described herein with respect to the various figures, and such modifications are contemplated as being within the scope of the present invention. For example, while a particular order of steps or arrangement of functional elements is presented in various embodiments described herein, various other orders / arrangements of steps or functional elements may be utilized within the context of the various embodiments. Additionally, while modifications to the embodiments may be described individually, various embodiments may employ multiple modifications simultaneously or in sequence, combined modifications, and the like.

[0106] Although certain systems, devices, methodologies, mechanisms, etc. have been disclosed as described above, it will be apparent to one skilled in the art that many more variations than those described above are possible without departing from the novel concepts described herein. Thus, the novel subject matter is not limited except to the gist of the present disclosure. Furthermore, in interpreting this disclosure, all terms should be interpreted in the broadest possible manner consistent with the context. In particular, the terms "comprise" and "comprising" should be interpreted as referring to elements, components, or steps in a non-exclusive manner, indicating that a referenced element, component, or step may be present, utilized, or combined with other elements, components, or steps not expressly mentioned. Furthermore, the references described herein are also incorporated by reference in their entirety as if fully set forth herein.

[0107] Various embodiment aspects are set forth in the claims and / or the following numbered clauses.

[0108] 1. A bit cell circuit configuration comprising: a bit cell storage circuit coupled to at least one bit cell operation unit; and a bit cell capacitor coupled to the bit cell operation unit, the bit cell capacitor being further coupled to one or more additional capacitors without a switch between the bit cell capacitor and the additional capacitor.

[0109] 2. The bit cell circuit configuration of claim 1, wherein a cathode plate of the bit cell capacitor is coupled to the bit cell operational device.

[0110] 3. The bit cell circuit configuration of claim 1, wherein an anode plate of the bit cell capacitor is coupled to the additional capacitor.

[0111] 4. The bit cell circuitry of claim 1, wherein the bit cell arithmetic unit is configured to perform an arithmetic operation on two operands.

[0112] 5. A bit cell circuit configuration as illustrated by FIG.

[0113] 6. A bit cell circuit configuration as recited in claim 5, wherein said configuration enables implementation of an XNOR or AND operation between stored data and input data.

[0114] 7. An interleaved layout for at least two bit cell configurations of paragraph 1, wherein the bit cell capacitors are coupled together into at least two sets.

[0115] 8. The interleaved layout of claim 7, wherein the set of coupled bit cell capacitors is disposed above one or more bit cell transistors.

[0116] 9. A charge domain in-memory computing bit cell that drives one plate of a local capacitor and coupling to the other bit cell capacitor is realized on the other plate.

[0117] 10. A charge domain in-memory computing bit cell capable of implementing an XNOR or AND operation between stored data and input data.

[0118] 11. An interleaved layout for charge-domain in-memory computing bit cells, where the bit cell capacitors are combined into multiple distinct sets.

[0119] 12. An interleaved layout for charge domain in memory computing bit cells, where the bit cell capacitors are combined into multiple distinct sets, such that there are X sets for interleaving in the X direction, where X is an integer greater than 1.

[0120] 13. A layout for a charge domain-in-memory computing bit cell, in which the distinct sets of coupled capacitors are disposed above the bit cell transistors.

[0121] 14. A multiplication bit cell (M-BC) configured to perform a charge domain operation between the stored data of the bit cell and a 1-b input signal.

[0122] 15. The M-BC of claim 14, wherein one or more NMOS transistors are utilized to perform the charge domain operation.

[0123] 16. The M-BC of claim 14, further comprising a capacitor that is a metal structure disposed over the bit cell.

[0124] 17. The M-BC of claim 14, wherein the M-BC is configured to implement a logical operation.

[0125] 18. The M-BC according to paragraph 17, wherein the logical operations include an XNOR operation, a NAND operation, an AND operation, and other logical operations.

[0126] 19. The M-BC of claim 14, further comprising a layout as illustrated in FIG. 12.

[0127] 20. The M-BC of clause 14, further comprising an extended layout of 6T cells.

[0128] 21. The M-BC of claim 20, further comprising a transistor having a regular poly structure.

[0129] 22. The M-BC of paragraph 14, further comprising a layout as illustrated in FIG. 13.

[0130] While various embodiments incorporating the teachings of the present invention have been shown and described in detail herein, those skilled in the art can readily devise numerous other variations which still incorporate these teachings. Thus, while the foregoing is directed to various embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof.

Claims

1. 1. An in-memory computing architecture configured to multiply an input matrix X by an input vector A to generate an output matrix Y, comprising: a CIM array of bit cells configured to receive at least a portion of an input matrix X as a plurality of parallel bit-wise input signals via bit cell rows and to receive said input vector A as a plurality of bit-wise input signals via respective bit cell columns, each of a plurality of bit cells within each bit cell column being associated with a common stored signal, each bit cell column forming a respective CIM channel configured to provide a respective output signal; a control circuit configured to cause the CIM array to perform multi-bit computing operations on the input matrix X, the input vector A, and the stored signals using single-bit internal circuits and signals; Equipped with Each bit cell in the CIM array of bit cells comprises: a bit cell storage circuit coupled to the at least one bit cell processing unit; a bit cell capacitor coupled to the bit cell processing unit, the bit cell capacitor being further coupled to one or more additional capacitors corresponding to other bit cell capacitors in the bit cell string; 1. An in-memory computing architecture having a bit cell circuit configuration comprising:

2. 2. The in-memory computing architecture of claim 1, further comprising a reshaping buffer configured to reshape a sequence of received data words to form the parallel bit-wise input signal.

3. 2. The in-memory computing architecture of claim 1, further comprising: an analog-to-digital converter (ADC) circuit configured to process the plurality of CIM channel output signals to provide the sequence of multi-bit output words.

4. 4. The in-memory computing architecture of claim 3, wherein the analog-to-digital converter (ADC) circuitry includes, for each CIM channel, a respective ADC configured to digitize the CIM channel output signal, and a respective shift register configured to form a respective portion of a multi-bit output word by applying a respective binary weight to the digitized CIM channel output signal.

5. 4. The in-memory computing architecture of claim 3, wherein the analog-to-digital converter (ADC) circuitry includes a respective ADC for each of a plurality of subsets of the CIM channels configured to form a respective portion of a multi-bit output word by digitizing a weighted analog sum of the CIM channel output signals of each subset of CIM channels.

6. 2. The in-memory computing architecture of claim 1, further comprising a sparsity controller configured to mask zero-valued elements of the parallel bit-wise input signals such that the multi-bit computing operations avoid processing zero-valued elements of the parallel bit-wise input signals.

7. 2. The in-memory computing architecture of claim 1, wherein the input signals and the stored signals are combined with other existing signals, such as wordlines and bitlines, within the memory.

8. 2. The in-memory computing architecture of claim 1, wherein the input signals and the store signals are isolated within the memory from other existing signals such as wordlines and bitlines.

9. 4. The in-memory computing architecture of claim 3, wherein each ADC and respective stored signal forms an in-memory computing channel.

10. An in-memory computing architecture as described in claim 1, including a near-memory computing path configured to provide a sequence of multi-bit output words as a computing result.

11. the multi-bit computing operation of the CIM array includes bit-parallel / bit-serial (BPBS) computing; The bit-parallel computing comprises: Load different matrix element bits into each in-memory computing channel, implementing corresponding bit weighting by barrel shifting the digitized output from said computing channel using a respective barrel shifter; performing digital accumulation for all of said computing channels using respective accumulators to generate multi-bit matrix element operation results; Including, The bit-serial computing method includes: applying each bit of the vector elements individually to the loaded matrix element bits and storing each resulting digitized output; using a respective barrel shifter to barrel shift the stored digitized output associated with each vector element bit prior to digital storage with the stored digitized output corresponding to a subsequent input vector bit; 10. The in-memory computing architecture of claim 9, comprising:

12. 5. The in-memory computing architecture of claim 4, further comprising one or more configurable finite state machines (FSMs), each FSM configured to control bit cycling of a respective one of the multiple CIM channel ADCs.

13. 13. The in-memory computing architecture of claim 12, wherein the FSM is configured to control parallel computing hardware used by some or all of a plurality of in-memory computing channels.

14. 13. The in-memory computing architecture of claim 12, wherein the FSM controls operations according to software instructions loaded into a local memory.

15. 3. The in-memory computing architecture of claim 2, wherein the reshaping buffer is configured to convert an external digital word of a first precision into a high dimensional input vector.

16. 16. The in-memory computing architecture of claim 15, wherein the reshaping buffer is configured to provide bits of elements of the input vector in order and in parallel to the CIM array.

17. 3. The in-memory computing architecture of claim 2, wherein the reshaping buffer is configured to align input data to ensure desired utilization and throughput of in-memory computing operations.

18. 3. The in-memory architecture of claim 2, wherein the reshaping buffer is configured to enable reuse and shifting of input data according to a convolutional neural network operation.

19. 2. The in-memory architecture of claim 1, wherein each bit cell capacitor comprises a cathode plate and an anode plate, the cathode plate being coupled to the bit cell processing unit and the anode plate being coupled to a bit cell capacitor of at least one other bit cell.

20. The in-memory architecture of claim 1 , wherein the bit cells are configured to perform an arithmetic operation with two operands.

21. 2. The in-memory architecture of claim 1, wherein the bit cells enable the implementation of an XNOR or AND operation between stored data and input data.

22. 2. The in-memory architecture of claim 1, wherein the bit cells include an interleaved layout for at least two bit cells, with respective bit cell capacitors coupled together into at least two separate sets to enable corresponding separate operations.

23. 23. The in-memory architecture of claim 22, wherein the set of coupled bit cell capacitors are disposed above one or more bit cell transistors.

24. 23. The in-memory architecture of claim 22, wherein the bit cells include a charge domain in-memory computing bit cell that drives one plate of a respective local bit cell capacitor, and coupling with another bit cell capacitor is realized at the other plate of the respective local bit cell capacitor.

25. 25. The in-memory architecture of claim 24, wherein the in-memory computing bit cells of the charge domain are configured to implement an XNOR or AND operation between the stored data and the input data.

26. 23. The in-memory architecture of claim 22, wherein an interleaved layout of in-memory computing bit cells of the charge domain is provided such that the bit cell capacitors are coupled into a plurality of distinct sets configured to be selectively coupled via one or more peripheral switches.

27. 23. The in-memory architecture of claim 22, wherein the bit cell capacitors are coupled into different sets to provide an interleaved layout of the in-memory computing bit cells of the charge domain such that there are X sets for an X-direction interleave, where X is an integer greater than one.

28. 24. The in-memory architecture of claim 23, wherein a charge domain in-memory computing bit cell layout is provided such that different sets of coupled capacitors are located above the bit cell transistors.

29. The in-memory computing architecture of claim 10, wherein the near-memory computing path includes one or more of a digital barrel shifter, a multiplier, an accumulator, a look-up table, and a non-linear function element.

Citation Information

Patent Citations

  • Nonionic azo compound and dichroic dye composed of the same for use in liquid crystal

    JP1987000555A

  • Product sum calculation circuit and method therefor

    JP2005122466A

  • High-precision matrix-vector multiplication on a charge-mode array with embedded dynamic memory and stochastic method thereof

    US20050125477A1

  • Neural network circuit and learning method thereof

    WO2015001697A1

Cited By

  • Hardware / software co-compressed computing method and system for static random access memory computing-in-memory-based processing unit

    US12260906B2

  • Hardware / software co-compressed computing method and system for static random access memory computing-in-memory-based processing unit

    US20220366971A1