A scalable array architecture for in-memory computing.

Abstract

Various embodiments include systems, methods, architectures, mechanisms, and apparatus for providing programmable or pre-programmed in-memory computation (IMC) operations via an array of configurable IMC cores interconnected by a configurable on-chip network to support scalable execution and data flow of applications mapped to the IMC cores.

Description

[Technical Field]

[0001] Government support This invention was made with government support under Contract No. NRO000-19-C-0014 awarded by the U.S. Department of Defense. The government has certain rights in this invention.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Patent Application No. 62 / 970,309, filed February 5, 2020, which is incorporated herein by reference in its entirety.

[0003] The present disclosure relates generally to the field of in-memory computation and matrix-vector multiplication. [Background technology]

[0004] This section is intended to introduce the reader to various aspects of art that may be related to various aspects of the present invention, which are described and / or claimed below. This discussion is believed to be helpful in providing the reader with background information to facilitate a better understanding of the various aspects of the present invention. As such, it should be understood that these statements are to be read in this light, and not as admissions of prior art.

[0005] Deep learning inference based on neural networks (NNs) has been deployed in a wide range of applications, motivated by breakthrough performance in cognitive tasks. However, it has driven the growth of NN complexity (number of layers, number of channels) and diversity (network architecture, internal variables / representations), which requires hardware acceleration for energy efficiency and throughput, albeit via flexibly programmable architectures.

[0006] The dominant operation in NNs is typically matrix-vector multiplication (MVM), which involves high-dimensional matrices. This makes data storage and movement in the architecture a major challenge. However, MVM also presents a structured data flow that motivates accelerator architectures in which hardware is explicitly arranged in a corresponding two-dimensional array. Such architectures are often referred to as spatial architectures, employing systolic arrays in which processing engines (PEs) perform simple operations (multiplication, addition) and pass the output to neighboring PEs for further processing. Many variants have been reported based on different ways of mapping MVM computation and data flow and providing support for different computational optimizations (e.g., sparsity, model compression).

[0007] An alternative architectural approach that has recently gained attention is in-memory computing (IMC). IMC can also be viewed as a spatial architecture, where PEs are memory bitcells. IMC typically employs analog operations to fit computational functionality in constrained bitcell circuits (i.e., for area efficiency) while performing computations with maximum energy efficiency. Recent demonstrations of NN accelerators based on IMC simultaneously achieved roughly 10x energy efficiency (TOPS / W) and 10x computational density (TOPS / mm2) compared to optimized digital accelerators.

[0008] While such gains make IMC attractive, recent demonstrations have also revealed several significant challenges, primarily stemming from analog non-idealities (variations, non-linearities). First, most demonstrations have been limited to small scale (less than 128Kb). Second, the use of advanced CMOS nodes, where analog non-idealities are expected to worsen, has not been demonstrated. Third, integration into larger computing systems (architectures and software stacks) is limited due to the difficulty of specifying functional abstractions for such analog operations.

[0009] Some recent research has begun to explore system integration. For example, ISAs have been developed, providing interfaces for domain-specific languages, but application mapping has been limited to small inference models and hardware architectures (single bank). Meanwhile, functional specifications for IMC computation have been developed, but the analog computations required for highly parallel IMC across many rows have been avoided in favor of digital forms of IMC with reduced parallelism. Thus, analog nonidealities remain a major obstacle to exploiting the full potential of IMC in scaled-up architectures for practical neural networks. Summary of the Invention

[0010] Various deficiencies in the prior art are addressed by a system, method, architecture, mechanism, or apparatus that provides programmable or pre-programmed in-memory computation (IMC) operations via an array of configurable IMC cores interconnected by a configurable on-chip network to support scalable execution and data flow of applications mapped to the IMC cores.

[0011] For example, various embodiments provide an integrated in-memory compute (IMC) architecture that is configurable to support scalable execution and data flow of applications mapped to the IMC, the IMC architecture comprising an array of configurable IMC cores, such as in-memory compute units (CIMUs), implemented on a semiconductor substrate and comprising IMC hardware and, optionally, other hardware such as digital compute hardware, buffers, control blocks, configuration registers, digital-to-analog converters (DACs), analog-to-digital converters (ADCs), etc., as described in more detail below.

[0012] The array of configurable IMC cores / CIMUs are interconnected via an on-chip network or on-chip network including inter-CIMU network portions, and are configured to communicate input data and calculated data (e.g., activity values ​​in a neural network embodiment) to / from other CIMUs or other structures within or outside the CIMU array via respective configurable inter-CIMU network portions disposed therebetween, and to communicate operand data (e.g., weights in a neural network embodiment) to / from other CIMUs or other structures within or outside the CIMU array via respective configurable operand loading network portions disposed therebetween.

[0013] Generally speaking, each IMC core / CIMU has a configurable input buffer for receiving computational data from the inter-CIMU network and composing the received computational data into an input vector for matrix vector multiplication (MVM) processing by the CIMU to generate an output vector.

[0014] Some embodiments comprise a neural network (NN) accelerator with an array-based architecture, where multiple computational in-memory computation units (CIMUs) are arranged and interconnected using a highly flexible on-chip network, where the output of one CIMU may be connected or flowed to the input of another CIMU or to multiple other CIMUs, the output of many CIMUs may be connected to the input of one CIMU, the output of one CIMU may be connected to the output of another CIMU, etc. The on-chip network may be implemented as a single on-chip network, as multiple on-chip network sections, or as a combination of on-chip and off-chip network sections.

[0015] One embodiment provides an integrated in-memory computing (IMC) architecture comprising a plurality of configurable in-memory compute units (CIMUs) forming an array of CIMUs, and a configurable on-chip network for communicating input data to the array of CIMUs, communicating computed data between the CIMUs, and communicating output data from the array of CIMUs, the integrated IMC architecture being configurable to support scalable execution and data flow of applications mapped to the IMCs.

[0016] One embodiment provides a computer-implemented method for mapping an application to configurable in-memory compute (IMC) hardware in an integrated IMC architecture, the IMC hardware comprising a plurality of configurable in-memory compute units (CIMUs) forming an array of CIMUs; and a configurable on-chip network for communicating input data to the array of CIMUs, communicating computed data between the CIMUs, and communicating output data from the array of CIMUs. The method includes: allocating the IMC hardware according to the application computation using parallelism and pipelining to generate an IMC hardware assignment configured to provide high-throughput application computation; defining placement of the allocated IMC hardware at locations within the array of CIMUs in a manner that tends to minimize the distance between the IMC hardware that generates output data and the IMC hardware that processes the generated output data; and configuring an on-chip network to route data between the IMC hardware. This application may include a network topology. The various steps may be implemented according to the mapping techniques discussed throughout this application.

[0017] Additional objects, advantages, and novel features of the invention will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following, or may be learned by the practice of the invention. The objects and advantages of the invention may be realized and attained by means of the instrumentalities and combinations particularly pointed out in the appended claims.

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the general description of the invention given above and the detailed description of the embodiments given below, serve to explain the principles of the invention. [Brief explanation of the drawings]

[0019] [Figure 1A] 1 depicts a diagrammatic representation of a conventional memory access architecture and an in-memory computing (IMC) architecture that is useful for understanding the present embodiment. [Figure 1B] 1 depicts a diagrammatic representation of a conventional memory access architecture and an in-memory computing (IMC) architecture that is useful for understanding the present embodiment. [Figure 2A] 1 depicts a diagrammatic representation of a capacitor-based high SNR charge-domain SRAM IMC that is useful for understanding the present embodiment. [Figure 2B] 1 depicts a diagrammatic representation of a capacitor-based high SNR charge-domain SRAM IMC that is useful for understanding the present embodiment. [Figure 2C] 1 depicts a diagrammatic representation of a capacitor-based high SNR charge-domain SRAM IMC that is useful for understanding the present embodiment. [Figure 3A] 1 shows a schematic depiction of a 3-bit binary input vector and matrix elements. [Figure 3B] 1 depicts an image of an implemented heterogeneous microprocessor chip, including a programmable heterogeneous architecture as well as the integration of software level interfaces. [Figure 4A] 1 illustrates a circuit diagram of an analog input voltage bitcell suitable for use in various embodiments. [Figure 4B] 4B illustrates a circuit diagram of a multi-level driver suitable for providing an analog input voltage to the analog input bitcell of FIG. 4A. [Figure 5]This is a graphical depiction of layer expansion by mapping multiple neural network layers to effectively form a pipeline. [Figure 6] 10 is a graphical depiction of pixel-level pipelining with input buffering of feature map rows. [Figure 7] 1 is a graphical depiction of replication for throughput matching in pixel-level pipelining. [Figure 8A] 1 depicts a graphical representation of row underutilization and mechanisms for addressing row underutilization that are useful in understanding various embodiments. [Figure 8B] 1 depicts a graphical representation of row underutilization and mechanisms for addressing row underutilization that are useful in understanding various embodiments. [Figure 8C] 1 depicts a graphical representation of row underutilization and mechanisms for addressing row underutilization that are useful in understanding various embodiments. [Figure 9] 10 is a graphical depiction of a sample of operations enabled by CIMU configurability via the software instruction library. [Figure 10] 1 is a graphical depiction of architectural support for spatial mapping within an application layer, such as the NN layer. [Figure 11] 10 is a graphical depiction of how NN filters are mapped to IMC banks, each having a dimension of N rows and M columns, by loading filter weights into memory as matrix elements, applying input activation values ​​as input vector elements, and calculating output preactivations as output vector elements. [Figure 12] 1 depicts a block diagram illustrating exemplary architecture support elements associated with layers and IMC banks for BPBS deployment. [Figure 13] 1 depicts a block diagram illustrating an exemplary near-memory computation SIMD engine. [Figure 14]1 depicts a graphical representation of an exemplary LSTM layer mapping function utilizing cross-element near-memory computation. [Figure 15] A graphical illustration of the mapping of a BERT layer using generated data as a loaded matrix. [Figure 16] 1 depicts a high-level block diagram of an IMC-based scalable NN accelerator architecture, according to some embodiments. [Figure 17] 17 depicts a high-level block diagram of a CIMU microarchitecture having a 1152×256 IMC bank suitable for use in the architecture of FIG. 16. [Figure 18] 1 depicts a high-level block diagram of a segment for obtaining input from a CIMU. [Figure 19] 1 depicts a high-level block diagram of a segment for providing output to a CIMU. [Figure 20] 1 depicts a high-level block diagram of an exemplary switch block for selecting which inputs are routed to which outputs. [Figure 21A] 1 illustrates a layout diagram of a CIMU architecture according to an embodiment implemented in 16 nm CMOS technology. [Figure 21B] 21B depicts a full-chip layout diagram consisting of a 4×4 tiling of CIMU as provided in FIG. 21A. [Figure 22] 10 is a graphical depiction of three stages of mapping software flow to an architecture, where, as an example, the NN mapping flow is mapped to an 8x8 array of CIMUs. [Figure 23A] 1 depicts a sample layout of layers from a pipeline segment. [Figure 23B] 1 depicts sample routing from a pipeline segment. [Figure 24]1 depicts a high-level block diagram of a computing device suitable for use in performing functions according to various embodiments. [Figure 25] 1 depicts a typical structure of an in-memory computing architecture. [Figure 26] 1 depicts a high-level block diagram of an exemplary architecture according to an embodiment. [Figure 27] 27 depicts a high-level block diagram of an exemplary in-memory computation unit (CIMU) suitable for use in the architecture of FIG. 26. [Figure 28] 3 depicts a high-level block diagram of an Input Activity Vector Reshaping Buffer (IA BUFF) suitable for use in the architecture of FIG. 2, according to an embodiment. [Figure 29] 27 depicts a high-level block diagram of a CIMA read / write buffer suitable for use in the architecture of FIG. 26, according to an embodiment. [Figure 30] 27 depicts a high-level block diagram of a near memory datapath (NMD) module suitable for use in the architecture of FIG. 26, according to an embodiment. [Figure 31] 27 depicts a high-level block diagram of a direct memory access (DMA) module suitable for use in the architecture of FIG. 26, according to an embodiment. [Figure 32A] 27 depicts a high-level block diagram of different embodiments of CIMA channel digitization / weighting suitable for use in the architecture of FIG. 26. [Figure 32B] 27 depicts a high-level block diagram of different embodiments of CIMA channel digitization / weighting suitable for use in the architecture of FIG. 26. [Figure 33] 1 depicts a flow diagram of a method according to an embodiment. [Figure 34] 1 depicts a flow diagram of a method according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] It should be understood that the accompanying drawings are not necessarily to scale and present somewhat simplified representations of various features illustrating the underlying principles of the present invention. Specific design features of the sequences of operations disclosed herein, including, for example, the specific dimensions, orientations, locations, and shapes of the various illustrated components, will be determined in part by the specific intended application and use environment. Certain features of the illustrated embodiments may be enlarged or distorted relative to other features to facilitate visualization and clear understanding. In particular, thin features may be thickened, for example, for clarity or illustration.

[0021] Before the present invention is described in further detail, it is to be understood that this invention is not limited to particular embodiments described, as such may, of course, vary. It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only, and is not intended to be limiting, since the scope of the present invention will be limited only by the appended claims.

[0022] Where a range of values ​​is provided, unless the context clearly dictates otherwise, it is understood that each intervening value, to the tenth of the unit of the lower limit, between the upper and lower limit of that range, and any other stated or intervening value in that stated range, is encompassed within the invention. The upper and lower limits of these smaller ranges may independently be included in the smaller ranges, subject to any specifically excluded limit in the stated range. Where the stated range includes one or both of the limits, ranges excluding either or both of those included limits are also included in the invention.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Although any methods and materials similar or equivalent to those described herein can also be used in the practice or testing of the present invention, a limited number of exemplary methods and materials are described herein. Please note that as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise.

[0024] The following description and drawings merely illustrate the principles of the present invention. It will thus be understood that those skilled in the art will be able to devise various arrangements, not explicitly described or shown herein, that embody the principles of the present invention and are within its scope. Furthermore, all examples recited herein are expressly intended for educational purposes only, primarily to help the reader understand the principles of the present invention and the concepts contributed by the inventors to further advance the art, and should not be construed as being limited to such specifically recited examples and conditions. Additionally, as used herein, the term "or" refers to a non-exclusive or (e.g., "or otherwise" or "or alternatively") unless otherwise indicated. Furthermore, the various embodiments described herein are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments.

[0025] Many of the innovative teachings of the present application are described herein with particular reference to preferred exemplary embodiments. However, it should be understood that this class of embodiments provides only a few examples of the many advantageous uses of the innovative teachings herein. In general, statements made in the specification of the present application do not necessarily limit any of the various claimed inventions. Moreover, some statements may apply to some inventive features but not to other features. Those skilled in the art, informed by the teachings herein, will recognize that the present invention is applicable to a variety of other technical areas or embodiments.

[0026] Various embodiments described herein are primarily directed to systems, methods, architectures, mechanisms, or apparatuses that provide programmable or pre-programmed in-memory computing (IMC) operations, as well as scalable dataflow architectures configured for in-memory computing.

[0027] For example, various embodiments provide an integrated in-memory computing (IMC) architecture that is configurable to support scalable execution and data flow of applications mapped to the IMC, the IMC architecture comprising an array of configurable IMC cores implemented on a semiconductor substrate and comprising IMC hardware and, optionally, other hardware such as digital computing hardware, buffers, control blocks, configuration registers, digital-to-analog converters (DACs), analog-to-digital converters (ADCs), etc.

[0028] The array of configurable IMC cores / CIMUs are interconnected via an on-chip network including inter-CIMU network portions and are configured to communicate input data and calculated data (e.g., activity values ​​in a neural network embodiment) to / from other CIMUs or other structures within or outside the CIMU array via respective configurable inter-CIMU network portions disposed therebetween, and to communicate operand data (e.g., weights in a neural network embodiment) to / from other CIMUs or other structures within or outside the CIMU array via respective configurable operand loading network portions disposed therebetween.

[0029] Generally speaking, each IMC core / CIMU has a configurable input buffer for receiving computational data from the inter-CIMU network and composing the received computational data into an input vector for matrix vector multiplication (MVM) processing by the CIMU to generate an output vector.

[0030] Additional embodiments described below are directed to scalable dataflow architectures for in-memory computing suitable for use independently of or in combination with the above-described embodiments.

[0031] Various embodiments address analog non-idealities by moving to charge-domain operations, where multiplication is digital but accumulation is analog and achieved by shorting together charge from capacitors localized in bit cells. These capacitors rely on geometric parameters that are well-controlled in advanced CMOS technologies, thus enabling much greater linearity and smaller variations (e.g., process, temperature) than semiconductor devices, e.g., transistors, resistive memory. This enables groundbreaking scales (e.g., 2.4 Mb) of single-shot, fully parallel IMC banks, as well as integration into larger computing systems (e.g., heterogeneous programmable architectures, software libraries), demonstrating practical neural networks (e.g., 10 layers).

[0032] Improvements to these embodiments address the architectural scaling up of IMC banks that is required to maintain high energy efficiency and throughput when running state-of-the-art NNs. These improvements adopt the proven approach of charge-domain IMC to develop an architecture and associated mapping approach for scaling up IMC while maintaining such efficiency and throughput.

[0033] Fundamental Trade-Offs in IMC IMC derives energy efficiency and throughput gains by performing analog computations and amortizing raw data movement into computational result movement, which leads to fundamental tradeoffs that ultimately shape the challenges of scaling up the architecture and application mapping.

[0034] Figure 1 depicts a diagrammatic representation of a conventional memory access architecture and an in-memory computation (IMC) architecture that is useful for understanding the present embodiment. In particular, the diagrammatic representation of Figure 1 illustrates the trade-offs by first comparing IMC (Figure 1B) with a conventional (digital) memory access architecture that separates memory and computation (Figure 1A), and then extending the insight to a comparison with a spatial digital architecture.

[0035]

number

number

number

number

[0036] However, the key trade-off is that while conventional architectures access single-bit data with BL / BLb, IMC

number

number

number

[0037] Digital space architectures reduce memory accesses and data movement by loading operands into PEs and exploiting opportunities for data reuse and short-range communication (i.e., between PEs). Typically, the computational cost of multiply-accumulate (MAC) operations dominates. IMC again introduces a trade-off between energy efficiency and throughput versus SNR. In this case, analog operations enable efficient MAC operations but also necessitate the need for subsequent analog-to-digital conversion (ADC). On the one hand, a large number of analog MAC operations (i.e., high row parallelism) amortizes the ADC overhead; on the other hand, more MAC operations increase the analog dynamic range and degrade the SNR.

[0038] Energy efficiency and throughput versus SNR tradeoffs have imposed major limitations on the scaling and integration of IMC in computing systems. Scaling ultimately results in severely compromised computational precision, limiting the energy / throughput gains that can be derived from row parallelism. Regarding integration in computing systems, noisy computations limit the ability to create robust abstractions needed for architecture design and interfacing to software. Previous efforts at integration in computing systems have required limiting row parallelism to four or two rows. As described below, charge-domain analog computing overcomes this, leading to both a substantial increase in row parallelism (4608 rows) and integration into heterogeneous architectures. However, while such high levels of row parallelism are advantageous for energy efficiency and throughput, they also limit the hardware granularity for flexible mapping of neural networks, necessitating specialized strategies explored in this work.

[0039] High SNR SRAM-based charge domain IMC This previous work moves to charge-domain computation rather than current-domain computation, where the bitcell output signal is a current induced by modulating the internal device resistance. Here, the bitcell output signal is the charge stored in a capacitor. While resistance depends on material and device properties and tends to exhibit substantial process and temperature variations, especially at advanced nodes, capacitance depends on geometric properties and can be very well controlled in advanced CMOS technologies.

[0040] 2 depicts a diagrammatic representation of a capacitor-based high SNR charge-domain SRAM IMC that is useful for understanding the present embodiment. In particular, the diagrammatic representation of FIG. 2 shows a logical representation of the charge-domain computation (FIG. 2A), a schematic representation of a bit cell (FIG. 2B), and an image of a 2.4 Mb integrated circuit implementation (FIG. 2C).

[0041] Figure 2A illustrates an approach to charge-domain computation. Each bit cell takes binary input data xn / xbn and performs a multiplication with binary storage data a,n / a,m,n. Treating binary 0 / 1 data as -1 / +1, this becomes a digital XNOR operation. The binary output result is then stored as a charge on a local capacitor. Accumulation is then implemented by shorting together the charges from all bit cell capacitors in the column to produce an analog output ym. Digital binary multiplication avoids analog noise sources and ensures perfect linearity (the two levels fit perfectly to the line), while capacitor-based charge accumulation avoids noise and also ensures high linearity (an inherent property of capacitors) due to excellent matching and temperature stability.

[0042] Figure 2B illustrates an SRAM-based bitcell circuit. In addition to the standard six transistors, two additional PMOS transistors are employed for XNOR-conditional capacitor charging, and two additional NMOS / PMOS transistors are employed outside the bitcell for charge accumulation (a single additional NMOS transistor is required for the entire column to pre-discharge all capacitors after accumulation). While the extra bitcell transistors impose a reported 80% area overhead, the local capacitors impose no area overhead because they are laid out using metal wiring above the bitcell. The dominant source of capacitor nonideality may be mismatch, allowing for row parallelism exceeding 100k before computational noise becomes comparable to minimal analog signal isolation. This enables the largest scale (2.4 Mb) previously reported for an IMC bank, overcoming the significant limitations of the SNR tradeoff that previously limited IMC (Figure 2C).

[0043] Charge domain IMC operations involve binary input vector and matrix elements, but extend it to multi-bit elements.

[0044] FIG. 3A shows a schematic representation of a 3-bit binary input vector and matrix elements. This is achieved through bit-parallel / bit-serial (BPBS) computation. Matrix element bits are mapped to parallel columns, while input vector elements are provided serially. Each column computation is then digitized using an 8-bit ADC selected to balance energy and area overhead. The digitized column outputs are finally summed after applying appropriate bit weighting (bit shifting) in the digital domain. This approach supports both two's complement number representations and special number representations optimized for bitwise XNOR computations.

[0045] Because the analog dynamic range of the column calculations can be larger than the dynamic range supported by an 8-bit ADC (256 levels), BPBS calculations result in calculation rounding that differs from standard integer calculations. However, accurate charge-domain operations on both the IMC columns and the ADC allow for robust modeling of rounding effects within the architectural and software abstractions. Figure 3B depicts an image of the realized heterogeneous microprocessor chip, including the integration of a programmable heterogeneous architecture and software-level interfaces. The current research extends this technology by developing a heterogeneous IMC architecture driven by application mapping for efficient and scalable execution. As described, we utilize a BPBS approach to overcome the hardware granularity constraints arising from the fundamental need for high row parallelism for energy efficiency and throughput in IMC.

[0046] FIG. 4A illustrates a circuit diagram of an analog input voltage bitcell suitable for use in various embodiments. The analog input voltage bitcell design of FIG. 4A can be used in place of the digital input (digital input voltage level) bitcell design illustrated above with respect to FIG. 2B. The bitcell design of FIG. 4A is configured to allow multiple voltage levels to be applied to input vector elements, rather than two digital voltage levels (e.g., VDD and GND). In various embodiments, use of the bitcell design of FIG. 4A allows for a reduction in the number of BPBS cycles, thereby correspondingly benefiting throughput and energy. Furthermore, by providing multiple voltage levels (e.g., x0, x1, x2, x3, and xb0, xb1, xb2, xb3) from dedicated power supplies, additional energy reductions are achieved, such as due to the use of lower voltage levels.

[0047] The illustrated bitcell circuit of FIG. 4A is depicted as having a switch-free coupling structure according to an embodiment. Note that other variations of this circuit are possible within the context of the disclosed embodiment. The bitcell circuit allows for the implementation of either an XNOR or AND operation between the stored data W / Wb (in the six-transistor cross-coupled circuit formed by MN1-3 / MP1-2) and the input data IA / IAb. For example, for an XNOR operation, after reset, IA / IAb can be driven in a complementary manner to pull up / pull down the bottom plates of the local capacitors according to IA XNOR W. Meanwhile, for an AND operation, after reset, only IA can be driven (and IAB kept low) to pull up / pull down the bottom plates of the local capacitors according to IA AND W. Advantageously, this structure allows for a reduction in the total capacitor switching energy due to the resulting series pull-up / pull-down charging structure between all coupling capacitors, as well as a reduction in the impact of switch charge injection errors due to the elimination of coupling switches at the output node.

[0048] Multi-Value Driver FIG. 4B illustrates a circuit diagram of a multi-level driver suitable for providing an analog input voltage to the analog input bitcell of FIG. 4A. Note that while the multi-level driver 1000 of FIG. 4B is illustrated as providing eight levels of output voltage, any number of output voltage levels may actually be used to support the processing of any number of bits of an input vector element in each cycle. The actual voltage levels of the dedicated power supplies may be fixed or selected using off-chip control. As an example, this may be useful for configuring into the bitcell either an XNOR calculation, which is required when multiple bits of an input vector element are chosen to be +1 / -1, or an AND calculation, which is required when multiple bits of an input vector element are chosen to be 0 / 1, as in standard two's complement format. In this case, the XNOR calculation requires using x3, x2, x1, x0, xb0, xb1, xb2, and xb3 to uniformly cover the input voltage range from VDD to 0 V, while the AND calculation requires using x3, x2, x1, and x0 to uniformly cover the input voltage range from VDD to 0 V, and setting xb0, xb1, xb2, and xb3 to 0 V. Various embodiments may be modified as necessary to provide a multi-level driver in which dedicated power supplies supporting numeric formats for XNOR calculations, AND calculations, etc., may be configured with off-chip / external control.

[0049] Note that dedicated voltages can be easily provided because the current from each power supply is correspondingly reduced, allowing for a corresponding reduction in the power grid density of each power supply (thus eliminating the need for additional power grid wiring resources). One challenge in some applications may be the need for multi-level repeaters, such as when many IMC columns must be driven (i.e., a number of IMC columns driven beyond the capabilities of a single driver circuit). In this case, digital input vector bits may be routed throughout the IMC array in addition to the analog driver / repeater outputs. Therefore, the number of levels should be selected based on routing resource availability.

[0050] In various embodiments, a bitcell is depicted in which a 1-bit input operand is represented by one of two values: binary zero (GND) and binary 1 (VDD). This operand is multiplied by another 1-bit value in the bitcell, which results in the storage of one of these two voltage levels in the sampling capacitor associated with that bitcell. When all of the capacitors in the column containing that bitcell are connected together to collect their stored values ​​(i.e., the charge stored in each capacitor), the resulting accumulated charge provides a voltage level representing the accumulation of all multiplication results for each bitcell in the column of bitcells.

[0051] Various embodiments contemplate the use of bit cells in which n-bit operands are used and the voltage level representing the n-bit operand necessarily includes one of n different voltage levels. For example, a 3-bit operand can be represented by eight different voltage levels. When the operand is multiplied in the bit cell, the resulting charge applied to the storage capacitors is such that n different voltage levels can exist during the accumulation phase (short-circuiting of the columns of capacitors). In this manner, a more precise and flexible system is provided. Therefore, the multi-value driver of FIG. 4 is used in various embodiments to provide such precision / flexibility. Specifically, in response to an n-bit operand, one of n voltage levels is selected and coupled to the bit cell for processing. Thus, multi-value input vector element signaling is provided by a multi-value driver employing dedicated voltage supplies that are selected by decoding multiple bits of the operand or input vector element.

[0052] The Challenges of Scalable IMC IMC presents three notable challenges to scalable mapping of NNs, stemming from its fundamental structure and tradeoffs: (1) matrix loading costs, (2) the inherent coupling between data storage and computational resources, and (3) large column dimensionality due to row parallelism, each of which is discussed below. This discussion is informed by Table I (which illustratively illustrates some of the IMC challenges of scalable application mapping for CNN benchmarks) and Algorithm 1 (which illustrates exemplary pseudocode for the execution loop in a typical CNN) that provide application context using a common convolutional NN (CNN) benchmark with 8-bit precision (the first layer was excluded from the analysis due to its characteristically low input channels). [Table 1]

number

[0053] Matrix Loading Costs. As noted above with regard to the basic tradeoffs, IMC reduces memory read and computation costs (energy, latency), but IMC does not reduce memory write costs. This can substantially degrade the overall gains in the execution of the complete application. A common approach in reported demonstrations is to load matrix data and keep it statically in memory. However, this becomes infeasible for practical-scale applications both in terms of the amount of storage required, exemplified by the large number of model parameters in the first row of Table I, and the replication required to ensure sufficient utilization, as described below.

[0054] Inherent coupling between data storage and computational resources. By combining memory and computation, IMC is constrained by allocating computational resources along with storage resources. The data involved in practical NNs is both large (first row of Table I), which can place substantial strain on storage resources, and also varies widely in terms of computational requirements. For example, the MAC operation with each weight is set by the number of pixels in the output feature map. As illustrated in the second row of Table I, this can vary significantly from layer to layer. This can lead to a significant loss of utilization unless a mapping strategy balances the operations.

[0055] Large column dimension for row parallelism. As discussed above with regard to the basic tradeoff, IMC derives its gain from a high level of row parallelism. However, a large column dimension to enable high row parallelism reduces the granularity of mapping matrix elements. As illustrated in the third row of Table I, the size of CNN filters varies widely both within and between applications. For layers with small filters, forming the filter weights into a matrix and mapping them to large IMC columns leads to low utilization and degradation of the gain from row parallelism.

[0056] To illustrate, two general strategies for mapping a CNN are considered next, showing how the above challenges manifest themselves. A CNN requires mapping nested loops, as shown in Algorithm 1. Mapping to hardware involves selecting a loop ordering and scheduling it in space (unrolling, replication) and time (blocking) for parallel hardware.

[0057] Static Mapping to IMC. Much of the current IMC research has primarily considered statically mapping the entire CNN to hardware (i.e., loops 2, 6-8) to avoid the relatively high matrix loading cost (challenge 1 above). As analyzed in Table II for the two approaches, this likely leads to very low utilization and / or very large hardware requirements. The first approach simply maps each weight to one IMC bit cell and further assumes that the IMC columns have different dimensionality to perfectly fit filters of various sizes throughout the layers (i.e., ignoring the utilization loss from challenge 3 above). This results in low utilization because each weight is allocated an equal amount of hardware, but the number of MAC operations, set by the number of pixels in the output feature map, varies widely (challenge 2 above). Alternatively, the second approach performs replication, mapping weights to multiple IMC bit cells according to the number of operations required. Again, ignoring the utilization loss from challenge 3 above, high utilization can be achieved, but a very large amount of IMC hardware is required. This may be practical for very small NNs, but is infeasible for NNs of any practical size. [Table 2]

[0058] Therefore, a more elaborate strategy for mapping CNN loops must be considered, involving non-static mapping of weights, thus incurring weight loading costs (the first challenge above). We note that this poses a further technical challenge when using NVM for IMC, as most NVM technologies face limitations in the number of write cycles.

[0059] Layer-by-layer mapping to IMC. A common approach taken in digital accelerators is to map the CNN layer-by-layer (i.e., unrolling loops 6-8). This provides an easy way to address the second challenge above, as the number of operations with each weight is equalized. However, the high level of parallelism often employed for high throughput in accelerators raises the need for replication to ensure high utilization. The main challenge here is the high weight loading cost (the first challenge above).

[0060] As an example, unrolling loops 6-8 and replicating filter weights across multiple PEs allows for parallel processing of input feature maps. However, here, each stored weight involves a smaller number of MAC operations, depending on the replication factor. Therefore, the relative total cost of weight loading (the first challenge above) becomes higher compared to MAC operations. While often feasible for digital architectures, this is problematic for IMC due to two reasons: (1) very high hardware density leads to significant weight replication to maintain utilization, thus significantly increasing matrix loading costs; and (2) as the cost of MAC operations decreases, the matrix loading costs become dominant, significantly mitigating the gains at the full application level.

[0061] Generally speaking, layer-by-layer mapping refers to a mapping where the next layer is not currently mapped to any CIMU where data needs to be buffered, whereas layer-expanded mapping refers to a mapping where the next layer is currently mapped to a CIMU where data will progress through the pipeline. Both layer-by-layer mapping and layer-expanded mapping are supported in various embodiments.

[0062] IMC Scalable Application Mapping Various embodiments contemplate an approach to scalable mapping that employs two ideas: (1) unrolling the layer loop (Loop 2) to achieve high utilization of parallel hardware, and (2) leveraging the emergence of two additional loops from the BPBS computation. These ideas are further described below.

[0063] Layer Unrolling. This approach involves unrolling loops 6-8, but instead of replicating to parallel hardware, it uses parallel hardware to map multiple NN layers, reducing the number of operations involved in each hardware unit and the weights loaded.

[0064] FIG. 5 graphically illustrates layer unfolding by mapping multiple NN layers to effectively form a pipeline. As described below, in various embodiments, filters within a NN layer are mapped to one or more physical IMC banks. If more IMC banks are required for a particular layer than can be physically supported, loop 5 and / or loop 6 are blocked, and the NN layer's filters are then mapped in time. This allows for scalability of both the NN input and NN output channels that can be supported. On the other hand, if more IMC banks are required to map the next layer than can be physically supported, loop 2 is blocked, and the layer is then mapped in time. This leads to NN layer pipeline segments, allowing for scalability of the NN depth that can be supported. However, such NN layer pipelines pose two challenges: latency and throughput.

[0065] Regarding latency, pipelines introduce delays in generating output feature maps. Due to the deep nature of NNs, some latency is inherently incurred. However, more traditional layer-by-layer mapping immediately utilizes all of the available hardware. Unrolling layer loops effectively delays hardware utilization in later layers. While such pipeline loading occurs only at startup, it becomes a significant concern given the emphasis on small-batch inference for a wide variety of latency-sensitive applications. Various embodiments mitigate latency using an approach referred to herein as pixel-level pipelining.

[0066] Figure 6 graphically depicts pixel-level pipelining with input buffering of feature map rows. Specifically, the goal of pixel-level pipelining is to start processing of subsequent layers as soon as possible. Feature map pixels represent the smallest granularity data structure processed through the pipeline. Therefore, pixels consisting of parallel output activation values ​​calculated from the hardware executing a given layer are immediately provided to the hardware executing the next layer. In CNNs, some pipeline latency beyond single-pixel latency must occur because the il × jl filter kernel requires a corresponding number of pixels to be available for calculation. This raises the need for local line buffers near the IMC to avoid the high cost of moving inter-layer activation values ​​to a global buffer. To ease the buffering complexity, various embodiments of the approach to pixel-level pipelining fill the input line buffer by receiving feature map pixels row by row, as illustrated in Figure 6.

[0067] In terms of throughput, pipelining requires throughput matching between CNN layers. The required operations vary widely between layers due to both the number of weights and the number of operations per weight. As mentioned above, IMC inherently combines data storage and computational resources. This provides a hardware allocation that handles operation scaling with the number of weights. However, the operations per weight are determined by the number of pixels in the output feature map, which itself varies widely (row 2 of Table I).

[0068] Figure 7 graphically depicts replication for throughput matching in pixel-level pipelining, where fewer operations in layer l+1 (e.g., due to larger convolution striding) require replication for layer l. Thus, as illustrated in Figure 7, throughput matching requires replication within the mapping of each CNN layer depending on the number of output feature map pixels (layer l has four times as many output pixels as layer l+1). Otherwise, a layer with fewer output pixels would incur a loss of utilization due to pipeline stalls.

[0069] As discussed above, replication reduces the number of operations associated with each weight stored in parallel hardware. This presents a problem for IMC, where the lower cost of MAC operations requires maintaining a large number of operations per stored weight to amortize the matrix loading cost. However, in practice, the replication required for throughput matching has proven acceptable for two reasons. First, such replication is explicitly scaled with the number of operations per weight, rather than being performed uniformly across all layers. Thus, the hardware used for replication can still effectively amortize the matrix loading cost. Second, a large amount of replication leads to the full utilization of the physical IMC bank. For subsequent layers, this forces new pipeline segments with independent throughput matching and replication requirements. Thus, the amount of replication is self-regulating with the amount of hardware.

[0070] Algorithm 2 depicts exemplary pseudocode for an execution loop in a CNN using bit-parallel / bit-serial (BPBS) computation, according to various embodiments.

number

[0071] BPBS Unfolding. As mentioned previously, the need for a high column dimension to maximize the gain from IMC results in a loss of utilization when used to map smaller filters. However, the BPBS computation effectively results in two additional loops, one for processing the input activation bits and one for processing the weight bits, as shown in Algorithm 2. These loops can be unfolded to increase the amount of column hardware used.

[0072] Figure 8 depicts a graphical representation of row underutilization and mechanisms for addressing row underutilization that are useful for understanding various embodiments. Specifically, Figure 8 depicts the row utilization challenge and the consequences of unrolling the BPBS computation loop to increase IMC column utilization.

[0073] Figure 8A graphically illustrates the row underutilization challenge, where, as an example, a small filter occupies only one-third of the IMC columns. Assuming 4-bit weights, the BPBS approach employs four parallel columns for each filter. Two alternative mapping approaches can be employed to increase the utilization to above 0.33. The first approach, illustrated in Figure 8b, merges two adjacent columns into one. However, because the original columns correspond to different matrix element bit positions, bits from the most significant positions must be duplicated in columns with corresponding binary weights; serially provided input vector elements are simply duplicated in the same way. This ensures proper capacitance charge shunting during the column accumulation operation.

[0074] FIG. 8A graphically depicts the effective utilization of columns. Specifically, column merging has two limitations. First, the duplication required to merge bits from higher matrix element positions leads to high physical utilization but a somewhat low effective utilization. For example, the effective utilization of the column in FIG. 8B is only 0.66, which is further limited as more columns are merged with corresponding binary-weighted duplication. Second, due to the need for binary-weighted duplication, the column dimensionality requirement increases exponentially with the number of columns being merged. This limits the cases in which column merging can be applied.

[0075] For example, two columns can be merged only if the original utilization is <0.33, three columns can be merged if the original utilization is <0.14, four columns can be merged only if the original utilization is <0.07, and so on. The second approach of overlapping and shifting is illustrated in Figure 8C. Specifically, matrix elements are duplicated and shifted, requiring an additional IMC column. In this case, two input vector bits are provided in parallel, with the most significant bit provided to the shifted matrix element. Unlike column merging, overlapping and shifting results in a high effective utilization that is equal to the physical utilization. Furthermore, the column dimensionality requirement does not increase exponentially with the effective utilization, making overlapping and shifting applicable in more cases. The main limitation is that while central columns achieve high utilization, columns toward either edge experience reduced utilization, and the first and last columns are limited to their original utilization levels, as shown in Figure 8C. Nevertheless, for weight precision of 4 to 8 bits, significant utilization gains are realized using various embodiments.

[0076] Multi-valued input activation values. The BPBS scheme allows the energy and throughput of the IMC computation to scale with the number of input vector bits that are applied serially. Multi-valued drivers are discussed above with respect to Figure 4.

[0077] Figure 9 graphically illustrates a sample of operations enabled by CIMU configurability via the software instruction library. In addition to temporal mapping of NN layers, the architecture provides extensive support for spatial mapping (loop unrolling). Given the high HW density / parallelism of IMC, this provides a wide range of mapping options for HW utilization beyond typical replication strategies that incur excessive state loading overhead due to state duplication between engines. To support spatial mapping of NN layers, various approaches for receiving and sequencing input activation values ​​for IMC computation are presented and enabled by the configurability of input and shortcut buffers, including: (1) high-bandwidth input for fully connected layers; (2) reduced-bandwidth input and line buffering for convolutional layers; (3) feedforward and regression input and output element computation for memory-expanded layers; and (4) parallel input and buffering of activation values ​​for NNs and shortcut paths, as well as summation of activation values. A wide range of other activation value reception / sequencing approaches, and configurability of the parameters of the above approaches, are supported.

[0078] Figure 10 graphically illustrates architectural support for spatial mapping within application layers, such as the NN layer, to both mitigate data swap / movement overhead and enable scalability of NN models. For example, output tensor depth (number of output channels) can be expanded through OCN routing of input activation values ​​to multiple CIMUs. Input tensor depth (number of input channels) can be expanded through short, high-bandwidth face-to-face connections between the outputs of adjacent CIMUs, and further expanded by summing partial pre-activation values ​​from two CIMUs via a third CIMU. In this way, efficient scale-up of layer computation enables a balance of IMC core dimensionality (found by mapping a wide range of NN benchmarks), where coarse grainedness benefits IMC parallelism and energy, and fine grainedness benefits efficient computational mapping.

[0079] General considerations of modular IMC for scalability. Both layer unfolding and BPBS unfolding introduce significant architectural challenges. For layer unfolding, the main challenge is the need to support diverse data flows and computations between layers in a NN application. This necessitates architectural configurability that can generalize to current and future NN designs. In contrast, within a single NN layer, MVM operations dominate, and the computation engine benefits from the relatively fixed data flows involved (although various optimizations that exploit attributes such as sparsity are gaining attention). Examples of data flow and computation configurability required between layers are discussed below.

[0080] In BPBS deployments, duplication and shifting in particular affect the bitwise sequencing of operations on input activations, posing additional complexities for throughput matching (column merging, binding input activations to bitwise computations, preserving sequencing for pixel-level pipelining). More generally, if different levels of input activation quantization are employed between layers, thus requiring different numbers of IMC cycles, this must also be considered within the replication approach discussed above for throughput matching in pixel-level pipelines.

[0081] 11 graphically illustrates how NN filters are mapped to IMC banks, each having a dimensionality of N rows and M columns, by loading filter weights into memory as matrix elements, applying input activation values ​​as input vector elements, and calculating output pre-activation values ​​as output vector elements. Specifically, FIG. 11 illustrates loading filter weights into memory as matrix elements for IMC banks, applying input activation values ​​as input vector elements, and calculating output pre-activation values ​​as output vector elements. Each bank is illustrated as having a dimensionality of N rows and M columns (i.e., processing an input vector of dimensionality N and providing an output vector of dimensionality M).

[0082] IMC implements the following forms of MVM:

number

[0083] This corresponds to a weight-fixed mapping. Alternative mappings are possible, such as input-fixed, where the input activation values ​​are stored in the IMC bank and the filter weights are stored in the input vector

number

number

[0084] Architecture Support Following the basic approach of mapping the NN layer onto the IMC array, various microarchitectural support around the IMC banks may be provided according to various embodiments.

[0085] FIG. 12 depicts a block diagram illustrating exemplary architecture support elements associated with layers and IMC banks for BPBS deployment.

[0086] Input Line Buffering for Convolution. In pixel-level pipelining, the output activity value for a pixel is generated by one IMC module and sent to the next. Furthermore, in the BPBS approach, incoming activity values ​​are processed at a time. However, convolution involves computation on multiple pixels at a time. This requires configurable buffering at the IMC input with support for stride steps of various sizes. While there are various ways to do this, the approach in Figure 12 buffers several rows of the input feature map corresponding to the height of the convolution kernel (exemplified in Figure 6). The row width supported by the buffer requires processing the input feature map in vertical segments (e.g., by performing blocking on loop 4). The kernel height / width supported by the buffer is an important architectural design parameter, but we can take advantage of the trend of 3x3 primary kernels to build larger kernels. With such buffering, incoming pixel data can be provided to the IMC one bit at a time, processed one bit at a time, and sent one bit at a time (according to the output BPBS operation).

[0087] The input line buffer can also support input pixel acquisition from various IMC modules by having additional input ports from the on-chip network. This allows for the allocation of multiple input IMC modules to equalize the number of operations performed by each IMC module in the pipeline, enabling the throughput matching required for pixel-level pipelining. For example, this may be required when using an IMC module to map a CNN layer with a larger stride step than the preceding CNN layer, or when the preceding CNN layer is followed by a pooling operation. In general, the kernel height / width determines the number of input ports that must be supported, since a stride step greater than or equal to the kernel height / width will not result in convolutional reuse of data and will require all new pixels for each IMC operation.

[0088] Note that the inventors contemplate various techniques by which the incoming (received) pixels may be suitably buffered. The approach depicted in Figure 12 assigns different input ports to different vertical segments of each row in the manner shown in Figure 7.

[0089] Near-memory element-wise operations. To feed data directly from the IMC hardware running one NN layer to the IMC hardware running the next, unified near-memory operations (NMC) are required for operations on individual elements, such as activation functions, batch normalization, scaling, and offsetting, as well as operations on small groups of elements, such as pooling. In general, such operations require a higher level of programmability than MVM and involve smaller amounts of input data.

[0090] FIG. 13 illustrates a block diagram of an exemplary near-memory computation SIMD engine. Specifically, FIG. 13 illustrates a programmable single instruction, multiple data (SIMD) digital engine integrated at the IMC output (i.e., following the ADC). The illustrated exemplary embodiment has two SIMD controllers: one for parallel control of BPBS near-memory computations and one for parallel control of other arithmetic near-memory computations. In general, SIMD controllers can be combined and / or other such controllers can be included. The illustrated NMC is grouped into eight blocks, each providing eight parallel computation channels (A / B, and 0-3) for IMC columns and for configuring columns in various ways. Each channel includes a local arithmetic logic unit (ALU) and register file (RF) and is multiplexed among four columns to accommodate throughput and layout pitch matching for IMC computations. In general, other architectures can also be employed. Additionally, a look-up table (LUT)-based implementation of a nonlinear function is illustrated. This can be used for any activation function. Here, a single LUT is shared across all parallel computation blocks, and the bits of the LUT entries are broadcast serially between the computation blocks. Each computation block then selects the desired entry and receives the bits serially over a number of cycles corresponding to the bit precision of the entry. This is controlled via a LUT client (FSM) in each parallel computation block, avoiding the area cost of having a LUT for every computation block at the expense of broadcast wiring.

[0091] Near-Memory Cross-Element Computation. Generally, operations are required not only on individual output elements from an MVM operation, but also between output elements. Examples include long short-term memory (LSTM), gated recurrent units (GRUs), and transformer networks. Therefore, the near-memory SIMD engine in Figure 10 supports subsequent digital operations between adjacent IMC columns, as well as reduction operations (adder, multiplier trees) between all columns.

[0092] As an example, to map LSTMs, GRUs, etc. where output elements from different MVM operations are combined via element-wise computation, matrices can be mapped to different interleaved IMC columns such that corresponding output vector elements are available in adjacent rows for near-memory cross-element computation.

[0093] Figure 14 depicts a graphical representation of an exemplary LSTM layer mapping function that leverages cross-element near-memory computation. Specifically, for a typical LSTM layer mapping to a CIMU for the example case of 2-bit weights (Bw=2), as illustrated in Figure 14. The GRU follows a similar mapping. Four MVM operations are performed to generate each output yt, and the intermediate outputs

number

number

number

[0094] In various embodiments, each CIMU is associated with a respective near-memory, programmable single instruction, multiple data (SIMD) digital engine, which may be contained within the CIMU, outside the CIMU, and / or in a separate element within an array that includes the CIMU. The SIMD digital engine is suitable for use in combining or time-aligning input buffer data, shortcut buffer data, and / or output feature vector data for inclusion within a feature vector map. Various embodiments enable computation across / among parallelized computation paths of the SIMD engine.

[0095] Shortcut Buffering and Merging. In pixel-level pipelining, spanning between NN layers requires special buffering in the shortcut path to align the pipeline latency with that of the NN path. In Figure 12, such buffering of the shortcut path is incorporated along with IMC input line buffering of the computed NN path so that the data flow and delay of the two paths are aligned. With the possibility of multiple overlapping shortcut paths (e.g., as in U-Net), the number of such buffers is an important architectural parameter. However, available buffers from any IMC bank can be used for this purpose, providing flexibility in mapping such overlapping shortcut paths. The final summation of the shortcut path and the NN computed path is supported by feeding the shortcut buffer output to a near-memory SIMD, as shown. The shortcut buffer can support input ports in a manner similar to the input line buffer. However, typically in CNNs, the layers through which shortcut connections pass maintain a fixed number of output pixels to enable the final pixel-wise summation, which leads to a fixed number of operations between layers, typically leading to an IMC module being fed by a single IMC module. Exceptions to this include U-Net, which potentially benefit from additional input ports in the shortcut buffer.

[0096] Input Feature Map Depth Expansion. The number of IMC rows limits the input feature map depth that can be processed, necessitating depth expansion through the use of multiple IMC banks. When multiple IMC banks are used to process deep input channels within a segment, FIG. 10 includes hardware for summing segments together into subsequent IMC banks. Previous segment data is provided to the local input buffer and shortcut buffer in parallel across output channels. The parallel segment data is then summed together via a custom adder between the two buffer outputs. By cascading IMC banks to perform such additions, arbitrary depth expansion can be performed.

[0097] The adder output is fed to a near-memory SIMD, allowing further element-wise and cross-element computations (eg, activation functions).

[0098] On-chip Network Interface for Weight Loading. In addition to an input interface for receiving input vector data from the on-chip network (i.e., for MVM computation), an interface for receiving weight data from the on-chip network (i.e., for storing matrix elements) may also be included. This allows matrices generated from MVM operations to be employed in IMC-based MVM operations, which is useful in various applications, such as, illustratively, mapping transformer networks. Specifically, FIG. 15 graphically illustrates the mapping of a bidirectional encoder representation from a transformer (BERT) layer using the generated data as the loaded matrix. In this example, both the input vector X and the generated matrix Yi,1 are loaded into the IMC module through the weight loading interface. The on-chip network may be implemented as a single on-chip network, as multiple on-chip network sections, or as a combination of on-chip and off-chip network sections.

[0099] Scalable IMC Architecture 16 illustrates a high-level block diagram of an IMC-based scalable NN accelerator architecture, according to some embodiments. Specifically, FIG. 16 illustrates an IMC-based scalable NN accelerator with integrated microarchitectural support for application mapping around IMC banks forming modules that enable scaling up the architecture through tiling and interconnection.

[0100] Figure 17 depicts a high-level block diagram of a CIMU microarchitecture with an 1152x256 IMC bank suitable for use in the architecture of Figure 16. That is, while the overall architecture is illustrated in Figure 16, a module with integrated IMC bank and microarchitectural support, referred to as an in-memory compute unit (CIMU), suitable for use in that architecture, is depicted in Figure 17. The inventors have determined that benchmark throughput, latency, and energy scale with the number of tiles (throughput / latency should scale proportionally, and energy remains substantially constant).

[0101] As depicted in Figure 16, the array-based architecture comprises: (1) a 4x4 array of in-memory computation unit (CIMU) cores, (2) an on-chip network (OCN) between the cores, (3) off-chip interface and control circuitry, and (4) an additional weight buffer with a dedicated weight loading network to the CIMU.

[0102] As depicted in Figure 17, each CIMU may include: (1) an IMC engine for MVM, denoted as an in-memory computational array (CIMA); (2) an NMC digital SIMD with a custom instruction set for flexible element-wise operations; and (3) buffering and control circuitry to enable a wide range of NN data flows. Each CIMU core provides a high level of configurability and can be abstracted into a software library of instructions for interfacing with the compiler (to assign / map applications, NNs, etc. to the architecture), so that instructions can be added in the future. That is, the library includes simple / fused instructions such as element mult / add, h(●) activation, (N-step convolutional stride + MVM + batch norm + h(●) activation + max.pool), (dense + MVM), etc.

[0103] The OCN consists of routing channels within the Network In / Out block and a switch block that provides flexibility through a dissociated architecture. The OCN works with configurable CIMU input / output ports to optimize data structures to / from the IMC engine, maximizing data locality across MVM dimensionality and tensor depth / pixel index. The OCN routing channel may include bidirectional wire pairs to mitigate repeater / pipeline-FF insertion while providing sufficient density.

[0104] An IMC architecture may be used to implement a neural network (NN) accelerator, where multiple computational in-memory computation units (CIMUs) are arranged and interconnected using a highly flexible on-chip network, where the output of one CIMU may be connected or flowed to the input of another CIMU or to multiple other CIMUs, the output of many CIMUs may be connected to the input of one CIMU, the output of one CIMU may be connected to the output of another CIMU, etc. The on-chip network may be implemented as a single on-chip network, as multiple on-chip network sections, or as a combination of on-chip and off-chip network sections.

[0105] Referring to Figure 17, CIMU data is received from the OCN via one of two buffers: (1) an input buffer that configurably provides data to the CIMA, and (2) a shortcut buffer that bypasses the CIMA and provides data directly to the NMC digital SIMD for element-wise computation in separate and / or converged NN activity paths. The central block is the CIMA, which consists of mixed-signal N (row) x M (column) (e.g., 1152 (row) x 256 (column)) IMC macros for multi-bit element MVM. In various embodiments, the CIMA employs a variant of full row / column parallel computation based on metal fringe capacitors. Each multiplication bit cell (M-BC) drives its capacitor with a 1-bit digital multiplication (XNOR / AND) with the input activity data (IA / IAb) and the stored weight data (W / Wb). This causes charge redistribution among the M-BC capacitors in the column to provide the dot product between the binary vectors on the computation line (CL). This results in low computational noise (nonlinearity, variability) because multiplication is digital and accumulation involves only capacitors defined with high lithographic precision. An 8-bit SAR ADC digitizes the CL and enables extension to multi-bit activation values / weights via bit-parallel / bit-serial (BP / BS) computation, where weight bits are mapped to parallel columns and activation value bits are input serially. Thus, each column performs a binary vector dot product, with multi-bit vector dot products achieved simply by digital bit shifting (for appropriate binary weighting) and summation between column ADC outputs. Digital BP / BS operations are performed in a dedicated NMC BPBS SIMD module, which can be optimized for 1- to 8-bit weight / activation values; further, programmable element-wise operations (e.g., arbitrary activation functions) are performed in the NMC CMPT SIMD module.

[0106] In the overall architecture, each CIMU is surrounded by an on-chip network (activation network) for transferring activation values ​​between CIMUs, as well as an on-chip network (weight loading interface) for transferring weights from the embedded L2 memory to the CIMUs. This has similarities to the architecture used in coarse-grained reconfigurable arrays (CGRAs), but with cores providing highly efficient MVM and element-wise computations targeted for NN acceleration.

[0107] There are various options for implementing an on-chip network. The approach of Figures 16-17 allows routing segments along a CIMU to obtain output from that CIMU and / or provide input to that CIMU. In this way, data originating from any CIMU can be routed to any CIMU and any number of CIMUs. This implementation is employed for purposes described herein.

[0108] Various embodiments contemplate an integrated in-memory computation (IMC) architecture that is configurable to support scalable execution and data flow of applications mapped to the IMC, comprising a plurality of configurable integrated in-memory computation units (CIMUs), forming an array of CIMUs; and a configurable on-chip network for communicating input operands from an input buffer to the CIMUs, for communicating input operands between the CIMUs, for communicating computed data between the CIMUs, and for communicating computed data from the CIMUs to an output buffer.

[0109] Each CIMU is associated with an input buffer for receiving computational data from the on-chip network and composing the received computational data into input vectors for matrix vector multiplication (MVM) processing by the CIMU to generate computed data including an output vector.

[0110] Each CIMU is associated with a shortcut buffer for receiving computational data from the on-chip network, applying a time delay to the received computational data, and forwarding the delayed computational data toward the next CIMU or output according to a data flow map so that data flow alignment among the multiple CIMUs is maintained. At least some of the input buffers may be configured to apply a time delay to computational data received from the on-chip network or from the shortcut buffer. The data flow map may support pixel-level pipelining to provide pipeline latency matching.

[0111] The time delays imparted by the shortcut buffer and the input buffer include at least one of an absolute time delay, a predetermined time delay, a time delay determined relative to the size of the input calculation data, a time delay determined relative to the expected calculation time of the CIMU, a control signal received from a data flow controller, a control signal received from another CIMU, and a control signal generated by the CIMU in response to the occurrence of an event within the CIMU.

[0112] In some embodiments, at least one of the input buffer and shortcut buffer of each of a plurality of CIMUs in the array of CIMUs is configured according to a data flow map that supports pixel-level pipelining to provide pipeline latency matching.

[0113] The array of CIMUs may also include parallelized computation hardware configured to process input data received from at least one of the respective input buffers and shortcut buffers.

[0114] At least a subset of the CIMUs may be associated with an on-chip network portion including an operand loading network portion configured according to the data flow of an application mapped to the IMC, the application including a neural network (NN) mapped to the IMC such that parallel output computed data of a configured CIMU executing at a given layer is provided to a configured CIMU executing at a next layer, the parallel output computed data forming respective NN feature map pixels.

[0115] The input buffer may be configured to transfer input NN feature map data to parallelized computation hardware within the CIMU according to a selected stride step. The NN may include a convolutional neural network (CNN), and the input buffer is used to buffer a number of rows of the input feature map corresponding to the size or height of the CNN kernel.

[0116] Each CIMU may include an in-memory computation (IMC) bank configured to perform matrix-vector multiplication (MVM) according to a bit-parallel-bit-serial (BPBS) computation process in which a single-bit computation is performed using iterative barrel shifting with a column weighting process, followed by a result accumulation process.

[0117] FIG. 18 depicts a high-level block diagram of a segment for obtaining input from a CIMU by employing multiplexers to select whether data on several parallel routing channels is obtained from an adjacent CIMU or provided from a previous network segment.

[0118] FIG. 19 depicts a high-level block diagram of a segment for providing output to a CIMU by employing multiplexers to select whether data from several parallel routing channels is provided to an adjacent CIMU.

[0119] 20 depicts a high-level block diagram of an exemplary switch block employing multiplexers (and optionally flip-flops for pipelining) to select which inputs are routed to which outputs. Thus, the number of parallel routing channels provided is an architectural parameter that can be selected to ensure perfect routability (between all points) or high probability routability between NNs of a desired class.

[0120] In various embodiments, L2 memory is located along the top and bottom and divided into separate blocks for each CIMU to reduce access costs and networking complexity. The amount of embedded L2 is an architectural parameter selected appropriately for the application; for example, this amount can be optimized for the number of NN model parameters typical in the application of interest. However, dividing into separate blocks for each CIMU requires additional buffering due to duplication within pipeline segments. Based on the benchmarks used in this study, a total of 35 MB of L2 is employed. Depending on the application, other configurations, or larger or smaller sizes, may be appropriate.

[0121] Each CIMU includes an IMC bank, a near-memory computation engine, and a data buffer, as described above. The IMC bank is selected to be a 1152 x 256 array, where 1152 is chosen to optimize the mapping of 3 x 3 filters up to 128 deep. The dimensionality of the IMC bank is selected to balance the amortization of energy overhead and area overhead of the peripheral circuitry.

[0122] Discussion of Some Embodiments Various embodiments described herein provide an array-based architecture (the array may be 1-dimensional, 2-dimensional, 3-dimensional, ... n-dimensional as needed / desired) formed using multiple CIMUs and operationally enhanced through the use of some or all of various configurable / programmable modules for flowing data between CIMUs, arranging data to be processed by the CIMUs in an efficient manner, delaying data being processed by a CIMU (or bypassing a particular CIMU) so as to maintain time alignment of the mapped NN (or other application), etc. Advantageously, various embodiments allow scalability such that n-dimensional CIMU arrays communicate over a network, allowing NNs, CNNs, and / or other problem spaces of various sizes / complexities where matrix multiplication is a key solution component to benefit from the various embodiments.

[0123] Generally speaking, a CIMU comprises various structural elements, including an in-memory computation array (CIMA), illustratively a bitcell CIMA configured via various configuration registers to provide programmable in-memory computation functions such as matrix-vector multiplication. In particular, a typical CIMU is tasked with multiplying an input matrix X by an input vector A to produce an output matrix Y. The CIMU is depicted as including an in-memory computation array (CIMA) 310, an input activity vector reshaping buffer (IA BUFF) 320, a sparsity / AND logic controller 330, a memory read / write interface 340, a row decoder / WL driver 350, a plurality of A / D converters 360, and a near-memory computation multiply-shift-accumulate data path (NMD) 370.

[0124] Regardless of how they are implemented, the CIMUs depicted herein are each surrounded by an on-chip network for moving activation values ​​between CIMUs (in the case of a NN implementation, an on-chip network such as an activation value network), as well as an on-chip network for moving weights from the embedded L2 memory to the CIMU (e.g., a weight loading interface), as discussed above with respect to architectural tradeoffs.

[0125] As noted above, the activity network comprises a configurable / programmable network for transmitting computational input and output data from, to, and between CIMUs, such that in various embodiments the activity network may be interpreted as an I / O data transfer network, an inter-CIMU data transfer network, etc. Thus, these terms are used somewhat interchangeably to encompass a configurable / programmable network directed to data transfer to / from a CIMU.

[0126] As noted above, a weight loading interface or network includes a configurable / programmable network for loading operands internal to a CIMU and may also be referred to as an operand loading network. Thus, these terms are used somewhat interchangeably to encompass a configurable / programmable interface or network directed to loading operands, such as weighting coefficients, into a CIMU.

[0127] As mentioned above, shortcut buffers are depicted as being associated with a CIMU, such as within the CIMU or external to the CIMU. Shortcut buffers may also be used as array elements depending on the application that is mapped to the shortcut buffer, such as a NN, a CNN, etc.

[0128] As mentioned above, a near-memory programmable single instruction multiple data (SIMD) digital engine (or near-memory buffer or accelerator) is depicted as being associated with the CIMU, such as within the CIMU or external to the CIMU. The near-memory programmable single instruction multiple data (SIMD) digital engine (or near-memory buffer or accelerator) buffer may also be used as an array element, depending on the application mapped to this buffer, such as a NN, a CNN, etc.

[0129] Note that in some embodiments, the input buffers described above may also provide data to the CIMA within the CIMU in a configurable manner, such as to provide configurable shifting corresponding to striding in convolutional NNs and the like. To implement nonlinear computations, lookup tables for mapping inputs to outputs according to various nonlinear functions may be provided individually to the SIMD digital engines of each CIMU or may be shared across multiple SIMD digital engines of a CIMU (e.g., a parallel lookup table implementation of the nonlinear function). In this manner, lookup table locations are broadcast among the SIMD digital engines so that each SIMD digital engine can selectively process specific bits appropriate to that SIMD digital engine.

[0130] Architecture Evaluation - Physical Design An evaluation of the IMC-based NN accelerator is performed in comparison with a conventional spatial accelerator composed of digital processing elements. Both designs allow for bit-precision scalability, but assume fixed-point 8-bit computation. The CIMU, digital processing elements, on-chip network blocks, and embedded L2 array are implemented in 16nm CMOS technology up to the physical design.

[0131] FIG. 21A illustrates a layout diagram of a CIMU architecture according to an embodiment implemented in 16 nm CMOS technology. FIG. 21B illustrates a full-chip layout diagram consisting of a 4×4 tiling of the CIMU as provided in FIG. 21A. The mixed-signal nature of the architecture requires both a full-custom transistor-level design as well as a standard cell-based RTL design (followed by synthesis and APR). For both designs, functional verification is performed at the RTL level. This requires employing a behavioral model of the IMC banks, which is itself verified via Spectre (a SPICE equivalent) simulation.

[0132] Architecture Evaluation - Energy and Speed ​​Modeling The physical design of the IMC-based architecture and digital architecture enables robust energy and speed modeling based on post-layout extraction of parasitic capacitances. Speed ​​is parameterized as the achievable clock cycle frequencies FCIMU and FPE of the IMC-based architecture and digital architecture, respectively (from both STA and Spectre simulations). Energy is parameterized as follows: Input Buffer (EBuff): This is the energy in the CIMU required to write and read input activation values ​​to / from the input buffer and shortcut buffer. IMC (EIMC). This is the energy in the CIMU required for the MVM calculation via the IMC bank (using 8-bit BPBS calculations). Near-memory computation (ENMC): This is the energy in the CIMU required for near-memory computation of all IMC column outputs. On-chip network (EOCN), which is an energy-efficient IMC-based architecture for moving activity data between CIMUs. Processing Engine (EPE): This is the energy in the digital PE for 8-bit MAC operations and output data movement to adjacent PEs. L2 read (EL2): This is the energy in both IMC-based and digital architectures to read weight data from L2 memory. ● Weight Loading Network (EWLN), which is the energy intensive in both IMC-based and digital architectures to move weight data from L2 memory to the CIMU and PEs, respectively. ● CIMU weight loading (EWL,CIMU), which is the energy in CIMU for writing weight data. PE Weight Loading (EWL,PE), which is the energy in the digital PE for writing the weight data.

[0133] Architecture Evaluation - Neural Network Mapping and Implementation To evaluate the impact of scaling up the architecture, different physical chip areas are considered for comparison between the IMC-based architecture and the digital architecture. The areas correspond to 4x4, 8x8, and 16x16 IMC banks. For benchmarking, a set of popular CNNs is employed to evaluate the metrics of energy efficiency, throughput, and latency for both small batch sizes (1) and large batch sizes (128).

[0134] Figure 22 graphically depicts three stages of mapping software flow to an architecture, where, for example, the NN mapping flow is mapped to an 8x8 array of CIMUs. Figure 23A depicts a sample placement of layers from a pipeline segment, and Figure 23B depicts a sample routing from a pipeline segment.

[0135] Specifically, the benchmarks are mapped to each architecture via software flows. For IMC-based architectures, the software flow mapping involves three stages, shown in Figure 22: allocation, placement, and routing.

[0136] The allocation corresponds to allocating CIMUs to NN layers in different pipeline segments based on filter mapping, layer expansion, and BPBS expansion as described above.

[0137] The placement corresponds to mapping the assigned CIMUs in each pipeline segment to physical CIMU locations in the architecture (as depicted in FIG. 23A). It employs a simulated annealing algorithm to minimize the activity network segments required between the sending and receiving CIMUs. A sample placement of layers from a pipeline segment is shown in FIG. 23A.

[0138] Routing corresponds to configuring routing resources in the on-chip network to move activity values ​​between CIMUs (e.g., on-chip network portions forming an inter-CIMU network). This employs dynamic programming to minimize the activity value network segments required between transmitting and receiving CIMUs, subject to routing resource constraints. Sample routing from a pipeline segment is shown in Figure 23B.

[0139] Following each stage of the mapping flow, functionality is verified using a behavioral model, which is also verified against the RTL design. After three stages, configuration data is output, which is loaded into an RTL simulation for final design verification. The behavioral model is cycle-accurate and allows for energy and speed characterization based on modeling the above parameters.

[0140] For digital architectures, the application mapping flow involves a typical layer-by-layer mapping, with replication maximizing hardware utilization. Again, cycle-accurate behavioral models are used to verify functionality and perform energy and speed characterization based on the above modeling.

[0141] Architecture Scalability Evaluation - Energy, Throughput, and Latency Analysis Compared to the digital architecture, the IMC-based architecture exhibits increased energy efficiency. In particular, across the benchmarks, gains of 12-25x and 17-27x are achieved for the IMC-based architecture for batch sizes of 1 and 128, respectively. This suggests that matrix loading energy is substantially amortized and column utilization is improved as a result of the deployment of layers and BPBS.

[0142] The IMC-based architecture achieves improved throughput compared to the digital architecture. In particular, across benchmarks, gains of 1.3x to 4.3x and 2.2x to 5.0x are achieved for the IMC-based architecture for batch sizes of 1 and 128, respectively. The throughput gains are not as large as the energy efficiency gains. This is because layer unfolding effectively results in lost utilization of the IMC hardware used to map subsequent layers in each pipeline segment. In fact, this effect is most pronounced for small batch sizes, where pipeline loading delays are amortized, and somewhat smaller for large batch sizes. However, even with large batches, CNNs require some delay to clear the pipeline between inputs to avoid overlapping convolution kernels across different inputs.

[0143] Compared to the digital architecture, the latency of the IMC-based architecture is reduced. The reduction observed tracks the throughput gain and follows the same rationale.

[0144] Scalability Evaluation of Architecture - Impact of Layer Deployment and BPBS Deployment To analyze the benefits of layer unfolding, the ratio of the total weight loading required in an IMC architecture with layer-by-layer mapping compared to layer unfolding is considered. We have determined that layer unfolding results in a substantial reduction in weight loading, especially as the architecture scales up. More specifically, for IMC bank scaling from 4x4, 8x8, and 16x16, weight loading accounts for 28%, 46%, and 73% of the average total energy with layer-by-layer mapping (batch size of 1). Meanwhile, weight loading accounts for only 23%, 24%, and 27% of the average total energy with layer unfolding (batch size of 1), allowing for much better scalability. In contrast, due to the significantly higher energy of MVM compared to IMC, traditional layer-by-layer mapping is tolerated in the digital architecture and accounts for 1.3%, 1.4%, and 1.9% of the average total energy (batch size of 1).

[0145] To analyze the benefits of BPBS deployment, the reduction factor of the unused IMC cell ratio is considered. This is shown in Figure 18 for both column merging (as physical and effective utilization gain) and overlap and shifting. As can be seen, a significant reduction in the unused bitcell ratio is achieved. The combined average bitcell utilization (effective) for column merging and overlap and shifting is 82.2% and 80.8%, respectively.

[0146] FIG. 24 depicts a high-level block diagram of a computing device suitable for use in implementing various control elements or portions thereof, and for use in performing functions described herein, such as the functions associated with the various elements described herein with respect to the figures.

[0147] For example, the NN and application mapping tools and various application programs depicted above may be implemented using a general-purpose computing architecture such as that depicted herein with respect to FIG.

[0148] As depicted in FIG. 24, the computing device 2400 includes a processor element 2402 (e.g., a central processing unit (CPU) or other suitable processor), memory 2404 (e.g., random access memory (RAM), read-only memory (ROM), etc.), cooperating modules / processes 2405, and various input / output devices 2406 (e.g., communication modules, network interface modules, receivers, transmitters, etc.).

[0149] It will be understood that the functions depicted and described herein may be implemented in hardware, or a combination of software and hardware, using, for example, a general purpose computer, one or more application specific integrated circuits (ASICs), or any other hardware equivalents. In one embodiment, cooperating process 2405 may be loaded into memory 2404 and executed by processor 2402 to implement the functions discussed herein. Thus, cooperating process 2405 (including associated data) may be stored on a computer-readable storage medium, such as RAM memory, a magnetic or optical drive or disk, etc.

[0150] It will be appreciated that the computing device 2400 depicted in FIG. 24 provides a general architecture and functionality suitable for implementing the functional elements described herein or portions of the functional elements described herein.

[0151] It is contemplated that some of the steps discussed herein may be implemented in hardware, for example, as circuitry that cooperates with a processor to perform various method steps. Portions of the functions / elements described herein may also be implemented as a computer program product, where computer instructions, when processed by a computing device, adapt the operation of the computing device such that a method or technique described herein is invoked or otherwise provided. Instructions for invoking the methods of the present invention may be stored on a tangible and non-transitory computer-readable medium, such as a fixed or removable medium or memory device, or stored in memory within a computing device that operates according to the instructions.

[0152] Various embodiments contemplate computer-implemented tools, application programs, systems, etc. configured for mapping, design, testing, operation, and / or other functions associated with the embodiments described herein. For example, the computing device of FIG. 24 may be used to provide a computer-implemented method for mapping an application, NN, or other function to an integrated in-memory computing (IMC) architecture as described herein.

[0153] As described above with respect to Figures 22-23, mapping a software flow or application, NN, or other function to the IMC hardware / architecture generally involves three stages: allocation, placement, and routing. Allocation corresponds to assigning CIMUs to NN layers in different pipeline segments based on the filter mapping, layer unfolding, and BPBS unfolding described above. Placement corresponds to mapping the assigned CIMUs in each pipeline segment to physical CIMU locations in the architecture. Routing corresponds to configuring routing resources in the on-chip network to move activity values ​​between CIMUs (e.g., portions of the on-chip network that form an inter-CIMU network).

[0154] Broadly, these computer-implemented methods may accept input data describing a desired / target application, NN, or other functionality, and in response generate output data in a form suitable for use in programming or configuring an IMC architecture such that the desired / target application, NN, or other functionality is realized, which may be provided for a default IMC architecture or a target IMC architecture (or a portion thereof).

[0155] The computer-implemented method may employ various known tools and techniques, such as computation graphs, data flow representations, high-, mid-, or low-level descriptors, to characterize, define, or describe a desired / target application, NN, or other functionality in terms of input data, operations, sequencing of operations, output data, etc.

[0156] A computer-implemented method may be configured to map a characterized, defined, or described application, NN, or other function to an IMC architecture by allocating IMC hardware accordingly, and to do so in a manner that substantially maximizes the throughput and energy efficiency of the IMC hardware executing the application (e.g., by using various techniques discussed herein, such as computational parallelism and pipelining using the IMC hardware). The computer-implemented method may be configured to utilize some or all of the functionality described herein, such as mapping a neural network to a tiled array of in-memory computing hardware, allocating in-memory computing hardware to particular computations required by the neural network, placing the allocated in-memory computing hardware to particular locations within the tiled array (optionally where the placement is configured to minimize the distance between in-memory computing hardware that provides particular outputs and in-memory computing hardware that obtains particular inputs), employing optimization methods (e.g., simulated annealing) to minimize such distances, configuring available routing resources to forward outputs from in-memory computing hardware to inputs to in-memory computing hardware within the tiled array, minimizing the total amount of routing resources required to achieve routing between the placed in-memory computing hardware, and / or employing optimization methods (e.g., dynamic programming) to minimize such routing resources.

[0157] Figure 34 depicts a flow diagram of a method according to an embodiment. Specifically, Figure 34 depicts a computer-implemented method for mapping an application to an integrated in-memory computing (IMC) architecture, the IMC architecture comprising a plurality of configurable in-memory computing units (CIMUs) forming an array of CIMUs, and a configurable on-chip network for communicating input data to the array of CIMUs, communicating computed data between the CIMUs, and communicating output data from the array of CIMUs.

[0158] The method of Figure 34 is directed to generating a computation graph, data flow map, and / or other mechanisms / tools suitable for use in programming an application or NN into an IMC architecture such as that discussed above. The method generally performs various configuration, mapping, optimization, and other steps, as described above. In particular, the method is depicted as allocating IMC hardware according to the computational requirements of the application or NN, defining the placement of the allocated IMC hardware to locations within the IMC core array in a manner that tends to minimize the distance between the IMC hardware that generates output data and the IMC hardware that processes the generated output data, configuring an on-chip network to route data between the IMC hardware, configuring input / output buffers, shortcut buffers, and other hardware, applying the BPBS developments discussed above (e.g., overlapping and shifting, column duplication, and other techniques), applying replication optimization, layering optimization, space optimization, time optimization, pipeline optimization, etc. The various calculations, optimizations, decisions, etc. may be implemented in any logical sequence and may be iterated or repeated to arrive at a solution, whereupon a data flow map may be generated for use in programming the IMC architecture.

[0159] One embodiment provides a computer-implemented method for mapping an application to configurable in-memory compute (IMC) hardware of an integrated IMC architecture, the IMC hardware comprising a plurality of configurable in-memory compute units (CIMUs) forming an array of CIMUs; and a configurable on-chip network for communicating input data to the array of CIMUs, communicating computed data between the CIMUs, and communicating output data from the array of CIMUs. The method includes: allocating the IMC hardware according to the application computation using parallelism and pipelining to generate an IMC hardware assignment configured to provide high-throughput application computation; defining placement of the allocated IMC hardware at locations within the array of CIMUs in a manner that tends to minimize distances between the IMC hardware that generates output data and the IMC hardware that processes the generated output data; and configuring the on-chip network to route data between the IMC hardware. This application may include a network. Various steps may be implemented according to the mapping techniques discussed throughout this application.

[0160] Various modifications may be made to the computer-implemented method, such as by using various mapping and optimization techniques described herein. For example, an application, NN, or function may be mapped to an IMC such that the parallel output computed data of a configured CIMU executing in a given layer is provided to a configured CIMU executing in a next layer, such as when the parallel output computed data forms each NN feature map pixel. Furthermore, pipelining of computations may be supported by allocating more configured CIMUs to execute in a given layer than in the next layer to compensate for more computation time in a given layer than in the next layer.

[0161] It will be understood that the functionality depicted and described herein may be implemented in hardware, or a combination of software and hardware, using, for example, a general-purpose computer, one or more application-specific integrated circuits (ASICs), or any other hardware equivalents. It is contemplated that some of the steps discussed herein may be implemented in hardware, for example, as circuitry that cooperates with a processor to perform various method steps. Portions of the functions / elements described herein may be implemented as a computer program product, where computer instructions, when processed by a computing device, adapt the operation of the computing device such that the methods or techniques described herein are invoked or otherwise provided. Instructions for invoking the methods of the present invention may be stored on a tangible and non-transitory computer-readable medium, such as a fixed or removable medium or memory, or stored in memory within a computing device that operates according to the instructions.

[0162] Various modifications may be made to the systems, methods, devices, mechanisms, techniques, and portions thereof described herein with respect to the various figures, and such modifications are contemplated as being within the scope of the present invention. For example, in various embodiments described herein, a particular order of steps or arrangement of functional elements is presented, but various other orders / arrangements of steps or functional elements may be utilized within the context of various embodiments. Furthermore, while modifications to embodiments may be discussed individually, various embodiments may employ multiple modifications simultaneously or sequentially, composite modifications, etc.

[0163] While certain systems, devices, methodologies, mechanisms, and the like have been disclosed as discussed above, it should be apparent to those skilled in the art that many modifications beyond those already described are possible without departing from the inventive concepts herein. Accordingly, the present subject matter is not intended to be limited except insofar as the spirit of the present disclosure prevails. Furthermore, in interpreting this disclosure, all terms should be interpreted in the broadest possible manner consistent with the context. In particular, the terms "comprises" and "comprising" should be interpreted as referring non-exclusively to elements, components, or steps indicating that a referenced element, component, or step may be present, utilized, or combined with other elements, components, or steps not expressly referenced. Additionally, the references recited herein are also part of this application and are incorporated by reference in their entireties as if fully set forth herein.

[0164] Example IMC Core / CIMU Discussion Various embodiments of an IMC core or CIMU may be used within the context of various embodiments. Such an IMC core / CIMU integrates configurability and hardware support around an in-memory computation accelerator, enabling the programmability and virtualization required for scaling to practical applications. Typically, the in-memory computation implements matrix-vector multiplication, where matrix elements are stored in a memory array and vector elements are broadcast in a parallel fashion across the memory array. Some aspects of the embodiments are directed to enabling programmability and configurability of such an architecture.

[0165] In-memory computations typically involve single-bit representations for either matrix elements, vector elements, or both. This is because memory stores data in independent bit cells, whereas broadcasting is done in a parallel, homogeneous manner without providing the different binary weighted combinations between bits required for multi-bit computations. In the present invention, extension to multi-bit matrix and vector elements is achieved via a bit-parallel / bit-serial (BPBS) scheme.

[0166] A highly configurable / programmable near-memory computation data path is included to enable common computational operations often surrounding matrix-vector multiplication. This both enables the computations needed to extend in-memory computation from bitwise computation to multi-bit computation, and generally, this supports multi-bit operations that are no longer constrained by the single-bit representation inherent in in-memory computation. Because programmable / configurable and multi-bit computation are more efficient in the digital domain, in the present invention, analog-to-digital conversion is performed following in-memory computation. In a specific embodiment, the configurable data path is multiplexed among eight ADC / in-memory computation channels, although other multiplexing ratios can be employed. This also aligns well with the BPBS scheme employed for multi-bit matrix element support; support for up to 8-bit operands is provided in embodiments.

[0167] Since input vector sparsity is common in many linear algebra applications, the present invention integrates support for enabling energy-proportional sparsity control. This is achieved by masking the broadcasting of bits from the input vector corresponding to zero-valued elements (such masking is performed for all bits in a bit-serial process). This saves broadcast energy as well as computational energy in the memory array.

[0168] Given the internal bitwise computation architecture for in-memory computation and the external digital word architecture of a typical microprocessor, data reshaping hardware is used both at the computation interface through which input vectors are provided, and at the memory interface through which matrix elements are written and read.

[0169] FIG. 25 illustrates a typical structure of an in-memory computation architecture. In-memory computation consisting of a memory array (which may be based on standard or modified bit cells) involves two additional “vertical” sets of signals: (1) input lines and (2) accumulation lines. Referring to FIG. 25, a two-dimensional array of bit cells is depicted, with each of a plurality of in-memory computation channels 110 comprising a respective column of bit cells, with each channel of bit cells associated with a common accumulation line and bit line (column), and a respective input line and word line (row). Note that to simply illustrate the row / column relationship within the context of an array of bit cells, such as the two-dimensional array of bit cells depicted in FIG. 25, the columns and rows of signals are labeled herein as being “perpendicular” to one another. The term “vertical” as used herein is not intended to convey any particular geometric relationship.

[0170] The input / bit and accumulate / bit sets of signals may be physically combined with existing signals in the memory (e.g., word lines, bit lines), or may be separate. To implement matrix-vector multiplication, matrix elements are first loaded into memory cells. Then, multiple input vector elements (possibly all) are applied at once via input lines. This causes a local arithmetic operation, typically some form of multiplication, to occur in each of the memory bit cells. The result of the arithmetic operation is then driven onto a shared accumulate line. In this way, the accumulate line represents the arithmetic result across multiple bit cells activated by the input vector elements. This is in contrast to standard memory access, where bit cells are accessed at a time via bit lines and activated by a single word line.

[0171] As mentioned above, in-memory computation has several important attributes. First, the computation is typically analog. This is because the constrained structure of memories and bitcells requires a richer computational model than is possible with simple digital switch-based abstractions. Second, local operations at bitcells typically involve computations on the single-bit representations stored in the bitcells. This is because bitcells in standard memory arrays do not couple to each other in any binary-weighted manner; any such coupling must be achieved by methods for accessing / reading the bitcells from the periphery. Below, we describe the extensions to in-memory computation proposed in this invention.

[0172] Extension to near-memory and multi-bit computation. While in-memory computation has the potential to address matrix-vector multiplication in a manner that conventional digital acceleration falls short of, a typical computation pipeline will involve a wide range of other operations surrounding matrix-vector multiplication. While such operations are typically well-addressed by conventional digital acceleration, locating such acceleration hardware near in-memory computation hardware can be highly valuable in appropriate architectures to address parallelism, high throughput (and thus the need for high communication bandwidth to and from), and common computation patterns associated with in-memory computation. Because many of the surrounding operations will preferably be performed in the digital domain, analog-to-digital conversion via an ADC is included following each in-memory computation accumulation line, hence the term in-memory computation channel. While a major challenge is integrating ADC hardware into the pitch of each in-memory computation channel, the appropriate layout approach taken in the present invention makes this possible.

[0173] Introducing an ADC following each computation channel enables an efficient way to extend in-memory computation to support multi-bit matrix and vector elements, respectively, via bit-parallel / bit-serial (BPBS) computation. Bit-parallel computation involves loading different matrix element bits into different in-memory computation columns. The ADC outputs from different columns are then appropriately bit-shifted to represent corresponding bit weights, and digital accumulation is performed across all of the columns to obtain the multi-bit matrix element computation result. Bit-serial computation, on the other hand, involves applying each bit of the vector element at a time, storing the ADC output each time and appropriately bit-shifting the stored output before digital accumulation with the next output corresponding to the subsequent input vector bit. Such a BPBS approach, enabling a hybrid of analog and digital computation, is highly efficient because it leverages the highly efficient, low-precision regime of analog (1-bit) with the highly efficient, high-precision regime of digital (multi-bit), while overcoming the access costs associated with traditional memory operations.

[0174] While a wide range of near-memory computing hardware can be considered, details of the hardware integrated in the current embodiment of the present invention are described below. To facilitate the physical layout of such multi-bit digital hardware, eight in-memory computing channels are multiplexed into each near-memory computing channel. Note that this enables highly parallel operations of in-memory computing with throughput that matches the high-frequency operation of digital near-memory computing (highly parallel analog in-memory computing operates at a lower clock frequency than digital near-memory computing). Each near-memory computing channel then includes digital barrel shifters, multipliers, accumulators, and implementations of look-up tables (LUTs) and fixed nonlinear functions. In addition, a configurable finite state machine (FSM) associated with the near-memory computing hardware is integrated to control the computation through the hardware.

[0175] Input Interfacing and Bit Scalability Control To integrate in-memory computation with programmable microprocessors, internal bitwise operations and representations must be properly interfaced with the external multi-bit representations employed in typical microprocessor architectures. Therefore, data reshaping buffers are included in both the input vector interface and the memory read / write interface, with matrix elements stored in a memory array. Details of the design employed in embodiments of the present invention are described below. The data reshaping buffers allow for bit-width scalability of input vector elements while maintaining maximum bandwidth for data transfers to and from external memory as well as other architectural blocks to the in-memory computation hardware. The data reshaping buffers consist of a register file that acts as a line buffer, receiving incoming parallel multi-bit data for each element of an input vector and providing outgoing parallel single-bit data for all vector elements.

[0176] In addition to word-wise / bit-wise interfacing, hardware support for convolution operations applied to input vectors is also included. Such operations are prominent in convolutional neural networks (CNNs). In this case, matrix-vector multiplication is performed on only a subset of the new vector elements that need to be provided (other input vector elements are stored in a buffer and simply shifted appropriately). This alleviates bandwidth constraints for retrieving data to high-throughput in-memory computation hardware. In embodiments of the present invention, convolution support hardware that must perform proper bit-serial sequencing of multi-bit input vector elements is implemented in a dedicated buffer, where the output read shifts the data appropriately for configurable convolution striding.

[0177] Dimensionality and sparsity control For programmability, two additional considerations must be addressed by the hardware: (1) the dimensions of the matrices / vectors may be variable between applications, and (2) in many applications, the vectors will be sparse.

[0178] Regarding dimensionality, in-memory computing hardware often integrates controls to enable / disable tiled portions of an array and consume energy only at the dimensionality level desired by the application. However, in the BPBS approach, the input vector dimensionality has a significant impact on computational energy and SNR. Regarding SNR, assuming that bit-wise computation in each in-memory computation channel between each input (provided on the input line) and the data stored in the bit cell produces a 1-bit output, the number of possible different levels on the accumulation line is equal to N+1, where N is the input vector dimensionality. This suggests the need for a log2(N+1)-bit ADC. However, ADCs have an energy cost that scales strongly with the number of bits. Therefore, to reduce the relative contribution of ADC energy, it may be beneficial to support a very large N, but smaller than log2(N+1) bits, in the ADC. As a result, the signal-to-quantization-noise ratio (SQNR) of the computational operation decreases with the number of ADC bits, unlike standard fixed-precision computation. Therefore, hardware support for configurable input vector dimensionality is essential to support various application-level dimensionality and SQNR requirements with corresponding energy consumption. For example, if a reduced SQNR is acceptable, input vector segments with large dimensionality should be supported, while if a high SQNR must be maintained, input vector segments with lower dimensionality should be supported, with dot product results from multiple input vector segments that can be combined from different in-memory computation banks. (Thus, in particular, the input vector dimensionality can be reduced to a level set by the number of ADC bits to ensure computations that ideally align with standard fixed-precision arithmetic.) The hybrid analog / digital approach taken in this invention makes this possible. That is, input vector elements can be masked to filter the broadcast to only the desired dimensionality. This saves broadcast energy and bitcell computation energy proportional to the input vector dimensionality.

[0179] For sparseness, the same masking approach can be applied across bit-serial operations to prevent broadcasting of all input vector element bits corresponding to zero-valued elements. Note that the employed BPBS approach is particularly conducive to this. This is because the expected number of nonzero elements is often known in sparse linear algebra applications, but the input vector dimensionality can be large. Thus, the BPBS approach allows for an increase in input vector dimensionality while ensuring that the number of levels required to be supported on the accumulation line is within the ADC resolution, thereby ensuring a high computational SQNR. While the expected number of nonzero elements is known, it is still essential to support a variable number of actual nonzero elements, which can vary from input vector to input vector. This is easily achieved with a hybrid analog / digital approach, since the masking hardware simply counts the number of zero-valued elements in a given vector and then applies a corresponding offset to the final dot-product result in the digital domain after the BPBS operation.

[0180] Exemplary Integrated Circuit Architecture Figure 26 depicts a high-level block diagram of an exemplary architecture according to an embodiment. Specifically, the exemplary architecture of Figure 26 has been implemented as an integrated circuit using VLSI processing techniques using specific components and functional elements to test various embodiments herein. It will be understood that additional embodiments having different components (e.g., larger or more powerful CPUs, memory elements, processing elements, etc.) are contemplated by the inventors to be within the scope of this disclosure.

[0181] As depicted in FIG. 26, architecture 200 includes a central processing unit (CPU) 210 (e.g., a 32-bit RISC-V CPU), a program memory (PMEM) 220 (e.g., 128 KB program memory), a data memory (DMEM) 230 (e.g., 128 KB data memory), an external memory interface 235 (e.g., illustratively configured to access one or more 32-bit external memory devices (not shown) to thereby expand accessible memory), a boot loader module 240 (e.g., configured to access an 8 KB off-chip EEPROM (not shown)), a computational memory unit (CIMU) 300 including various configuration registers 255 and configured to perform in-memory calculations and various other functions in accordance with embodiments described herein, a direct memory access (DMA) module 260 including various configuration registers 265, and various support / peripheral modules such as a universal asynchronous receiver / transmitter (UART) module 271 for receiving / transmitting data, a general-purpose input / output (GPIO) module 273, various timers 274, etc. Other elements not depicted herein may also be included in the architecture 200 of FIG. 26, such as an SoC configuration module (not shown).

[0182] While CIMU 300 is well suited for matrix-vector multiplication and the like, other types of calculations / computations may be better performed by non-CIMU computing devices. Thus, in various embodiments, a close coupling between CIMU 300 and near memory is provided so that the selection of computing devices tasked with particular calculations and / or functions can be controlled to provide more efficient computing capabilities.

[0183] Figure 27 depicts a high-level block diagram of an exemplary in-memory computation unit (CIMU) 300 suitable for use in the architecture of Figure 26. The following description relates to the architecture 200 of Figure 26 as well as an exemplary CIMU 300 suitable for use within the context of that architecture 200.

[0184] Generally speaking, CIMU 300 is an in-memory computation array (CIMA) comprising various structural elements, illustratively including a bitcell CIMA configured via various configuration registers to provide programmable in-memory computation functions such as matrix vector multiplication. In particular, exemplary CIMU 300 is configured as a 590 kb, 16-bank CIMU tasked with multiplying an input matrix X by an input vector A to produce an output matrix Y.

[0185] Referring to FIG. 27, CIMU 300 is depicted as including an in-memory computation array (CIMA) 310, an input activity vector reshaping buffer (IA BUFF) 320, a sparsity / AND logic controller 330, a memory read / write interface 340, a row decoder / WL driver 350, a plurality of A / D converters 360, and a near-memory computation multiply-shift-accumulate data path (NMD) 370.

[0186] The exemplary in-memory computation array (CIMA) 310 comprises a 256×(3×3×256) in-memory computation array arranged as a 4×4 clock gateable 64×(3×3×64) in-memory computation array, thus having a total of 256 in-memory computation channels (e.g., memory columns), in which case 256 ADCs 360 are also included to support the in-memory computation channels.

[0187] IA BUFF 320 illustratively operates to receive a sequence of 32-bit data words and reshape these 32-bit data words into a sequence of high-dimensional vectors suitable for processing by CIMA 310. Note that 32-bit, 64-bit, or any other width data words may be reshaped to fit the available or selected size of the computations in memory array 310, and that memory array 310 itself is configured to operate on high-dimensional vectors, including elements that may be 2-8 bits, 1-8 bits, or some other size, and apply them in parallel across the entire array. Note also that while the matrix-vector multiplication operations described herein are depicted as utilizing the entire CIMA 310, in various embodiments, only a portion of CIMA 310 is used. Furthermore, in various other embodiments, CIMA 310 and associated logic circuitry are adapted to provide interleaved matrix-vector multiplication operations, in which parallel portions of a matrix are processed simultaneously by respective portions of CIMA 310.

[0188] In particular, IA BUFF 320 reshapes sequences of 32-bit data words into highly parallel data structures that can be added to CIMA 310 at once (or at least in larger chunks) and properly sequenced in a bit-serial manner. For example, a 4-bit calculation with eight vector elements can be associated with a high-dimensional vector of over 2000 n-bit data elements. IA BUFF 320 shapes this data structure.

[0189] As depicted herein, IA BUFF 320 is illustratively configured to receive input matrix X as a sequence of 32-bit data words and resize / rearrange the received sequence of data words according to the size of CIMA 310 to provide a data structure illustratively including 2303 n-bit data elements. Each of these 2303 n-bit data elements, along with a respective masking bit, is communicated from IA BUFF 320 to sparsity / AND logic controller 330.

[0190] Sparsity / AND-logic controller 330 is illustratively configured to receive 2303 n-bit data elements and their respective masking bits, and responsively invoke a sparsity function such that zero-valued data elements (as indicated by their respective masking bits) are not propagated to CIMA 310 for processing. In this manner, energy otherwise required for processing such bits by CIMA 310 is conserved.

[0191] During operation, CPU 210 reads PMEM 220 and boot loader 240 through direct data paths implemented in a standard manner. CPU 210 may also access DMEM 230, IA BUFF 320, and memory read / write buffer 340 through direct data paths implemented in a standard manner. All these memory modules / buffers, CPU 210, and DMA module 260 are connected by AXI bus 281. Chip configuration modules and other peripheral modules are grouped by APB bus 282, which is attached as a slave to AXI bus 281. CPU 210 is configured to write to PMEM 220 through AXI bus 281. DMA module 260 is configured to access DMEM 230, IA BUFF 320, memory read / write buffer 340, and NMD 370 through dedicated data paths, and to access all other accessible memory spaces via the AXI / APB bus, such as per DMA controller 265. CIMU 300 performs the BPBS matrix-vector multiplication described above. Further details of these and other embodiments are provided below.

[0192] Thus, in various embodiments, the CIMA operates in a bit-serial-bit-parallel (BSBP) manner to receive vector information, perform matrix-vector multiplication, and provide a digitized output signal (i.e., Y=AX), which may be further processed by another computational function as appropriate to provide a combined matrix-vector multiplication function. Generally speaking, embodiments described herein provide an in-memory computing architecture comprising: a reshaping buffer configured to reshape a received sequence of data words to form a massively parallel bit-wise input signal; an in-memory computing (CIM) array of bit cells configured to receive the massively parallel bit-wise input signal via a first CIM array dimension and to receive one or more accumulate signals via a second CIM array dimension, wherein each of a plurality of bit cells associated with a common accumulate signal forms a respective CIM channel configured to provide a respective output signal; an analog-to-digital converter (ADC) circuit configured to process the plurality of CIM channel output signals, thereby providing a sequence of multi-bit output words; a control circuit configured to cause the CIM array to perform multi-bit computation operations on the input and accumulate signals using single-bit internal circuits and signals; and a near-memory computation path configured to provide a sequence of multi-bit output words as a result of the computation.

[0193] Memory Map and Programming Model Because CPU 210 is configured to directly access IA BUFF 320 and memory read / write buffer 340, these two memory spaces resemble DMEM 230 from the perspective of a user program and in terms of latency and energy, especially for structured data such as array / matrix data. In various embodiments, when the in-memory computation feature is not activated or partially activated, memory read / write buffer 340 and CIMA 310 can be used as regular data memory.

[0194] FIG. 28 illustrates a high-level block diagram of an input activity vector reshaping buffer (IA BUFF) 320 suitable for use in the architecture of FIG. 26 , according to an embodiment. The illustrated IA BUFF 320 supports input activity vectors with element precision from 1 bit to 8 bits, although other precisions may be supported in various embodiments. According to the bit-serial flow mechanism discussed herein, specific bits of all elements of the input activity vector are broadcast to the CIMA 310 at once for the matrix-vector multiplication operation. However, the high parallelism of this operation requires providing maximum bandwidth and minimum energy for elements of high-dimensional input activity vectors; otherwise, the throughput and energy-efficiency benefits of in-memory computation would not be utilized. To achieve this, the Input Activity Value Reshaping Buffer (IA BUFF) 320 may be constructed in such a way that in-memory computation can be integrated into the 32-bit (or other bit width) architecture of the microprocessor, thereby maximizing the use of the corresponding hardware for 32-bit data transfers for the highly parallel internal organization of the in-memory computation.

[0195] Referring to FIG. 28, IA BUFF 320 receives a 32-bit input signal, which may contain input vector elements with bit precisions ranging from 1 to 8 bits. Therefore, the 32-bit input signal is first stored in 4×8-bit registers 410, of which there are 24 in total (referred to herein as registers 410-0 through 410-23). ​​These registers 410 provide their contents to eight register files (referred to herein as register files 420-0 through 420-8), each with 96 columns, so that input vectors with a maximum dimensionality of 3×3×256=2304 have their elements arranged in parallel columns. For 8-bit input elements, this is accomplished by 24 4×8-bit registers 410 providing 96 parallel outputs across one of the register files 420; for 1-bit input elements, this is accomplished by 24 4×8-bit registers 410 providing 1536 parallel outputs across all eight register files 420 (or some other intermediate configuration of bit precision). The height of each register file column is 2x4x8 bits, which allows each input vector (with up to 8-bit element precision) to be stored in four segments when all input vector elements are loaded, enabling double buffering. On the other hand, when only one-third of the input vector elements are loaded (i.e., a CNN with a stride of 1), one out of every four register file columns acts as a buffer, allowing data from the other three columns to be forward propagated to the CIMU for computation.

[0196] Thus, of the 96 columns output by each register file 420, only 72 are selected by the respective circular barrel shifting interface 430, giving a total of 576 outputs at a time across the eight register files 420. These outputs correspond to one of the four input vector segments stored in the register files. Thus, four cycles are required to load all input vector elements into the sparsity / AND logic controller 330 in 1-bit registers.

[0197] To exploit sparsity in the input activity vector, a mask bit is generated for each data element while the CPU 210 or DMA 260 writes to the reshaping buffer 320. The masked input activity values ​​prevent charge-based calculation operations in the CIMA 310, thereby saving computational energy. The mask vector is also stored in an SRAM block and is organized similarly to the input activity vector, but in a single-bit representation.

[0198] A 4-to-3 barrel shifter 430 is used to support VGG-style (3x3 filter) CNN computations. When moving to the next filtering operation (convolutional reuse), only one out of three input activation vectors need to be updated, which saves energy and improves throughput.

[0199] Figure 29 depicts a high-level block diagram of a CIMA read / write buffer 340 suitable for use in the architecture of Figure 26, according to an embodiment. The depicted CIMA read / write buffer 340 is illustratively organized as a 768-bit wide static random access memory (SRAM) block 510, while the depicted CPU's word width is 32 bits in this example, and the read / write buffer 340 is used to interface therebetween.

[0200] The depicted read / write buffer 340 contains a 768-bit write register 511 and a 768-bit read register 512. The read / write buffer 340 generally operates like a cache to wide SRAM blocks in CIMA 310, although some details differ. For example, the read / write buffer 340 writes back to CIMA 310 only when CPU 210 writes to a different row, while reading a different row does not trigger a writeback. When a read address matches the tag of a write register, the modified byte (indicated by the dirty bit) in write register 511 is bypassed to read register 512 rather than read from CIMA 310.

[0201] Accumulation Line Analog-to-Digital Converters (ADCs). Each accumulation line from the CIMA310 has an 8-bit SAR ADC that matches the pitch of the in-memory computation channel. To save area, the finite state machine (FSM) that controls the bit cycling of the SAR ADCs is shared among the 64 ADCs required for each in-memory computation tile. The FSM control logic consists of 8+2 shift registers, generating pulses that cycle through the reset, sample, and subsequent 8-bit decision phases. The shift register pulses are broadcast to the 64 ADCs, buffered locally, trigger local comparator decisions, store the corresponding bit decisions in local ADC code registers, and are used to trigger the next capacitor DAC configuration. Precision metal-oxide-metal (MOM) caps may be used to enable the small size of each ADC's capacitor array.

[0202] Figure 30 depicts a high-level block diagram of a near-memory datapath (NMD) module 600 suitable for use in the architecture of Figure 26, according to one embodiment, which can employ digital near-memory computation with other features. The depicted NMD module 600 depicted in Figure 30 shows a digital computation datapath after the ADC output that supports multi-bit matrix multiplication via a BPBS scheme.

[0203] In a particular embodiment, the 256 ADC outputs are organized into groups of 8 for digital computation flow, allowing support for matrix element configurations up to 8 bits. Thus, the NMD module 600 contains 32 identical NMD units. Each NMD unit consists of a multiplexer 610 / 620 for selecting among eight ADC outputs 610 and corresponding biases 621, a multiplicand 622 / 623, a shift number 624 and an accumulation register, an adder 631 with an 8-bit unsigned input and a 9-bit signed input for subtracting the global bias and mask count, a signed adder 632 for calculating the local bias for neural network tasks, a fixed-point multiplier 633 for performing scaling, a barrel shifter 634 for calculating the exponent of the multiplicand and performing shifts for different bits of the weight element, a 32-bit signed adder 635 for performing accumulation, eight 32-bit signed accumulation registers 640 for supporting weights with 1, 2, 4, and 8-bit configurations, and a ReLU unit 650 for neural network applications.

[0204] Figure 31 depicts a high-level block diagram of a direct memory access (DMA) module 700 suitable for use in the architecture of Figure 26, according to an embodiment. The depicted DMA module 700 illustratively comprises two channels to support simultaneous data transfers to / from different hardware resources, and five independent data paths to / from the DMEM, IA BUFF, CIMU R / W BUFF, NMD result, and AXI4 bus, respectively.

[0205] Bit-Parallel / Bit-Serial (BPBS) Matrix Vector Multiplication Multi-bit MVM

number

[0206] The bitwise AND can support multi-bit matrix and standard two's complement representation of input vector elements. This involves appropriately applying a negative sign corresponding to the most significant bit (MSB) element to the column calculation in the digital domain after the ADC before adding the digitized output to the output of the other column calculation.

[0207] Bitwise XNOR requires a slight modification of the numeric representation: the constituent bits map to +1 / -1 instead of 1 / 0, and two bits with equal LSB weighting are required to properly represent zero. This is done as follows: First, each B-bit operand (in standard two's complement representation) is decomposed into a B+1-bit signed integer. For example, y is represented as B+1 plus / minus 1 bits.

number

number

[0208] If the bits of 1 / 0 values ​​map to +1 / -1 mathematical values, bitwise in-memory computation multiplication may be realized via a logical XNOR operation. Thus, an M-BC that performs logical XNOR using differential signals of input vector elements can enable signed multi-bit multiplication by bit-weighting and summing the digitized outputs from column computations.

[0209] While AND-based M-BC multiplication and XNOR-based M-BC multiplication present two options, other options are possible by using appropriate number representations with the logical operations possible in M-BC. Such alternatives are beneficial. For example, XNOR-based M-BC multiplication is preferable for binary (1-bit) calculations, while AND-based M-BC multiplication allows for more standard number representations for easier integration within digital architectures. Furthermore, the two approaches result in slightly different signal-to-quantization-noise ratios (SQNRs), which can therefore be selected based on application needs.

[0210] Heterogeneous Computing Architecture and Interface Various embodiments described herein contemplate different aspects of charge-domain in-memory computation in which a bitcell (or multiplication bitcell, M-BC) drives an output voltage corresponding to the computation result onto a local capacitor. Capacitors from an in-memory computation channel (column) are then coupled to produce an accumulation via charge redistribution. As noted above, such capacitors can be formed using specific geometries that are very easy to replicate, such as in VLSI processes, via wiring that is simply adjacent to each other and thus coupled via an electric field. Thus, a local bitcell formed as a capacitor stores a charge representing a one or zero, while the local summation of all of the charges of a number of these capacitors or bitcells enables implementation aspects of the multiplication and accumulate / sum functions that are core operations in matrix-vector multiplication.

[0211] The various embodiments described above advantageously provide improved bitcell-based architectures, computational engines, and platforms. Matrix-vector multiplication is one operation that is not efficiently performed by standard digital processing or digital acceleration. Therefore, this one type of in-memory computation offers enormous advantages over existing digital designs. However, various other types of operations are efficiently performed using digital designs.

[0212] Various embodiments contemplate mechanisms for connecting / interfacing these bitcell-based architectures, computational engines, platforms, etc. to more conventional digital computing architectures and platforms to form heterogeneous computing architectures. In this way, those computational operations well suited to bitcell architecture processing (e.g., matrix-vector processing) are processed as described above, while those other computational operations well suited to traditional computer processing are processed via traditional computer architectures. That is, various embodiments provide a computing architecture that includes the highly parallel processing mechanisms described herein, which are connected to multiple interfaces so that they can be externally coupled to more conventional digital computing architectures. In this way, digital computing architectures can be directly and efficiently aligned with in-memory computing architectures, allowing the two to be located in close proximity and minimizing data movement overhead between them. For example, a machine learning application may involve 80%-90% matrix-vector computations, yet still have 10%-20% of other types of computations / operations remaining to be performed. By combining the in-memory computing discussed herein with near-memory computing, which is more conventional in architecture, the resulting system offers exceptional configurability for performing many types of processing. Thus, various embodiments contemplate near-memory digital computing in conjunction with the in-memory computing described herein.

[0213] The in-memory operations discussed herein are massively parallel, but single-bit operations. For example, in a bit cell, only one bit can be stored: a one or a zero. The signal driven into the bit cell is typically an input vector (i.e., in a 2D vector multiplication operation, each matrix element is multiplied by each vector element). The vector elements are placed on signals that are both digital and only one bit, so that the vector elements are also one bit.

[0214] Various embodiments use a bit-parallel / bit-serial approach to expand matrices / vectors from single-bit elements to multi-bit elements.

[0215] Figures 8A-8B depict high-level block diagrams of different CIMA channel digitization / weighting embodiments suitable for use in the architecture of Figure 26. Specifically, Figure 32A depicts a digital binary weighting and summing embodiment similar to that described above with respect to various other figures. Figure 32B depicts an analog binary weighting and summing embodiment with modifications to various circuit elements to enable the use of fewer analog-to-digital converters than the embodiment of Figure 32A and / or other embodiments described herein.

[0216] As discussed above, various embodiments contemplate a computational in-memory (CIM) array of bit cells configured to receive massively parallel bit-wise input signals via a first CIM array dimension (e.g., rows of a 2D CIM array) and one or more accumulation signals via a second CIM array dimension (e.g., columns of a 2D CIM array), with each of a plurality of bit cells (e.g., depicted as columns of bit cells) associated with a common accumulation signal forming a respective CIM channel configured to provide a respective output signal. An analog-to-digital converter (ADC) circuit is configured to process the plurality of CIM channel output signals, thereby providing a sequence of multi-bit output words. A control circuit is configured to cause the CIM array to perform multi-bit computation operations on the input signals and accumulation signals using single-bit internal circuits and signals, such that a near-memory computation path operatively associated with the control circuit can be configured to provide a sequence of multi-bit output words as a computation result.

[0217] 32A, a digital binary weighting and summing embodiment for performing ADC circuit functions is depicted. In particular, a two-dimensional CIMA 810A receives matrix input values ​​along a first (row) dimension (i.e., via multiple buffers 805) and vector input values ​​along a second (column) dimension, which CIMA 810A operates on in accordance with control circuitry or the like (not shown) to provide various channel output signals CH-OUT.

[0218] The ADC circuit of FIG. 32A provides, for each CIM channel, a respective ADC 760 configured to digitize the CIM channel output signal CH-OUT, and a respective shift register 865 configured to apply a respective binary weighting to the digitized CIM channel output signal CH-OUT, thereby forming a respective portion of a multi-bit output word 870.

[0219] 32B, an analog binary weighting and summing embodiment for performing ADC circuit functions is depicted. In particular, a two-dimensional CIMA 810B receives matrix input values ​​in a first (row) dimension (i.e., via multiple buffers 805) and vector input values ​​in a second (column) dimension, which CIMA 810B operates in accordance with control circuitry or the like (not shown) to provide various channel output signals CH-OUT.

[0220] The ADC circuit of FIG. 32B provides four controllable (or pre-set) banks of switches 815-1, 815-2, etc. within CIMA 810B to couple and / or isolate capacitors formed in CIMA 810B, thereby implementing an analog binary weighting scheme for each of one or more subgroups of channels, each of which digitizes the weighted analog sum of the CIM channel output signals of a respective subset of CIM channels, thereby providing a single output signal such that only one ADC 860B is required to form each portion of a multi-bit output word.

[0221] Figure 33 depicts a flow diagram of a method according to an embodiment. Specifically, method 900 of Figure 33 is directed to various processing operations implemented by architectures, systems, etc. described herein where input matrices / vectors are augmented to be computed in a bit-parallel / bit-serial approach.

[0222] In step 910, the matrix and vector data is loaded into the appropriate memory locations.

[0223] In step 920, each of the vector bits (from MSB to LSB) is processed sequentially. Specifically, the MSB of the vector is multiplied by the MSB of the matrix, the MSB of the vector is multiplied by MSB-1 of the matrix, the MSB of the vector is multiplied by MSB-2 of the matrix, and so on until the MSB of the vector is multiplied by the LSB of the matrix. The resulting analog charge results are then digitized for each vector multiplication from MSB to LSB to obtain a result, which is latched. This process is repeated for vector MSB-1, vector MSB-2, etc., up to vector LSB, until each of vectors MSB-LSB has been multiplied by each of the MSB-LSB elements of the matrix.

[0224] In step 930, the bits are shifted to apply the appropriate weighting and the results are added together. Note that in some embodiments where analog weighting is used, the shifting operation of step 930 is not necessary.

[0225] Various embodiments enable highly stable and robust computation within circuits used to store data in high-density memories. Furthermore, various embodiments advance the computational engines and platforms described herein by enabling higher densities for memory bitcell circuits. Density can be increased both due to more compact layouts and due to improved compatibility of layouts with highly aggressive design rules (i.e., push rules) used for memory circuits. Various embodiments substantially improve processor performance for machine learning and other linear algebra applications.

[0226] We have disclosed a bitcell circuit that can be used within in-memory computing architectures. The disclosed approach allows for highly stable / robust computations to be performed within circuits used to store data in high-density memories. The disclosed approach for robustness in memory computing allows for higher density memory bitcell circuits than known approaches. Density can be increased both due to a more compact layout and due to improved compatibility of the layout with highly aggressive design rules (i.e., push rules) used for memory circuits. The disclosed devices can be fabricated using standard CMOS integrated circuit processing.

[0227] Partial List of Disclosed Embodiments Aspects of various embodiments are set forth in the claims. These and other aspects of at least a subset of various embodiments are set forth in the following numbered clauses.

[0228] 1. An integrated in-memory computing (IMC) architecture, the IMC architecture being configurable to support the data flow of an application mapped to the IMC, comprising a plurality of configurable Compute-In-Memory Units (CIMUs) forming an array of CIMUs, the CIMUs configured to communicate activation values ​​to / from other CIMUs or other structures within or outside the CIMUs via respective configurable inter-CIMU network portions disposed therebetween, and to communicate weights to / from other CIMUs or other structures within or outside the CIMUs via respective configurable operand loading network portions disposed therebetween.

[0229] 2. The integrated IMC architecture described in clause 1, wherein each CIMU comprises a configurable input buffer for receiving computational data from the inter-CIMU network and composing the received computational data into an input vector for generating an output feature vector through matrix vector multiplication (MVM) processing by the CIMU.

[0230] 3. The integrated IMC architecture described in clause 1, wherein each CIMU comprises a configurable input buffer for receiving computational data from an inter-CIMU network, and each CIMU configures the received computational data into an input vector for generating an output feature vector by a matrix vector multiplication (MVM) process.

[0231] 4. The integrated IMC architecture of clause 2 or 3, wherein each CIMU is associated with a configurable shortcut buffer for receiving computational data from the inter-CIMU network according to a data flow map, imparting a time delay to the received computational data, and forwarding the delayed computational data to the next CIMU.

[0232] 5. The integrated IMC architecture of clause 2 or 3, wherein each CIMU is associated with a configurable shortcut buffer for receiving computational data from an inter-CIMU network, imparting a time delay to the received computational data, and forwarding the delayed computational data towards a configurable input buffer.

[0233] 6. The integrated IMC architecture of clause 2 or 3, wherein each CIMU includes parallelized computation hardware configured to process input data received from at least one of the respective input buffers and shortcut buffers.

[0234] 7. The integrated IMC architecture of clause 4 or 5, wherein each CIMU shortcut buffer is configured according to a data flow map such that data flow alignment between multiple CIMUs is maintained.

[0235] 8. The integrated IMC architecture of clause 4 or 5, wherein the shortcut buffers of each of a plurality of CIMUs in the array of CIMUs are configured according to a data flow map that supports pixel-level pipelining to provide pipeline latency matching.

[0236] 9. The integrated IMC architecture of clause 4 or 5, wherein the time delay imparted by the shortcut buffer of the CIMU includes at least one of an absolute time delay, a predetermined time delay, a time delay determined with respect to the size of the input calculation data, a time delay determined with respect to the expected calculation time of the CIMU, a control signal received from a data flow controller, a control signal received from another CIMU, and a control signal generated by the CIMU in response to the occurrence of an event within the CIMU.

[0237] 10. The integrated IMC architecture of clause 4, 5 or 6, wherein each configurable input buffer is capable of imparting a time delay to computational data received from the inter-CIMU network or shortcut buffer.

[0238] 11. The integrated IMC architecture of clause 10, wherein the time delay imparted by the configurable input buffer of the CIMU comprises at least one of an absolute time delay, a predetermined time delay, a time delay determined with respect to the size of the input calculation data, a time delay determined with respect to the expected calculation time of the CIMU, a control signal received from a data flow controller, a control signal received from another CIMU, and a control signal generated by the CIMU in response to the occurrence of an event within the CIMU.

[0239] 12. The integrated IMC architecture of claim 1, wherein at least a subset of the CIMUs, the inter-CIMU network portion, and the operand loading network portion are configured according to the data flow of the application mapped to the IMC.

[0240] 13. The integrated IMC architecture of clause 9, wherein at least a subset of the CIMUs, inter-CIMU network portions, and operand loading network portions are configured according to the layer data flow by layer mapping of a neural network (NN) to the IMC, whereby parallel output activation values ​​calculated by a configured CIMU executing in a given layer are provided to a configured CIMU executing in a next layer, and the parallel output activation values ​​form respective NN feature map pixels.

[0241] 14. The integrated IMC architecture of clause 13, wherein the configurable input buffer is configured to transfer input NN feature map data to parallelized computation hardware within the CIMU according to a selected stride step.

[0242] 15. The integrated IMC architecture of clause 14, wherein the NN includes a convolutional neural network (CNN) and the input line buffer is used to buffer a number of rows of the input feature map corresponding to the size of the CNN kernel.

[0243] 16. The integrated IMC architecture of clause 2 or 3, wherein each CIMU comprises an in-memory computation (IMC) bank configured to perform matrix vector multiplication (MVM) according to a bit-parallel bit-serial (BPBS) computation process in which a single-bit computation is performed using iterative barrel shifting with a column weighting process, followed by a result accumulation process.

[0244] 17. The integrated IMC architecture of clause 2 or 3, wherein each CIMU comprises an in-memory computation (IMC) bank configured to perform matrix vector multiplication (MVM) according to a bit-parallel bit-serial (BPBS) computation process in which a single-bit computation is performed using an iterative column merge with a column weighting process, followed by a result accumulation process.

[0245] 18. The integrated IMC architecture of clause 2 or 3, wherein each CIMU comprises an in-memory computation (IMC) bank configured to perform matrix vector multiplication (MVM) according to a bit-parallel bit-serial (BPBS) computation process, in which elements of the IMC bank are allocated using a BPBS expansion process.

[0246] 19. The integrated IMC architecture of clause 18, wherein the IMC bank elements are further configured to perform said MVM using a duplication and shifting process.

[0247] 20. The integrated IMC architecture of clause 4 or 5, wherein each CIMU is associated with a respective near memory, programmable single instruction multiple data (SIMD) digital engine, and the SIMD digital engine is suitable for use in combining or temporally aligning input buffer data, shortcut buffer data, and / or output feature vector data for inclusion within a feature vector map.

[0248] 21. The integrated IMC architecture of clause 20, wherein at least a portion of the CIMUs are associated with respective lookup tables for mapping inputs to outputs according to a plurality of nonlinear functions, and wherein the nonlinear function output data is provided to a SIMD digital engine associated with each CIMU.

[0249] 22. The integrated IMC architecture of clause 20, wherein at least a portion of the CIMUs are associated with parallel lookup tables for mapping inputs to outputs according to a plurality of nonlinear functions, and wherein the nonlinear function output data is provided to a SIMD digital engine associated with each CIMU.

[0250] 23. An in-memory computation (IMC) architecture for mapping a neural network (NN) onto the IMC architecture; an on-chip array of in-memory computation units (CIMUs), logically configurable as elements within a layer of a NN mapped to the CMIU, each CIMU output activation value including a respective feature vector supporting a respective portion of a data flow associated with the mapped NN, and the parallel output activation values ​​computed by the CIMUs executing in a given layer forming feature map pixels; an on-chip activity network configured to communicate CIMU output activation values ​​between adjacent CIMUs, such that the parallel output activation values ​​calculated by the CIMUs executing in a given layer form feature map pixels; and an on-chip operand loading network for communicating the weights to adjacent CIMUs via respective weight loading interfaces between the adjacent CIMUs;

[0251] 24. Any of the above clauses, modified as necessary to provide a dataflow architecture for in-memory computation in which computational inputs and outputs are passed from one in-memory computation block to the next via a configurable on-chip network.

[0252] 25. Any of the above clauses, modified as necessary to provide a dataflow architecture for in-memory computing in which an in-memory computing module may receive input from, and provide output to, multiple in-memory computing modules.

[0253] 26. Any of the above clauses, modified as necessary to provide a dataflow architecture for in-memory computation in which appropriate buffering is provided to the inputs or outputs of the in-memory computation modules to allow the inputs and outputs to flow between the modules in a synchronized manner.

[0254] 27. Any of the above clauses, modified as necessary to provide a dataflow architecture in which parallel data corresponding to an output channel of a particular pixel in an output feature map of a neural network is passed from one in-memory computation block to the next.

[0255] 28. Any of the above clauses, modified as necessary to provide a method of mapping neural network computations to in-memory computations in which neural network weights are stored in memory as matrix elements, with memory columns corresponding to different output channels.

[0256] 29. Any of the above clauses, modified as necessary to provide a method for mapping neural network computations onto in-memory computation hardware in which matrix elements stored in memory may be modified over the course of the computation.

[0257] 30. Any of the above clauses, modified as necessary to provide a method for mapping neural network computations to in-memory computation hardware in which matrix elements stored in memory may be stored in multiple in-memory computation modules or locations.

[0258] 31. Any of the above clauses, as modified as necessary to provide a method for mapping neural network computations to in-memory computational hardware where multiple neural network layers are mapped at once (layer unfolding).

[0259] 32. Any of the above clauses, modified as necessary to provide a method of mapping neural network computations onto in-memory computing hardware that performs bitwise operations, where different matrix element bits are mapped to the same column (BPBS expansion).

[0260] 33. Any of the above clauses, modified as necessary to provide a method for mapping multiple matrix element bits to the same column where higher order bits are duplicated to allow for proper analog weighting (column merging).

[0261] 34. Any of the above clauses, modified as necessary to provide a method of mapping multiple matrix element bits to the same column where elements are duplicated and shifted, and higher order input vector elements are provided in rows with shifted elements (duplicating and shifting).

[0262] 35. Any of the above clauses, modified as necessary to provide a method for mapping neural network computations onto in-memory computation hardware that performs bitwise operations but where multiple input vector bits are provided simultaneously as multi-valued (analog) signals.

[0263] 36. Any of the above clauses, modified as necessary to provide a method for multi-value input vector element signaling in which a multi-value driver employs dedicated voltage supplies that are selected by decoding multiple bits of the input vector element.

[0264] 37. Any of the above clauses, modified as necessary to provide a multi-value driver whose dedicated power supply may be configured off-chip (e.g., to support XNOR calculations and numeric formats for calculations).

[0265] 38. Any of the above clauses, as modified as necessary, to provide a modular architecture for in-memory computation in which modular tiles are arranged in groups to achieve scale-up.

[0266] 39. Any of the above clauses, as modified as necessary, to provide a modular architecture for in-memory computation in which modules are connected by a configurable on-chip network.

[0267] 40. Any of the above clauses, as modified as necessary to provide a modular architecture for in-memory computation, wherein the modules include any one or combination of the modules described herein.

[0268] 41. Any of the above clauses, as modified as necessary to provide control and configuration logic for appropriately configuring modules and providing appropriate localized control.

[0269] 42. Any of the above clauses, as modified as necessary, to provide an input buffer for receiving data to be calculated by the module.

[0270] 43. Any of the above provisions, as modified as necessary to provide buffers to provide delays in input data to properly synchronize data flow through the architecture.

[0271] 44. Any of the above provisions, as modified as necessary to provide for local, near-memory computation.

[0272] 45. Any of the above clauses, modified as necessary to provide buffers within a module or as a separate module for synchronizing data flow through the architecture.

[0273] 46. ​​Any of the above clauses, as modified as necessary, to provide near-memory digital computing that is located near the in-memory computing hardware and provides programmable / configurable parallel computing on output data from the in-memory computing.

[0274] 47. Any of the above clauses, modified as necessary to provide a computation data path between parallel output data paths to provide computation between different in-memory computation outputs (e.g., between adjacent in-memory computation outputs).

[0275] 48. Any of the above clauses, modified as necessary to provide a computational data path for data reduction between all parallel output data paths in a hierarchical fashion down to a single output.

[0276] 49. Any of the above clauses, modified as necessary to provide a computational data path that can obtain input from auxiliary sources in addition to in-memory computational output (e.g., shortcut buffers, computation units between input buffers and shortcut buffers, etc.).

[0277] 50. Any of the above provisions, as modified as necessary to provide near-memory digital computation employing instruction decoding and controlling hardware shared among parallel data paths adapted to output data from in-memory computation.

[0278] 51. Any of the above clauses, modified as necessary to provide a near-memory data path providing configurable / controllable multiplication / division, addition / subtraction, bitwise shifting, etc. operations.

[0279] 52. Any of the above clauses, modified as necessary to provide a near-memory data path with local registers for intermediate computation results (scratchpad) and parameters.

[0280] 53. Any of the above clauses, modified as necessary to provide a method for computing arbitrary nonlinear functions between parallel data paths via a shared look-up table (LUT).

[0281] 54. Any of the above clauses, modified as necessary to provide for sequential bit-wise broadcast of look-up table (LUT) bits with a local decoder for LUT decoding.

[0282] 55. Any of the above clauses, modified as necessary to provide an input buffer located proximate to the in-memory computation hardware to provide storage of input data to be processed by the in-memory computation hardware.

[0283] 56. Any of the above clauses, modified as necessary to provide input buffering to enable reuse of data for in-memory computations (e.g., as required for convolution operations).

[0284] 57. Any of the above clauses, modified as necessary to provide input buffering that buffers rows of input feature maps to enable reuse of convolutions in two dimensions of the filter kernel (across rows and across multiple rows).

[0285] 58. Any of the above clauses, modified as necessary to provide input buffering that allows input to be obtained from multiple input ports so that incoming data may be provided from multiple different sources.

[0286] 59. Any of the above clauses, as modified as necessary to provide for a plurality of different ways of arranging data from a plurality of different input ports, for example, one way could be to arrange data from different input ports into different vertical segments of a buffered row.

[0287] 60. Any of the above provisions, modified as necessary to provide for in-memory computational hardware, the ability to access data from an input buffer at a multiple of the clock frequency.

[0288] 61. Any of the above clauses, modified as necessary to provide additional buffering located near the in-memory computation hardware or in a separate location within a tiled array of in-memory computation hardware, but not necessarily providing data directly to the in-memory computation hardware.

[0289] 62. Any of the above clauses, modified as necessary to provide additional buffering to provide appropriate delays for data so that data from different in-memory computational hardware may be properly synchronized (e.g., as in the case of shortcut connections in a neural network).

[0290] 63. Any of the above clauses, modified as necessary to provide additional buffering to enable data reuse for in-memory computations (e.g., required for convolution operations), and optionally to provide input buffering to buffer rows of input feature maps to enable reuse for convolutions in two dimensions of the filter kernel (across rows and across multiple rows).

[0291] 64. Any of the above clauses, as modified as necessary to provide additional buffering to allow input to be obtained from multiple input ports so that incoming data may be provided from multiple different sources.

[0292] 65. Any of the above clauses, as modified as necessary to provide for a plurality of different ways of arranging data from a plurality of different input ports, for example, one way could be to arrange data from different input ports into different vertical segments of a buffered row.

[0293] 66. Any of the above clauses, as modified as necessary, to provide an input interface for in-memory computation hardware for retrieving matrix elements stored in bit cells through an on-chip network.

[0294] 67. Any of the above clauses, modified as necessary to provide an input interface for matrix element data that allows use of the same on-chip network for input vector data.

[0295] 68. Any of the above provisions, modified as necessary to provide computational hardware between the input buffering and additional buffers proximate the in-memory computational hardware.

[0296] 69. Any of the above clauses, modified as necessary to provide computing hardware capable of providing parallel computation between input buffering and output from additional buffering.

[0297] 70. Any of the above clauses, as modified as necessary to provide computing hardware capable of providing computation between the output of input buffering and additional buffering.

[0298] 71. Any of the above clauses, modified as necessary to provide computing hardware, the output of which can feed in-memory computing hardware.

[0299] 72. Any of the above clauses, modified as necessary to provide computing hardware, the output of which can feed near-memory computing hardware subsequent to the in-memory computing hardware.

[0300] 73. Any of the above clauses, modified as necessary to provide an on-chip network between in-memory compute tiles having a modular structure in which segments with parallel routing channels surround the CIMU tiles.

[0301] 74. Any of the above clauses, modified as necessary to provide an on-chip network comprising several routing channels, each capable of obtaining input from and / or providing output to in-memory computational hardware.

[0302] 75. Any of the above clauses, as modified as necessary, to provide an on-chip network with routing resources that can be used to provide data originating from any in-memory computational hardware to any other in-memory computational hardware in the tiled array, and possibly to multiple different in-memory computational hardware.

[0303] 76. Any of the above clauses, as modified as necessary to provide an on-chip network implementation in which in-memory compute hardware provides data to routing resources or obtains data from routing resources via multiplexing between routing resources.

[0304] 77. Any of the above clauses, as modified as necessary to provide for an on-chip network implementation in which connections between routing resources are made via switching blocks at the intersections of the routing resources.

[0305] 78. Any of the above provisions, as modified as necessary to provide for switching blocks capable of providing complete switching between intersecting routing resources, or a subset of complete switching between intersecting routing resources.

[0306] 79. Any of the above provisions, as modified as necessary, to provide software for mapping a neural network onto a tiled array of in-memory computational hardware.

[0307] 80. Any of the above provisions, as modified as necessary, to provide software tools for performing the allocation of in-memory computational hardware to the specific computations required by a neural network.

[0308] 81. Any of the above provisions, as modified as necessary, to provide a software tool that performs placement of allocated in-memory compute hardware to specific locations within a tiled array.

[0309] 82. Any of the above provisions, as modified as necessary, to provide a software tool whose placement is configured to minimize the distance between in-memory computational hardware that provides a particular output and in-memory computational hardware that obtains a particular input.

[0310] 83. Any of the above provisions, as modified as necessary, to provide a software tool that employs optimization methods to minimize such distances (e.g., simulated annealing).

[0311] 84. Any of the above provisions, as modified as necessary, to provide a software tool that performs configuration of available routing resources to route output from in-memory compute hardware to input to in-memory compute hardware in a tiled array.

[0312] 85. Any of the above provisions, as modified as necessary, to provide software tools that minimize the total amount of routing resources required to achieve routing between collocated in-memory compute hardware.

[0313] 86. Any of the above provisions, as modified as necessary, to provide software tools employing optimization methods (e.g., dynamic programming) to minimize such routing resources.

[0314] Various modifications may be made to the systems, methods, devices, mechanisms, techniques, and portions thereof described herein with respect to the various figures, and such modifications are contemplated as being within the scope of the present invention. For example, in various embodiments described herein, while a particular order of steps or arrangement of functional elements is presented, various other orders / arrangements of steps or functional elements may be utilized within the context of various embodiments. Furthermore, while modifications to embodiments may be discussed individually, various embodiments may employ multiple modifications simultaneously or sequentially, may employ composite modifications, etc. As used herein, the term "or" will be understood to refer to a non-exclusive or (e.g., use of "or otherwise" or "or alternatively") unless otherwise indicated.

[0315] While various embodiments incorporating the teachings of the present invention have been shown and described in detail herein, those skilled in the art can readily devise many other various embodiments which still incorporate these teachings. Thus, while the foregoing is directed to various embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof.

Claims

1. Integrated in-memory computing (IMC) hardware, the IMC hardware configurable to support scalable execution and data flow of applications mapped to the IMC hardware; a plurality of configurable Compute-In-Memory Units (CIMUs), the plurality of CIMUs forming an array of CIMUs; a configurable on-chip network for communicating input data to the array of CIMUs, communicating calculated data between CIMUs, and communicating output data from the array of CIMUs, the on-chip network including at least one switch block that provides selective routing of data between one or more CIMUs.

2. 2. The integrated IMC hardware of claim 1, wherein each CIMU comprises an input buffer for receiving computational data from the on-chip network and composing the received computational data into input vectors for matrix vector multiplication (MVM) processing by the CIMU to generate computed data including an output vector.

3. 3. The integrated IMC hardware of claim 2, wherein each CIMU is associated with a shortcut buffer for receiving computational data from the on-chip network according to a data flow map, imparting a time delay to the received computational data, and forwarding the delayed computational data to a next CIMU or output so that data flow alignment among multiple CIMUs is maintained.

4. The integrated IMC hardware of claim 2 , wherein each CIMU includes parallelized computation hardware configured to process input data received from at least one of a respective input buffer and shortcut buffer.

5. 4. The integrated IMC hardware of claim 3, wherein at least one of the input buffer and shortcut buffer of each of the plurality of CIMUs in the array of CIMUs is configured according to a data flow map that supports pixel-level pipelining to provide pipeline latency matching.

6. 4. The integrated IMC hardware of claim 3, wherein the time delay provided by the shortcut buffer of the CIMU includes at least one of an absolute time delay, a predetermined time delay, a time delay determined with respect to a size of input calculation data, a time delay determined with respect to an expected calculation time of the CIMU, a control signal received from a data flow controller, a control signal received from another CIMU, and a control signal generated by the CIMU in response to an occurrence of an event within the CIMU.

7. The integrated IMC hardware of claim 3 , wherein at least some of the input buffers can be configured to impart a time delay to computational data received from the on-chip network or from a shortcut buffer.

8. 8. The integrated IMC hardware of claim 7, wherein the time delay imparted by the input buffer of a CIMU comprises at least one of an absolute time delay, a predetermined time delay, a time delay determined relative to the size of input calculation data, a time delay determined relative to the expected calculation time of the CIMU, a control signal received from a data flow controller, a control signal received from another CIMU, and a control signal generated by the CIMU in response to the occurrence of an event within the CIMU.

9. The integrated IMC hardware of claim 8 , wherein at least a subset of the CIMUs are associated with an on-chip network portion including an operand loading network portion configured according to the data flow of the application mapped to the IMC hardware.

10. 10. The integrated IMC hardware of claim 9, wherein the application mapped to the IMC hardware includes a neural network (NN) mapped to the IMC hardware such that parallel output computed data of a configured CIMU executing at a given layer is provided to a configured CIMU executing at a next layer, the parallel output computed data forming respective NN feature map pixels.

11. The integrated IMC hardware of claim 10 , wherein the input buffer is configured to transfer input NN feature map data to parallelized computation hardware within the CIMU according to a selected stride step.

12. 12. The integrated IMC hardware of claim 11, wherein the NN includes a convolutional neural network (CNN), and the input buffer is used to buffer a number of rows of an input feature map corresponding to a size of a CNN kernel.

13. 3. The integrated IMC hardware of claim 2, wherein each CIMU comprises an in-memory computation (IMC) bank configured to perform matrix vector multiplication (MVM) according to a bit-parallel bit-serial (BPBS) computation process in which a single-bit computation is performed using iterative barrel shifting with a column weighting process, followed by a result accumulation process.

14. 3. The integrated IMC hardware of claim 2, wherein each CIMU comprises an in-memory computation (IMC) bank configured to perform matrix vector multiplication (MVM) according to a bit-parallel bit-serial (BPBS) computation process in which a single-bit computation is performed using an iterative column merge with a column weighting process, followed by a result accumulation process.

15. 3. The integrated IMC hardware of claim 2, wherein each CIMU comprises an in-memory computation (IMC) bank configured to perform matrix vector multiplication (MVM) according to a bit-parallel bit-serial (BPBS) computation process in which elements of the IMC bank are allocated using a BPBS unfolding process.

16. The unified IMC hardware of claim 15 , wherein the IMC bank elements are further configured to execute MVM using a duplication and shifting process.

17. 16. The integrated IMC hardware of claim 15, wherein each CIMU is associated with a respective near memory, programmable single instruction, multiple data (SIMD) digital engine, the SIMD digital engine suitable for use in combining or time-aligning input buffer data, shortcut buffer data, and / or output feature vector data for inclusion within a feature vector map.

18. 16. The integrated IMC hardware of claim 15, wherein at least a portion of the CIMUs include respective lookup tables for mapping inputs to outputs according to a plurality of nonlinear functions, and nonlinear function output data is provided to the SIMD digital engine associated with the respective CIMU.

19. 16. The integrated IMC hardware of claim 15, wherein at least a portion of the CIMUs are associated with parallel lookup tables for mapping inputs to outputs according to a plurality of nonlinear functions, and nonlinear function output data is provided to the SIMD digital engines associated with the respective CIMUs.

20. 2. The integrated IMC hardware of claim 1, wherein each input comprises a multi-bit input, each multi-bit input value being represented by a respective voltage level.

21. The integrated IMC hardware of claim 1, wherein the application mapped onto the IMC hardware includes a neural network (NN), and layers of the NN are mapped onto one or more CIMUs of the array of CIMUs.

22. The integrated IMC hardware of claim 1, wherein the multiple CIMUs and the on-chip network are integrated within a single semiconductor substrate.

23. Integrated in-memory computing (IMC) hardware, configurable to support scalable execution and data flow of neural networks (NNs) mapped onto the IMC hardware; a plurality of configurable Compute-In-Memory Units (CIMUs) forming an array of CIMUs logically configured as elements in a layer of the NN being mapped, each CIMU providing a computed data output representing a respective portion of a vector in a data flow associated with the mapped NN, the parallel output computed data of the CIMUs executing in a given layer forming feature map pixels; and a configurable on-chip network for communicating input data to the array of CIMUs, communicating calculated data between the CIMUs, and communicating output data from the array of CIMUs, the on-chip network including an on-chip operand loading network for communicating operands between the CIMUs via respective interfaces between the CIMUs, the on-chip network including at least one switch block providing selective routing of data between one or more CIMUs associated with one or more layers of the NN.

24. 24. The IMC hardware of claim 23, wherein the mapping of neural network computations to in-memory computation hardware operates to perform bitwise operations, and wherein multiple input vector bits are provided simultaneously and represented via selected voltage levels of analog signals.

25. 24. The IMC hardware of claim 23, wherein the multi-valued driver conveys an output signal from a selected one of a plurality of voltage sources, the voltage source being selected by decoding a plurality of bits of an input vector element.

26. The integrated IMC hardware of claim 23, wherein each input comprises a multi-bit input, the value of each multi-bit input being represented by a respective voltage level.

27. ​​The integrated IMC hardware described in claim 23, wherein the multiple CIMUs and the on-chip network are integrated within a single semiconductor substrate.

28. A computer-implemented method for mapping an application onto configurable in-memory computing (IMC) hardware, the IMC hardware comprising a plurality of configurable Compute-In-Memory Units (CIMUs) forming an array of CIMUs, and a configurable on-chip network for communicating input data to the array of CIMUs, communicating computed data between CIMUs, and communicating output data from the array of CIMUs, the method comprising: allocating the IMC hardware according to the application computation using parallelism and pipelining of the IMC hardware to generate an IMC hardware allocation configured to provide high throughput application computation; defining an arrangement of allocated IMC hardware to locations within the array of CIMUs in a manner that tends to minimize distance between IMC hardware that generates output data and IMC hardware that processes the generated output data; and configuring the on-chip network to route the data between IMC hardware.

29. 29. The computer-implemented method of claim 28, wherein the application mapped to the IMC hardware includes a neural network (NN) mapped to the IMC hardware such that parallel output computed data of a configured CIMU executing at a given layer is provided to a configured CIMU executing at a next layer, the parallel output computed data forming respective NN feature map pixels.

30. 29. The computer-implemented method of claim 28, wherein pipelining of computations is supported by allocating more configured CIMUs to execute at the given layer than configured CIMUs to execute at the next layer to compensate for greater computation time at a given layer than at a next layer.

Images