2D Meshes for Compute-in-Memory Accelerator Architectures

Abstract

The embodiments disclosed herein include a compute-in-memory (CIM) accelerator architecture for deep neural networks (DNNs). The CIM accelerator architecture may include a first analog fabric engine having a plurality of compute-in-memory (CIM) analog tiles. Each CIM analog tile may be configured to store a matrix of weight operands that create a vector of outputs from a vector of inputs and perform in-memory computations. The first analog fabric may also include a plurality of compute cores. Each CIM analog tile and each compute core may include a microcontroller configured to execute a sequence of instructions. The first analog fabric may also include an on-chip interconnect communicatively connecting all the CIM analog tiles in the plurality of CIM analog tiles to the compute cores.

Description

[Technical field]

[0001] The present invention relates generally to the fabrication of deep neural networks (DNNs), and more particularly to a compute-in-memory accelerator architecture for DNNs. [Background technology]

[0002] Rapid improvements in AI hardware accelerators have been an unacknowledged but crucial driver of progress in deep neural networks (DNNs). Better hardware has enabled the training of very large networks with enormous datasets, and the rapid inference of the resulting large, highly functional DNN models. Current DNN hardware ranges from state-of-the-art GPUs (graphics processing units) with numerous features specifically designed for DNN training and inference, to specialized digital CMOS accelerators incorporating precision reduction, sparsity, dataflow architectures, hardware and software optimizations, and very large area accelerator chips. In general, such accelerators must carefully orchestrate the flow of vast amounts of data between on-chip or off-chip volatile memory (SRAM and DRAM) and highly specialized SIMD (single instruction multiple data) units. These units execute multiply-add instructions that account for the majority of DNN compute workloads. This dataflow encompasses not only the many neuron activations produced by each DNN layer, but also the weights and partial sums of the DNN models.

[0003] Recently, compute-in-memory (CIM) designs can improve energy efficiency (e.g., by performing multiply-and-accumulate operations with time complexity in on-chip memory) and do so by reducing the movement of weights and partial sums in DNN models. By exploiting such weight stationarity over short periods in volatile memories such as SRAM or DRAM, or over longer periods in slower, finite-endurance nonvolatile memories (NVMs) such as flash, resistive RAM (RRAM), magnetic random access memory (MRAM), or phase-change memory (PCM), CIM approaches can provide both high speed and high energy efficiency. These benefits are seen across all DNN workloads, but are particularly pronounced for workloads that exhibit large fully-connected layers with low weight reuse. However, most of these memories only offer binary or a few bits of storage, and therefore implementing multi-bit weights required for state-of-the-art DNN performance requires invoking spatial multiplexing across multiple word lines or bit lines. This spatial multiplexing trades off area and energy to achieve the required product-sum accuracy, and is usually paired with time multiplexing on the word lines or bit lines to support multi-bit activation.

[0004] Some emerging non-volatile memories, such as PCM and RRAM, exhibit a wide range and continuous analog conductance states, providing a path to dense weight storage. Such devices also raise additional considerations such as weight programming errors, read noise, and conductance drift. This analog AI paradigm, in which energy-efficient MAC operations are performed on area-efficient crossbar array tiles of analog non-volatile memory, represents a particularly attractive form of compute-in-memory for hardware acceleration of DNN workloads.

[0005] While it is relatively easy to implement a single or a few tile demonstrator at the macro level, there are many challenges to implementing a complete CIM accelerator system. First, while MAC operations tend to dominate the compute requirements, there are many non-MAC operations that need to be performed and are embedded throughout all DNN workloads. These tasks do not always map well to CIM tiles and therefore require traditional SIMD digital compute cores. For a system to be efficient, multiple types of such compute cores may be required. For example, a simple ReLU or scaling operation requires a much simpler compute core than the complex matrix-matrix multiplication of attention computes in a transformer network. Dedicated on-chip memory cores are frequently required to store intermediate signals in local scratchpad SRAM for use in later computations. This could be the previous token in a sentence, or neighboring pixels in the image being processed or an intermediate image. In addition to providing an adequate amount of on-chip scratchpads, and the right number and mix of compute cores, the spatial arrangement of tiles, compute cores, and memory cores must be carefully organized, and data transfers between cores and tiles must be fast and energy-efficient to minimize communication-related penalties. Despite the clear need for holistic system-level optimizations such as those just described, most previous work has focused on building block level designs involving novel materials, devices, and circuits, where novel crossbar tiles are simply bolted onto traditional digital accelerator architectures, or on algorithmic advances to bring the application-level accuracy of analog AI systems to functional parity with digital AI systems. While these efforts have addressed many architectural challenges, some important design considerations have been overlooked.As a result, the cap between the reported peak performance and energy efficiency of these architectures and their actual sustained performance and energy efficiency can be significantly larger due to insufficient system-level utilization. These system-level inefficiencies found in previous studies stem from an overemphasis on MAC operations, as well as insufficient attention to the implementation of auxiliary operations, on-chip interconnect and data transfer, and power management. Furthermore, previous studies have focused on convolutional neural networks (CNNs) with little effort to support a broader class of important DNN workloads such as language and speech networks, which have also been ignored in most previous digital accelerator studies. Summary of the Invention

[0006] The present invention provides a microelectronic structure. The microelectronic structure may include a first compute-in-memory (CIM) analog tile configured to store a matrix of weight operands that create a vector of outputs from a vector of inputs. The first CIM analog tile may include a first microcontroller configured to execute instructions to perform in-memory operations. The microelectronic structure may include a first compute core communicatively coupled to the first CIM analog tile. The first compute core may include a second microcontroller configured to execute instructions. The microcontroller provides the advantage of locally controlling the operation of each analog tile or each compute core such that it is efficiently executed and communication times between the analog tile and the compute core are minimized.

[0007] An embodiment of the present invention includes a compute-in-memory (CIM) accelerator architecture for deep neural networks (DNNs). The CIM accelerator architecture may include a first analog fabric engine. The first analog fabric engine may include a plurality of compute-in-memory (CIM) analog tiles. Each CIM analog tile may be configured to store a matrix of weight operands that create a vector of outputs from a vector of inputs and perform in-memory computations. The first analog fabric may also include a plurality of compute cores. Each CIM analog tile and each compute core comprises a microcontroller configured to execute a sequence of instructions. The first analog fabric may also include an on-chip interconnect communicatively connecting all the CIM analog tiles in the plurality of CIM analog tiles to the compute cores.

[0008] An embodiment of the present invention includes a heterogeneous and programmable compute-in-memory (CIM) accelerator architecture including a plurality of compute-in-memory (CIM) analog tiles configured to store matrices of weight operands. Each CIM analog tile may be configured to store a matrix of weight operands and perform in-memory computations that create a vector of outputs from a vector of inputs. The CIM accelerator architecture may include a first plurality of compute cores, each of which includes a scratchpad memory configured to perform a self-attention auxiliary operation. The CIM accelerator architecture may include a second plurality of compute cores configured to perform a complex LSTM-aux operation, as well as an operation selected from the group consisting of tanh, sigmoid, multiplication, and addition. The CIM accelerator architecture may also include a microcontroller embedded in at least one CIM analog tile in the first plurality of analog tiles and at least one compute core in the second plurality of compute cores, and an on-chip interconnect communicatively connecting all CIM analog tiles in the plurality of CIM analog tiles to the first plurality of compute cores and the second plurality of compute cores.

[0009] An embodiment of the present invention includes a heterogeneous and programmable compute-in-memory (CIM) accelerator architecture including a plurality of compute-in-memory (CIM) analog tiles configured to store a matrix of weight operands. Each CIM analog tile may be configured to store a matrix of weight operands and perform in-memory computations that create a vector of outputs from a vector of inputs. The CIM accelerator architecture may include a first plurality of compute cores, each compute core including a scratchpad memory configured to perform a self-attention auxiliary operation. The CIM accelerator architecture may include a second plurality of compute cores configured to perform a complex LSTM-aux operation, as well as an operation selected from the group consisting of tanh, sigmoid, multiplication, and addition. The CIM accelerator architecture may also include an on-chip interconnect communicatively connecting all the CIM analog tiles in the plurality of CIM analog tiles to the first plurality of compute cores and the second plurality of compute cores.

[0010] A more complete understanding of the present invention, as well as further features and advantages of the present invention, will be obtained by reference to the following detailed description and drawings.

[0011] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which: [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a functional block diagram illustrating a deep neural network (DNN) inference environment 100, according to one embodiment of the present invention. [Diagram 2] 2 is a schematic diagram illustrating an analog fabric 200 for use in a CIM accelerator architecture, according to one embodiment of the present invention. [Diagram 3]FIG. 3 illustrates a two-dimensional (2D) mesh incorporated into the analog fabric of FIG. 2 in accordance with one embodiment of the present invention. [Figure 4] FIG. 2 is a schematic diagram of a 2D mesh, according to one embodiment of the present invention. [Diagram 5] FIG. 2 is a schematic diagram of a CIM architecture having one analog fabric engine, according to one embodiment of the present invention. [Figure 6] FIG. 1 is a schematic diagram of a CIM architecture having an Analog Fabric (AF) chip with four Analog Fabric engines, according to one embodiment of the present invention. [Figure 7] FIG. 1 is a schematic diagram of a CIM architecture having four AF chips, each with four analog fabric engines, according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Provided herein is a heterogeneous and programmable compute-in-memory (CIM) accelerator architecture and microelectronic structure for deep neural network (DNN) inference. The architecture combines spatially distributed CIM memory array "tiles" for weight-stationary, energy-efficient multiply-accumulate (MAC) operations with heterogeneous special-function "compute cores" for auxiliary digital computation. Massively parallel vectors of neuron activation data are exchanged over short distances using a two-dimensional (2D) mesh, providing full end-to-end support for a wide range of DNN workloads, including convolutional neural networks (CNNs), long short-term memories (LSTMs), and transformers. The embodiments disclosed herein include the design of an "analog fabric" (i.e., a 2D grid of tiles and compute cores interconnected by a 2D mesh) that addresses efficiency in both mapping DNNs to hardware and pipelining various DNN workloads over a range of mini-batch sizes. Embodiments may also include a system-level evaluation using projected component parameters of a realistic "analog AI" system based on a high-density crossbar array of low-power non-volatile analog memory elements and incorporating a single common analog fabric design that can be scaled to a large-scale network by introducing data transfers between multiple analog AI chips.

[0014]

[0023] Referring now to the drawings, Figure 1 is a functional block diagram illustrating a deep neural network (DNN) inference environment 100 in accordance with one embodiment of the present invention. Figure 1 is intended to be merely an illustration of one implementation and is not intended to suggest any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made by one of ordinary skill in the art without departing from the scope of the invention as set forth in the claims.

[0015] The DNN inference environment 100 includes an accelerator architecture 104 and a host device 106 connected via a network 102. The network 102 can be, for example, a telecommunications network, a local area network (LAN), a wide area network (WAN) such as the Internet, or a combination of the three, and can include wired, wireless, or fiber optic connections. The network 102 can include one or more wired and / or wireless networks capable of transmitting and receiving data signals, audio signals, or video signals, or combinations thereof, including multimedia signals including voice, data, and video information. In general, the network 102 can be any combination of connections and protocols that support communication between the accelerator architecture 104, the host device 106, and other computing devices (not shown) in the DNN inference environment 100. In various embodiments, network 102 operates locally over wired, wireless, or optical connections and can be any combination of connections and protocols (e.g., personal area network (PAN), near field communication (NFC), laser, infrared, ultrasonic, etc.).

[0016] The host device 106 may be a standalone computing device, a management server, a web server, a mobile computing device, or any other electronic device or computing system capable of receiving, transmitting, and processing data. In other embodiments, the host device 106 may represent a server computing system utilizing multiple computers as a server system, such as in a cloud computing environment. In another embodiment, the host device 106 may be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smart phone, or any programmable electronic device capable of communicating with other computing devices (not shown) in the DNN inference environment 100 via the network 102. In another embodiment, the host device 106 represents a computing system utilizing connected computers and components (e.g., database server computers, application server computers, etc.) that act as a single pool of seamless resources when accessed within the DNN inference environment 100. In the illustrated embodiment, the host device 106 includes a data set 112, which may include raw stock price or volume data for stocks collected from a data source 108. In other embodiments, the host device 106 may include other applications, databases, programs, etc. not shown in the DNN inference environment 100.

[0017] FIG. 2 is a schematic diagram illustrating an analog fabric 200 used in a CIM accelerator architecture. The analog fabric 200 includes a microelectronic structure having unit cells, such as CIM analog tiles 202 and compute cores 204, arranged in rows and columns. To improve communication speed and reduce energy usage in communication between the analog tiles 202 and the compute cores 204, the analog tiles 202 and compute cores 204 are distributed heterogeneously throughout the analog fabric 200. The heterogeneous distribution includes a combination of various types of analog tiles 202 and compute cores 204 that may be manufactured to have distinct properties for performing specific tasks that improve the overall efficiency and speed of the DNN inference environment 100. Each compute core 204 may be located between and communicatively coupled to the analog tiles 202. Each CIM analog tile and each compute core is communicatively connected to all immediately adjacent CIM analog tiles or compute cores. In particular embodiments, compute cores 204 may be located between and communicatively coupled to other compute cores 204. Similarly, each analog tile 202 may be located between and communicatively coupled to analog tiles 202 and / or compute cores 204. In other words, each analog tile 202 and each compute core 204 may be communicatively connected to all of its immediately neighboring CIM analog tiles 202 or compute cores 204.

[0018] Analog tiles 202 are the primary multiply-accumulate (MAC) compute and weight storage units configured to store matrices of weight operands for DNN inference workloads. Activations arrive as parallel bit streams and are converted to analog pulse durations, enabling parallel excitation of the analog memory arrays embodied within analog tiles 202. The multiply-accumulate is performed using a combination of Ohm's law and Kirchhoff's current law. The resulting raw sums are converted from analog currents and integrated charges to digital bit streams, which can be transmitted on a two-dimensional (2D) mesh, described in more detail below. The conversions are performed in parallel using analog-to-digital converters (ADCs). Because the DNN model computation weight operands stored in analog tiles 202 are preprogrammed into analog tiles 202 before inference begins, the input and output bit precision of analog tiles 202 must be carefully selected. Too high an input precision results in long read times and high read energy. Too high an output precision requires higher resolution in the ADC, resulting in additional area and energy consumption. The required precision depends on both the accuracy requirements of the DNN workload and the non-idealities of both the unit cell and the surrounding circuitry.

[0019] The analog tiles 202 can perform a variety of operations, with some analog tiles 202 being manufactured to perform certain operations more efficiently and other analog tiles 202 being manufactured to perform other operations more efficiently. For example, the operations may include analog MAC operations that can be easily performed on an analog crossbar.

[0020] The operations performed by analog fabric 200 may also include auxiliary operations (e.g., ReLU, Tanh, softmax) that cannot be easily performed in the analog crossbar. For these operations, analog fabric 200 includes compute core 204, which maps all auxiliary operations to digital compute cores that contain specialized units that support these calculations. These operations may include one or more of batch normalization, ReLU, GeLU, element-wise addition, element-wise multiplication, mean pooling, max pooling, layer normalization, activation functions with lookup tables, LSTM-aux, tanh, sigmoid, softmax, and self-attention.

[0021] For LSTM-aux, the following computation generates the next hidden (h) state and cell (c) state from the MAC output: h i,o,f (t) = σ[W i,o,f *x(t)+U i,o,f *h(t-1)+b i,o,f ] h c (t) = tanh[W c *x(t)+U c *h(t-1)+b c ] c(t) = h f (t).*c(t-1)+h i (t).*h c (t) h(t)=h0(t).*tanh[c(t)] where W, U, and b represent the DNN weight matrices that store the weight operands.

[0022] For self-attention, the embedding activations are typically split into multiple heads of vector size 64. For each set of attention heads, including the query (Q) matrix, key (K) matrix, and value (V) matrix, the following computations are required: Attention(Q,K,V) = softmax(QK T / sqrt(d k ))V where d k is the dimension of the key, typically 64. The illustrated embodiment of analog fabric 200 includes three types of specialized compute cores to handle these auxiliary operations: H 206 includes a dual-port SRAM for staging the activations of the self-attention computation (transformer) and for the aligner operation that rearranges the gradual activations of the weight stationary convolution (CNN). H The 206 uses 10-bit integer precision for self-attention calculations, enough to achieve near software-level accuracy for BERT. L The MC208 is designed to perform lightweight yet massively parallel computations such as activation functions, fused LSTM-aux Ops, and LN computations. The MC210 memory core is designed to perform CC without attention compute. H It is similar to the 206 and has a single-port SRAM that provides higher capacity in the same footprint, and supports activation staging for CNN and bidirectional LSTM networks.

[0023] These specialized compute cores 206, 208, 210 and their various compute operations allow auxiliary operations to be performed close to where data is generated and later consumed by MAC operations within the analog tiles 202. The compute cores 206, 208, 210 may vary based on specializations such as precision, data format, data flow, storage capacity, and programmability. The analog fabric 200 may also include input landing pads 212 and output landing pads 214 that communicate with a DRAM 216 and may be configured to perform off-chip input / output (I / O) data transfer operations for ultimate communication to a host device 218.

[0024] Analog Fabric 200 may also perform additional operations that are not part of the original DNN graph. These additional operations are added to the analog AI system to perform noise management and error compensation functions. Therefore, these operations do not count toward the performance metrics, and the energy and time spent performing the operations must be amortized by the higher energy efficiency and speed at which the “real” DNN operations are performed. Nonetheless, these additional operations help achieve functional equivalence in DNN accuracy and are therefore supported by Analog Fabric 200.

[0025] To support and orchestrate each operation performed by the analog fabric 200, each of the tiles 202 and cores 204 may include a microcontroller 220 that controls the operation of the particular component (i.e., the tile 202 or core 204). The analog fabric 200 of FIG. 2 includes tiles 202, CCs, and TCs. H 206, C.C. L Only one microcontroller 220 is shown for each of the tiles 202, CC 208, and MC 210. However, in certain embodiments, the tiles 202, CC 208, and MC 210 of the analog fabric 200 may each include a single microcontroller 220. H 206, C.C. L All of the CCs 208, 210, and MCs 210 may include a microcontroller 220. The microcontroller 220 controls the operation of a particular tile 202 or core 204. For example, a tile 202 or core 204 may require scratchpad memory when a DNN workload requires staging of activation data. In the analog fabric 200 of the illustrated embodiment, each CC HSRAM is provided within each microcontroller 220 of the MC 206 and MC 210. How many scratch pads are needed per core depends on the maximum sequence length, image resolution (input size), and the ratio of the number of layers (network depth) to the number of supporting cores 204. To cover the most common use cases, the microcontroller 220 provides a scratch pad for full local support of sequence lengths of increasing size and complexity within each CC 206. H 206 and MC 210 may include SRAM. Multiple CCs per DNN layer H By using MC206 and MC210, or by storing the data in off-chip main memory, longer sequences can be supported at the expense of some loss in throughput per area or energy efficiency.

[0026] The microcontroller 220 may also execute a set of instructions stored in a dedicated i-mem memory. Instructions and synchronization sequences are compiled during program preparation and preloaded into the appropriate i-mem along with the DNN weight programming for the new workload. During inference, each component (i.e., tile 202 or core 204) independently executes a specific set of instructions with periodic global synchronization to ensure successful execution of the entire DNN workload. Control and synchronization are managed in software to minimize additional hardware area and power budgets.

[0027] FIG. 3 illustrates a two-dimensional (2D) mesh 250 incorporated into the analog fabric 200 of FIG. 2. The 2D mesh 250 includes parallel metal row wires 252 and parallel metal column wires 254 formed within each CIM analog tile 202 and on a corresponding portion of each compute core 204. Although only a limited number of row wires 252 and column wires 254 are illustrated, each row and column of unit cells may be fabricated with its own two wires, allowing transfers in either a horizontal, vertical, or plus sign pattern (pairs of horizontal and vertical wires connected). The compute cores 204 may be configured to operate functions on adjacent analog tiles 202 simultaneously. The 2D mesh 250 may also enable multicasting, where a subset of receivers can get the same data at the same time. That is, the 2D mesh allows multiple data transfers involving multiple sender cores 204 or transmitting tiles 202 and multiple cores 204 or receiving tiles, all participating at the same time.

[0028] The spatial architecture has many components that are active for only a portion of the total pipeline latency to provide power saving functionality that performs at least one of the operations selected from the group consisting of coarse-grained clock gating, fine-grained clock gating, coarse-grained power gating, and fine-grained power gating. There may also be many unmapped components that are completely idle throughout a given DNN workload. In this situation, proper power gating is a key aspect to achieve high energy efficiency. Thus, the analog fabric 200 may include various power gating and clock gating options. The baseline option for comparison is no power gating, where all components consume energy continuously throughout the workload based on the idle power specification of that component. The second option is coarse-grained power gating, where if a component remains unmapped throughout the workload, the component is powered down and / or clock gated. Coarse-grained power gating also includes partial tile power and clock gating based on tile 202 segment utilization, where segments that are not fully mapped are clock gated and their associated peripherals are powered down, while partially used segments consume their full idle power. A third option is fine-grained power gating. Fine-grained power gating includes all of the capabilities of coarse-grained power gating to turn off unmapped cores 204, tiles 202, and tile segments, and in addition, each component is clock gated and power gated except during the first and last times the component is used throughout the workload execution time.

[0029] In a particular embodiment, an analog tile 202 may include unit cells arranged in 512 rows and 512 columns. Each 512×512 analog tile 202 is hierarchically designed to have 8 rows and 8 columns of segments, each containing 64×64 unit cells. Power and clock gating, defined herein as coarse-grained power gating, can be performed independently on the rows and columns of each segment, which helps mitigate efficiency losses due to reduced mapping efficiency.

[0030] FIG. 4 is a schematic diagram of a 2D mesh 400 according to one embodiment of the present invention. The 2D mesh 400 is formed over component blocks 403, which may be either tiles (e.g., tiles 202) or cores (e.g., cores 204) as described above. The 2D mesh 400 includes parallel metal row wires 452 and parallel metal column wires 454 in a manner similar to the 2D mesh 250 also described above. However, the 2D mesh 400 of FIG. 4 also illustrates boundaries 460 between component blocks 403. At all four boundaries 460 of each component block 403, the 2D mesh 400 includes border protection circuitry (BG) 462 that controls data transfer. The BG 462 includes signal buffers and pass gates that are configured by a vector of digital bits stored in each component block 403's local microcontroller (e.g., microcontroller 220 of FIG. 2) to establish a circuit-switched communication path before the component block 403 broadcasts data to one or more destination component blocks 403.

[0031] Each BG 462 can independently configure each segment of its mesh wiring (e.g., row wires 452 or column wires 454). It can therefore perform efficient manipulation operations on data vectors during transfer, including concatenation, splitting, gathering, and gathering all, which are useful within DNN workloads. For example, in an LSTM network with a hidden size of 512, a common use case is to concatenate four adjacent CCs. LThe concatenation of four vectors of length 128 across two segments produced by the cores. This concatenation can be efficiently performed in one parallel 2D mesh operation. To overlap data communication and computation, the transmit and receive circuits are each equipped with two sets of data registers. This allows the 2D mesh to transmit data and simultaneously consume data inside the component blocks 403 (e.g., individual cores 204 or tiles 202), enhancing scheduling without pipeline stalls.

[0032] FIG. 5 is a schematic diagram of a CIM architecture 500 having one analog fabric engine 570, according to an embodiment of the present invention. The analog fabric engine 570 may include multiple CIM analog tiles formed into a two-dimensional (2D) mesh and multiple compute cores connected to the 2D mesh. For ease of illustration, tiles and cores are not specifically illustrated in FIG. 5. However, each analog tile and each compute core may be communicatively connected to all immediately adjacent CIM analog tiles or compute cores. The analog fabric engine 570 may be surrounded by high bandwidth inter-chip inputs / outputs 572. Additionally, the high bandwidth inter-chip inputs / outputs 570 may be separated and individualized per boundary of the analog fabric engine 500.

[0033] FIG. 6 is a schematic diagram of a CIM architecture 600 having an analog fabric (AF) chip 674 with four analog fabric engines 670, according to one embodiment of the present invention. The four analog fabric engines 670 may include tiles and cores in a manner similar to the analog fabric engine 570 of FIG. 5. All of the analog fabric engines 670 of FIG. 6 may be connected with a single 2D mesh. Additionally, the analog fabric engines 670 of FIG. 6 may include a single high bandwidth inter-chip input / output 672 around all four analog fabric engines 670, rather than including a separate high bandwidth inter-chip input / output for each analog fabric engine 670.

[0034] FIG. 7 is a schematic diagram of a CIM architecture 700 having four AF chips 774, each with four analog fabric engines 770, according to an embodiment of the present invention. To scale to larger workloads, multiple analog fabric engines 770 can be tiled together to implement large-scale weight stationarity systems. Multiple AF chips 774 can be connected together by extending 2D mesh wires on the same chip or by introducing suitable inter-chip inputs / outputs that enable multi-chip module systems. As analog CIM and CMOS technologies continue to scale to higher areal densities, the footprint of each AF chip 774 is expected to shrink. This will result in more AF engines on a single chip, minimizing cross-chip communication while providing enough analog CIM tiles (and supporting cores) to store all DNN weight operands, and further allowing reuse of the modular design structure to facilitate scalability.

[0035] The embodiments disclosed above may be used to complete matrix vector multiplication (MVM) operations. For example, Analog Fabric may be used to complete long short-term memory (LSTM) workloads. An LSTM network consists of recursively connected memory blocks, each containing self-connected memory cells and three multiplication gates (input, output, and forget) that control cell activation. The Analog Fabric microelectronic structure disclosed above is fed with an input vector x(t) at time t, and the cell state c(t) and hidden state h(t) are updated from the previous time step (t-1) through four sets of MAC operations involving weight operands followed by auxiliary operations. In a simple LSTM network, the dimensions of the input vector (x), cell state (c), and hidden state (h) are all equal and typically range from 50 to 2000. Thus, a one-layer LSTM network with input and hidden vectors of size 512 requires 8 tiles of unit cells of 512 x 512 each. At each time step, the input vector x(t) is first concatenated with the hidden vector h(t-1) from the previous time step before being sent to the tile. Weight operands with the same input dimensions are mapped to horizontally adjacent tiles to take advantage of multicasting on the 2D mesh. By mapping weight operands with the same output dimensions to vertically adjacent tiles, the output vectors from each column of the tile are mapped one by one to the CCs in the column for element-wise addition and auxiliary operations. L The analog fabric also provides a single tile for each CC. L Each column is h f (t), h i (t), h o (t), and h c The weights of the MAC operations of the cells and gates may be interleaved to create 128 elements of CC(t), L The core will be able to construct 128 elements of c(t) and h(t) using fused LSTM-aux Ops. Four separate CCs LThe four sets of h(t) elements from the core are concatenated on the mesh to create the complete h(t) vector, and the c(t) elements are then reconstructed into individual CCs for the next time step. L The digital compute is stored locally in the core. L Distributing across cores reduces core latency, balances cores, tile workload, and avoids unnecessary mesh contention. These throughput gains become even more pronounced as networks get larger and mini-batch sizes get larger.

[0036] The embodiments disclosed above may also be used to complete matrix vector multiplication (MVM) operations such as the Transformer workload. The Transformer workload layer consists of a self-attention block followed by two large fully connected layers. The self-attention block includes an in-projection MAC operation to generate a query (Q), key (K), and value (V) vector of the same length for each token in the sequence, and an out-of-projection MAC operation to prepare the attention data for input. When the embedding size of the vector is between 512 and 2048, the analog fabric in the disclosed embodiments maps the in-projection tiles horizontally to leverage multicasting on the 2D mesh. The multi-head self-attention computation is performed by selecting the closest available CC. H , each processing several heads in parallel.

[0037] After this self-attention block, we then focus on nearby CCs before broadcasting the input activation to many other tiles. L The remaining element-wise addition and LayerNorm (LN) operations are performed in the core. The partial outputs from all tiles are added together to form the final CC. L The vertices are collected into the core, where the final remaining summation and LN operations are performed. For larger transformer models, the tiles, CC H , and C.C. LThe required number of cores scales directly based on the embedding size of the network.

[0038] Convolutional Neural Networks (CNNs) are activation-heavy networks that contain many banks of small weight kernel filters. Weight and activation reuse is important, but the importance of reuse varies with the depth of the network, as a few filters convolved over large images in early layers are replaced by many filters and small intermediate images deep in the network. In an environment of weight stationarity, accessing the same weight kernels in different patterns requires reorganizing and sending out stored activations. This results in a large number of SRAM accesses, requiring careful memory management for good energy efficiency. Because the latency of MAC operations is determined by the number of analog CIM tile reads required rather than the total number of Ops, weight reuse reduces the total number of tiles required but increases the total latency. The embodiments disclosed herein focus on precisely considering the energy efficiency of CNNs to further optimize throughput, and do not implement deliberate weight duplication.

[0039] Even within the same part of a CNN layer (e.g., a residual bottleneck block), the size and shape of the weight layers can vary widely, resulting in variance in the number of tiles required. Furthermore, the fine-grained pipelining of CNN activations allows each CNN layer to start processing as soon as it has generated enough output from the previous layer. Here, we consider the corresponding core (CC) that performs activation alignment. H To ensure a dedicated 2D mesh path to the MC (or MC), each weight layer is organized horizontally across tiles as much as possible.

[0040] A computer readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. A non-exhaustive list of more specific examples of computer readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or raised structures in grooves with instructions recorded thereon, and any suitable combination of the above. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses through a fiber optic cable), or electrical signals transmitted over electrical wires.

[0041] The computer readable program instructions described herein may be downloaded from a computer readable storage medium to each computing / processing device or to an external computer or storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical transmission fiber, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0042] The computer readable program instructions for carrying out the operations of the present invention may be either source or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or conventional procedural programming languages, such as object oriented programming languages, such as Smalltalk®, C++, and the like, as well as the "C" programming language or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer as a stand-alone software package, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through an unspecified type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer readable program instructions to perform aspects of the invention by utilizing state information of the computer readable program instructions to personalize the electronic circuitry.

[0043] The description of various embodiments of the present invention is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used in this specification have been selected to best explain the principles of the embodiments, practical applications, or technical improvements over the technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A microelectronic structure comprising: a first compute-in-memory (CIM) analog tile configured to store a matrix of weight operands that create a vector of outputs from a vector of inputs, the first CIM analog tile comprising a first microcontroller configured to execute instructions to perform in-memory operations; a first compute core communicatively coupled to the first CIM analog tile, the first compute core comprising a second microcontroller configured to execute instructions; A microelectronic structure comprising:

2. 10. The microelectronic structure of claim 1, wherein the first compute core comprises a scratchpad memory.

3. 10. The microelectronic structure of claim 1, wherein the first compute core is configured to perform operations selected from the group consisting of batch normalization, rectified linear activation function, Gaussian error linear unit, element-wise addition, element-wise multiplication, mean pooling, max pooling, layer normalization, look up table activation function, long short term memory -aux, tanh, sigmoid, softmax, and self-attention.

4. 10. The microelectronic structure of claim 1, wherein the first compute core includes power saving functionality that performs at least one of the operations selected from the group consisting of coarse grain clock gating, fine grain clock gating, coarse grain power gating, and fine grain power gating.

5. 10. The microelectronic structure of claim 1, wherein the first compute core is configured to perform off-chip input / output (I / O) data transfer operations.

6. The microelectronic structure of claim 1 , wherein the first CIM analog tile is configured to perform vector matrix multiplication (VMM) operations.

7. 10. The microelectronic structure of claim 1, further comprising a second compute core communicatively coupled to the first CIM analog tile and the first compute core, the second compute core comprising a third microcontroller configured to execute a sequence of instructions.

8. 8. The microelectronic structure of claim 7, wherein the first compute core and the second compute core differ based on a specialization selected from the group consisting of precision, data format, data flow, storage capacity, and programmability.

9. A compute-in-memory (CIM) accelerator architecture for deep neural networks (DNNs), comprising: a first analog fabric engine, the first analog fabric engine comprising: a plurality of compute-in-memory (CIM) analog tiles, each CIM analog tile configured to store a matrix of weight operands that create a vector of outputs from a vector of inputs and to perform in-memory computations; a plurality of compute cores, each CIM analog tile and each compute core comprising a microcontroller configured to execute a sequence of instructions; an on-chip interconnect communicatively connecting all of the CIM analog tiles in the plurality of CIM analog tiles to the compute core; A CIM accelerator architecture comprising:

10. the plurality of compute cores, a first subset of compute cores comprising a scratchpad memory of a first capacity; a second subset of the compute cores comprising a scratchpad memory of a second capacity; and 10. The CIM accelerator architecture of claim 9, comprising:

11. 10. The CIM accelerator architecture of claim 9, wherein the plurality of compute cores includes a subset of compute cores configured to perform off-chip input / output (I / O) data transfer operations.

12. 10. The CIM accelerator architecture of claim 9, wherein each compute core in the plurality of compute cores is configured to perform an operation selected from the group consisting of batch normalization, ReLU, GeLU, element-wise addition, element-wise multiplication, mean and max pooling, layer normalization, activation functions with look-up tables, LSTM-aux, tanh, sigmoid, softmax, and self-attention.

13. 10. The CIM accelerator architecture of claim 9, wherein each CIM analog tile and each compute core includes power saving functionality that performs operations selected from the group consisting of coarse grain clock gating, fine grain clock gating, coarse grain power gating, and fine grain power gating.

14. 10. The CIM accelerator architecture of claim 9, wherein a first compute core from the plurality of compute cores and a second compute core from the plurality of compute cores differ based on a specialization selected from the group consisting of precision, data format, data flow, storage capacity, and programmability.

15. a first four engine system, the first four engine system comprising: a second analog fabric engine; and a third analog fabric engine; and A fourth analog fabric engine 10. The CIM accelerator architecture of claim 9, wherein the second analog fabric engine, the third analog fabric engine, and the fourth analog fabric engine are communicatively connected to the first analog fabric engine by a 2D mesh, the 2D mesh being surrounded by high bandwidth chip-to-chip inputs / outputs.

16. a second four-engine system; and a third four-engine system; a fourth four-engine system communicatively connected as a multi-chip system; and 20. The CIM accelerator architecture of claim 15 further comprising:

17. 10. The CIM accelerator architecture of claim 9, wherein a plurality of CIM analog tiles and compute cores are connected using a two-dimensional (2D) mesh with parallel row and parallel wires that enables multicast between the plurality of CIM analog tiles and the plurality of compute cores.

18. 10. The CIM accelerator architecture of claim 9, further comprising a boundary guard circuit (BG) on each parallel row wire and each parallel row wire on each boundary of said plurality of CIM analog tiles and said plurality of compute cores.

19. 20. The CIM accelerator architecture of claim 18, wherein the BG comprises signal buffers and pass gates controlled by a microcontroller on each compute core and CIM analog tile.

20. A heterogeneous and programmable compute-in-memory (CIM) accelerator architecture, comprising: a plurality of compute-in-memory (CIM) analog tiles configured to store matrices of weight operands, each CIM analog tile configured to store a matrix of weight operands that create a vector of outputs from a vector of inputs and to perform in-memory computations; a first plurality of compute cores, each of the compute cores comprising a scratchpad memory configured to perform self-attention auxiliary operations; a second plurality of compute cores configured to perform complex LSTM-aux operations; a microcontroller embedded in at least one CIM analog tile in the first plurality of analog tiles and at least one compute core in the second plurality of compute cores; an on-chip interconnect communicatively connecting all CIM analog tiles within the plurality of CIM analog tiles to the first plurality of compute cores and to the second plurality of compute cores; A CIM accelerator architecture comprising:

21. 21. The CIM accelerator architecture of claim 20, further comprising a third plurality of compute cores that can be configured to perform off-chip input / output (I / O) data transfer operations.

22. 21. The CIM accelerator architecture of claim 20, further comprising a third plurality of compute cores, each compute core of the third plurality of compute cores including a power saving function that performs at least one of the operations selected from the group consisting of coarse grain clock gating, fine grain clock gating, coarse grain power gating, and fine grain power gating.

23. 21. The CIM accelerator architecture of claim 20, wherein the first plurality of compute cores and the second plurality of compute cores are configured to perform operations selected from the group consisting of batch normalization, ReLU, GeLU, element-wise addition, element-wise multiplication, mean and max pooling, layer normalization, activation functions with look-up tables, tanh, sigmoid, and LSTM-aux.

24. 21. The CIM accelerator architecture of claim 20, further comprising border protection circuitry (BG) on parallel row and column wires of the on-chip interconnect between the first plurality of compute cores, the second plurality of compute cores, and the plurality of CIM analog tiles.

25. A heterogeneous and programmable compute-in-memory (CIM) accelerator architecture, comprising: a plurality of compute-in-memory (CIM) analog tiles configured to store matrices of weight operands, each CIM analog tile configured to store a matrix of weight operands that create a vector of outputs from a vector of inputs and to perform in-memory computations; a first plurality of compute cores, each of the compute cores comprising a scratchpad memory configured to perform self-attention auxiliary operations; a second plurality of compute cores configured to perform complex LSTM-aux operations, as well as operations selected from the group consisting of tanh, sigmoid, multiplication, and addition; an on-chip interconnect communicatively connecting all CIM analog tiles within the plurality of CIM analog tiles to the first plurality of compute cores and to the second plurality of compute cores; A CIM accelerator architecture comprising:

Images