Two-dimensional mesh for in-memory computing accelerator architecture

Through a heterogeneous and programmable in-memory computing accelerator architecture, combined with the 2D grid interconnection of simulated tile and computing core, the problems of low system-level utilization and low auxiliary operation efficiency in the prior art are solved. Efficient support and system-level performance improvements for a variety of DNN workloads are achieved.

CN118974745APending Publication Date: 2024-11-15INTERNATIONAL BUSINESS MACHINE CORPORATION

Patent Information

Application Number
CN202380030169.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-03-31
Filing Date
2023-03-13
Publication Date
2024-11-15

AI Technical Summary

Technical Problem

When existing deep neural network (DNN) hardware accelerators realize high-speed and high-energy-efficient in memory computing accelerator systems, they face the problems of low system-level utilization, low auxiliary operation and low data transmission efficiency, especially for non-convolutional nerves. Inadequate support for network (CNN) and other important DNN workloads.

Method used

A heterogeneous and programmable in-memory computing accelerator architecture is designed, combining spatially distributed analog tile and heterogeneous computing cores to achieve efficient data transmission and operational execution through 2D grid interconnection, supporting a wide range of DNN workloads , such as CNN, LSTM and Transformer networks. The architecture includes multiple analog tile and computing cores, each equipped with a microcontroller for optimizing communications and local operations, providing comprehensive end-to-end support.

Benefits of technology

It improves the overall efficiency and speed of the DNN inference environment, reduces communication energy consumption, supports multiple DNN workloads, achieves higher system-level performance and energy efficiency, and fills the gap in support for non-convolutional neural networks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118974745A_ABST
    Figure CN118974745A_ABST
Patent Text Reader

Abstract

Embodiments disclosed herein include an in-memory computing (CIM) accelerator architecture for a deep neural network (DNN). The CIM accelerator architecture may include a first simulation fabric engine having a plurality of in-memory computing (CIM) simulation tiles. Each CIM simulation tile may be configured to store a matrix of weight operands that produce an output vector from an input vector, and perform in-memory calculations. The first simulation structure may also include a plurality of computing cores. Each CIM simulation tile and each computing core may include a microcontroller configured to execute an instruction set. The first simulation fabric may also include an on-chip interconnect that communicatively connects all of the plurality of CIM simulation tiles to the computing core.
Need to check novelty before this filing date? Find Prior Art

Description

Background Art

[0001] The present invention relates generally to fabricating deep neural networks (DNNs), and more particularly, to in-memory compute accelerator architectures for DNNs.

[0002] Rapid improvements in AI hardware accelerators have been a hidden but critical driver of progress in deep neural networks (DNNs). Better hardware enables the training of very large networks with huge datasets, as well as the rapid inference of the resulting large and therefore highly capable DNN models. Current DNN hardware ranges from modern GPUs (graphics processing units) with a large number of features designed specifically for DNN training and inference to specialized digital CMOS accelerators that incorporate reduced precision, sparsity, dataflow architectures, hardware-software optimizations, and ultra-large area accelerator chips. Typically, such accelerators must carefully orchestrate the flow of large amounts of data between on-chip or off-chip volatile memories (SRAM and DRAM) and highly specialized SIMD (single instruction multiple data) units. These units perform the multiply-accumulate instructions that dominate most DNN computational workloads. This data flow includes not only the many neuron activations produced by each DNN layer, but also the DNN model weights and partial sums.

[0003] Recently, computation-in-memory (CIM) designs that can improve energy efficiency (e.g., by performing multiply-accumulate operations with time complexity within on-chip memory) have been implemented by reducing the actions of DNN model weights and partial sums. CIM approaches can provide both high speed and high energy efficiency by exploiting this weight stationarity over short time spans with volatile memories (such as SRAM or DRAM), or over longer time spans with slower and limited endurance non-volatile memories (NVMs) (such as flash, resistive RAM (RRAM), magnetic random access memory (MRAM), or phase change memory (PCM)). These benefits can be seen in all DNN workloads, but are particularly evident for workloads that exhibit large fully connected layers with low weight reuse. However, since most of these memories only provide binary or a few bits of storage, spatial multiplexing across multiple wordlines or bitlines must be invoked to achieve the multi-bit weights required for state-of-the-art DNN performance. This trades off area and energy to achieve the necessary multiply-accumulate accuracy, which is often paired with temporal multiplexing on wordlines or bitlines to support multi-bit activations.

[0004] Some emerging non-volatile memories, such as PCM and RRAM, exhibit a wide and continuous range of analog conductance states, providing a path to high-density weight storage. Such devices also introduce additional considerations such as weight programming errors, readout noise, and conductance drift. This analog AI paradigm, in which energy-efficient MAC operations are performed on area-efficient crossbar array tiles of analog non-volatile memory, represents a particularly attractive form of in-memory computing for hardware acceleration of DNN workloads.

[0005] While it is relatively trivial to implement a single-tile or few-tile demonstrator at the macro level, there are many challenges to implementing a full CIM accelerator system. First, while MAC operations easily dominate the computational requirements, many non-MAC operations must be performed, which are embedded in each DNN workload. These tasks do not always map well to CIM tiles, and therefore require conventional SIMD numeric computation cores. In order for the system to be efficient, multiple types of such computation cores may be required. For example, a simple ReLU or scaling operation requires a much simpler computation core than the complex matrix-matrix product of the attention calculation in a Transformer network. Dedicated on-chip memory cores are often required to store intermediate signals in local scratchpad SRAM for later computation. This can be previous tokens in a sentence, or neighboring pixels in an image being processed or an intermediate image. In addition to providing the right amount of on-chip scratchpad and the right number and mix of computation cores, the spatial arrangement of tiles, computation cores, and memory cores must be carefully organized, and inter-core and inter-tile data transfers must be fast and energy-efficient in order to minimize the penalties associated with communication. Despite the clear need for holistic system-level optimizations such as the one we have just described, most prior work has either focused on component-level designs that include novel materials, devices, and circuits, in which case novel crossbar switch tiles are simply bolted into traditional digital accelerator architectures, or on algorithmic improvements used to make application-level accuracy in analog AI systems functionally equivalent to their digital counterparts. Although many architectural challenges have been addressed through these efforts, several important design considerations have been overlooked. As a result, the gap between the reported peak performance and energy efficiency of these architectures and the actual sustained performance and energy efficiency can be large due to poor system-level utilization. These system-level inefficiencies found in prior work stem from an overemphasis on MAC operations and insufficient attention to the implementation of auxiliary operations, on-chip interconnect and data transfer, and power management. In addition, early work has focused primarily on convolutional neural networks (CNNs), and little effort has been made to support a broader class of important DNN workloads, such as language and speech networks—workloads that have also been ignored by most traditional digital accelerator work. Summary of the invention

[0006] The present invention provides a microelectronic structure. The microelectronic structure may include a first computation-in-memory (CIM) simulation tile, the first CIM simulation tile being configured to store a weight operand matrix that generates an output vector based on an input vector. The first CIM simulation tile may include a first microcontroller configured to execute instructions and perform in-memory operations. The microelectronic structure may include a first computing core communicatively coupled to the first CIM simulation tile. The first computing core may include a second microcontroller configured to execute instructions. The microcontroller provides the benefit of local control operations, so that each simulation tile or each computing core can run efficiently and have the shortest communication time between the simulation tile and the computing core.

[0007] Embodiments of the present invention include a compute-in-memory (CIM) accelerator architecture for deep neural networks (DNNs). The CIM accelerator architecture may include a first analog structure engine. The first analog structure engine may include a plurality of compute-in-memory (CIM) simulation tiles. Each CIM simulation tile may be configured to store a weight operand matrix that generates an output vector based on an input vector and perform in-memory calculations. The first simulation structure may also include a plurality of computing cores. Each CIM simulation tile and each computing core include a microcontroller configured to execute an instruction set. The first simulation structure may also include an on-chip interconnect that communicatively connects all CIM simulation tiles in the plurality of CIM simulation tiles to the computing core.

[0008] Embodiments of the present invention include a heterogeneous and programmable in-memory computing (CIM) accelerator architecture, which includes a plurality of in-memory computing (CIM) simulation tiles configured to store weight operand matrices. Each CIM simulation tile can be configured to store a weight operand matrix that generates an output vector according to an input vector, and perform in-memory calculations. The CIM accelerator architecture may include a first plurality of computing cores, wherein each computing core includes a temporary storage memory configured to perform self-attention auxiliary operations. The CIM accelerator architecture may include a second plurality of computing cores, the second plurality of computing cores being configured to perform complex LSTM auxiliary (LSTM-aux) operations, and operations selected from the group consisting of the following items: hyperbolic tangent, sigmoid, multiplication, and addition. The CIM accelerator architecture may also include a microcontroller embedded in at least one CIM simulation tile in the first plurality of simulation tiles and at least one computing core in the second plurality of computing cores, and an on-chip interconnect that communicatively connects all CIM simulation tiles in the plurality of CIM simulation tiles to the first plurality of computing cores and the second plurality of computing cores.

[0009] Embodiments of the present invention include a heterogeneous and programmable in-memory computing (CIM) accelerator architecture, which includes a plurality of in-memory computing (CIM) simulation tiles configured to store weight operand matrices. Each CIM simulation tile can be configured to store a weight operand matrix that generates an output vector based on an input vector, and perform in-memory calculations. The CIM accelerator architecture may include a first plurality of computing cores, wherein each computing core includes a temporary storage memory configured to perform self-attention auxiliary operations. The CIM accelerator architecture may include a second plurality of computing cores, the second plurality of computing cores being configured to perform complex LSTM auxiliary (LSTM-aux) operations, and operations selected from the group consisting of the following items: hyperbolic tangent, sigmoid, multiplication, and addition. The CIM accelerator architecture may also include an on-chip interconnect that communicatively connects all CIM simulation tiles in the plurality of CIM simulation tiles to the first plurality of computing cores and the second plurality of computing cores.

[0010] A more complete understanding of the present invention, as well as other features and advantages of the present invention, will be obtained by referring to the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Embodiments of the present invention will now be described, by way of example only, with reference to the accompanying drawings, in which:

[0012] FIG1 depicts a functional block diagram illustrating a deep neural network (DNN) inference environment 100 according to one embodiment of the present invention;

[0013] FIG. 2 depicts a schematic diagram showing a simulation structure 200 for a CIM accelerator architecture according to one embodiment of the present invention;

[0014] FIG. 3 depicts a two-dimensional (2D) grid incorporated into the simulation structure of FIG. 2 according to one embodiment of the present invention;

[0015] FIG4 depicts a schematic diagram of a 2D grid according to one embodiment of the present invention;

[0016] FIG5 depicts a schematic diagram of a CIM architecture having a simulation structure engine according to an embodiment of the present invention;

[0017] FIG. 6 depicts a schematic diagram of a CIM architecture having four analog fabric engines of an analog fabric (AF) chip according to one embodiment of the present invention; and

[0018] FIG. 7 depicts a schematic diagram of a CIM architecture with four AF chips, each with four analog fabric engines, according to one embodiment of the present invention. DETAILED DESCRIPTION

[0019] A heterogeneous and programmable compute-in-memory (CIM) accelerator architecture and microelectronic structure for deep neural network (DNN) inference is provided herein. The architecture combines spatially distributed CIM memory array "tiles" for weight-fixed, energy-efficient multiply-accumulate (MAC) operations with heterogeneous special function "compute cores" for auxiliary digital computations. A two-dimensional (2D) grid is used to exchange massively parallel vectors of neuron activation data over short distances, providing comprehensive end-to-end support for a wide range of DNN workloads, including convolutional neural networks (CNNs), long short-term memory (LSTM), and Transformers. The embodiments disclosed herein include a design for an "analog structure" (i.e., a 2D mesh of tiles and compute cores interconnected by a 2D grid) that addresses the efficiency of mapping DNNs to hardware and pipelining a variety of DNN workloads across a range of mini-batch sizes. Embodiments may also include system-level evaluation of projected component parameters of actual "analog AI" systems using dense crossbar arrays based on low-power non-volatile analog memory elements, combined with a single common analog fabric design that can scale to large networks by introducing data transfers between multiple analog AI chips.

[0020] Turning now to the drawings, FIG1 depicts a functional block diagram showing a deep neural network (DNN) reasoning environment 100 according to one embodiment of the present invention. FIG1 merely provides an illustration of one embodiment and does not imply any limitations on the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made by those skilled in the art without departing from the scope of the present invention as described in the claims.

[0021] The DNN inference environment 100 includes an accelerator architecture 104 and a host device 106 connected via a network 102. The network 102 may be, for example, a telecommunications network, a local area network (LAN), a wide area network (WAN) (such as the Internet), or a combination of the three, and may include wired, wireless, or fiber optic connections. The network 102 may include one or more wired and / or wireless networks capable of receiving and sending data, voice, and / or video signals (including multimedia signals containing voice, data, and video information). In general, the network 102 may be any combination of connections and protocols that will support communications between the accelerator architecture 104, the host device 106, and other computing devices (not shown) within the DNN inference environment 100. In various embodiments, the network 102 operates locally via a wired, wireless, or optical connection, and may be any combination of connections and protocols (e.g., personal area network (PAN), near field communication (NFC), laser, infrared, ultrasonic, etc.).

[0022] The host device 106 may be a stand-alone computing device, a management server, a web server, a mobile computing device, or any other electronic device or computing system capable of receiving, sending, and processing data. In other embodiments, the host device 106 may represent a server computing system such as a server system utilizing multiple computers in a cloud computing environment. In another embodiment, the host device 106 may be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a personal digital assistant (PDA), a smart phone, or any programmable electronic device capable of communicating with other computing devices (not shown) within the DNN reasoning environment 100 via the network 102. In another embodiment, the host device 106 represents a computing system utilizing connected computers and components (e.g., database server computers, application server computers, etc.) that act as a single seamless resource pool when accessed within the DNN reasoning environment 100. In the depicted embodiment, the host device 106 includes a data set 112, which may include raw stock price data or quantity data for stocks collected from the data source 108. In other embodiments, the host device 106 may contain other applications, databases, programs, etc. that are not depicted in the DNN reasoning environment 100.

[0023] FIG. 2 depicts a schematic diagram showing an analog structure 200 for a CIM accelerator architecture. The analog structure 200 includes a microelectronic structure having cells arranged in rows and columns, such as CIM analog tiles 202 and computing cores 204. The analog tiles 202 and computing cores 204 are heterogeneously distributed throughout the analog structure 200 to increase communication speed and reduce energy usage in communications between the analog tiles 202 and the computing cores 204. The heterogeneous distribution includes combinations of various types of analog tiles 202 and computing cores 204, which can be manufactured to have different properties for performing specific tasks that improve the overall efficiency and speed of the DNN reasoning environment 100. Each computing core 204 can be located between the analog tiles 202 and communicatively coupled to the analog tiles 202. Each CIM analog tile and each computing core is communicatively connected to all adjacent CIM analog tiles or computing cores. In some embodiments, the computing core 204 can be located between other computing cores 204 and communicatively coupled to other computing cores 204. Similarly, each simulation tile 202 may be located between and communicatively coupled to simulation tiles 202 and / or computing cores 204. In other words, each simulation tile 202 and each computing core 204 may be communicatively connected to all immediately adjacent CIM simulation tiles 202 or computing cores 204.

[0024] The analog tile 202 is the primary multiply-accumulate (MAC) computation and weight storage unit, which is configured to store weight operand matrices for DNN inference workloads. Activations arrive as parallel bitstreams and are converted into analog pulse durations, enabling parallel excitation of the analog memory array embodied within the analog tile 202. The multiply-accumulate is performed by a combination of Ohm's law and Kirchhoff's current law. The resulting raw sum is converted from the analog current and integrated charge back into a digital bitstream, which can be emitted on a two-dimensional (2D) grid as described in detail below. The conversion is performed in parallel using an analog-to-digital converter (ADC). Since the DNN model operation weight operands stored on the analog tile 202 are pre-programmed into the analog tile 202 before inference begins, the input and output bit precision of the analog tile 202 must be carefully selected. Excessive input precision results in longer read times and higher read energy. Excessive output precision requires the ADC to have a higher resolution, which results in additional area and energy consumption. The required precision depends on the accuracy requirements of the DNN workload and the non-idealities of both the cell and the peripheral circuitry.

[0025] The analog tiles 202 may perform various operations, with some analog tiles 202 being fabricated to perform certain operations more efficiently and other analog tiles being fabricated to perform other operations more efficiently. For example, the operations may include analog MAC operations that may be readily performed on an analog crossbar.

[0026] The operations performed by the analog structure 200 may also include auxiliary operations that cannot be easily performed on the analog crossbar (e.g., ReLU, hyperbolic tangent, softmax). For these operations, the analog structure 200 includes a computing core 204, and all auxiliary operations are mapped to a digital computing core containing dedicated units to support these calculations. These operations may include one or more of the following: batch normalization, ReLU, GeLU, element-by-element addition, element-by-element multiplication, average pooling, max pooling, layer normalization, activation functions via lookup tables, LSTM-aux, hyperbolic tangent, sigmoid, softmax, and self-attention.

[0027] For LSTM-aux, the following calculation generates the next hidden (h) state and cell (c) state based on the MAC output:

[0028]

[0029]

[0030]

[0031]

[0032] Among them, W, U, and b represent the DNN weight matrix storing weight operands.

[0033] For self-attention, the embedding activations are usually split into multiple heads with a vector size of 64. For each set of attention heads containing a query (Q) matrix, a key (K) matrix, and a value (V) matrix, the following computations are required:

[0034] Attention(Q,K,V) = softmax(QK T / sqrt(d k ))V

[0035] where d k is the size of the key, which is typically 64. The illustrated embodiment of simulation architecture 200 includes three types of specialized compute cores to handle these auxiliary operations. H 206 includes a dual-port SRAM for hierarchical activations for self-attention computation (Transformer) and an aligner operation for reorganizing hierarchical activations for weight-fixed convolutions (CNN). In addition to softmax, CC H 206 uses 10-bit integer precision for self-attention calculations, which is sufficient to achieve approximate software equivalent accuracy on BERT. "Light" computing core CC L 208 is designed for light-weight but massively parallel computations such as activation functions, fused LSTM-aux Ops, and LN computations. Memory Core MC 210 is similar to CC without attention computations H 206, with higher capacity single-port SRAM in the same area footprint to support CNN and activation classification for bidirectional LSTM networks.

[0036] These specialized compute cores 206, 208, 210 and their different compute operations enable auxiliary operations to be performed close to where data is generated in the analog tile 202 and later consumed by the MAC operation. The compute cores 206, 208, 210 can differ based on specializations such as precision, data format, data flow, storage capacity, and programmability. The analog structure 200 can also include an input buffer (landing pad) 212 and an output buffer 214 that communicate with a DRAM 216, and can be configured to perform off-chip input-output (I / O) data transfer operations to ultimately communicate with a host device 218.

[0037] The simulation structure 200 can also perform additional operations that are not part of the original DNN graph. These additional operations are added to the simulated AI system to perform noise management and error compensation functions. Therefore, these operations cannot be counted in the performance metric, and the energy and time spent executing them must be amortized by the high energy efficiency and speed of executing "real" DNN operations. However, these additional operations help provide functional equivalence in terms of DNN accuracy and are therefore supported by the simulation structure 200.

[0038] To support and organize each of the operations performed by the simulation structure 200, each of the tiles 202 and the core 204 may include a microcontroller 220 that controls the operation of a particular component (ie, tile 202 or core 204). H 206.CC L 208 and MC 210. However, in some embodiments, all tiles 202, CC 208 and MC 210 of the simulation structure 200 may be configured to be connected to the same microcontroller 220. H 206.CC L 208 and MC 210 may include a microcontroller 220. The microcontroller 220 controls the operation of a particular tile 202 or core 204. For example, when a DNN workload calls for the staging of activation data, a tile 202 or core 204 may require temporary storage. In the simulation structure 200 of the illustrated embodiment, in each CC H SRAM is provided in each microcontroller 220 of the MC 206 and MC 210. The scratchpad requirement of each core is determined by the maximum sequence length, image resolution (input size), and the ratio of the number of layers (network depth) to the number of supported cores 204. To cover the most common use cases, the microcontroller 220 can be H 206 and MC 210 include SRAM for fully natively supporting sequence lengths of increasing size and complexity. By using multiple CCs for each DNN layer H 206 and MC 210, or by placing the data in off-chip main memory, longer sequences can be supported at some degradation in throughput or energy efficiency per area.

[0039] The microcontroller 220 can also execute instruction sets stored in dedicated i-mem memory. Instructions and synchronization sequences are compiled during program preparation and then preloaded to the appropriate i-mem along with the DNN weight programming for the new workload. During inference, each component (i.e., tile 202 or core 204) independently executes a specific instruction set regulated by periodic global synchronization to ensure successful execution of the entire DNN workload. Control and synchronization are managed by software to minimize additional hardware area budget and power budget.

[0040] FIG3 depicts a two-dimensional (2D) grid 250 incorporated into the analog structure 200 of FIG2 . The 2D grid 250 includes parallel metal row conductors 252 and parallel metal column conductors 254 formed in each CIM analog tile 202 and above the corresponding portion of each computing core 204. Although only a limited number of row conductors 252 and column conductors 254 are shown, each of the cell rows and columns can be manufactured with two conductors dedicated to it, allowing transmission in horizontal, vertical, or plus-sign patterns (connected by horizontal and vertical conductor pairs). The computing core 204 can be configured to operate functions on adjacent analog tiles 202 simultaneously. The 2D grid 250 can also implement multicasting, in which a subset of receivers can obtain the same data at the same time. That is, the 2D grid implements multiple data transmissions involving multiple transmitter cores 204 or transmitting tiles 202; and multiple cores 204 or receiver tiles that all participate simultaneously.

[0041] The spatial architecture will have many components that are only active within a portion of the total pipeline delay to provide power saving functionality to perform at least one of the operations selected from the group consisting of: coarse-grained clock gating, fine-grained clock gating, coarse-grained power gating, fine-grained power gating. There may also be many unmapped components that are completely idle throughout a given DNN workload. In this case, proper power gating is a key aspect of achieving high energy efficiency. Therefore, the simulation structure 200 may include a variety of power gating and clock gating options. For comparison purposes, the baseline option is no power gating, in which all components consume energy continuously throughout the workload based on the idle power specification of the component. The second option is coarse-grained power gating, in which the component is powered off and / or clock gated if it remains unmapped throughout the workload. Coarse-grained power gating also includes partial tile power gating and clock gating based on tile 202 fragment utilization, where completely unmapped fragments are clock-gated and their associated peripheral circuits are powered off, but partially utilized fragments consume all of their idle power. The third option is fine-grained power gating. Fine-grained power gating includes all the features of coarse-grained power gating for shutting down unmapped cores 204, tiles 202, and tile fragments, and in addition, each component is clock-gated and power-gated outside of the first and last time the component is used during the overall workload execution time.

[0042] In some embodiments, the analog tile 202 may include cells arranged in 512 rows and 512 columns. Each 512×512 analog tile 202 is hierarchically designed with eight rows and eight columns of fragments, each fragment containing 64×64 cells. Each fragment row and column can be independently power gated and clock gated, defined herein as coarse-grained power gating, and this helps mitigate efficiency losses due to poor mapping efficiency.

[0043] FIG. 4 depicts a schematic diagram of a 2D grid 400 according to one embodiment of the present invention. The 2D grid 400 is formed on a component block 403, which may be any of the tiles (e.g., tile 202) or cores (e.g., core 204) described above. The 2D grid 400 includes parallel metal row conductors 452 and parallel metal column conductors 454 in a manner similar to the 2D grid 250 described above. However, the 2D grid 400 in FIG. 4 also shows boundaries 460 between component blocks 403. On all four boundaries 460 of each component block 403, the 2D grid 400 includes boundary protection circuits (BGs) 462 that control data transmission. The BGs 462 include signal buffers and transmission gates configured by digital bit vectors stored in a local microcontroller (e.g., microcontroller 220 of FIG. 2) of each component block 403 to establish a circuit-switched communication path before the component block 403 broadcasts data to one or more destination component blocks 403.

[0044] Each BG 462 can independently configure each segment of its grid wiring (e.g., row conductors 452 or column conductors 454). As a result, efficient manipulation operations useful within DNN workloads can be performed on data vectors during transmission, including concatenation, splitting, gathering, and all-gathering. For example, in an LSTM network with a hidden size of 512, a common use case is a network consisting of four adjacent CCs. L The core generates a concatenation of four vectors, each of length 128 spanning two tiles. This concatenation can be performed efficiently using a parallel 2D grid operation. To overlap data communication and computation, the transmit and receive circuits are each equipped with two sets of data registers. This allows for tighter scheduling without causing pipeline stalls, as the 2D grid can simultaneously transmit and consume data internally within component blocks 403 (e.g., individual cores 204 or tiles 202).

[0045] FIG5 depicts a schematic diagram of a CIM architecture 500 having a simulation structure engine 570 according to one embodiment of the present invention. The simulation structure engine 570 may include a plurality of CIM simulation tiles formed into a two-dimensional (2D) grid and a plurality of computing cores connected to the 2D grid. To simplify the description, the tiles and cores are not specifically shown in FIG5 . However, each simulation tile and each computing core may be communicatively connected to all adjacent CIM simulation tiles or computing cores. The simulation structure engine 570 may be surrounded by high-bandwidth chip-to-chip inputs / outputs 572. In addition, the high-bandwidth chip-to-chip inputs / outputs 570 may be separated and personalized for each boundary of the simulation structure engine 500.

[0046] FIG6 depicts a schematic diagram of a CIM architecture 600 having an analog fabric (AF) chip 674 and four analog fabric engines 670 according to one embodiment of the present invention. The four analog fabric engines 670 may include tiles and cores in a manner similar to the analog fabric engine 570 of FIG5 . The analog fabric engines 670 of FIG6 may all be connected to a single 2D grid. In addition, the analog fabric engines 670 of FIG6 may include a single high-bandwidth chip-to-chip input / output 672 around the periphery of all four analog fabric engines 670, rather than individually including high-bandwidth chip-to-chip input / output for each analog fabric engine 670.

[0047] FIG7 depicts a schematic diagram of a CIM architecture 700 with four AF chips 774, each AF chip 774 having four analog fabric engines 770, according to one embodiment of the present invention. To scale to larger workloads, multiple analog fabric engines 770 can be tiled together to implement a large weight fixed system. Multiple AF chips 774 can be connected together by extending 2D grid lines on the same chip or by introducing appropriate chip-to-chip inputs / outputs to implement a multi-chip module system. As analog CIM and CMOS technology continue to scale to higher surface densities, the footprint of each AF chip 774 is expected to shrink. This will result in more AF engines on a chip, minimizing cross-chip communication while providing enough analog CIM tiles (and supporting cores) to store all DNN weight operands, but allowing reuse of modular design constructs for easy scalability.

[0048] The embodiments disclosed above can be used to perform matrix-vector multiplication (MVM) operations. For example, the analog structure can be used to perform long short-term memory (LSTM) workloads. The LSTM network includes recurrently connected memory blocks, each of which contains self-connected memory cells and three multiplication gates (input, output, and forget) to control cell activation. The analog structure microelectronic structure disclosed above can be given an input vector x(t) at time t, and the cell state c(t) and hidden state h(t) are updated from the previous time step (t-1) through four sets of MAC operations with weight operands. In a simple LSTM network, the dimensions of the input vector (x), cell state (c), and hidden state (h) are all equal, typically in the range of 50 to 2000. Therefore, a 1-layer LSTM network with an input vector and hidden vector of size 512 requires 8 tiles with 512×512 cells each. In each time step, the input vector x(t) is first concatenated with the hidden vector h(t-1) from the previous time step before being sent to the tile. Weight operands with the same input dimension are mapped to horizontally adjacent tiles to exploit multicast on the 2D grid. By mapping weight operands with the same output dimension to vertically adjacent tiles, the output vectors from each column of tiles can be efficiently moved one by one to the CCs in its column. L The analog structure can further interleave the weights for unit and gate MAC operations so that each tile + CC L Each column generates h f (t), h i (t), h o (t) and h c (t) has 128 elements, thus allowing CC L The core uses a fused LSTM-aux Op to construct 128 elements of c(t) and h(t). From 4 separate CCs L The four sets of core h(t) elements are concatenated on the grid to produce the complete h(t) vector, and the c(t) elements are added to each individual CC L The core is kept locally for the next time step. L Distributing numerical computations across cores reduces core latency, balances core and tile workloads, and avoids unnecessary grid contention. These throughput gains become larger with larger networks and larger mini-batch sizes.

[0049] The embodiments disclosed above can also be used to complete matrix-vector multiplication (MVM) operations, such as transformer workloads. The Transformer workload layer includes a self-attention block followed by two large fully connected layers. The self-attention block includes an input projection MAC operation for generating a query (Q) vector, a key (K) vector, and a value (V) vector of equal length for each token in the sequence, and an output projection MAC operation for the attention data prepared for the input. Given a vector embedding size between 512 and 2048, the analog structure in the disclosed embodiments maps the input projection tiles horizontally to utilize multicast on a 2D grid. The multi-head self-attention computation occurs at the closest available CC H In each CC H Both process multiple heads in parallel.

[0050] After this self-attention block, before broadcasting the input activation to a large number of other tiles, the nearby CC L The core performs element-wise addition of residuals and LayerNorm (LN) operations. The partial outputs from all tiles are collected into the final CC where the final residual addition and LN operations are performed. L Core. For larger Transformer models, based on the embedding size of the network, tiles, CC H and CC L The required number of cores increases directly.

[0051] Convolutional Neural Networks (CNNs) are activation-heavy networks that include a large number of small-weight kernel filter banks. While weight and activation reuse is significant, it does vary with network depth as the few filters convolved across large images in the initial layers give way to many filters and small intermediate images deeper into the network. In a weight-fixed environment, stored activations must be reorganized and issued to access the same weight kernel in different patterns. This incurs a large number of SRAM accesses and requires careful memory management to achieve good energy efficiency. Since the latency of a MAC operation is determined by the number of analog CIM tile reads required, rather than the number of total OPs, weight reuse can reduce the total number of tiles required but increase the overall latency. The embodiments disclosed herein focus on accurately accounting for the energy efficiency of CNNs and not implementing deliberate weight duplication in order to further optimize throughput.

[0052] Even within the same part of a CNN layer (e.g., a residual bottleneck block), the weight layer size and shape can vary widely, resulting in variations in the number of tiles required. Furthermore, with fine-grained pipelining of CNN activations, each CNN layer can start processing once enough outputs from previous layers have been generated. Here, each weight layer is arranged horizontally across tiles as much as possible to ensure that the weights are aligned to the corresponding core (CC) that performs activation alignment.H or MC).

[0053] A computer-readable storage medium may be a tangible device that can retain and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disc (DVD), a memory stick, a floppy disk, a mechanical encoding device (such as a punched card or a raised structure in a groove on which instructions are recorded), and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated through a waveguide or other transmission medium (e.g., a light pulse transmitted through a fiber optic cable), or an electrical signal transmitted through a wire.

[0054] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network). The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0055] The computer-readable program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits including, for example, programmable logic circuits, field programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions to personalize the electronic circuits by utilizing the state information of the computer-readable program instructions so as to perform various aspects of the present invention.

[0056] The description of various embodiments of the present invention has been presented for the purpose of illustration, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, practical applications, or technical improvements over technologies found in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A microelectronic structure comprising: a first in-memory computation CIM simulation tile configured to store a weight operand matrix for generating an output vector from an input vector, wherein the first CIM simulation tile comprises a first microcontroller configured to execute instructions and perform in-memory operations; as well as A first computing core is communicatively coupled to the first CIM emulation tile, wherein the first computing core includes a second microcontroller configured to execute instructions.

2. The microelectronic structure of claim 1, wherein the first computing core includes a scratchpad memory.

3. A microelectronic structure according to claim 1, wherein the first computing core is configured to perform an operation selected from the group consisting of: batch normalization, rectified linear activation function, Gaussian error linear unit, element-wise addition, element-wise multiplication, average pooling, max pooling, layer normalization, activation function via lookup table, long short-term memory assistance, hyperbolic tangent, sigmoid, softmax and self-attention.

4. The microelectronic structure of claim 1 , wherein the first computing core includes a power saving function to perform at least one of operations selected from the group consisting of: coarse-grained clock gating, fine-grained clock gating, coarse-grained power gating, and fine-grained power gating.

5. The microelectronic structure of claim 1, wherein the first computing core is configured to perform off-chip input-output (I / O) data transfer operations.

6. The microelectronic structure of claim 1, wherein the first CIM simulation tile is configured to perform a vector-matrix multiplication VMM operation.

7. The microelectronic structure of claim 1, further comprising a second computing core communicatively coupled to the first CIM simulation tile and the first computing core, wherein the second computing core comprises a third microcontroller configured to execute an instruction set.

8. The microelectronic structure of claim 7, wherein the first computing core and the second computing core differ based on a specialization selected from the group consisting of: precision, data format, data flow, storage capacity, and programmability.

9. A computing-in-memory (CIM) accelerator architecture for a deep neural network (DNN), comprising: The first simulation structure engine, including: a plurality of in-memory computation CIM simulation tiles, wherein each CIM simulation tile is configured to store a weight operand matrix for generating an output vector from an input vector and to perform in-memory computation; a plurality of computing cores, wherein each CIM emulation tile and each computing core comprises a microcontroller configured to execute an instruction set; and An on-chip interconnect communicatively connects all of the plurality of CIM emulation tiles to the computing core.

10. The CIM accelerator architecture of claim 9, wherein the plurality of computing cores comprises: a first subset of computing cores, the first subset of computing cores comprising a scratchpad memory of a first capacity; as well as A second subset of computing cores includes a second capacity of scratch pad memory.

11. The CIM accelerator architecture of claim 9, wherein the plurality of computing cores comprises a subset of computing cores configured to perform off-chip input-output (I / O) data transfer operations.

12. The CIM accelerator architecture of claim 9, wherein each of the plurality of computing cores is configured to perform an operation selected from the group consisting of batch normalization, ReLU, GeLU, element-wise addition, element-wise multiplication, average and max pooling, layer normalization, activation function via lookup table, LSTM-aux, hyperbolic tangent, sigmoid, softmax, and self-attention.

13. The CIM accelerator architecture of claim 9, wherein each CIM simulation tile and each computing core includes a power saving function to perform an operation selected from the group consisting of: coarse-grained clock gating, fine-grained clock gating, coarse-grained power gating, and fine-grained power gating.

14. The CIM accelerator architecture of claim 9, wherein a first computing core from the plurality of computing cores and a second computing core from the plurality of computing cores differ based on a specialization selected from the group consisting of: precision, data format, data flow, storage capacity, and programmability.

15. The CIM accelerator architecture according to claim 9, further comprising: The first four-engine system, including: The second simulation structure engine; A third simulation structure engine; and A fourth simulation structure engine, wherein the second simulation structure engine, the third simulation structure engine, and the fourth simulation structure engine are communicatively connected to the first simulation structure engine through a 2D grid, and the 2D grid is surrounded by high bandwidth chip-to-chip input / output.

16. The CIM accelerator architecture of claim 15, further comprising: 24 engine system; 3rd and 4th engine systems; as well as A fourth four-engine system, wherein the second four-engine system, the third four-engine system and the fourth four-engine system are communicatively connected as a multi-chip system.

17. The CIM accelerator architecture of claim 9, wherein a plurality of CIM simulation tiles and computing cores are connected using a two-dimensional (2D) grid comprising parallel row lines and parallel column lines, the parallel row lines and parallel column lines enabling multicasting between the plurality of CIM simulation tiles and the plurality of computing cores. 18 . The CIM accelerator architecture of claim 9 , further comprising a boundary protection circuit BG on each parallel row line and each parallel column line at each boundary of the plurality of CIM simulation tiles and the plurality of computing cores.

19. The CIM accelerator architecture of claim 18, wherein the BG includes signal buffers and transmission gates controlled by a microcontroller on each compute core and CIM analog tile.

20. A heterogeneous and programmable in-memory computing CIM accelerator architecture, comprising: a plurality of in-memory computation CIM simulation tiles, the plurality of CIM simulation tiles being configured to store weight operand matrices, wherein each CIM simulation tile is configured to store a weight operand matrix for generating an output vector from an input vector, and to perform in-memory computation; a first plurality of computing cores, wherein each computing core includes a scratchpad memory configured to perform a self-attention-assisted operation; a second plurality of computing cores configured to perform complex LSTM-auxiliary LSTM-aux operations; a microcontroller embedded in at least one CIM analog tile of the first plurality of analog tiles and at least one computing core of the second plurality of computing cores; as well as An on-chip interconnect communicatively connects all of the plurality of CIM emulation tiles to the first plurality of computing cores and the second plurality of computing cores.

21. The CIM accelerator architecture of claim 20, further comprising a third plurality of computing cores configurable to perform off-chip input-output (I / O) data transfer operations.

22. The CIM accelerator architecture of claim 20, further comprising a third plurality of computing cores, wherein each computing core of the third plurality of computing cores comprises a power saving function to perform at least one of operations selected from the group consisting of: coarse-grained clock gating, fine-grained clock gating, coarse-grained power gating, and fine-grained power gating.

23. The CIM accelerator architecture of claim 20, wherein the first plurality of computing cores and the second plurality of computing cores are configured to perform operations selected from the group consisting of batch normalization, ReLU, GeLU, element-wise addition, element-wise multiplication, average and max pooling, layer normalization, activation function via lookup table, hyperbolic tangent, sigmoid, and LSTM-aux.

24. The CIM accelerator architecture of claim 20, further comprising boundary protection circuits (BG) on parallel row lines and parallel column lines of the on-chip interconnects between the first plurality of computing cores, the second plurality of computing cores, and the plurality of CIM analog tiles.

25. A heterogeneous and programmable in-memory computing CIM accelerator architecture, comprising: a plurality of in-memory computation CIM simulation tiles, the plurality of CIM simulation tiles being configured to store weight operand matrices, wherein each CIM simulation tile is configured to store a weight operand matrix for generating an output vector from an input vector, and to perform in-memory computation; a first plurality of computing cores, wherein each computing core includes a scratchpad memory configured to perform a self-attention-assisted operation; a second plurality of computational cores configured to perform complex LSTM auxiliary LSTM-aux operations and operations selected from the group consisting of: hyperbolic tangent, sigmoid, multiplication, and addition; as well as An on-chip interconnect communicatively connects all of the plurality of CIM emulation tiles to the first plurality of computing cores and the second plurality of computing cores.

Citation Information

Patent Citations

  • In-memory compute array with integrated bias elements

    US20200388330A1

  • Efficient Neural Network Accelerator Dataflows

    US20220076110A1

  • Scalable array architecture for in-memory computing

    US20230074229A1

Cited By

  • Transform model accelerator for ocean drifting buoy

    CN121683911A