Shared routing and sensing in multi-block digital in-block computing (DIMC) neural processing unit (NPU)

Abstract

The invention relates to shared routing and sensing in a multi-block digital in-memory computing (DIMC) neural processing unit (NPU). A first in-memory computing (IMC) circuit includes a first IMC processing block coupled for data communication with a first interface circuit. The second IMC circuit includes a second IMC processing block coupled for data communication with the second interface circuit. A shared resource bus connects the first interface circuit and the second interface circuit for data communication of feature data, weight data, or input computational data.

Description

Cross-references to related applications

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 705,746, filed October 10, 2024, the contents of which are incorporated herein by reference. Technical Field

[0002] The embodiments described herein relate to a neural processing unit (NPU) that utilizes multiple interconnected digital in-memory computation (DIMC) processing tiles. Background Technology

[0003] Data communication between Digital In-Memory Computation (DIMC) blocks is a critical issue within the Neural Processing Unit (NPU). Data transferred between DIMC blocks can include feature data, weight data, and computational data (such as sums, partial sums, partial products, and / or partial computational data). Significant routing resources are required to support high-bandwidth operations.

[0004] There is a need in this field for more efficient data communication interconnection between DIMC processing blocks. Summary of the Invention

[0005] In one embodiment, a circuit includes: a first in-memory computing (IMC) circuit including a first IMC processing block coupled to a first interface circuit for data communication; a second IMC circuit including a second IMC processing block coupled to a second interface circuit for data communication; and a shared resource bus connecting the first interface circuit to the second interface circuit to support data communication between the first IMC processing block and the second IMC processing block.

[0006] Data communication via the shared resource bus includes the transmission of feature data for in-memory computation operations provided to the first IMC processing block and the second IMC processing block.

[0007] Data communication via the shared resource bus includes the transmission of weight data for in-memory computation operations provided to the first IMC processing block and the second IMC processing block.

[0008] Data communication via the shared resource bus includes the transmission of output computation data, which is generated by one of the first IMC processing blocks and the second IMC processing block performing in-memory computation operations, and is provided as input computation data to the other of the first IMC processing block and the second IMC processing block. Attached Figure Description

[0009] To better understand the embodiments, reference will now be made to the accompanying drawings by way of example only, in which:

[0010] Figure 1A This is a block diagram of the multi-island in-memory computing (IMC) neural processing unit (NPU);

[0011] Figure 1B It is a block diagram of individual IMC NPU islands that include multiple IMC tile clusters;

[0012] Figure 1C It is a block diagram of an IMC block cluster that includes multiple digital in-memory computing (DIMC) circuits;

[0013] Figure 1D This is a block diagram of one embodiment of a DIMC circuit;

[0014] Figure 2 This is a more detailed schematic diagram of the DIMC circuit;

[0015] Figure 3 This illustrates a block cluster configuration that supports IMC blocks with decompressor logic and shared computation logic;

[0016] Figure 4 A schematic diagram of an example implementation of a DIMC processing block is shown; and

[0017] Figures 5A-5B A neural network graph timeline is shown for in-memory computation operations using shared resource bus interconnects between DIMC circuits. Detailed Implementation

[0018] Now for reference Figure 1AThe diagram illustrates a processing system block diagram comprising a multi-island in-memory computation (IMC) neural processing unit (NPU) 10. The multi-island IMC NPU 10 includes multiple IMC NPU islands 12 arranged in an array and interconnected with each other via a data interconnect network 13. The multiple IMC NPU islands 12 of the multi-island IMC NPU 10 are also connected to memory circuitry 16 (including, for example, flash memory or random access memory (RAM)) via a memory bus 14. Data stored in the memory circuitry 16 includes computational weights of the network. Prior to in-memory computation execution, the weights of the processing layer to be computed are transferred to digital IMC blocks (discussed in detail below) within a given IMC NPU island 12. The system RAM may also store sums, partial sums, partial products, and / or partial computational outputs from the IMC blocks of the IMC NPU islands 12, which will be used for computation in the next processing layer. The multiple IMC NPU islands 12 are also coupled to a host processing unit 22 and external interface (IF) circuitry 24 via a system bus 20. The host processing unit 22 (also known as the central processing unit (CPU)) is responsible for executing instructions from the program and managing the overall operation of the system. It coordinates the activities of all other hardware components and ensures that tasks are executed efficiently. Data storage memory 26 is also coupled to the system bus 20 for access by the host processing unit 22. Data storage memory 26 can store programming and application data required by the main processor. One or more functional (IP) circuits 28 are also connected to the system bus 20. Functional (IP) circuits can be any intellectual property circuit or block used in the system. Examples include: Direct Memory Access (DMA) circuitry, Serial Peripheral Interface (SPI) circuitry, Universal Asynchronous Receiver Transmitter (UART) circuitry, Universal Serial Bus (USB) circuitry, clock and reset generator circuitry, top-level register interface circuitry, data converter circuitry, etc. Data bridge circuitry 36 interconnects the system bus 20 and the memory bus 14 to support data communication between them.

[0019] In summary, a neural processing unit (NPU) is an accelerator designed to improve the performance of neural processing tasks. Within the system, it communicates with various components, including system and external memory, to retrieve weights and store sums, partial sums, partial products, and / or partial computations. Furthermore, it interacts with various sensor function (IP) circuitry and memory to obtain input features.

[0020] Now for reference Figure 1BThis diagram illustrates a block diagram of an individual IMC NPU island 12. Each IMC NPU island 12 includes a bus interface 40 for supporting connections between the island 12 and one or both of the system bus 20 and memory bus 14. Multiple Direct Memory Access (DMA) circuits 42 are connected to the bus interface 40. The DMA circuits 42 act as data movers and operate to move data from one memory to another. In this case, the DMA circuits 42 are used to transfer data from external flash memory / non-volatile memory to system memory, or from system memory to IMC memory, and from IMC output to system memory. Multiple IMC block clusters 46 are interconnected to the DMA circuits 42 via local router circuits 48. Control circuitry 50 for NPU operation is connected to the bus interface 40 and the DMA circuits 42. The NPU control circuitry 50 controls different modules of the NPU subsystem. All NPU programming registers are part of the NPU control. Tensor cache and reshape circuitry 54 is coupled to the local router circuitry 48. Tensor cache and remodeling module 54 is used to remodel input features and weights according to the computational requirements of the DIMC blocks. Program accelerator circuitry 58 is coupled to local router circuitry 48 and configured to perform various scalar operations within the NPU. System non-volatile memory circuitry 62 is also coupled to local router circuitry 48. This memory circuitry 62 is configured to store weight data for in-memory computational operations, which is selectively accessed and delivered to the IMC block cluster 46 via local router circuitry 48.

[0021] In summary, the IMC NPU island 12 comprises (e.g., one or more) collections of IMC block clusters 46. The IMC NPU island 12 has control circuitry 50 for managing the NPU, data reforming blocks 54 for regulating input data to the IMC clusters, data movers 42 for facilitating data transmission, and accelerators 58 for performing various scalar operations within the NPU. All these different blocks coordinate and communicate with each other via local router circuitry 48.

[0022] Now for reference Figure 1C This diagram illustrates a block diagram of an IMC block cluster 46. Each block cluster 46 includes multiple Digital In-Memory Computation Operations (DIMC) circuits 70 arranged in an array. Adjacent circuits 70 are interconnected for data communication via a shared resource bus 72. The block clusters 46 are connected to a router 48 of the IMC NPU island 12. The arrangement of the DIMC circuits 70 can be programmed according to processing requirements, such that specific DIMC circuits 70 are connected to the router 48 of the IMC NPU island 12. The connection between the block clusters 46 and the router 48 is achieved through a set of buffer circuits (…). Figure 1C(An example is shown) to implement these buffer circuits as part of block cluster 46. Shared resource bus 72 can be used by DIMC circuit 70 for transferring feature data, weight data, and / or computation data (such as sums and partial sums, partial products, and / or partial computation data) from one circuit 70 to an adjacent circuit 70.

[0023] The advantage of using the shared resource bus 72 is that it eliminates the need for separate buses or communication links to carry different types of data (such as feature data, weight data, and / or computed data). Shared compute resources between two or more DIMC circuits 70 are also supported. This also facilitates configuring specific DIMC circuits 70 within a given block cluster 46 to have certain compute logic and / or decompressor logic, which are shared by all DIMC circuits 70 within the block cluster 46 in a time-division multiplexing manner. The decompressor logic within a specific DIMC circuit 70 can be used to process compressed computed weights stored in the processing block memory to obtain decompressed weight data and output it to other DIMC circuits 70 within the block cluster 46. The presence of structured and unstructured sparsity in both weight data and feature data provides opportunities for compressed data and processing blocks that use DIMC circuits 70 in a dense manner. Including decompressor logic can be costly, and therefore providing a solution for sharing decompressor logic across blocks can be significantly advantageous.

[0024] Therefore, the above implementation supports compressed data storage and decompression computation. Computational resources can be shared by many DIMC circuits 70 in sparse mode.

[0025] One or more sideband communication channels can be provided to connect to the DIMC circuits 70 of the IMC block cluster 46. An example of such a sideband communication channel is a power management (PM) channel, where power management control signaling is transmitted via the sideband communication channel. The granularity of power management control functions is on a per-block basis. Therefore, the system can perform independent power management control for each DIMC circuit 70 via power management control signaling, such as specifying an active mode, sleep mode, data retention mode, etc.

[0026] Therefore, the IMC block cluster 46 includes one or more DIMC circuits 70. Within the cluster, these DIMC circuits 70 can be used independently or linked in various configurations to handle any neural network workload.

[0027] Figure 1DA block diagram of one embodiment of a DIMC circuit 70 is shown. Each DIMC circuit 70 includes a DIMC processing block 80. The block 80 is configured to perform digital in-memory computation (DIMC) operations based on stored weight data and received feature data. (See also: U.S. Patent Application Publication No. 2024 / 0071439) Figure 4 An example of such a DIMC processing block is shown in the example (in the example). As described above, the DIMC processing block 80 may include computation logic that provides processing resources that can be shared by processing blocks 80 of other IMC circuits 70. The DIMC processing block 80 may include decompressor logic that provides additional processing resources related to the decompression of stored weight data that can be shared by processing blocks 80 of other DIMC circuits 70.

[0028] Power management control signaling (PM) is received by DIMC processing block 80 via a sideband communication channel to selectively control the operating mode of DIMC processing block 80 (e.g., active mode, sleep mode, data retention mode, etc.).

[0029] Each DIMC circuit 70 is coupled to the shared resource bus 72 via an interface circuit (IF) 86 for data communication with adjacent DIMC circuits 70 (via its corresponding interface circuit 86). In an example array configuration of the block cluster 46, there is an interface circuit 86 associated with each basic compass direction (north, south, east, west). The DIMC processing block 80 of this DIMC circuit 70 is coupled to a given interface circuit in the interface circuit 86 via a router circuit 88 for data communication. In an example embodiment, the router circuit 88 may be implemented using a packet-switched network or a circuit-switched network. Only those DIMC processing blocks 80 involved in a given functional operation (such as data transmission and processing) are in active mode controlled by power management control signaling.

[0030] Each DIMC processing block 80 is coupled to router circuitry 88 to receive feature data of an in-memory computation operation being performed. For example, this feature data may be transmitted to IMC processing block 80 via router 48 of IMC NPU island 12 through a shared resource bus 72 interconnecting DIMC processing block 80 and router 88. Each DIMC processing block 80 is also coupled to router circuitry 88 to receive weight data of an in-memory computation operation being performed. For example, this weight data may be transmitted to DIMC processing block 80 via router 48 of IMC NPU island 12 (e.g., retrieved from ePCM memory 62) through a shared resource bus 72 interconnecting IMC processing block 80 and router 88. DIMC processing block 80 can also be a source of weight data (compressed or uncompressed) transmitted via router circuitry 88 to other IMC processing blocks 80 via shared resource bus 72. Furthermore, each DIMC processing block 80 is coupled to router circuitry 88 to output processing data (e.g., sums and partial sums, partial products, and / or partial computation outputs) of the in-memory computation operation being performed. For example, this processing data may be transmitted from the DIMC processing blocks 80 via a shared resource bus 72 that interconnects the DIMC processing blocks 80 and router 88. The DIMC processing block 80 may also receive, via router circuitry 88, input processing data (e.g., sums and partial sums, partial products, and / or partial computation outputs) of in-memory computation operations being performed from other DIMC processing blocks 80, transmitted via the shared resource bus 72.

[0031] Now for reference Figure 2 It shows a more detailed block diagram of the DIMC circuit 70.

[0032] The DIMC processing block 80 includes a data buffer circuit configured to buffer data related to transmission via router 88 and the shared resource bus 72. The input buffer circuit can store weight data, feature data, and / or computational data received on the shared resource bus 72 via interface 86 and routed to the DIMC processing block 80 by router 88. The output buffer circuit can store weight data, feature data, and / or computational data generated by the DIMC processing block 80, which will be routed by router 88 and transmitted on the shared resource bus 72 via interface 86.

[0033] This allows, for example, feature data to be broadcast via the shared resource bus 72 and input to DIMC processing blocks 80 of multiple DIMC circuits 70. This is important, for example, for supporting in-memory computation operations where the same feature data is applied in computations targeting different weight datasets stored in different DIMC circuits 70. In this case, power management control signaling transmitted via the sideband communication channel can specifically select the DIMC processing blocks 80 of the multiple DIMC circuits 70 to receive the feature data as operating modes to access the shared resource bus 72 and use their input buffer circuitry (acting as feature buffers) to receive the broadcast feature data. In an alternative implementation, the feature data can be passed directly from the shared resource bus 72 for use by the DIMC processing blocks 80 without processing by the buffer circuitry.

[0034] This allows weight data to be read from a DIMC processing block 80 of a DIMC circuit 70 and transmitted to DIMC processing blocks 80 of multiple DIMC lines 70. In this context, power management control signaling transmitted via the sideband communication channel can specifically select a source DIMC processing block 80 of a DIMC circuit 70 providing weight data in an operating mode where an output buffer circuit acting as a weight buffer outputs the weight data to the shared resource bus 72; and specifically select a destination DIMC processing block 80(one or more) of DIMC circuits 70 receiving weight data in an operating mode where its input buffer circuit acting as a weight buffer receives the transmitted weight data. In an alternative implementation, the weight data can be directly transmitted from the shared resource bus 72 for use by the DIMC processing blocks 80 without the need for buffer circuitry.

[0035] This also allows computational data generated by an in-memory computation operation performed by a DIMC processing block 80 of one DIMC circuit 70 to be transmitted for further processing by a DIMC processing block 80 of another DIMC circuit 70. In this case, the power management control signaling transmitted via the sideband communication channel can specifically select the source DIMC processing block 80 of the DIMC circuit 70 providing the computational data as operating in a mode where an output buffer circuit, acting as a partial and / or partial product buffer, outputs the computational data to the shared resource bus 72; and specifically select the destination DIMC processing block 80(one or more) of the DIMC circuit 70 receiving the computational data as operating in a mode where its input buffer circuit, acting as a partial and / or partial product buffer, receives the transmitted computational data. In an alternative implementation, the computational data can be directly transmitted from the shared resource bus 72 for use by the DIMC processing block 80 without the need for buffer circuitry.

[0036] Now for reference Figure 3 This illustrates the configuration of block cluster 46, where a specific DIMC circuit within DIMC circuit 70 of block cluster 46 includes decompressor logic, and a specific DIMC circuit within DIMC circuit 70 of block cluster 46 includes shared computation logic. It should be understood that a given DIMC circuit 70 may include both decompressor logic and shared computation logic. For a DIMC circuit 70 having a DIMC processing block 80, a shared resource bus 72 can be used to transfer weights and partial computation results (e.g., sums and partial sums, partial products, and / or partial computations) between multiple DIMC circuits 70. The shared computation logic is available to the DIMC circuit 70 on a time-division multiplexing basis, where weights and partial computation data are transmitted via bus 72. Compressed weight data can also be stored in the DIMC processing block 80 of a given DIMC circuit 70, retrieved from memory for processing in the decompressor logic, and then decompressed weight data can be delivered via the shared resource bus 72 for use in the computations of the DIMC processing blocks 80 of other DIMC circuits 70 in block cluster 46. Furthermore, feature data can be received by block cluster 46 (e.g., via a buffer circuit connection to router 48). Figure 1B (), and is delivered to one or more DIMC circuits 70 via the shared resource bus 72 to the DIMC processing block 80.

[0037] As described above, router circuit 88 can be implemented using either a packet-switched network or a circuit-switched network. In a packet-switched network implementation, data to be transmitted on the shared resource bus 72 is multiplexed into data packets at different time intervals within the system clock cycle. Control logic specifies packet access for data communication at given time intervals. In a circuit-switched network, tri-state buffers drive the signal lines of the shared resource bus 72 at different time intervals within the system clock cycle. Control logic circuitry specifies access for data communication at given time intervals.

[0038] Now for reference Figure 4The diagram illustrates a schematic of an example implementation of DIMC processing block 80. See also U.S. Patent Application Publication No. 2024 / 0071439, which is incorporated herein by reference. DIMC processing block 80 is implemented using memory circuitry comprising a static random access memory (SRAM) array 112 formed by a plurality of SRAM memory cells 114 arranged in a matrix format having N rows and M columns. Each memory cell 114 is programmed to store one bit of data. In digital in-memory computation processing, the data stored in memory array 112 includes computational weights or kernel data for digital in-memory computation operations. In this context, digital in-memory computation operations are understood as a form of high-dimensional matrix-vector multiplication (MVM) that supports multi-bit weights stored in multiple bit cells of memory. Groups of bit cells (in the case of multi-bit weights) can be considered as virtual synaptic elements. Each data bit (whether user data or weight data) stored in the memory array has a logic "1" or logic "0" value.

[0039] Each SRAM memory cell 114 may include a 6T-type memory cell. Cell 114 may include two cross-coupled CMOS inverters, the inputs and outputs of which are coupled to form a latch circuit having a true data storage node and a complementary data storage node for storing complementary logic states of the stored data bits. Cell 114 also includes two transfer (transfer gate) transistors, the gate terminals of which are driven by word lines WL, and the source-drain paths of which are coupled between the true data storage node and a node associated with the true bit line BLT, and between the complementary data storage node and a node associated with the complementary bit line BLC.

[0040] Alternatively, each SRAM memory cell 114 may include an 8T-type memory cell. Cell 114 may include two cross-coupled CMOS inverters, the inputs and outputs of which are coupled to form a latch circuit having a true data storage node and a complementary data storage node for storing complementary logic states of the stored data bits. Cell 114 also includes two transfer (transfer gate) transistors, the gate terminals of which are driven by word lines WL, and the source-drain paths of which are coupled between the true data storage node and the node associated with the true bit line BLT, and between the complementary data storage node and the node associated with the complementary bit line BLC. A signal path between the read bit line RBL and a reference voltage reference forms a read circuit with a read transistor, which is gate-controlled by a signal at the complementary memory node QC and selected by the read word line RWL.

[0041] It should be understood that the DIMC processing block 80 can be modified to use different types of memory cells, such as any form of bit cell, storage element, or synaptic element used to generate deterministic reads arranged in an array. As a non-limiting example, consider the use of non-volatile memory (NVM) cells, such as magnetoresistive RAM (MRAM) cells, flash memory cells, phase-change memory (PCM) cells, or resistive RAM (RRAM) cells. In the following discussion, the focus is on an implementation using an 8T-type SRAM cell 114, but this is by way of a non-limiting example, and it should be understood that any suitable memory element (e.g., binary (two-level) memory element or m-level (multi-level) memory element) can be used.

[0042] Each cell 114 includes a word line WL, complementary bit line pairs BLT and BLC, a read word line RWL, and a read bit line RBL. SRAM memory cells in the common rows of the matrix are interconnected via common word lines WL and common read word lines RFL. Each word line (WL and / or RWL) is driven by word line driver circuitry 116 during read / write operations using word line signals generated by row decoder circuitry 118. SRAM memory cells in the common columns of the matrix of the entire array 112 are interconnected via common complementary (write) bit line pairs BLT and BLC. Array 112 is divided into P subarrays 1130 to 1130. P-1 Each subarray 113 includes M columns and N / P rows of memory cells 114. The SRAM memory cells in the common columns of each subarray 113 are connected to each other via local read bit lines RBL.

[0043] P local read bit lines RBL0 from subarray 113 for column x in array 112 <x>To RBL P-1 <x>With the common complementary bit line pair BLT for column x in array 112 <x>and BLC <x>Together, they are coupled to column input / output (I / O) circuit 120 (x). Here, x = 0 to M-1. The data input port (D) of column I / O circuit 120 receives input data (user or weight data) from the input buffer circuit. In conventional memory access operation mode, in response to an assertion of the word line signal, the received input data is written to the SRAM memory cell 114 in the column via complementary bit line pairs BLT, BLC. The data output port (Q) of column I / O circuit 120 generates output data to be stored in the output buffer circuit. In conventional memory access operation mode, in response to an assertion of the read word line signal, the output data is read from the SRAM memory cell 114 in the column via the read bit line RBL. In addition, column I / O circuit 120 also includes P subarray data output ports R0 to R1 for generating output data. P-1 In digital in-memory computing operation mode, in response to simultaneous assertions of multiple read word line signals (one read word line signal for each subarray 113), the output data is transmitted from the corresponding subarrays 1130 to 1130. P-1 The memory cell 114 on the local read bit line RBL is read. The digital computation processing circuit 123 performs digital computation on the output data from the subarray data output port R based on the feature data. The feature data is received by the digital computation processing circuit 123 from the input buffer circuit. Alternatively or additionally, the digital computation processing circuit 123 may receive input computation data from the input buffer circuit (e.g., the received computation data may involve sums and partial sums, partial products, and / or partial computations performed by another DIMC processing block 80 in a pipelined processing operation). The digital computation processing circuit 123 is used to generate output computation data for digital in-memory computation operations. This output computation data is stored in the output buffer circuit (e.g., and may involve sums and partial sums, partial products, and / or partial computations to be further processed by another DIMC processing block 80 in a pipelined processing operation).

[0044] The processing circuit 123 can implement the computational logic of digital signal processing in various ways, including: fully supporting Boolean operations (XOR, XNOR, NAND, NOR, etc.) and vector operations, depending on system and application requirements; accumulation pipelined operations, where vector multiplication is supported in memory; and matrix-vector multiplication pipelined operations, where the memory output is a vector of the multiplication and accumulation (MAC) function. The processing circuit 123 can also be used to perform decompression operations (e.g., to decompress compressed weighted data read from memory 112). It should be noted that the processing circuit 123 is a component of the digital in-memory computation circuit 80.

[0045] The input / output buffer circuitry enables the DIMC processing block 80 to interconnect with the shared resource bus 72 via the router 88 of the DIMC circuit 70. The shared resource bus 72 allows weight data (WD) to be transferred to the DIMC processing block 80 of a given DIMC circuit 70 (via the input buffer circuitry), where the weight data can be written to memory 112 via the D port in data write mode, or passed to digital computation logic 123 for in-memory computation operations. The DIMC processing block 80 of a given DIMC circuit 70 can also use the shared resource bus 72 to support the transfer of weight data read from memory 112 (via the output buffer circuitry) to another DIMC circuit 70 (note that the read weight data can come directly from the R<> port used for transfer, or first via digital computation logic 123 before transfer). The shared resource bus 72 also allows feature data (FD) to be transferred to the DIMC processing block 80 of a given DIMC circuit 70 (via the input buffer circuitry), where the feature data is passed to digital computation logic 123 for in-memory computation operations. The shared resource bus 72 also enables the transfer of input computation data (Comp) to the DIMC processing block 80 (via input buffer circuitry), where the characteristic data is passed to the digital computation logic 123 for performing in-memory computation operations. The DIMC processing block 80 of a given DIMC circuit 70 can also use the shared resource bus 72 to support the transfer of output computation data (Comp) from the DIMC processing block 80 (via output buffer circuitry) to another DIMC circuit 70.

[0046] The computational logic for digital signal processing, along with the input / output circuitry and subarray data output ports R0 to R123, are executed by the processing circuit 123. P-1 Tight integration to support the utilization of wide (e.g., P times) vector access. This solution brings many quality factor (FOM) advantages, including: enabling multi-word access in the same cycle; amortizing the common logic switching capability within the SRAM during wide vector access; reducing bit-line switching power consumption using subarray 113 (i.e., where P word lines are asserted in parallel to access P corresponding subarrays); supporting both, and having the opportunity to switch between traditional memory access operation modes and digital in-memory compute operation modes; and improving the on / off current ratio on the same bit line, which is a key issue when implementing circuits using fully depleted silicon-on-insulator (FDSOI) technology with aggressively positive body bias.

[0047] It should be noted that, according to conventional memory access operation modes, the DIMC processing block 80 presents a conventional SRAM interface via data input port D and data output port Q. In response to an applied memory address (Addr), the circuit supports read (via data output port Q) and write (via data input port D) access to a single row of memory cells 114 in array 112 via selected assertions on a single word line WL or RWL. The circuit also supports read (via data output ports R0 to R1) according to digital in-memory computation operation modes. P-1 Presents a subarray processing interface. In response to the applied memory address (Addr), this circuitry supports processing of each subarray 1130 to 113 by simultaneously asserting the corresponding read word line RWL. P-1 Simultaneous reading of single-row memory cell 114 (via data output ports R0 to R) P-1 Access can be performed by decoding a single address to select multiple word lines (one word line per subarray 113) for assertion, or by decoding multiple addresses to select multiple word lines (one word line per subarray 113) for assertion. Using multiple subarrays 113 in this mode enables parallelism to support a very wide range of computational processing accesses without sacrificing density. Advantageously, this digital in-memory computation operation mode leverages the resources of a conventional SRAM design with modified control, decoding, and input / output circuitry (as will be discussed in detail herein) to achieve parallel access in the digital in-memory computation operation mode, and switches between the conventional memory access operation mode and the digital in-memory computation mode via additional control as needed by the system application. This architecture brings parallelism to the use of push-rule bit cells, thereby achieving high density / computational density when configured for in-memory computation operation mode. Despite the foregoing, other bit cell types can also be used, as mentioned above.

[0048] Control circuit 119 controls the mode operation of circuits within DIMC processing block 80 in response to the logic state of control signal IMC. When control signal IMC is in a first logic state (e.g., logic low), circuit 80 operates according to a conventional memory access operation mode (for writing data from data input port D to the memory array or reading data from the memory array to data output port Q). Conversely, when control signal IMC is in a second logic state (e.g., logic high), DIMC processing block 80 operates according to a digital in-memory computation operation mode (for reading weighted data from the memory array to the subarray data output port R).

[0049] When the DIMC processing block 80 operates in the conventional memory access operation mode, the row decoder circuit 118 decodes the received address (Addr) and selectively actuates only one word line WL (during write) or one read word line RWL (during read) of the entire array 112 using word line signal pulses to access the corresponding row in the row of memory cell 114. During write, the logical state of the data at input port D is written by column I / O circuit 120 through complementary bit lines paired BLT, BLC to the single row memory cell coupled to the accessed word line WL. During read, the logical state of the data stored in the single row memory cell coupled to the accessed word line WL is output from the read bit line RBL to column I / O circuit 120 for output at data output port Q.

[0050] When the DIMC processing block 80 operates in digital in-memory computation mode, the row decoder circuit 118 decodes the received address (Addr) and selectively (simultaneously) actuates a read word line RWL in each subarray 113 of the memory array 112 using word line signal pulses to access the corresponding row memory cell 114 in each subarray 113. The logical state of the weighted data stored in the row of memory cells coupled to the accessed read word line RWL in each subarray 113 is read from the bit line RBL0. <x>To RBL P-1 <x>to column I / O circuitry 120 for output at corresponding subarray data output ports RO to R P-1

[0051] It should be noted that the output of each subarray 113 can be considered as a sub-tensor / tensor for processing operations. Further, the outputs of multiple subarrays 113 can be grouped into larger tensors. The grouping of subarray outputs can be across columns, across rows, or both. Such processing is supported by the configuration and operation of processing circuitry 123.

[0052] Figure 4 The illustrated architecture provides many advantages for digital in-memory computing, including: very wide vector access is implemented to support high-dimensional tensor processing for artificial neural networks (ANNs); super-dimensional computing is also supported for artificial intelligence (AI) training and inference workloads; the computations are deterministic with a wide range of weight data and feature data precisions and number formats allowed for neural network applications (note that this is a significant distinction compared to analog in-memory computing, which is limited to a simplified signed / unsigned integer format); and the solution is scalable to include additional stochastic computing modes to increase area and power efficiency.

[0053] Reference is now made to Figures 5A-5B which illustrates a neural network graph schedule for in-memory computing operations.

[0054] In Figure 5A the block cluster 46 includes multiple DIMC circuits 70 that utilize DIMC processing blocks 80. Figure 5A ​The neural network graph temporal representation 1000 shows an example of a layer flow pipeline (which includes mapping different layers of a given neural network onto different DIMC circuits; such mapping is managed by the compiler). The layer flow pipeline includes: layer (n-1) utilizing DIMC circuit 70(1) and its DIMC processing block 80, layer (n) utilizing DIMC circuit 70(3) and its DIMC processing block 80, and layer (n+1) utilizing DIMC circuit 70(5) and its DIMC processing block 80. For processing scenarios that provide output of layer (n-1) as input to layer (n), DIMC circuit 70(1) and 70(3) will be communicatively interconnected through shared resource bus 72. The computational output of DIMC processing block 80 will pass through an output buffer circuit, then through router 88 of DIMC circuit 70(1), through shared resource bus 72, through router 88 of DIMC circuit 70(3), to an input buffer circuit of DIMC processing block 80 for further processing. For processing scenarios that provide output of layer (n) as input to layer (n+1), DIMC circuit 70(3) and 70(5) will be communicatively interconnected through shared resource bus 72. The computational output of DIMC processing block 80 will pass through an output buffer circuit, then through router 88 of DIMC circuit 70(3), through shared resource bus 72, through router 88 of DIMC circuit 70(5), to an input buffer circuit of DIMC processing block 80 for further processing. This layer flow pipeline processing operation can also involve providing feature data to the DIMC processing blocks 80 of each of the DIMC circuits 70(1), 70(3), and 70(5). This broadcast of feature data occurs over shared resource bus 72, with the feature data distributed over bus 72 being routed by router 88 of each DIMC circuit 70(1), 70(3), and 70(5) to the input buffer circuits of the connected DIMC processing blocks 80 for application to processing circuit 123 (which also receives subarray data from subarray data output ports R0 through R P-1 The weight data read from memory 112 is received), where computational processing is performed to generate output computational data for storage in the output buffer circuit.

[0055] In Figure 5B In this example, block cluster 46 includes multiple IMC circuits 70, where IMC circuits 70(1), 70(2), 70(3), 70(4), 70(5), and 70(6) each utilize an AIMC processing block 80. Figure 5B A neural network graph time representation 1000 of one example of a tensor pipeline (implemented in the case where the fully unrolled tensor cannot be fully mapped in one block but is pipelined between multiple blocks; this is also managed by the compiler). The tensor pipeline includes: layer (n-1) with a first part of tensor operations using DIMC circuit 70(1) and its DIMC processing block 80, and a second part of tensor operations using DIMC circuit 70(2) and its DIMC processing block 80; layer (n) with a first part of tensor operations using DIMC circuit 70(3) and its DIMC processing block 80, and a second part of tensor operations using DIMC circuit 70(4) and its DIMC processing block 80; and layer (n+1) with a first part of tensor operations using DIMC circuit 70(5) and its DIMC processing block 80, and a second part of tensor operations using DIMC circuit 70(6) and its DIMC processing block 80. For the processing scenario where the output of layer (n-1) is provided as input to layer (n), the first part of tensor operations between DIMC circuits 70(1) and 70(3) and the second part of tensor operations between DIMC circuits 70(2) and 70(4) will be communicatively interconnected through shared resource bus 72. The computational output of DIMC processing block 80 for the first part of tensor operations will pass through an output buffer circuit, then through router 88 of DIMC circuit 70(1), through shared resource bus 72, through router 88 of DIMC circuit 70(3), to an input buffer circuit of DIMC processing block 80 for further processing. Likewise, the computational output of DIMC processing block 80 for the second part of tensor operations will pass through an output buffer circuit, then through router 88 of DIMC circuit 70(2), through shared resource bus 72, through router 88 of DIMC circuit 70(4), to an input buffer circuit of DIMC processing block 80 for further processing. For the processing scenario where the output of layer (n-1) is provided as input to layer (n), the first part of tensor operations between DIMC circuits 70(3) and 70(5) and the second part of tensor operations between DIMC circuits 70(4) and 70(6) will be communicatively interconnected through shared resource bus 72. The computational output of DIMC processing block 80 for the first part of tensor operations will pass through an output buffer circuit, then through router 88 of DIMC circuit 70(3), through shared resource bus 72, through router 88 of DIMC circuit 70(5), to an input buffer circuit of DIMC processing block 80 for further processing.Similarly, the computational output of the DIMC processing block 80 for the 2ndpart of the tensor operation will pass through the output buffer circuit, then through the router 88 of the DIMC circuit 70(4), through the shared resource bus 72, through the router 88 of the DIMC circuit 70(6), to the input buffer circuit of the DIMC processing block 80 for further processing. This tensor pipeline processing operation can also involve providing feature data to the DIMC processing block 80 of each of the DIMC circuits 70(1), 70(2), 70(3), 70(4), 70(5), and 70(6). This broadcast of feature data occurs through the shared resource bus 72, where the feature data for the 1stpart of the tensor operation is distributed through the bus 72 and routed by the router 88 of each of the DIMC circuits 70(1), 70(3), and 70(5) to the input buffer circuit of the connected DIMC processing block 80 for application to the processing circuit 123 (which is also fed through the subarray data output ports R0 through R P-1 receiving weight data read from the memory 112), where computational processing is performed to generate output computational data for storage in the output buffer circuit. Similarly, the feature data for the 2ndpart of the tensor operation is distributed through the bus 72 and routed by the router 88 of each of the DIMC circuits 70(2), 70(4), and 70(6) to the input buffer circuit of the connected DIMC processing block 80 for application to the processing circuit 123 (which is also fed through the subarray data output ports R0 through R P-1 receiving weight data read from the memory 112), where computational processing is performed to generate output computational data for storage in the output buffer circuit.

[0056] The above description provides a complete and informative description of exemplary embodiments of the application based on specific examples. However, various modifications and adaptations to the foregoing illustrative embodiments can be made by those skilled in the relevant art having the benefit of the foregoing description. However, all such and similar modifications will still fall within the scope of the application as defined in the appended claims.< / x> < / x> < / x> < / x> < / x> < / x>

Claims

1. A circuit comprising: The first in-memory computation IMC circuit includes a first IMC processing block coupled to a first interface circuit for data communication. The second IMC circuit includes a second IMC processing block coupled for data communication with the second interface circuit; A shared resource bus connects the first interface circuit to the second interface circuit to support data communication between the first IMC processing block and the second IMC processing block. The data communication conducted via the shared resource bus includes one or more of the following transmissions: Feature data for in-memory computation operations provided to the first IMC processing block and the second IMC processing block; Weight data for in-memory computation operations is provided to the first IMC processing block and the second IMC processing block. as well as Output computational data is generated by performing the in-memory computation operation by one of the first IMC processing block and the second IMC processing block, and is provided as input computational data to the other of the first IMC processing block and the second IMC processing block.

2. The circuit of claim 1, wherein each of the first IMC processing block and the second IMC processing block includes an input buffer circuit configured to receive feature data, weight data, or input calculation data of the data communication transmitted via the shared resource bus.

3. The circuit of claim 2, wherein the IMC processing block includes a processing circuit configured to receive the feature data from the input buffer circuit and weight data read from the memory of the IMC processing block, and the processing circuit is configured to generate output computation data to be stored in the output buffer circuit.

4. The circuit of claim 3, wherein the weight data from the input buffer circuit is written to the memory of the IMC processing block.

5. The circuit of claim 2, wherein the IMC processing block includes a processing circuit configured to receive the input calculation data from the input buffer circuit and configured to generate output calculation data to be stored in the output buffer circuit.

6. The circuit of claim 1, wherein each of the first IMC processing block and the second IMC processing block includes an output buffer circuit configured to receive weight data or output calculated data for data communication transmission via the shared resource bus.

7. The circuit of claim 6, wherein the IMC processing block includes processing circuitry configured to receive feature data and weight data read from the memory of the IMC processing block, the processing circuitry being configured to generate output computation data to be stored in the output buffer circuitry.

8. The circuit of claim 6, wherein the weight data read from the memory of the IMC processing block is output to be stored in the output buffer circuit.

9. The circuit of claim 1, wherein the data communication via the shared resource bus includes the transmission of feature data to the first IMC circuit and the second IMC circuit for in-memory computation operations performed by the first IMC processing block and the second IMC processing block.

10. The circuit of claim 1, wherein the data communication via the shared resource bus includes the transmission of weight data to the first IMC circuit and the second IMC circuit for in-memory computation operations performed by the first IMC processing block and the second IMC processing block.

11. The circuit of claim 1, wherein the data communication via the shared resource bus includes the transmission of computational data generated by the first IMC processing block from the first IMC circuit to the second IMC circuit for further processing by the second IMC processing block.

12. The circuit of claim 1, wherein the first IMC circuit includes decompressor logic configured to decompress compressed weight data for in-memory computation operations, and wherein the data communication via the shared resource bus is a transmission of the decompressed weight data from the first IMC circuit to the second IMC circuit.

13. The circuit of claim 1, wherein the first IMC processing block of the first IMC circuit includes shared computing logic, the shared computing logic being configured to receive data from the second IMC processing block via the shared resource bus and perform computing operations on the received data.

14. The circuit of claim 1, wherein the first IMC processing block of the first IMC circuit includes shared computing logic configured to generate output computing data, the output computing data being transmitted from the first IMC processing block to the second IMC processing block via the shared resource bus.

15. The circuit of claim 1, wherein the first IMC circuit and the second IMC circuit are layers in a layered pipelined processing operation.

16. The circuit of claim 1, wherein the first IMC circuit and the second IMC circuit are part of a layer in a tensor pipelined processing operation.

17. The circuit of claim 1, wherein each of the first IMC circuit and the second IMC circuit includes a router circuit coupled to the shared resource bus and configured for packet switching operations to route data communication between the interface and the IMC processing block.

18. The circuit of claim 1, wherein each of the first IMC circuit and the second IMC circuit includes a router circuit coupled to the shared resource bus and configured for circuit-switching operations to route data communication between the interface and the IMC processing block.

Images