Memory circuit and method for operating computing circuit in memory

Abstract

A memory circuit includes an array, a first buffer, a second buffer, a fetch circuit, and a controller. The fetch circuitry may be configured to write, during a first period, a first subset of first data elements to a first subset of a plurality of processing elements (PEs) arranged along a first row in an array, extract the first subset of first data elements from a first buffer, and temporarily store the first subset of first data elements. The controller may be configured to write a second subset of the first data elements to a second subset of the PEs arranged along the second row during a subsequent second period, controlling the fetch circuitry to selectively limit extraction of the second subset of the first data elements from the first buffer. The embodiment of the invention also relates to a method for operating the computing circuit in the memory.

Description

Technical Field

[0001] Embodiments of the present application relate to memory circuits and methods of operating computing circuits in memories. Background Art

[0002] Memory devices are components of electronic systems that store data in a way that allows for rapid access and modification. Traditionally, memory devices are designed to store binary information in the form of "0"s and "1"s in a large number of memory cells. Computing-in-memory (CIM) technology integrates processing power directly into the memory array, enabling faster data computation by reducing the distance data must travel between storage and processing elements. Multi-storage row CIMs are designed with a fixed data mapping. Summary of the invention

[0003] According to one aspect of an embodiment of the present application, a memory circuit is provided, comprising: an array, comprising a plurality of processing elements (PEs) arranged on a plurality of columns and a plurality of rows, wherein each PE is configured to perform a multiplication-accumulation (MAC) operation on a corresponding one of a plurality of first data elements and a corresponding one of a plurality of second data elements; a first buffer, configured to store the first data element and output the first data element to the array; a second buffer, configured to store the second data element and output the second data element to the array; an acquisition circuit, coupled between the first buffer and the array, wherein the acquisition circuit is configured to extract a first subset of the first data elements from the first buffer during a first cycle and temporarily store the first subset of the first data elements, the first cycle being used to write the first subset of the first data elements to a first subset of PEs arranged along a first row in a row; and a controller, coupled to the acquisition circuit and configured to control the acquisition circuit to selectively restrict the extraction of a second subset of the first data elements from the first buffer during a subsequent second cycle, the second cycle being used to write the second subset of the first data elements to a second subset of PEs arranged along a second row in a row.

[0004] According to another aspect of an embodiment of the present application, a memory circuit is provided, comprising: an array, comprising a plurality of processing elements (PEs) arranged on a plurality of columns and a plurality of rows, wherein each PE is configured to perform a multiplication-accumulation (MAC) operation on a corresponding one of a plurality of first data elements and a corresponding one of a plurality of second data elements; an acquisition circuit, coupled to the array and configured to extract a first subset of first data elements from a corresponding buffer during a first cycle, the first cycle being used to write the first subset of first data elements to a first subset of PEs arranged along a first row in a row; and a controller, coupled to the acquisition circuit and configured to cause the acquisition circuit to restrict fetching of a second subset of first data elements from the buffer during a subsequent second cycle, the second cycle being used to write the second subset of first data elements to a second subset of PEs arranged along a second row in a row.

[0005] According to another aspect of an embodiment of the present application, a method for operating a compute-in-memory (CIM) circuit is provided, comprising: receiving a plurality of first data elements by a buffer; extracting a first subset of the first data elements by the buffer during a first cycle, and temporarily storing the first subset of the first data elements to write the first subset of the first data elements to a first subset of a plurality of processing elements (PEs); determining by the buffer whether to reuse at least one of the first subset of the first data elements based on a configuration of an artificial intelligence (AI) neural network associated with the plurality of first data elements; and selectively restricting extraction of a second subset of the first data elements by the buffer during a subsequent second cycle based on the determination of whether to reuse at least one of the first subset of the first data elements to write the second subset of the first data elements to a second subset of the PEs. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Various aspects of the present disclosure can be best understood from the following detailed description when read in conjunction with the accompanying drawings. It should be emphasized that, in accordance with standard practice in the industry, the various components are not drawn to scale and are only used for illustrative purposes. In fact, the size of the various components may be arbitrarily increased or reduced for clarity of discussion.

[0007] Figure 1 is a block diagram illustrating an example of a memory circuit according to some embodiments.

[0008] Figure 2 An example mapping scheme in a single-bank compute-in-memory (CIM) circuit is shown in accordance with some embodiments.

[0009] Figure 3 An example mapping scheme in a compute-in-multiple-storage-line-memory (CIM) circuit is shown in accordance with some embodiments.

[0010] Figure 4An example mapping scheme in a compute-in-multiple-storage-line-memory (CIM) circuit is shown in accordance with some embodiments.

[0011] Figure 5 An example mapping scheme in a compute-in-multiple-storage-line-memory (CIM) circuit is shown in accordance with some embodiments.

[0012] Figure 6 is a block diagram illustrating an example of a memory circuit according to some embodiments.

[0013] Figure 7 is a block diagram illustrating an example of a memory circuit according to some embodiments.

[0014] Figure 8 An example data mapping in a single-bank compute-in-memory (CIM) array is shown in accordance with some embodiments.

[0015] Fig. 9 An example data mapping in a compute-in-memory (CIM) array is shown in accordance with some embodiments.

[0016] Fig.10 An example data mapping in a compute-in-memory (CIM) array is shown in accordance with some embodiments.

[0017] Fig.11 An example data mapping in a compute-in-memory (CIM) array is shown in accordance with some embodiments.

[0018] Fig.12 A flow chart illustrating an example method of operating a compute-in-memory (CIM) circuit in accordance with some embodiments is shown. DETAILED DESCRIPTION

[0019] The following disclosure provides many different embodiments or examples for realizing different features of the present disclosure. Specific embodiments or examples of components and arrangements are described below to simplify the present disclosure. Of course, these are only examples and are not intended to be limiting. For example, in the following description, forming a first component above or on a second component may include an embodiment in which the first component and the second component are directly contacted, and may also include an embodiment in which an additional component may be formed between the first component and the second component so that the first component and the second component may not be in direct contact. In addition, the present disclosure may repeat reference numbers and / or letters in various examples. This repetition is for the purpose of simplicity and clarity, and does not itself indicate the relationship between the various embodiments and / or configurations discussed.

[0020] Additionally, for ease of description, spacing relation terms such as "below," "beneath," "lower," "above," "upper," etc. may be used herein to describe the relationship of one element or component to another element or component as shown in the figures. The spacing relation terms are intended to encompass different orientations of the device in use or in the process of operation in addition to the orientation shown in the figures. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spacing relation descriptors used herein may likewise be interpreted accordingly.

[0021] Deep learning uses neural networks to achieve artificial intelligence. These networks consist of many interconnected processing nodes that facilitate machine learning by analyzing example data. For example, a system designed to recognize objects might process thousands of images of objects, such as trucks, to discern and learn the visual patterns that correspond to objects in new images. Neural networks are typically structured in layers, with data flowing through the layers in a single forward pass. Each node in the network may be connected to multiple nodes in subsequent layers to which it sends data, as well as multiple nodes in previous layers from which it receives data.

[0022] In a neural network, nodes assign numerical values, called "weights," to their connections. Once activated, a node can multiply incoming data by this weight and sum the products of all its connections to produce a single numerical output. If the output is below a certain threshold, the node can prevent it from entering the next layer. Conversely, if the output exceeds the threshold, the node can transmit the sum to the connected nodes in the next layer. In a deep learning system, the neural network model is stored in memory, and the computational logic in the processor performs multiply-accumulate (MAC) calculations on the parameters (e.g., weights) stored in memory.

[0023] Traditional digital multipliers receive two operands in digital form and produce a digital result, taking care of the sign and scaling. Computation in memory (CIM) uses a different approach, storing the weight coefficients as analog values ​​in specially designed subarrays of transistor cells with rows and columns. The input digital data word enters the rows of the CIM array, triggering analog voltage multiples, which then sum analog currents along the columns. The analog-to-digital converter creates the final digital word output based on the summed analog values.

[0024] In a multi-row memory computing (CIM) architecture, data mapping can be a fixed process where weights are mapped to memory rows. During this process, the same data in the buffer can be accessed multiple times. This inflexible mapping strategy may lead to severe underutilization of memory rows, especially when processing neural network layers with different characteristics and requirements, resulting in inefficient memory usage and suboptimal performance.

[0025] Furthermore, the processing of input activations in convolutional neural networks (CNNs) often exacerbates energy consumption due to redundant data processing. During computation, input activations within a convolution window are read from a buffer, converted into a single vector, and then dispatched to a CIM macro for processing. However, this approach is inefficient; overlapping activations shared between different convolution windows are retrieved multiple times from the buffer. This repeated fetching process not only increases the computational load, but also results in additional energy consumption for buffer accesses, thereby reducing the overall energy efficiency of the memory circuit. Optimization methods that reduce redundant buffer accesses can significantly improve the energy profile of CIM operations.

[0026] The present disclosure provides various embodiments of memory circuits that address these problems (e.g., repeated acquisition). For example, the memory circuit disclosed herein includes an array, a first buffer, a second buffer, an acquisition circuit, and a controller. The acquisition circuit can be configured to extract a first data element from the first buffer and temporarily store a first subset of the first data element during a first cycle, and the first cycle is used to write the first subset of the first data element to a first subset of multiple processing elements (PEs) arranged along a first row in a row of the array. The controller can be configured to control the acquisition circuit to selectively limit the extraction of a second subset of the first data element from the first buffer during a subsequent second cycle to write the second subset of the first data element to a second subset of the PEs arranged along a second row in the row.

[0027] The present disclosure outlines a method for improving the energy efficiency and processing throughput of convolutional neural networks (CNNs), which are critical in artificial intelligence (AI) driven tasks such as computer vision applications. The present disclosure significantly improves computational performance by dynamically adjusting the data mapping and operation flow within the compute in multi-storage line memory (CIM) macro to accommodate the unique needs of various neural network layers. In addition, the present disclosure introduces an optimized method to reduce activation buffer accesses. This is achieved by exploiting the inherent data reuse in stride-based convolution operations (a common feature in CNNs), thereby minimizing the energy cost typically associated with repeated data retrieval. The advances presented in the present disclosure will set a new benchmark for energy and operational efficiency in the field of artificial intelligence, especially in applications involving convolutional layers.

[0028] The present disclosure introduces an adaptive data mapping protocol that intelligently reconfigures the allocation of data and operation sequences to align with the different characteristics of various neural network layers, thereby optimizing the utilization of compute in memory (CIM) resources. This flexibility is achieved by implementing customized peripheral circuits that are carefully designed to support this dynamic data mapping. These circuits are designed to facilitate shift-based data extraction, an approach that is consistent with the inherent data reuse pattern of stride-based convolution operations, which is a characteristic of many deep learning workloads.

[0029] In addition, the present disclosure includes a specific write sequence protocol that tailors memory interactions to the specific hardware configuration and unique requirements of the workload in question. This protocol is particularly beneficial when used in conjunction with the input static data stream paradigm in a multi-storage row CIM setting. By doing so, it greatly improves the reuse of activation data, which is critical to improving the efficiency and throughput of the system. This tailored approach not only reduces operational overhead, but also simplifies the computational process, ensuring that energy consumption is kept to a minimum while maximizing performance.

[0030] The present disclosure proposes an advanced adaptive data mapping technique that aims to significantly improve the utilization of storage rows, thereby improving the throughput and energy efficiency of different layers of neural networks. By intelligently adjusting the allocation of data to storage rows according to the unique requirements of each layer, the method ensures more efficient utilization of memory resources. As a complement to this, the present disclosure introduces a shift-based write operation that is optimized to reduce the number of times activation data must be accessed from the buffer. This approach not only reduces the computational load of the memory circuit, but also significantly improves energy efficiency. This operation is critical for high-performance computing tasks, where the balance between speed and power consumption is critical.

[0031] Figure 1 1 is a block diagram illustrating an example of a memory circuit 100 according to some embodiments. The memory circuit 100 may include an array 110, a first buffer 102, a second buffer 104, an acquisition circuit 106, a controller 108, and a third buffer 120. Figure 1 1 and 2. Some components are shown in FIG. 1 , but the embodiment is not limited thereto, and more or fewer components may be included in the memory circuit 100. In some embodiments, the memory circuit 100 may be used as a building block of an artificial intelligence (AI) accelerator.

[0032] In some embodiments, the first buffer 102 (e.g., input buffer) may include one or more memories (e.g., registers) that may receive and store inputs (e.g., input activation data) of an artificial intelligence (AI) neural network. For example, these inputs may be received as outputs from, for example, different memory circuits 100 (not shown), global buffers (not shown), or different devices. Inputs from the input buffer 102 may be provided to the acquisition circuit 106 and / or the PE array 110 for processing as described below. In some embodiments, the first buffer 102 may be configured to store a first data element (e.g., input activation or weight) and output the first data element to the array 110. In some embodiments, the first buffer 102 may be coupled to a memory array (not shown). The memory array may include a plurality of memory cells. The plurality of memory cells may store inputs or weights of a neural network. One or more peripheral circuits (not shown) may be located in one or more regions peripheral to or internal to the memory array. The memory cells and the peripheral circuits may be coupled via word lines and / or complementary bit lines BL and BLB, and data may be read and written from the memory bit cells via the complementary bit lines BL and BLB. Different combinations of voltages applied to the word lines and bit lines can define a read, erase, or write (program) operation to a memory bit cell. In some embodiments, the memory array architecture can incorporate various types of non-volatile or volatile memory technologies, including but not limited to static random access memory (SRAM), resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), and phase change random access memory (PCRAM).

[0033] In some embodiments, the second buffer 104 (e.g., a weight buffer) may include one or more memories (e.g., registers) that may receive and store weights of an artificial intelligence (AI) neural network. The weight buffer 104 may receive and store weights from, for example, different storage circuits 100 (not shown), a global buffer (not shown), or different devices. The weights from the weight buffer 104 may be provided to the acquisition circuit 106 and / or the PE array 110 for processing as described below. In some embodiments, the second buffer 104 may be configured to store a second data element (e.g., an input activation or weight) and output the second data element to the array 110. In some embodiments, the second buffer 104 may be coupled to a memory array (not shown). The memory array may include a plurality of memory cells. The plurality of memory cells may store inputs or weights of a neural network.

[0034] In some embodiments, the array 110 may include a plurality of processing elements (PEs) (e.g., 111, 112, 113, 114, 115, 116, 117, 118, and 119) arranged in a plurality of columns and a plurality of rows. Each of the PEs 111, 112, 113, 114, 115, 116, 117, 118, and 119 may include at least one of a register (or memory), a multiplexer (MUX), a multiplier, or an adder. A register may be a storage space of a memory unit for transmitting data for immediate use by a CPU (central processing unit) for data processing. A multiplexer (MUX) may be a network device that allows one or more analog or digital input signals to be transmitted together on the same communication transmission link. A multiplier may perform a multiplication operation of the output of a register and / or MUX. An adder may add the output of a multiplier to the output of a multiplexer. A PE may receive a data signal including an input, a weight, and a previous output. Each of PEs 111, 112, 113, 114, 115, 116, 117, 118, 119 may be configured to perform a multiplication and accumulation (MAC) operation on a corresponding one of a plurality of first data elements (e.g., input activations) and a corresponding one of a plurality of second data elements (e.g., weights). In some embodiments, array 110 may be a compute-in-memory (CIM) array. A first row may include PEs 111-113, a second row may include PEs 114-116, and a third row may include PEs 117-119. A first column may include PEs 111, 114, 117, a second column may include PEs 112, 115, 118, and a third column may include PEs 113, 116, 119. Although memory circuit 100 includes 9 PEs 111-119, embodiments are not limited thereto, and memory circuit 100 may include more or fewer PEs. PEs 111-119 may perform multiplication and accumulation (e.g., summation) operations (MAC operations) based on inputs and weights received and / or stored in first buffer 102 (e.g., input buffer), second buffer 104 (e.g., weight buffer), acquisition circuit 106, or inputs and weights received from different PEs (e.g., PEs 111-119). The output of a PE (e.g., PE 111) may be provided to one or more different PEs (e.g., PEs 112, 114) in the same CIM array 110 for multiplication and / or summation operations.

[0035] For example, PE 111 may receive a first input (e.g., a first data element) from first buffer 102 (via fetch circuit 106), receive a first weight (e.g., a second data element) from second buffer 104, and may perform a multiplication and / or summation operation based on the first input and the first weight. PE 114 may receive an output of PE 111, a second input from first buffer 102 (via fetch circuit 106), and a second weight from weight buffer 104, and may perform a multiplication and / or summation operation based on the output of PE 111, the second input, and the second weight. PE 117 may receive an output of PE 114, a third input from first buffer 102 (via fetch circuit 106), and a third weight from weight buffer 104, and may perform a multiplication and / or summation operation based on the output of PE 114, the third input, and the third weight. PE 112 may receive the output of PE 111, the fourth input from first buffer 102 (via acquisition circuit 106), the fourth weight from weight buffer 104, and perform a multiplication and / or summation operation based on the output, the fourth input, and the fourth weight of PE 111. PE 115 may receive the output of PEs 112 and 114, the fifth input from first buffer 102 (via acquisition circuit 106), and the fifth weight from weight buffer 104, and perform a multiplication and / or summation operation based on the output, the fifth input, and the fifth weight of PEs 112 and 114. PE 118 may receive the output of PEs 115 and 117, the sixth input from first buffer 102 (via acquisition circuit 106), and the sixth weight from weight buffer 104, and may perform a multiplication and / or summation operation based on the output, the sixth input, and the sixth weight of PEs 115 and 115. PE 113 may receive the output of PE 112, the seventh input from the first buffer 102 (via acquisition circuit 106), the seventh weight from the weight buffer 104, and perform a multiplication and / or summation operation based on the output of PE 112, the seventh input, and the seventh weight. PE 116 may receive the output of PE 113 and 115, the eighth input from the first buffer 102 (via acquisition circuit 106), and the eighth weight from the weight buffer 104, and may perform a multiplication and / or summation operation based on the output of PE 113 and 115, the eighth input, and the eighth weight. PE 119 may receive the output of PE 116 and 118, the ninth input from the first buffer 102 (via acquisition circuit 106), and the ninth weight from the weight buffer 104, and may perform a multiplication and / or summation operation based on the output of PE 116 and 118, the ninth input, and the ninth weight. For the bottom row of PEs of the PE array (eg, PEs 117 - 119 ), the outputs may also be provided to one or more accumulators (not shown) or a third buffer 120 .According to an embodiment, the first to ninth inputs and / or the first to ninth weights and / or the outputs of PEs 111-119 may be forwarded to some or all of PEs 111-119. These operations may be performed in parallel so that the outputs of PEs 111-119 are provided in each cycle. In some embodiments, CIM array 110 may be a multi-storage row compute-in-memory (CIM).

[0036] In some embodiments, the CIM array 110 may include one or more accumulators. The accumulator may sum partial sum values ​​of the results of the PEs 111-119. For example, the accumulator may sum three outputs provided by the PE 117 for a set of inputs provided by the input buffer 102. Each accumulator may include one or more registers for storing the outputs of the PEs 117-119, and a counter for tracking how many times the sum operation has been performed before the sum is output to the output buffer 120. For example, the accumulator may perform three sum operations on the output of the PE 117 (e.g., considering the outputs from the three PEs 111, 114, 117) before the accumulator provides the sum to the output buffer 120. Once the accumulators in the CIM array complete summing all partial values, the output may be provided to the output buffer 120. In some embodiments, the CIM array 110 may include a digital adder circuit (or adder tree). The adder tree may sum MAC elements to provide a final MAC result through one output channel.

[0037] In some embodiments, the acquisition circuit 106 may include a plurality of multiplexers and a plurality of registers. Each of the plurality of multiplexers controlled by the controller may include a first input, a second input, and an output coupled to a corresponding one of the registers, the register also being coupled to a corresponding one of the columns. In some embodiments, the multiplexers and the corresponding registers may be connected to each other in a shift-based manner. A detailed description of the acquisition circuit 106 may be found in Figure 5-Figure 6 . In some embodiments, the acquisition circuit 106 can be coupled between the first buffer 102 and the array 110. The acquisition circuit 106 can be configured to acquire and store a first subset of the first data elements from the first buffer 102. For example, during a first cycle of writing the first subset of the first data elements to the first subset of PEs arranged along the first row in the row, the acquisition circuit 106 can extract the first data elements from the first buffer 102 and temporarily store the first data elements. In some embodiments, the acquisition circuit 106 can be a shift-based acquisition module. Considering the regular pattern of overlapping activations of the CNN, the shift-based acquisition module utilizes a shift register to reuse overlapping data, thereby avoiding repeated accesses to the activation buffer.

[0038] In some embodiments, the third buffer 120 (e.g., output buffer) may include one or more memories (e.g., registers) that may receive and store the outputs (e.g., partial sums) of the artificial intelligence (AI) neural network. The third buffer 120 may store the outputs of the CIM array 110 and provide these outputs as inputs to different memory circuits (e.g., processing cores) or to a global output buffer (not shown) for further processing and / or analysis and / or prediction. In some embodiments, the third buffer 120 may be a customized accumulation circuit to reduce accumulation buffer accesses in a particular mode.

[0039] In some embodiments, the controller 108 may include a hardware component that can control coupled components (e.g., the first buffer 102, the second buffer 104, the third buffer 120, the acquisition circuit 106, and the CIM array 110). The controller 108 may be coupled to the acquisition circuit 106 and configured to control the acquisition circuit 106 to selectively limit the reading of the first data element from the first buffer 102. For example, during a subsequent second cycle in which a second subset of the first data element is written into a second subset of PEs arranged along a second row, the controller 108 may extract a portion of the second subset of the first data element from the first buffer 102. In some embodiments, the controller 108 may be configured to control signals that cater to various data mapping schemes within a memory architecture (e.g., the memory circuit 100). These schemes are critical to optimizing the spatial allocation of data. One strategy is to distribute the weights of the same filter across multiple storage rows, thereby allowing parallel processing and increasing data access speed. This mapping strategy may result in inefficiencies, especially for layers with relatively few weights per filter. Another approach is to map the weights of different filters onto multiple storage rows, which helps to calculate various filter outputs simultaneously. In addition, input activations are also mapped to multiple storage rows, an approach that ensures rapid retrieval and processing of data required for neural network computations. By implementing these flexible mapping strategies, the system is designed to improve storage row utilization, increase throughput, and improve energy efficiency. In some embodiments, before acquiring any new data from the first buffer 102, the controller 108 (e.g., a write scheduler) can determine whether to reuse old data in the acquisition circuit 106 (e.g., a shift register) based on recognizing that the old data will be overlapped by the filter in one or more subsequent cycles in the CNN. Such reused data can be configured to generate a new output (or PS) in the next row.

[0040] In the context of AI workloads, neural network layers often exhibit large variations in their structural dimensions, particularly in terms of the number of output channels and the number of weights per filter. This diversity is evident when examining models such as ResNet-50 and MobileNet-v2, whose layers vary widely in these dimensions, as shown in Table 1. In compute-in-multiple-memory (CIM) macros, a one-size-fits-all approach to data mapping fails to accommodate this diversity, resulting in suboptimal utilization of memory resources. For example, considering a 64x32 CIM array with 16 rows per MAC operation, mapping the weights of the same filter exclusively to these memory rows can result in a mapping dimension of 1024x32. Under this scheme, the memory rows may be underutilized, particularly in the case of MobileNet-v2, where the maximum weight per filter does not even reach the 1024 memory rows available in each column.

[0041] Table 1

[0042]

[0043] This disclosure proposes a solution by introducing three different mapping schemes ( Figure 3-Figure 5 ), enabling multiple rows of CIM macros to dynamically adapt to the requirements of different layers. These schemes allow for flexible use of storage rows, which can be configured to map weights from the same or different filters, and input activations corresponding to different convolution windows. By doing so, the present disclosure ensures more efficient use of storage rows, thereby improving the overall utilization and performance of CIM macros in various neural network architectures.

[0044] Figure 2 1 shows an example mapping scheme in a single-storage row computing-in-memory (CIM) circuit according to some embodiments. In some embodiments, CIM array 110 may be a single-storage row computing-in-memory (CIM) circuit. Figure 2 In the example mapping scheme in , there are 3 input channels and 4 output channels, with a total of 4 filters used (e.g. 210, 215, 220, 225). Each filter is defined by a 2x2x3 weight structure, meaning that each filter kernel operates on a 2x2 region of 3 input channels. The stride of 2 specifies the stride of the filter when convolving in the input space. This setup is typical in convolutional neural networks, where multiple filters are used to extract different features from the input data, each filter producing its own unique output channel. The described architecture utilizes multiple filters to transform the 3D input data into a 4-channel output feature map.

[0045] In some embodiments, CIM array 110 may be a 4x2 CIM array. Figure 2In , the data flow operates under the weight-fixed data flow paradigm. This configuration allows generating 9 activation vectors from each input channel after applying the core to a 2x2 region on these channels. Within this framework, the task of the accumulator buffer 202 is to calculate the initial partial sum (PS).

[0046] In the initial stage (e.g., step (a)), K1 to K4 of the first channel of the first filter 210 are assigned to the first column of the CIM array 110, and K5 to K8 of the first channel of the second filter 215 are assigned to the second column of the CIM array. The CIM array 110 may perform multiplication-accumulation (e.g., summation) operations based on the mapped inputs (e.g., 9 activation vectors of the first channel) and weights (e.g., K1-K8 of the first channel). After these operations, the accumulator buffer 202 may be responsible for calculating and temporarily saving the first partial sum (PS), which is then stored in the CIM array 110 for further processing. In step (b), K1 to K4 of the second channel of the first filter 210 are assigned to the first column of the CIM array 110, and K5 to K8 of the second channel of the second filter 215 are assigned to the second column of the CIM array 110. The CIM array 110 may perform multiplication-accumulation (e.g., summation) operations based on the mapped inputs (e.g., 9 activation vectors) and weights (e.g., K1-K8). After these operations, the accumulator buffer 202 may be responsible for calculating and temporarily saving the second partial sum (PS), which is then stored in the CIM array 110 for further processing.

[0047] In step (c), K1 to K4 of the third channel of the first filter 210 are assigned to the first column of the CIM array 110, and K5 to K8 of the third channel of the second filter 215 are assigned to the second column of the CIM array 110. The CIM array 110 may perform multiplication-accumulation (e.g., summation) operations based on the mapped inputs (e.g., 9 activation vectors of the third channel) and weights (e.g., K1-K8 of the first channel). After these operations, the accumulator buffer 202 may be responsible for calculating and temporarily saving the third partial sum (PS), which is then stored in the CIM array 110 for further processing. The accumulator buffer may frequently update the partial sum, as shown by the three rounds of access shown in this scenario. In step (d), K9 to K12 of the first channel of the third filter 220 are assigned to the first column of the CIM array 110, and K13 to K16 of the first channel of the fourth filter 225 are assigned to the second column of the CIM array 110. The CIM array 110 may perform multiplication-accumulation (e.g., summation) operations based on the mapped inputs (e.g., the 9 activation vectors for the first channel) and weights (e.g., K1-K4 for the first channel of the first filter 210). After these operations, the accumulator buffer 202 may be responsible for calculating and temporarily saving the first partial sum (PS) of the new output, which is then stored in the CIM array 110 for further processing. In some embodiments, when the CIM array 110 is updated with weights for different filters, it may be necessary to reload the activations from the buffer multiple times.

[0048] In some embodiments, in a weighted fixed data flow, weights are pre-populated and stored in each PE 110 before computation begins, so that all PEs for a given filter are distributed along PE columns. Then, an input feature map (e.g., IFMAP) can flow in through the left edge of the CIM array 110, while weights are fixed in each PE, and each PE generates one partial sum per cycle. The generated partial sums can then be reduced in parallel along the rows and along each column to generate one output feature map (e.g., OFMAP) pixel per column. The input fixed data flow is similar to the weighted fixed data flow, except for the mapping order. Instead of pre-populating the CIM array 110 with weights, the expanded IFMAP is stored in each PE. Then, weights flow in from the edge, and each PE generates one partial sum per cycle. The generated partial sums are also reduced in parallel across the rows and along each column to generate one output feature map pixel per column. The output fixed data flow refers to a mapping where each PE performs all computations on one OFMAP, while weights and IFMAPs are fed from the edge of the array and distributed to the PEs using PE-to-PE interconnects. The partial sums are generated and reduced within each PE. Once all PEs in the array have completed the generation of the OFMAP, the result is data transferred out of the array via the PE-to-PE interconnect.

[0049] Figure 3 An example mapping scheme in a compute-in-multiple-storage-line-memory (CIM) circuit is shown in accordance with some embodiments. Figure 3 An example mapping scheme for multiple storage rows of weights for the same filter is shown. In some embodiments, CIM array 110 may be a multi-storage row compute-in-memory (CIM) circuit. In some embodiments, CIM array 110 may be a 4x2 CIM array with multiple storage rows (e.g., two rows per MAC unit). Figure 3 In the example mapping scheme in , there are 3 input channels and 4 output channels, with a total of 4 filters used. This configuration allows multiple interleaved activation vectors to be generated from the input channels after applying the core to the 2x2 region on these channels. Within this framework, the task of the accumulator buffer 202 is to calculate the initial partial sum (PS). In some embodiments, the accumulator buffer 202 may include an adder, a register, a multiplexer (MUX), and a memory unit (e.g., SRAM).

[0050] In this case, the interleaved input activation vector 310 may correspond to weights stored in two storage rows 320 of the CIM array 110. The two storage rows 320 of the CIM array 110 may store weights belonging to the same filter (e.g., the first filter 210, the second filter 215, the third filter 220, or the fourth filter 225). Each column of the CIM array 110 includes two storage rows. In step (a), K1 to K4 of the first channel of the first filter 210 are assigned to the first column of the first row of the CIM array 110. K1 to K4 of the second channel of the first filter 210 are assigned to the first column of the second row of the CIM array 110. K5 to K8 of the first channel of the second filter 215 are assigned to the second column of the first row of the CIM array 110. K5 to K8 of the second channel of the second filter 215 are assigned to the second column of the second row of the CIM array 110. The CIM array 110 may perform a multiplication-accumulation (e.g., summation) operation based on the mapped inputs (e.g., A1, A2, A7, A8 of the first channel) and weights (e.g., K1, K2, K3, and K4 of the first channel). In step (a), the activation row 321 is the first storage row in the computation-in-memory (CIM) circuit. After these operations, the accumulator buffer 202 may be responsible for calculating and temporarily saving the first partial sum (PS) (e.g., A1×K1+A2×K2+A7×K3+A8×K4), which is then stored in a register for further processing.

[0051] In step (b), the CIM array 110 may perform a multiply-accumulate (e.g., sum) operation based on the mapped inputs (e.g., A1, A2, A7, A8 of the second channel) and weights (e.g., K1, K2, K3, and K4 of the first channel). In step (b), the activated row 322 is the second storage row in the compute-in-memory (CIM) circuit. After these operations, the accumulator buffer 202 may be responsible for calculating and temporarily saving the second partial sum (PS) (e.g., A1×K1+A2×K2+A7×K3+A8×K4), which is then stored in the CIM array 110 for further processing. This mapping scheme effectively expands the row dimension of the CIM array 110 to improve partial sum reuse. Increasing the number of rows of the CIM array 110 (e.g., per MAC unit) may further reduce the need for accumulator buffer accesses, although this may come with a trade-off of increased control overhead and increased array area.

[0052] Traditionally, accumulation is performed by combining a new partial sum (PS) with a previous PS retrieved from an SRAM buffer. However, the present disclosure introduces a different approach. SEL signal 302 is used to dynamically select a data source for accumulation. Selection can be made between PS 304 read from SRAM and PS stored in a register from a previous cycle. By providing flexibility in data source selection, the present disclosure improves the efficiency and adaptability of the accumulation process, meets different computing requirements and optimizes resource utilization.

[0053] Figure 4 An example mapping scheme in a compute-in-multiple-storage-line-memory (CIM) circuit is shown in accordance with some embodiments. Figure 4 An example mapping scheme for multiple storage rows for different filters is shown. In some embodiments, CIM array 110 may be a multi-storage row compute-in-memory (CIM) circuit. In some embodiments, CIM array 110 may be a 4x2 CIM array with multiple storage rows (e.g., two rows per MAC unit). Figure 4 In the example mapping scheme in , there are 3 input channels and 4 output channels, and a total of 4 filters are used. In this configuration, each input activation vector 410 can remain activated for two consecutive cycles to multiply with two sets of filters stored in two storage rows. In this framework, the task of the accumulator buffer 202 is to calculate the initial partial sum (PS). In some embodiments, the accumulator buffer 202 may include an adder, a register, a multiplexer (MUX), and a memory unit (e.g., SRAM).

[0054] In this case, the input activation vector 410 may correspond to weights stored in two storage rows 420 of the CIM array 110. The two storage rows 420 of the CIM array 110 may store weights belonging to different filters (e.g., the first filter 210, the second filter 215, the third filter 220, or the fourth filter 225). In step (a), K1 to K4 of the first channel of the first filter 210 are assigned to the first column of the first row of the CIM array 110. K9 to K12 of the second channel of the third filter 220 are assigned to the first column of the second row of the CIM array 110. K5 to K8 of the first channel of the second filter 215 are assigned to the second column of the first row of the CIM array 110. K5 to K8 of the second channel of the second filter 215 are assigned to the second column of the second row of the CIM array 110. The CIM array 110 may perform a multiplication-accumulation (e.g., summation) operation based on the mapped inputs (e.g., A1, A2, A7, A8) and the weights (e.g., K1, K2, K3, and K4). In step (a), the activated row 421 is the first storage row in the computation in memory (CIM) circuit. After these operations, the accumulator buffer 202 can be responsible for calculating and temporarily saving the first partial sum (PS) (e.g., A1×K1+A2×K2+A7×K3+A8×K4), and then storing it in a register for further processing.

[0055] In step (b), the CIM array 110 may perform a multiplication-accumulation (e.g., summation) operation based on the mapped inputs (e.g., A1, A2, A7, A8) and weights (e.g., K9, K10, K11, and K12). In this configuration, each input activation vector may remain activated for two consecutive cycles to be multiplied with two sets of filters stored in two storage rows. In step (b), the activation row 422 is the second storage row in the computation-in-memory (CIM) circuit. After these operations, the accumulator buffer 202 may be responsible for calculating and temporarily saving the second partial sum (PS) (e.g., A1×K9+A2×K10+A7×K11+A8×K12), which is then stored in the CIM array 110 for further processing. This mapping scheme effectively expands the column dimension of the CIM array 110 to enhance activation reuse. Implementing more rows in the CIM array 110 (e.g., per MAC unit) can further reduce the number of accesses to the activation buffer, although at the expense of requiring larger accumulator buffer capacity, in addition to the control and array area overhead. In some embodiments, the activation of the two accumulator buffer pools 202 can be alternating. By reusing activation vectors, the number of accesses to the activation buffer can be halved.

[0056] Figure 5 An example mapping scheme in a compute-in-multiple-storage-line-memory (CIM) circuit is shown in accordance with some embodiments. Figure 5 An example mapping scheme for multiple storage rows for input activation is shown. In some embodiments, CIM array 110 may be a multiple storage row compute-in-memory (CIM) circuit. In some embodiments, CIM array 110 may be a 4x2 CIM array with multiple storage rows (e.g., two rows per MAC unit). Figure 5 In the example mapping scheme in , there are 3 input channels and 4 output channels, and a total of 4 filters are used. In this configuration, each expanded weight vector can remain activated for two consecutive cycles to be multiplied with two sets of activations stored in two storage rows in the CIM array 110. In this framework, the task of the accumulator buffer 202 is to calculate the initial partial sum (PS). In some embodiments, the accumulator buffer 202 may include an adder, a register, a multiplexer (MUX), and a memory unit (e.g., SRAM).

[0057] In this case, the weights 510 may correspond to input activation vectors stored in two storage rows 521, 522 of the CIM array 110. The two storage rows 521, 522 of the CIM array 110 may store input activations (e.g., input fixations) associated with two convolution windows. In step (a), A1, A2, A7, A8 of the input activations 250 are assigned to the first column of the first row of the CIM array 110. A5, A6, A11, A12 of the input activations 250 are assigned to the first column of the second row of the CIM array 110. A3, A4, A9, A10 of the input activations 250 are assigned to the second column of the first row of the CIM array 110. A13, A14, A19, A20 of the input activations 250 are assigned to the second column of the second row of the CIM array 110. The CIM array 110 may perform multiplication-accumulation (e.g., summation) operations based on the mapped weights (e.g., K1, K2, K3, K4) and inputs (e.g., A1, A2, A7, and A8). In step (a), the activated row 521 is the first storage row in the computation-in-memory (CIM) circuit. After these operations, the accumulator buffer 202 may be responsible for calculating and temporarily saving the first partial sum (PS) (e.g., K1×A1+K2×A2+K3×A7+K4×A8), which is then stored in a register or accumulator 202 for further processing.

[0058] In step (b), the CIM array 110 may perform a multiplication-accumulation (e.g., summation) operation based on the mapped inputs (e.g., K1, K2, K3, K4) and weights (e.g., A5, A6, A11, and A12). In this configuration, each expanded weight vector may remain activated for two consecutive cycles to be multiplied with two sets of activations stored in two storage rows in the CIM array 110. In step (b), the activation row 522 is the second storage row in the computation-in-memory (CIM) circuit. After these operations, the accumulator buffer 202 may be responsible for calculating and temporarily saving the second partial sum (PS) (e.g., A1×K9+A2×K10+A7×K11+A8×K12), which is then stored in the CIM array 110 for further processing. This mapping scheme effectively expands the column dimension of the CIM array 110 to enhance activation reuse. Implementing more rows in the CIM array 110 (e.g., per MAC unit) can further reduce the number of activation buffer accesses, albeit at the expense of requiring larger accumulator buffer capacity, in addition to the control and array area overhead. In some embodiments, the activation of the two sets of accumulator buffers 202 can be alternating. By combining this mapping with a shift-based acquisition module, the reuse of activation data can be further exploited by applying specific write sequences (e.g., Figure 8-Figure 10 ).

[0059] Figure 6 is a block diagram illustrating an example of a memory circuit 100 according to some embodiments. Figure 7 1 is a block diagram illustrating an example of a memory circuit 100 according to some embodiments. The memory circuit may include a CIM array 110, an activation buffer 102, an acquisition circuit 106, and a write scheduler 108. Figure 6 and Figure 7 1 and 2. Some components are shown in FIG. 1 , but the embodiment is not limited thereto, and more or fewer components may be included in the memory circuit 100. In some embodiments, the memory circuit 100 may be used as a building block of an artificial intelligence (AI) accelerator. Figure 6 and Figure 7 The memory circuit 100 is substantially similar to Figure 1 For the sake of brevity, this article omits the specific operations of similar components that have been discussed in detail in the above paragraphs, unless they are required to be introduced in conjunction with Figure 6 and Figure 7 To simplify the demonstration, assume that 4 data points are written to the Nx4 CIM array.

[0060] In some embodiments, the acquisition circuit 106 may include a plurality of multiplexers 602, 604, 606, 608 (MUX) and a plurality of registers 612, 614, 616, 618. The MUX 602, 604, 606, 608 may select a data source between the activation buffer 102 and an adjacent register. The register may receive the output of the MUX. The output of the register may be provided to the CIM array. The acquisition circuit 106 may output data signals to D[0], D[1], D[2], and D[3] through the registers 612, 614, 616, 618. In some embodiments, the acquisition circuit 106 may be a shift-based acquisition module. Considering the regular pattern of overlapping activations of CNN, the shift-based acquisition module utilizes shift registers to reuse overlapping data, thereby avoiding repeated access to the activation buffer. In some embodiments, the acquisition circuit 106 may include fewer multiplexers (MUX) 604, 606, 608 and fewer registers 614, 616, 618. The acquisition circuit 106 may output data signals to D[1], D[2], and D[3] through registers 614 , 616 , 618 .

[0061] In some embodiments, the write scheduler 108 may include a hardware component that can control coupled components (e.g., the activation buffer 102, the acquisition circuit 106, and the CIM array 110). The write scheduler 108 may be coupled to the acquisition circuit 106 and configured to control the acquisition circuit 106 to selectively limit the acquisition of the first data element from the activation buffer 102. For example, during a subsequent second cycle in which a second subset of the first data element is written into a second subset of PEs arranged along a second row, the write scheduler 108 may extract a portion of the second subset of the first data element from the activation buffer 102. In some embodiments, the write scheduler 108 may be configured to control signals that cater to various data mapping schemes within a memory architecture (e.g., the memory circuit 100). The write scheduler may include a read mask and a write mask. The read mask may be used to filter out unnecessary SRAM accesses, for example, to only acquire new data from the SRAM. The write scheduler 108 may coordinate the process by providing a read / write mask signal and selecting a signal for the MUX. The write mask may select a target row to be written. By properly configuring the order of row updates, the reuse of shifted data can be maximized.

[0062] Figure 8 An example data mapping in a single storage row compute-in-memory (CIM) array without shift-based writes according to some embodiments is shown. In some embodiments, the CIM array 110 may be a 4x4 CIM array with a single storage row (e.g., one row per MAC unit). Figure 8In the example data mapping in FIG. 1 , a 5x5x1 input feature map 810 (e.g., IFMAP), a 2x2x1 filter 820, and a 4x4x1 output feature map 830 (e.g., OFMAP) are used. Before the calculation begins, the input activations (e.g., IFMAP) are pre-filled and stored in each PE 110 of the CIM array 110 so that all PEs for a given filter are configured along the PE columns. Then, weights flow in from the edges, and each PE generates a partial sum per cycle. The generated partial sums can then be reduced in parallel along each column across the rows to generate one output feature map (e.g., OFMAP) pixel per column.

[0063] In some embodiments, the acquisition circuit 106 may acquire 4 data from the activation buffer 102 (e.g., IFMAP) during each cycle. For example, during a first cycle (e.g., cycle #1), the acquisition circuit 106 may acquire A1, A2, A3, A4 from the IFMAP 810. The acquisition circuit 106 may write A1, A2, A3, A4 to a first storage row in the CIM array 110. During a second cycle (e.g., cycle #2), the acquisition circuit 106 may acquire A2, A3, A4, A5 from the IFMAP 810. The acquisition circuit 106 may write A2, A3, A4, A5 to a second storage row in the CIM array 110. During a third cycle (e.g., cycle #3), the acquisition circuit 106 may acquire A6, A7, A8, A9 from the IFMAP 810. The acquisition circuit 106 may write A6, A7, A8, A9 to a third storage row in the CIM array 110. During a fourth cycle (e.g., cycle #4), the acquisition circuit 106 may acquire A7, A8, A9, A10 from the IFMAP 810. The acquisition circuit 106 may write A7, A8, A9, A10 to a fourth storage row in the CIM array 110. In cycle #4, each column of the CIM array may correspond to a CONV window (e.g., a 2x2 CONV window) in the 5x5x1 input feature map 810 (e.g., IFMAP). The CIM array 110 may perform a multiplication-accumulation (e.g., summation) operation based on the inputs (e.g., A1, A2, A6, A7) and weights (e.g., K1, K2, K3, K4). The accumulator buffer 202 in the CIM array 110 may calculate and generate a partial sum (PS) (e.g., A1×K1+A2×K2+A6×K3+A7×K4), which is then stored in the output feature map (e.g., O1 in the OFMAP 830).

[0064] After the calculation is completed with the previous set of activations, new data is fetched from the activation buffer 102 to overwrite the previous CIM array 110. For example, during the Nth cycle (e.g., cycle #N), the fetch circuit 106 may fetch A6, A7, A8, A9 from the IFMAP 810. The fetch circuit 106 may write A6, A7, A8, A9 to the first storage row in the CIM array 110. During the N+1th cycle (e.g., cycle #N+1), the fetch circuit 106 may fetch A7, A8, A9, A10 from the IFMAP 810. The fetch circuit 106 may write A7, A8, A9, A10 to the second storage row in the CIM array 110. During the N+2th cycle (e.g., cycle #N+2), the fetch circuit 106 may fetch A11, A12, A13, A14 from the IFMAP 810. The acquisition circuit 106 may write A11, A12, A13, A14 to a third storage row in the CIM array 110. During the N+3th cycle (e.g., cycle #N+3), the acquisition circuit 106 may acquire A12, A13, A14, A15 from the IFMAP 810. The acquisition circuit 106 may write A12, A13, A14, A15 to a fourth storage row in the CIM array 110. In cycle #N+3, each column of the CIM array may correspond to a CONV window (e.g., a 2x2 CONV window) in the 5x5x1 input feature map 810 (e.g., an IFMAP). The CIM array 110 may perform a multiplication-accumulation (e.g., summation) operation based on the inputs (e.g., A6, A7, A11, A12) and the weights (e.g., K1, K2, K3, K4). The accumulator buffer 202 in the CIM array 110 may calculate and generate a partial sum (PS) (e.g., A6×K1+A7×K2+A11×K3+A12×K4) and then store it in the output feature map (e.g., O5 in OFMAP 830). There are a total of 32 data accesses from the activation buffer 102.

[0065] Fig. 9 1 shows an example data mapping in a multi-storage row compute-in-memory (CIM) array with shift-based writes according to some embodiments. In some embodiments, the CIM array 110 may be a 4x4 CIM array with two storage rows (e.g., two rows per MAC unit). Fig. 9In the example data mapping in , a 5x5x1 input feature map 810 (e.g., IFMAP), a 2x2x1 filter 820 (stride size = 1), and a 4x4x1 output feature map 830 (e.g., OFMAP) are used. Before the calculation begins, the input activations (e.g., IFMAP) are pre-filled and stored in each PE 110 of the CIM array 110 so that all PEs for a given filter are distributed along the PE columns. Then, weights flow in from the edges, and each PE generates a partial sum per cycle. The generated partial sums can then be reduced in parallel along each column across the rows to generate one output feature map (e.g., OFMAP) pixel per column.

[0066] In some embodiments, the acquisition circuit 106 may acquire 4 data from the activation buffer 102 (e.g., IFMAP) during each cycle. Given an array configuration and a specific workload (e.g., filter size, stride size, activation buffer size, weight buffer size, core size, or size of multiple PEs), the data acquisition pattern may be predetermined. Each column of the CIM array 110 includes two storage rows (e.g., 902 and 904). For example, during the first cycle (e.g., cycle #1), the acquisition circuit 106 may extract A1, A2, A3, A4 from the IFMAP 810. The acquisition circuit 106 may write A1, A2, A3, A4 to the first storage row 902 (e.g., storage row #1.1 (MAC row number, storage row #)) in the CIM array 110. The CIM array 110 may include a multiplier 903 (not shown in the following cycles for simplicity).

[0067] In some embodiments, before acquiring any new data from the first buffer 102, the controller 108 (e.g., a write scheduler) may determine whether to reuse old data in the acquisition circuit 106 (e.g., a shift register) based on recognizing that the old data will be overlapped by the filter in one or more subsequent cycles in the CNN. For example, during the second cycle (e.g., cycle #2), the acquisition circuit 106 may reuse A2, A3, A4, and acquire A5 from the IFMAP 810. The acquisition circuit 106 may write A2, A3, A4, A5 to the second storage row 904 (e.g., storage row #1.2 (MAC row number, storage row #)) in the CIM array 110. During the third cycle (e.g., cycle #3), the acquisition circuit 106 may read A6, A7, A8, A9 from the IFMAP 810. The acquisition circuit 106 may write A6, A7, A8, A9 to the first storage row 906 (e.g., storage row #2.1 (MAC row number, storage row #)) in the CIM array 110.

[0068] During a fourth cycle (e.g., cycle #4), the acquisition circuit 106 may reuse A6, A7, A8, A9. The acquisition circuit 106 may write A6, A7, A8, A9 to a second storage row 904 (e.g., storage row #1.2 (MAC row number, storage row #)) in the CIM array 110. Such reused data may be configured to generate a new output (or PS) in a next row (e.g., O5 in the OFMAP 830). During a fifth cycle (e.g., cycle #5), the acquisition circuit 106 may reuse A7, A8, A9 and read A10 from the IFMAP 810. The acquisition circuit 106 may write A7, A8, A9, A10 to a second storage row 908 (e.g., storage row #2.2 (MAC row number, storage row #)) in the CIM array 110.

[0069] During the sixth cycle (e.g., cycle #6), the acquisition circuit 106 may reuse A7, A8, A9, A10. The acquisition circuit 106 may write A7, A8, A9, A10 to the first storage row 914 (e.g., storage row #4.1 (MAC row number, storage row #)) in the CIM array 110. Such reused data may be configured to generate a new output (or PS) in the next row (e.g., O5 in the OFMAP 830). During the seventh cycle (e.g., cycle #7), the acquisition circuit 106 may acquire A11, A12, A13, A14 from the IFMAP 810. The acquisition circuit 106 may write A11, A12, A13, A14 to the second storage row 912 (e.g., storage row #3.2 (MAC row number, storage row #)) in the CIM array 110. During the eighth cycle (e.g., cycle #8), the acquisition circuit 106 may reuse A12, A13, A14, and acquire A15 from the IFMAP 810. The acquisition circuit 106 may write A12, A13, A14, A15 to the second storage row 916 (e.g., storage row #4.2 (MAC row number, storage row #)) in the CIM array 110. Fig. 9In the example of FIG. 830 , the same weights may be multiplied with the two sets of activations, respectively (by row switching), to generate two partial sums from each column of the CIM array 110. For example, the CIM array 110 may perform a multiplication-accumulation (e.g., summation) operation based on the inputs (e.g., A1, A2, A6, A7) and weights (e.g., K1, K2, K3, K4) in the first storage row. The accumulator buffer 202 in the CIM array 110 may calculate and generate a partial sum (PS) (e.g., A1×K1+A2×K2+A6×K3+A7×K4) and then store it in the output feature map (e.g., O1 in OFMAP 830). The CIM array 110 may perform a multiplication-accumulation (e.g., summation) operation based on the inputs (e.g., A6, A7, A11, A12) and weights (e.g., K1, K2, K3, K4) in the second storage row. The accumulator buffer 202 in the CIM array 110 can calculate and generate the partial sum (PS) (e.g., A6×K1+A7×K2+A11×K3+A12×K4) and then store it in the output feature map (e.g., O5 in OFMAP 830). There are a total of 15 accesses to the activation buffer 102, which is more than a 2x reduction from the previous 32 counts, and 9 accesses to the shift register 106.

[0070] Fig.10 1 shows an example data mapping in a multi-storage row compute-in-memory (CIM) array with shift-based writes according to some embodiments. In some embodiments, the CIM array 110 may be a 4x4 CIM array with three storage rows (e.g., three rows per MAC unit). Fig.10 In the example data mapping in , a 5x5x1 input feature map 810 (e.g., IFMAP), a 2x2x1 filter 820 (stride size = 1), and a 4x4x1 output feature map 830 (e.g., OFMAP) are used. Before the calculation begins, the input activations (e.g., IFMAP) are pre-filled and stored in each PE 110 of the CIM array 110 so that all PEs for a given filter are distributed along the PE columns. Then, weights flow in from the edges, and each PE generates a partial sum per cycle. The generated partial sums can then be reduced in parallel along each column across the rows to generate one output feature map (e.g., OFMAP) pixel per column.

[0071] In some embodiments, the acquisition circuit 106 may acquire 4 data from the activation buffer 102 (e.g., IFMAP) during each cycle. Given an array configuration and a specific workload (e.g., filter size, stride size, activation buffer size, weight buffer size, core size, or size of multiple PEs), the data acquisition pattern may be predetermined. Each column of the CIM array 110 includes three storage rows (e.g., 1002, 1004, and 1006). For example, during a first cycle (e.g., cycle #1), the acquisition circuit 106 may acquire A1, A2, A3, A4 from the IFMAP 810. The acquisition circuit 106 may write A1, A2, A3, A4 to the first storage row 1002 (e.g., storage row #1.1 (MAC row number, storage row #)) in the CIM array 110. The CIM array 110 may include a multiplier (not shown for simplicity).

[0072] In some embodiments, before acquiring any new data from the first buffer 102, the controller 108 (e.g., a write scheduler) may determine whether to reuse old data in the acquisition circuit 106 (e.g., a shift register) based on recognizing that the old data will be overlapped by the filter in one or more subsequent cycles in the CNN. For example, during the second cycle (e.g., cycle #2), the acquisition circuit 106 may reuse A2, A3, A4, and acquire A5 from the IFMAP 810. The acquisition circuit 106 may write A2, A3, A4, A5 to the first storage row 1008 (e.g., storage row #2.1 (MAC row number, storage row #)) in the CIM array 110. During the third cycle (e.g., cycle #3), the acquisition circuit 106 may read A6, A7, A8, A9 from the IFMAP 810. The acquisition circuit 106 may write A6, A7, A8, A9 to the first storage row 1014 (e.g., storage row #3.1 (MAC row number, storage row #)) in the CIM array 110.

[0073] During a fourth cycle (e.g., cycle #4), the acquisition circuit 106 may reuse A6, A7, A8, A9. The acquisition circuit 106 may write A6, A7, A8, A9 to a second storage row 1004 (e.g., storage row #1.2 (MAC row number, storage row #)) in the CIM array 110. Such reused data may be configured to generate a new output (or PS) in a next row (e.g., O5 in the OFMAP 830). During a fifth cycle (e.g., cycle #5), the acquisition circuit 106 may reuse A7, A8, A9 and acquire A10 from the IFMAP 810. The acquisition circuit 106 may write A7, A8, A9, A10 to a second storage row 1010 (e.g., storage row #2.2 (MAC row number, storage row #)) in the CIM array 110.

[0074] During the sixth cycle (e.g., cycle #6), the acquisition circuit 106 may reuse A7, A8, A9, A10. The acquisition circuit 106 may write A7, A8, A9, A10 to the first storage row 1020 (e.g., storage row #4.1 (MAC row number, storage row #)) in the CIM array 110. Such reused data may be configured to generate a new output (or PS) in the next row (e.g., O5 in the OFMAP 830). During the seventh cycle (e.g., cycle #7), the acquisition circuit 106 may acquire A11, A12, A13, A14 from the IFMAP 810. The acquisition circuit 106 may write A11, A12, A13, A14 to the second storage row 1016 (e.g., storage row #3.2 (MAC row number, storage row #)) in the CIM array 110. During the eighth cycle (e.g., cycle #8), the fetch circuit 106 may reuse A12, A13, A14, and fetch A15 from the IFMAP 810. The fetch circuit 106 may write A12, A13, A14, A15 to the third storage row 1006 (e.g., storage row #1.3 (MAC row number, storage row #)) in the CIM array 110.

[0075] During a ninth cycle (e.g., cycle #9), the acquisition circuit 106 may reuse A12, A13, A14, and extract A15 from the IFMAP 810. The acquisition circuit 106 may write A12, A13, A14, A15 to a third storage row 1012 (e.g., storage row #2.3 (MAC row number, storage row #)) in the CIM array 110. During a tenth cycle (e.g., cycle #10), the acquisition circuit 106 may reuse A12, A13, A14, A15. The acquisition circuit 106 may write A12, A13, A14, A15 to a second storage row 1022 (e.g., storage row #4.2 (MAC row number, storage row #)) in the CIM array 110. During an eleventh cycle (e.g., cycle #11), the acquisition circuit 106 may read A16, A17, A18, A19 from the IFMAP 810. The acquisition circuit 106 may write A16, A17, A18, A19 to the third storage row 1018 (e.g., storage row #3.3 (MAC row number, storage row #)) in the CIM array 110. During the twelfth cycle (e.g., cycle #12), the acquisition circuit 106 may reuse A17, A18, A19 and acquire A20 from the IFMAP 810. The acquisition circuit 106 may write A17, A18, A19, A20 to the third storage row 1024 (e.g., storage row #4.3 (MAC row number, storage row #)) in the CIM array 110. Fig.10, the same weights may be multiplied with the three sets of activations, respectively (by row switching), to generate three partial sums from each column of the CIM array 110. For example, the CIM array 110 may perform a multiplication-accumulation (e.g., summation) operation based on the inputs (e.g., A1, A2, A6, A7) and weights (e.g., K1, K2, K3, K4) in the first storage row. The accumulator buffer 202 in the CIM array 110 may calculate and generate a partial sum (PS) (e.g., A1×K1+A2×K2+A6×K3+A7×K4), which is then stored in the output feature map (e.g., O1 in OFMAP 830). The CIM array 110 may perform a multiplication-accumulation (e.g., summation) operation based on the inputs (e.g., A6, A7, A11, A12) and weights (e.g., K1, K2, K3, K4) in the second storage row. The accumulator buffer 202 in the CIM array 110 may calculate and generate a partial sum (PS) (e.g., A6×K1+A7×K2+A11×K3+A12×K4) and then store it in an output feature map (e.g., O5 in OFMAP 830). The CIM array 110 may perform a multiplication-accumulation (e.g., summation) operation based on the inputs (e.g., A11, A12, A16, A17) and the weights (e.g., K1, K2, K3, K4) in the third storage row. The accumulator buffer 202 in the CIM array 110 may calculate and generate a partial sum (PS) (e.g., A11×K1+A12×K2+A16×K3+A17×K4) and then store it in an output feature map (e.g., O9 in OFMAP 830). Adding a storage row (row #3) allows the reuse of data acquired for row #2 to write row #3, thereby further saving cost. This approach ensures more efficient use of memory resources by intelligently adjusting the allocation of data to storage rows based on the unique requirements of each tier.

[0076] Fig.11 1 shows an example data mapping in a multi-storage row compute-in-memory (CIM) array according to some embodiments. In some embodiments, the CIM array 110 may be a 9x4 CIM array with two storage rows (eg, two rows per MAC unit). Fig.11In the example data mapping in , a 9x9x1 input feature map 1110 (e.g., IFMAP), a 3x3x1 filter 1120 (stride size = 2), and a 4x4x1 output feature map 830 (e.g., OFMAP) are used. Before the calculation begins, the input activations (e.g., IFMAP) are pre-filled and stored in each PE 110 of the CIM array 110 so that all PEs for a given filter are distributed along the PE columns. Then, weights flow in from the edges, and each PE generates a partial sum per cycle. The generated partial sums can then be reduced in parallel along each column across the rows to generate one output feature map (e.g., OFMAP) pixel per column.

[0077] In some embodiments, the fetch circuit 106 may fetch 4 data from the activation buffer 102 (e.g., IFMAP) during each cycle. Given the array configuration and a particular workload (e.g., filter size, stride size, activation buffer size, weight buffer size, core size, or size of multiple PEs), the data fetch pattern may be predetermined. Each column of the CIM array 110 includes two storage rows. In some embodiments, the shift register hardware remains unchanged as it still requires a one-step shift. Fig.11 In , the writing protocol varies according to the situation, e.g. Figure 8 to suit different scenarios.

[0078] Fig.12 1 shows a flow chart of an example method of operating a compute-in-memory (CIM) circuit 100 according to some embodiments. It should be noted that the method 1200 is merely an example and is not intended to limit the present disclosure. Therefore, it should be understood that the method 1200 may be used in Fig.12 Any additional operations are provided during, before, and after the method 1200, and some other operations may only be briefly described herein. The method 1200 may be used to operate the memory circuit 110, and therefore, will be combined with Figure 1-Figure 11 The operations of method 1200 are discussed with reference to the components discussed in .

[0079] In brief, the method 1200 begins with an operation 1202 of receiving a plurality of first data elements. In some embodiments, the fetch circuit may receive the plurality of first data elements from the first buffer 102. Next, the method 1200 continues to an operation 1204 of fetching a first subset of the first data elements and temporarily storing the first subset of the first data elements during a first cycle of writing a first subset of the first data elements to a first subset of the plurality of processing elements (PEs). For example, during a first cycle of writing a first subset of the first data elements to a first subset of the PEs arranged along a first row, the fetch circuit 106 may fetch a first subset of the first data elements from the first buffer 102 and temporarily store the first subset of the first data elements. Considering the regular pattern of overlapping activations of the CNN, the fetch circuit 106 may utilize a shift register to reuse overlapping data.

[0080] Next, the method 1200 proceeds to operation 1206 to determine whether to reuse at least one of the first subset of the first data elements based on a configuration of an artificial intelligence (AI) neural network associated with the plurality of first data elements. In some embodiments, the configuration of the AI ​​neural network includes at least one of the following: a filter size, a stride size, an activation buffer size, a weight buffer size, a core size, or a size of a plurality of PEs. In some embodiments, before acquiring any new data from the first buffer 102, the controller 108 (e.g., a write scheduler) may determine whether to reuse old data in the acquisition circuit 106 (e.g., a shift register) based on recognizing that the old data may be overlapped by the filter in one or more subsequent cycles in the CNN.

[0081] Next, the method 1200 proceeds to operation 1208, and based on the determination of whether to reuse at least one of the first subset, selectively restricting the acquisition of the second subset of the first data elements during a subsequent second cycle in which the second subset of the first data elements is written to the second subset of PEs. For example, during a certain cycle (e.g., cycle #2), the acquisition circuit 106 may reuse a portion of the first subset (e.g., A2, A3, A4) and acquire new data (e.g., A5) from the first buffer 102.

[0082] This disclosure outlines a method to improve the energy efficiency and processing throughput of convolutional neural networks (CNNs), which is critical in artificial intelligence (AI) driven tasks such as computer vision applications. By dynamically adjusting the data mapping and operation flow within the compute in multi-storage line memory (CIM) macro to adapt to the unique needs of various neural network layers, this disclosure significantly improves computational performance.

[0083] As used herein, the terms "about" and "approximately" generally refer to plus or minus 10% of the stated value. For example, about 0.5 would include 0.45 and 0.55, about 10 would include 9 to 11, and about 1000 would include 900 to 1100.

[0084] According to one aspect of an embodiment of the present application, a memory circuit is provided, comprising: an array, comprising a plurality of processing elements (PEs) arranged on a plurality of columns and a plurality of rows, wherein each PE is configured to perform a multiplication-accumulation (MAC) operation on a corresponding one of a plurality of first data elements and a corresponding one of a plurality of second data elements; a first buffer, configured to store the first data element and output the first data element to the array; a second buffer, configured to store the second data element and output the second data element to the array; an acquisition circuit, coupled between the first buffer and the array, wherein the acquisition circuit is configured to extract a first subset of the first data elements from the first buffer during a first cycle and temporarily store the first subset of the first data elements, the first cycle being used to write the first subset of the first data elements to a first subset of PEs arranged along a first row in a row; and a controller, coupled to the acquisition circuit and configured to control the acquisition circuit to selectively restrict the extraction of a second subset of the first data elements from the first buffer during a subsequent second cycle, the second cycle being used to write the second subset of the first data elements to a second subset of PEs arranged along a second row in a row.

[0085] In some embodiments, the acquisition circuit includes a plurality of multiplexers and a plurality of registers.

[0086] In some embodiments, each of a plurality of multiplexers controlled by the controller includes a first input, a second input, and an output coupled to a corresponding one of the registers that is also coupled to a corresponding one of the columns.

[0087] In some embodiments, the multiplexers and corresponding registers are connected to each other in a shift-based manner such that the first input of a first multiplexer of the multiplexers is coupled to the first buffer, the second input of the first multiplexer is coupled to the output of a second register of the registers, the first input of a second multiplexer of the multiplexers is coupled to the first buffer, the second input of the second multiplexer is coupled to the output of a third register of the registers, and the first input of a third multiplexer of the multiplexers is coupled to the first buffer, and the second input of the third multiplexer is coupled to the output of a fourth register of the registers.

[0088] In some embodiments, during a first cycle, a first subset of first data elements are temporarily stored in ones of the registers.

[0089] In some embodiments, during the second cycle, one or more of the second subset of the first data elements remain stored in ones of the registers.

[0090] In some embodiments, the first subset of the first data elements and the second subset of the first data elements differ from each other by one of the first data elements.

[0091] In some embodiments, the first subset of the first data elements and the second subset of the first data elements are completely different from each other.

[0092] In some embodiments, the first subset of first data elements and the second subset of first data elements are identical to each other.

[0093] In some embodiments, at least one of the second data elements is configured to be multiplied with the third subset and the fourth subset of the first data elements.

[0094] In some embodiments, the third subset and the fourth subset of the first data elements are written to a third row and a fourth row of the rows, respectively, and wherein the third row is disposed immediately adjacent to the fourth row.

[0095] According to another aspect of an embodiment of the present application, a memory circuit is provided, comprising: an array, comprising a plurality of processing elements (PEs) arranged on a plurality of columns and a plurality of rows, wherein each PE is configured to perform a multiplication-accumulation (MAC) operation on a corresponding one of a plurality of first data elements and a corresponding one of a plurality of second data elements; an acquisition circuit, coupled to the array and configured to extract a first subset of first data elements from a corresponding buffer during a first cycle, the first cycle being used to write the first subset of first data elements to a first subset of PEs arranged along a first row in a row; and a controller, coupled to the acquisition circuit and configured to cause the acquisition circuit to restrict fetching of a second subset of first data elements from the buffer during a subsequent second cycle, the second cycle being used to write the second subset of first data elements to a second subset of PEs arranged along a second row in a row.

[0096] In some embodiments, the buffer is an activation buffer configured to store a first data element and output the first data element to the array, or is a weight buffer configured to store a second data element and output the second data element to the array.

[0097] In some embodiments, the acquisition circuit includes a plurality of multiplexers and a plurality of registers.

[0098] In some embodiments, each of a plurality of multiplexers controlled by the controller includes a first input, a second input, and an output coupled to a corresponding one of the registers that is also coupled to a corresponding one of the columns.

[0099] In some embodiments, the multiplexers and corresponding registers are connected to each other in a shift-based manner such that the first input of a first multiplexer of the multiplexers is coupled to the buffer, the second input of the first multiplexer is coupled to the output of a second register of the registers, the first input of a second multiplexer of the multiplexers is coupled to the buffer, the second input of the second multiplexer is coupled to the output of a third register of the registers, and the first input of a third multiplexer of the multiplexers is coupled to the buffer, and the second input of the third multiplexer is coupled to the output of a fourth register of the registers.

[0100] In some embodiments, during a first cycle, a first subset of first data elements are temporarily stored in ones of the registers.

[0101] In some embodiments, during the second cycle, one or more of the second subset of the first data elements remain stored in ones of the registers.

[0102] According to another aspect of an embodiment of the present application, a method for operating a compute-in-memory (CIM) circuit is provided, comprising: receiving a plurality of first data elements by a buffer; extracting a first subset of the first data elements by the buffer during a first cycle, and temporarily storing the first subset of the first data elements to write the first subset of the first data elements to a first subset of a plurality of processing elements (PEs); determining by the buffer whether to reuse at least one of the first subset of the first data elements based on a configuration of an artificial intelligence (AI) neural network associated with the plurality of first data elements; and selectively restricting, during a subsequent second cycle, extraction of a second subset of the first data elements by the buffer to write the second subset of the first data elements to a second subset of the PEs based on the determination of whether to reuse at least one of the first subset.

[0103] In some embodiments, the configuration of the AI ​​neural network includes at least one of: a filter size, a stride size, an activation buffer size, a weight buffer size, a kernel size, or a size of a plurality of PEs.

[0104] The features of several embodiments are summarized above so that those skilled in the art can better understand the various aspects of the present disclosure. Those skilled in the art will appreciate that they can easily use the present disclosure as a basis for designing or modifying other processes and structures for achieving the same purpose of the embodiments introduced herein and / or achieving the same advantages thereof. Those skilled in the art will also appreciate that such equivalent structures do not deviate from the spirit and scope of the present disclosure, and that they can make various changes, substitutions, and changes in the present disclosure without departing from the spirit and scope of the present disclosure.

Claims

1. A memory circuit, comprising: an array comprising a plurality of processing elements arranged in a plurality of columns and a plurality of rows, wherein each of the processing elements is configured to perform a multiply-accumulate operation on a corresponding one of a plurality of first data elements and a corresponding one of a plurality of second data elements; a first buffer configured to store the first data element and output the first data element to the array; a second buffer configured to store the second data element and output the second data element to the array; a fetch circuit coupled between the first buffer and the array, wherein the fetch circuit is configured to extract a first subset of the first data elements from the first buffer and temporarily store the first subset of the first data elements during a first cycle for writing the first subset of the first data elements to a first subset of the processing elements arranged along a first one of the rows; and a controller coupled to the fetch circuit and configured to control the fetch circuit to selectively restrict fetching of a second subset of the first data elements from the first buffer during a subsequent second cycle for writing the second subset of the first data elements to a second subset of the processing elements arranged along a second one of the rows.

2. The memory circuit according to claim 1, wherein: The acquisition circuit includes a plurality of multiplexers and a plurality of registers.

3. The memory circuit according to claim 2, wherein: Each of the plurality of multiplexers controlled by the controller includes a first input, a second input, and an output coupled to a corresponding one of the registers that is also coupled to a corresponding one of the columns.

4. The memory circuit according to claim 3, wherein: The multiplexers and the corresponding registers are connected to each other in a shift-based manner such that the first input of a first multiplexer among the multiplexers is coupled to the first buffer, the second input of the first multiplexer is coupled to the output of a second register among the registers, the first input of a second multiplexer among the multiplexers is coupled to the first buffer, the second input of the second multiplexer is coupled to the output of a third register among the registers, and the first input of a third multiplexer among the multiplexers is coupled to the first buffer, and the second input of the third multiplexer is coupled to the output of a fourth register among the registers.

5. The memory circuit according to claim 1, wherein: At least one of the second data elements is configured to be multiplied with a third subset and a fourth subset of the first data elements.

6. A memory circuit comprising: an array comprising a plurality of processing elements arranged in a plurality of columns and a plurality of rows, wherein each of the processing elements is configured to perform a multiply-accumulate operation on a corresponding one of a plurality of first data elements and a corresponding one of a plurality of second data elements; fetch circuitry coupled to the array and configured to extract a first subset of the first data elements from corresponding buffers during a first cycle for writing the first subset of the first data elements to a first subset of the processing elements arranged along a first one of the rows; as well as A controller is coupled to the fetch circuit and is configured to cause the fetch circuit to restrict fetching of a second subset of the first data elements from the buffer during a subsequent second cycle for writing the second subset of the first data elements to a second subset of the processing elements arranged along a second one of the rows.

7. The memory circuit according to claim 6, wherein: The buffer is an activation buffer configured to store the first data element and output the first data element to the array, or is a weight buffer configured to store the second data element and output the second data element to the array.

8. The memory circuit according to claim 6, wherein: The acquisition circuit includes a plurality of multiplexers and a plurality of registers.

9. The memory circuit according to claim 8, wherein: Each of the plurality of multiplexers controlled by the controller includes a first input, a second input, and an output coupled to a corresponding one of the registers that is also coupled to a corresponding one of the columns.

10. A method of operating a computing circuit in a memory, comprising: receiving a plurality of first data elements from a buffer; fetching a first subset of the first data elements by the buffer and temporarily storing the first subset of the first data elements during a first cycle to write the first subset of the first data elements to a first subset of the plurality of processing elements; determining, by the buffer, whether to reuse at least one of the first subset of the first data elements based on a configuration of an artificial intelligence neural network associated with the plurality of first data elements; as well as Based on the determination of whether to reuse the at least one of the first subset, fetching of a second subset of the first data elements by the buffer is selectively restricted during a subsequent second cycle to write the second subset of the first data elements to a second subset of the processing elements.

Images