MEMORY CIRCUITS AND METHOD FOR ENCODER / DECODER DUAL MODE FOR COMPUTE-IN-MEMORY

Abstract

An integrated circuit can have multiple compute-in-memory (CIM) circuits physically implemented on a substrate. Each of the multiple CIM circuits can have: an input circuit configured to receive multiple first data elements; a memory array coupled to the input circuit and configured to store the multiple first data elements; a data multiplexer configured to output the multiple first data elements via a first data path or a second data path; and multiple arithmetic cells coupled to the data multiplexer and configured to perform multiplication-accumulation (MAC) operations on the multiple first data elements and multiple second data elements.

Description

CROSS-REFERENCE TO RELATED REGISTRATION

[0001] This application claims the priority and benefit of the preliminary US application No. 63 / 702,329, filed on October 2, 2024, entitled “Encoder / Decoder Dual Mode CIM Macro”, which is incorporated herein by cross-reference in its entirety for all purposes. BACKGROUND

[0002] Storage devices are integral components of electronic systems that store data in a way that allows for fast access and modification. Traditionally, storage devices are designed to store binary information in the form of "0s" and "1s" in a large array of memory cells. Due to manufacturing variations and design constraints, these cells often have unbalanced physical structures, resulting in differences in their electrical characteristics. Compute-in-memory (CIM) technology integrates processing capabilities directly into memory arrays, enabling faster data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Aspects of this disclosure are best understood from the detailed description below, when read together with the accompanying figures. It should be noted that, in accordance with standard industry practice, various features are not drawn to scale. In fact, the dimensions of the various features may have been arbitrarily enlarged or reduced for the sake of clarity. Fig. Figure 1 shows a block diagram of a compute-in-memory (CIM) based accelerator, according to some embodiments of the present disclosure. Fig. Figure 2 shows a detailed schematic diagram of a compute-in-memory (CIM) circuit. Fig. 1, according to some embodiments of the present disclosure. Fig. Figure 3 shows a detailed schematic diagram of a compute-in-memory (CIM) circuit. Fig. 1, according to some embodiments of the present disclosure. Fig. Figure 4 shows a detailed schematic diagram of a compute-in-memory (CIM) circuit. Fig. 1, according to some embodiments of the present disclosure. Fig. Figure 5 shows a block diagram of a compute-in-memory (CIM) based accelerator, according to some embodiments of the present disclosure. Fig. Figure 6 shows a detailed schematic diagram of a compute-in-memory (CIM) circuit. Fig. 5, according to some embodiments of the present disclosure. Fig. Figure 7 shows a detailed schematic diagram of a compute-in-memory (CIM) circuit. Fig. 5, according to some embodiments of the present disclosure. Fig. Figure 8 is a flowchart of an example method for operating a compute-in-memory (CIM) circuit, according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0004] The following disclosure provides many different embodiments, or examples, for implementing various features of the present subject matter. Specific examples of components and arrangements are described below to simplify the present disclosure. These are, of course, merely examples and are not intended to be limiting. For example, forming a first element over or on top of a second element in the following description may include embodiments in which the first and second elements are in direct contact, and may also include embodiments in which additional elements may be formed between the first and second elements, so that the first and second elements may not be in direct contact. Furthermore, the present disclosure may repeat reference numbers and / or letters in the various examples.This repetition is done for the sake of simplicity and clarity and does not in itself prescribe any relationship between the various designs and / or configurations discussed.

[0005] Furthermore, terms relating to spatial relativity, such as "below," "under," "lower," "above," "upper," "above," "below," and the like, may be used herein for the convenience of discussion to describe the relationship of one element or feature to another element or feature (or other elements or features), as illustrated in the figures. The terms relating to spatial relativity are intended to encompass various orientations of the apparatus used or operated in addition to the orientation illustrated in the figures. The apparatus may be oriented in a different way (rotated by 90 degrees or otherwise), and the terms relating to spatial relativity used herein may likewise be interpreted accordingly.

[0006] In a compute-in-memory (CIM) architecture, the CIM macro for an encoder structure can be equipped with a memory array (e.g., a latch array), which facilitates weight reuse—a feature for efficient processing in encoder tasks. In contrast, the CIM macro designed for a decoder structure does not require a memory array (e.g., a latch array) for weight reuse, thus addressing a different set of operational efficiencies and limitations. A conventional CIM macro can only support either an encoder structure or a decoder structure. In the proposed compute-in-memory (CIM) architecture of the present application, the CIM circuit is designed to support both encoder and decoder functionality of a transformer model that, in principle, has separate encoder and decoder structures.The present application enables the CIM architecture to effectively accommodate the distinct functionalities of both the encoder and the decoder, thereby overcoming the limitations of conventional CIM macros, which typically support only one of these structures. This dual-functionality design improves overall processing efficiency and adaptability when handling the varying computational requirements of transformer models.

[0007] The transformer architecture / model can be divided into an encoder component and a decoder component. The input to the encoder component can include the summation of the input embedding and positional encoding of the input tokens. Positional encoding is necessary because, unlike sequential architectures such as recurrent neural networks, where input tokens are inserted sequentially and thus the order of the input tokens is maintained, the transformer has no concept of the word order. The encoder layer architecture can have two sublayers. The first sublayer can contain a multi-head attention component followed by an addition and normalization component. The second sublayer can contain a feedforward neural network component followed by an addition and normalization component.A multi-head attention component can have multiple instances of scaled scalar product attention, each instance having its own weights to improve the generalization of the model. The output matrix of each instance {zo, ... , zn} is concatenated and multiplied by a weight matrix, resulting in an output matrix.

[0008] The architecture of the decoder layer in the transformer architecture can have three sublayers. The first sublayer has a masked multi-head attention component, followed by an addition and normalization component. The second sublayer has a multi-head attention component (encoder-decoder component), followed by an addition and normalization component. The third sublayer has a feedforward mesh component, followed by an addition and normalization component. The encoder-decoder attention component is similar to the multi-head attention component; however, the query vector Q comes from the previous sublayer of the decoder layer, and the key vectors K and value vectors V are retrieved from the output of the final encoder layer.The masked multi-head attention component is a multi-head attention component with a modification such that the self-attention layer is only allowed to consider previous positions of the input tokens. The output of the decoder layer can be connected to a linear layer, followed by the SoftMax calculation to generate the probabilities of the output vocabulary, thus representing the predicted tokens. The input to the decoder component can include the token embeddings of the output tokens and the position encoding.

[0009] A core component of the transformer architecture is the attention component. A transformer can have three types of attention mechanisms: encoder self-attention, decoder self-attention, and encoder-decoder attention. The input for encoder self-attention consists of the transformer's source input tokens or the output of the previous encoder layer. The encoder self-attention component has no masking, and each token has a global dependency on the other input tokens. The decoder self-attention component uses the transformer's output tokens as the input tokens or the output of the previous decoder layer. In decoder self-attention, the input tokens depend on the previous input tokens.In the encoder-decoder attention component, queries are retrieved from the preceding decoder layer component, and keys and values ​​are retrieved from the encoder's output. In some embodiments, the encoder reads and processes the input data simultaneously using self-attention and positional feedforward networks. The encoder transforms the input into a set of attention vectors representing different aspects of the input. In some embodiments, the decoder generates the output sequence step by step. The encoder uses self-attention to consider previously unseen words in the output and encoder-decoder attention to focus on relevant parts of the input.

[0010] The compute-in-memory (CIM) circuits presented in the present application provide significant improvements in hardware efficiency and area utilization for neural network accelerators, such as tensor processing units (TPUs), graphics processing units (GPUs), and neural network processing units (NPUs). In some embodiments, the present application addresses the inefficiencies encountered in conventional transformer-based models (e.g., ChatGPT) that use dedicated CIM hardware for both encoder and decoder structures. These models face utilization problems because only one hardware set (either encoder or decoder) is active at any given time, resulting in significant idle times. In the conventional setup, the CIM encoder (e.g.,The memory-intensive CIM decoder (e.g., computationally intensive), which is not computationally intensive and requires a memory array for data reuse, is underutilized when the computationally intensive CIM decoder (e.g., computationally intensive), which does not require a memory array for data reuse, is in operation, and vice versa.

[0011] The proposed dual-mode encoder / decoder CIM architecture significantly improves upon the above situation by allowing a single CIM macro to dynamically switch between encoder and decoder functions (by incorporating at least one data multiplexer), thereby maintaining high utilization across both processing tasks. This dual functionality not only results in a smaller footprint but also improves the system's operational efficiency. By enabling the CIM circuitry to support both encoder and decoder functions, this flexible approach addresses the resource underutilization problems in traditional transformer models, paving the way for more compact and efficient neural network accelerators.

[0012] The present disclosure provides various embodiments of an integrated circuit that address such underutilization problems for encoders and decoders. For example, the integrated circuit, as disclosed herein, includes several compute-in-memory (CIM) circuits physically implemented on a substrate. Each of the multiple CIM circuits can include: an input circuit, a memory array, a data multiplexer, and multiple arithmetic cells. The input circuit can be configured to receive multiple first data elements. The memory array can be coupled to the input circuit and can be configured to store the multiple first data elements. The data multiplexer can be configured to output the multiple first data elements via a first data path or via a second data path.The multiple computing cells can be coupled to the data multiplexer and can be configured to perform multiplication-accumulation operations (MAC operations) on the multiple first data elements and multiple second data elements.

[0013] Fig. Figure 1 shows a block diagram of a compute-in-memory (CIM) based accelerator, according to some embodiments of the present disclosure. It is understood that Fig. 1 has been simplified for better understanding of the concepts of the present disclosure. The CIM-based accelerator 100 can have multiple compute-in-memory circuits (CIM circuits) 110 (e.g., CIM cores). Each of the multiple CIM circuits can have a data multiplexer 112 and multiple compute cells 114. In some embodiments, an activation signal can be received by the multiple CIM circuits 110. In certain embodiments, multiple activation signals can each be received by the multiple CIM circuits 110. In some embodiments, the multiple CIM circuits can be physically formed on a substrate.

[0014] In some embodiments, the data multiplexer 112 can receive an activation signal. The data multiplexer 112 can be configured to output multiple first data elements according to the activation signal via a first data path or via a second data path. In response to the activation signal being configured with a first logical state (e.g., 1, encoder mode), the data multiplexer 112 can select the first data path. In response to the activation signal being configured with a second logical state (e.g., 0, decoder mode), the data multiplexer 112 can select the second data path. In some embodiments, the first data path can operatively extend from an input circuit, through a memory array and the data multiplexer 112, to the multiple arithmetic cells 114.In some embodiments, the second data path can operatively extend from an input circuit, via the data multiplexer 112, and to the multiple computing cells 114.

[0015] By processing via the first data path, the CIM circuit 110 can function as an encoder (e.g., encoder mode). The encoder can include a multi-head attentional component and a feedforward neural network component. The multi-head attentional component is configured to perform self-attentional processes in parallel using different weight matrices, enabling the model to capture various types of relationships in the data simultaneously. The feedforward neural network component is configured to further refine the attentionally processed output. Each position in the input sequence independently undergoes the same neural network process. There is no need for masking in the encoder's self-attentional layers, as all inputs are available at the time of processing.

[0016] By processing via the second data path, the CIM circuit 110 can function as a decoder (e.g., decoder mode). The decoder can include a masked multi-head attention component, a multi-head attention component, and a feedforward network component. The masked multi-head attention component can be configured to selectively prevent certain positions in an input sequence from influencing output positions during attentional computations. The multi-head attention component can be configured to process multiple attentional mechanisms in parallel, each applying attention to different representational subspaces of the input sequence. The feedforward network component can be structured to apply the same neural network configuration to all positions in a sequence independently.Both the encoder and decoder layers use layer normalization and residual connections around each sublayer (self-attention, feedforward networks, and in the decoder encoder-decoder attention) to facilitate training and improve the flow of gradients through the network.

[0017] Within a neural network, a node assigns a numerical value, called a "weight," to its connections. When enabled, a node can multiply incoming data by this weight and sum the products of all its connections, resulting in a single numerical output. In a deep learning system, a model of the neural network is stored in memory, and computational logic in a processor performs multiplication-accumulation (MAC) calculations on the parameters (e.g., weights) stored in memory. In some implementations, the weights can be stored in multiple memory cells within a memory array.

[0018] In some embodiments, the multiple compute cells 114 can be coupled to the data multiplexer 112. In some embodiments, the multiple compute cells 114 can be configured to receive multiple inputs (e.g., first data elements, second data elements). The multiple compute cells 114 can be configured to perform multiplication-accumulation operations (MAC operations) on the multiple first data elements and multiple second data elements. In some embodiments, the first data elements can have multiple weight data elements. The second data elements can have multiple input data elements. In certain embodiments, the first data elements can have multiple input data elements. The second data elements can have multiple weight data elements.

[0019] Fig. Figure 2 shows a detailed schematic diagram of a compute-in-memory (CIM) circuit. Fig. 1, according to some embodiments of the present disclosure. It is understood that Fig. 2 has been simplified for a better understanding of the concepts of the present disclosure. The CIM circuit 110 (e.g., CIM core) can comprise an input circuit 202, a memory array 204, a data multiplexer 112, and several arithmetic cells 114.

[0020] In some embodiments, the input circuit 202 can be configured to receive multiple first data elements. In some embodiments, the input circuit 202 can be a data latch that allows weight reuse. The input circuit 202 can have a data input (e.g., W) and a clock input (e.g., CLK). In some embodiments, the input circuit 202 detects the value at the data input during a specific portion of the clock cycle and holds this value until the next clock pulse. In some embodiments, the first data elements can include multiple weight data elements. In certain embodiments, the first data elements can include multiple input data elements.

[0021] In some embodiments, the memory array 204 can be coupled to the input circuit 202. The memory array 204 can be configured to store the multiple first data elements. In some embodiments, the memory array 204 can have multiple memory cells. Each of the multiple memory cells can accommodate various types of non-volatile or volatile memory technologies, including, but not limited to, static random-access memory (SRAM), dynamic random-access memory (DRAM), resistive random-access memory (ReRAM), magnetoresistive random-access memory (MRAM), and phase-change random-access memory (PCRAM). One or more peripheral circuits (not shown) can be arranged in one or more regions located peripherally with respect to the memory array 204 or within it.The memory cells and peripheral circuits can be coupled by word lines and / or complementary bit lines BL and BLB, and data can be read from and written to the memory bit cells via the complementary bit lines BL and BLB. Different voltage combinations applied to the word lines and bit lines can define a read, erase, or write (programming) operation on the memory bit cells.

[0022] In some embodiments, the data multiplexer 112 can receive an activation signal 212. In some embodiments, the activation signal 212 can be configured with a first logical state (e.g., 1) or a second logical state (e.g., 0). The data multiplexer 112 can be configured to output multiple first data elements according to the activation signal via a first data path or via a second data path. In response to the activation signal being configured with a first logical state (e.g., 1, encoder mode), the data multiplexer 112 can select the first data path. In response to the activation signal being configured with a second logical state (e.g., 0, decoder mode), the data multiplexer 112 can select the second data path.In some embodiments, the first data path can operatively extend from the input circuit 202, via the memory array 204 and the data multiplexer 112, to the multiple arithmetic cells 114. In some embodiments, the second data path can operatively extend from the input circuit 202, via the data multiplexer 112, to the multiple arithmetic cells 114.

[0023] By processing via the first data path, the CIM circuit 110 can function as an encoder (e.g., encoder mode). The encoder can include a multi-head attentional component and a feedforward neural network component. The multi-head attentional component is configured to perform self-attentional processes in parallel using different weight matrices, enabling the model to capture various types of relationships in the data simultaneously. The feedforward neural network component is configured to further refine the attentionally processed output. Each position in the input sequence independently undergoes the same neural network process. There is no need for masking in the encoder's self-attentional layers, as all inputs are available at the time of processing.

[0024] By processing via the second data path, the CIM circuit 110 can function as a decoder (e.g., decoder mode). The decoder can include a masked multi-head attention component, a multi-head attention component, and a feedforward network component. The masked multi-head attention component can be configured to selectively prevent certain positions in an input sequence from influencing output positions during attentional computations. The multi-head attention component can be configured to process multiple attentional mechanisms in parallel, each applying attention to different representational subspaces of the input sequence. The feedforward network component can be structured to apply the same neural network configuration to all positions in a sequence independently.Both the encoder and decoder layers use layer normalization and residual connections around each sublayer (self-attention, feedforward networks, and in the decoder encoder-decoder attention) to facilitate training and improve the flow of gradients through the network.

[0025] In some embodiments, the multiple compute cells 114 can be coupled to the data multiplexer 112. In some embodiments, the multiple compute cells 114 can be configured to receive multiple inputs (e.g., first data elements 208a, second data elements 208b). The multiple compute cells 114 can be configured to perform multiplication-accumulation operations (MAC operations) on the multiple first data elements 208a and the multiple second data elements 208b. In some embodiments, the first data elements 208a can have multiple weight data elements (e.g., W). The second data elements 208b can have multiple input data elements (e.g., Xin). In certain embodiments, the first data elements 208a can have multiple input data elements (e.g., Xin). The second data elements 208b can have multiple weight data elements (e.g., W).

[0026] The present application provides an additional data path for an encoder-CIM circuit. This new data path can be designed to bypass the memory array, thereby enabling the encoder-CIM circuit to support decoder functions. This approach results in a minimal footprint for the overall design of the CIM circuit. Furthermore, the integration of an additional data multiplexer (MUX) along this new data path optimizes data routing between encoder and decoder modes. This dual functionality not only maximizes the utility and efficiency of the CIM architecture but also preserves the compactness essential for integrated circuit design.

[0027] Fig. Figure 3 shows a detailed schematic diagram of a compute-in-memory (CIM) circuit. Fig. 1, according to some embodiments of the present disclosure. Fig. Figure 3 shows an example of a first data path in the CIM circuit 110, according to some embodiments of the present disclosure. The CIM circuit 110 of Fig. 3 is the CIM circuit 110 of Fig. 2 essentially similar, except that the activation signal is configured with a first logical state (e.g. EN=1).

[0028] In some embodiments, the data multiplexer 112 can select the first data path in response to the activation signal being configured with a first logical state (e.g., 1, encoder mode). In encoder mode (EN=1) of the CIM circuit 110, the first data path is designed for efficient processing and computation. Specifically, the data flow begins at the data latch 202, which temporarily holds the input data, ensuring stability before the input data is transferred to the memory array 204. The memory array 204 serves as the primary storage location for data for subsequent computational tasks. After the memory array 204, the data passes through a weight D flip-flop (W-DFF) 208a, which synchronizes the data timing for the next processing stage. The final component in the first data path is the multiple computation cells 114 (e.g.,Multiplication-accumulation unit (MAC unit)), which performs the core arithmetic operations using the data retrieved from memory array 204.

[0029] By processing via the first data path, the CIM circuit 110 can function as an encoder (e.g., encoder mode). The encoder can include a multi-head attentional component and a feedforward neural network component. The multi-head attentional component is configured to perform self-attentional processes in parallel using different weight matrices, enabling the model to capture various types of relationships in the data simultaneously. The feedforward neural network component is configured to further refine the attentionally processed output. Each position in the input sequence independently undergoes the same neural network process. There is no need for masking in the encoder's self-attentional layers, as all inputs are available at the time of processing.

[0030] Fig. Figure 4 shows a detailed schematic diagram of a compute-in-memory (CIM) circuit. Fig. 1, according to some embodiments of the present disclosure. Fig. Figure 4 shows an example of a second data path in the CIM circuit 110, according to some embodiments of the present disclosure. The CIM circuit 110 of Fig. 4 is the CIM circuit 110 of Fig. 2 essentially similar, except that the activation signal is configured with a second logical state (e.g. EN=0).

[0031] In some embodiments, the data multiplexer 112 can select the second data path in response to the activation signal being configured with a second logical state (e.g., 0, decoder mode). In decoder mode (EN=0) of the CIM circuit 110, the second data path is advanced to accelerate processing by bypassing the memory array 204. Starting at the data latch 202, input data is temporarily stored and stabilized before being passed directly to the weight D flip-flop (W-DFF) 208a. The W-DFF efficiently aligns the data clocking during the transition to the final stage, which comprises the multiple arithmetic cells 114 (e.g., multiplication-accumulation unit (MAC unit)). This design eliminates the need to access the memory array 204.By directly forwarding data from the data latch 202 to the MAC 114, the decoder mode optimizes processing speed and efficiency, making them ideal for tasks requiring fast data manipulation (e.g., computationally intensive) and output generation without the additional overhead of memory access.

[0032] By processing via the second data path, the CIM circuit 110 can function as a decoder (e.g., decoder mode). The decoder can include a masked multi-head attention component, a multi-head attention component, and a feedforward network component. The masked multi-head attention component can be configured to selectively prevent certain positions in an input sequence from influencing output positions during attentional computations. The multi-head attention component can be configured to process multiple attentional mechanisms in parallel, each applying attention to different representational subspaces of the input sequence. The feedforward network component can be structured to apply the same neural network configuration to all positions in a sequence independently.Both the encoder and decoder layers use layer normalization and residual connections around each sublayer (self-attention, feedforward networks, and in the decoder encoder-decoder attention) to facilitate training and improve the flow of gradients through the network.

[0033] Fig. Figure 5 shows a block diagram of a compute-in-memory (CIM) based accelerator 100, according to some embodiments of the present disclosure. The CIM-based accelerator 100 can have multiple compute-in-memory circuits (CIM circuits) 110 (e.g., CIM cores). In some embodiments, multiple activation signals (e.g., ENs) can be received by each of the multiple CIM circuits 110. In some embodiments, the multiple CIM circuits 110 can be physically formed on a substrate. The CIM-based accelerator 100 of Fig. 5 is the CIM-based accelerator 100 of Fig. 1 essentially similar, except that multiple activation signals are received.

[0034] In some embodiments, each of the multiple CIM circuits 110 can include a data multiplexer 112 and multiple compute cells 114. Each of the multiple CIM circuits 110 can receive an activation signal. The data multiplexer 112 can be configured to output multiple first data elements according to the activation signal (e.g., 1 or 0) via a first data path or via a second data path. In response to the activation signal being configured with a first logical state (e.g., 1, encoder mode), the data multiplexer 112 can select the first data path. In response to the activation signal being configured with a second logical state (e.g., 0, decoder mode), the data multiplexer 112 can select the second data path. In some embodiments, the first data path can operatively extend from the input circuit 202, via the memory array 204 and the data multiplexer 112, and to the multiple computing cells 114.In some embodiments, the second data path can operatively extend from the input circuit 202, via the data multiplexer 112, and to the multiple computing cells 114.

[0035] In Fig. Each CIM circuit 110 is equipped with its own data multiplexer 112 and a set of computation cells 114. These CIM circuits 110 are capable of receiving individual activation signals that specify operating modes (e.g., an encoder mode or decoder mode). The data multiplexer 112 in each circuit plays a crucial role in directing the data flow. Based on the state of the activation signal—either 1 or 0—the data multiplexer 112 can forward multiple initial data elements either via a primary data path or a secondary data path. This flexible data routing allows each CIM circuit to dynamically switch between different computational tasks or modes, thereby improving the overall functionality and efficiency of the accelerator.By integrating multiple activation signals corresponding to different cores within the accelerator, the integrated circuit design enables precise control and synchronization across the entire array of CIM circuits, tailored to specific processing requirements.

[0036] Fig. Figure 6 shows a detailed schematic diagram of a compute-in-memory (CIM) circuit. Fig. 5, according to some embodiments of the present disclosure. Fig. Figure 7 shows a detailed schematic diagram of a compute-in-memory (CIM) circuit. Fig. 5, according to some embodiments of the present disclosure. The CIM-based accelerator 100 can comprise an input circuit 602, a memory array 604, and multiple compute-in-memory circuits (CIM circuits) 110 (e.g., CIM cores). Each of the multiple CIM circuits can comprise a data multiplexer 112 and multiple compute cells 114. In some embodiments, multiple activation signals (e.g., ENs) can be received by each of the multiple CIM circuits 110. In some embodiments, the multiple CIM circuits 110 can be physically formed on a substrate. The CIM-based accelerator 100 of Fig. 6 and Fig. 7 is the CIM-based accelerator 100 of Fig. 1 essentially similar, with the main difference being the single memory array that stores weights shared by the CIM cores.

[0037] In some embodiments, the input circuit 602 can be configured to receive multiple first data elements (e.g., W). In some embodiments, the input circuit 602 can be a data latch that allows weights to be reused. The input circuit 602 can have a data input (e.g., W) and a clock input (e.g., CLK). In some embodiments, the input circuit 602 detects the value at the data input during a specific portion of the clock cycle and holds this value until the next clock pulse. In some embodiments, the first data elements can include multiple weight data elements. In certain embodiments, the first data elements can include multiple input data elements.

[0038] In some embodiments, the memory array 604 can be coupled to the input circuit 602. The memory array 604 can be configured to store the multiple first data elements. In some embodiments, the memory array 604 can have multiple memory cells. Each of the multiple memory cells can accommodate various types of non-volatile or volatile memory technologies, including, but not limited to, static random-access memory (SRAM), dynamic random-access memory (DRAM), resistive random-access memory (ReRAM), magnetoresistive random-access memory (MRAM), and phase-change random-access memory (PCRAM). One or more peripheral circuits (not shown) can be arranged in one or more regions located peripherally with respect to the memory array 604 or within it.The memory cells and peripheral circuits can be coupled by word lines and / or complementary bit lines BL and BLB, and data can be read from and written to the memory bit cells via the complementary bit lines BL and BLB. Different voltage combinations applied to the word lines and bit lines can define a read, erase, or write (programming) operation on the memory bit cells.

[0039] In some embodiments, each of the multiple CIM circuits 110 can be coupled to the memory array 604 by means of a first data path. In some embodiments, each of the multiple CIM circuits 110 can be coupled to the input circuit 602 by means of a second data path. In some embodiments, the first data path can operatively extend from an input circuit 602, via a memory array 604 and a data multiplexer 112, to multiple arithmetic cells 114. In some embodiments, the second data path can operatively extend from an input circuit 602, via a data multiplexer 112, to multiple arithmetic cells 114.

[0040] In some embodiments, each of the multiple CIM circuits 110 can have a data multiplexer 112 and multiple arithmetic cells 114. Each of the multiple CIM circuits 110 can receive an activation signal. The data multiplexer 112 can be configured to output multiple first data elements according to the activation signal (e.g., 1 or 0) via a first data path or via a second data path. Fig. 6. In response to the activation signal being configured with a first logical state (e.g., 1, encoder mode), the data multiplexer 112 can select the first data path. Fig. 7. In response to the activation signal being configured with a second logical state (e.g. 0, decoder mode), the data multiplexer 112 can select the second data path.

[0041] Fig. Figure 8 is a flowchart of an example method for operating a compute-in-memory (CIM) circuit, according to some embodiments of the present disclosure. It is understood that Fig. 8 has been simplified for a better understanding of the concepts of this disclosure. Accordingly, it should be noted that additional processes before, during, and after the procedure of Fig. 8 can be provided, and some other processes can only be briefly described here.

[0042] Referring to Procedure 805, and in some embodiments, a compute-in-memory (CIM) circuit 110 can be configured to receive multiple first data elements (e.g., W), multiple second data elements (e.g., Xin), and an activation signal (e.g., EN). In some embodiments, the data multiplexer 112 of the CIM circuit 110 can receive an activation signal. The data multiplexer 112 can be configured to output multiple first data elements according to the activation signal via a first data path or via a second data path. In some embodiments, the multiple compute cells 114 can be coupled to the data multiplexer 112. In some embodiments, the multiple compute cells 114 can be configured to receive multiple inputs (e.g., first data elements, second data elements).The multiple computation cells 114 can be configured to perform multiplication-accumulation (MAC) operations on the multiple first data elements and multiple second data elements. In some embodiments, the first data elements can have multiple weight data elements. The second data elements can have multiple input data elements. In certain embodiments, the first data elements can have multiple input data elements. The second data elements can have multiple weight data elements.

[0043] Next, procedure 800 proceeds to operation 810, in which, in response to a determination that the activation signal is equal to a first logical state, a first data path is selected to forward the multiple first data elements, received via an input circuit and a memory array, to multiple arithmetic cells. The multiple arithmetic cells can be configured to perform multiplication-accumulation (MAC) operations on the multiple first data elements and multiple second data elements. In some embodiments, by processing via the first data path, the CIM circuit 110 can function as an encoder (e.g., encoder mode). The encoder can include a multi-head attention component and a feedforward neural network component.The multi-head attention component is configured to perform self-attention processes in parallel using different weight matrices, enabling the model to capture various types of relationships in the data simultaneously. The feedforward neural network component is configured to further refine the attention-processed output. Each position in the input sequence independently undergoes the same neural network process. There is no need for masking in the encoder's self-attention layers, as all inputs are available at processing time.

[0044] Next, procedure 800 proceeds to operation 815, in which, in response to a determination that the activation signal corresponds to a second logical state, a second data path is selected to forward the multiple first data elements received via the input circuit to the multiple computation cells. In some embodiments, by processing via the second data path, the CIM circuit 110 can function as a decoder (e.g., decoder mode). The decoder can include a masked multi-head attention component, a multi-head attention component, and a feedforward net component. The masked multi-head attention component can be configured to selectively prevent certain positions in an input sequence from influencing output positions during attention calculations.The multi-head attention component can be configured to process multiple attentional mechanisms in parallel, each applying attention to different representational subspaces of the input sequence. The feedforward network component can be structured to apply the same neural network configuration to all positions in a sequence independently. Both the encoder and decoder layers employ layer normalization and residual connections around each sublayer (self-attention, feedforward networks, and, in the decoder, encoder-decoder attention) to facilitate training and improve gradient flow through the network.

[0045] The present application provides a CIM-based accelerator that incorporates advanced features to enhance its versatility and processing efficiency. The application enables the encoder to support CIM decoder functions, achieving this flexibility with a negligible footprint, thus maintaining a compact and efficient circuit design. The present application introduces a data multiplexer (MUX) within the integrated circuit. This MUX is strategically positioned to select between data sourced directly from memory or from a data latch, effectively enabling the option of bypassing memory when necessary. This capability is particularly advantageous in scenarios where speed and response time take precedence over memory reads.The present application supports all types of memory technologies, including DRAM, ReRAM, and MRAM, across various technology nodes. This universal compatibility ensures that the accelerator can be integrated into diverse system environments and optimized for a wide range of applications, from mobile devices to large data centers, thus providing a robust solution that can be adapted to future technological advancements.

[0046] In one aspect of the present disclosure, an integrated circuit is disclosed. The integrated circuit may comprise multiple compute-in-memory (CIM) circuits physically formed on a substrate. Each of the multiple CIM circuits may comprise: an input circuit configured to receive multiple first data elements; a memory array coupled to the input circuit and configured to store the multiple first data elements; a data multiplexer configured to output the multiple first data elements via a first data path or via a second data path; and multiple arithmetic cells coupled to the data multiplexer and configured to perform multiplication-accumulation (MAC) operations on the multiple first data elements and multiple second data elements.

[0047] In another aspect of the present disclosure, an integrated circuit is disclosed. The integrated circuit can include multiple compute-in-memory (CIM) circuits. Each of the multiple CIM circuits can be configured to output multiple multiplication-accumulation (MAC) results. Each of the multiple CIM circuits can include at least one data multiplexer and multiple arithmetic cells. The data multiplexer can be configured to: forward multiple first data elements over a first data path in response to receiving an activation signal configured with a first logical state; or forward the multiple first data elements over a second data path in response to receiving the activation signal configured with a second logical state. The multiple arithmetic cells can be configured to receive multiple second data elements.The multiple computing cells can be configured to output the respective MAC results based on the multiple received first data elements and the multiple second data elements forwarded by the data multiplexer.

[0048] In a further aspect of the present disclosure, a method for operating an integrated circuit is disclosed. The method may include receiving several first data elements, several second data elements, and an activation signal. Upon determining that the activation signal corresponds to a first logical state, the method may include selecting a first data path for forwarding the several first data elements, received via an input circuit and a memory array, to several arithmetic cells. The several arithmetic cells may be configured to perform multiplication-accumulation (MAC) operations on the several first data elements and several second data elements.The procedure can, in response to a finding that the activation signal is equivalent to a second logical state, include selecting a second data path to forward the multiple first data elements received via the input circuit to the multiple computing cells.

[0049] As used here, the terms "approximately" and "about" generally indicate the value of a given quantity, which may vary based on a specific technology node associated with the semiconductor device in question. Based on that specific technology node, the term "approximately" may indicate a value of a given quantity that varies, for example, within 10 to 30% of the value (e.g., ±10%, ±20%, or ±30% of the value).

[0050] The foregoing outlines features of several embodiments so that a person skilled in the art can better understand the aspects of the present disclosure. A person skilled in the art should recognize that they can readily use the present disclosure as a basis for designing or modifying other processes and structures to accomplish the same tasks and / or achieve the same advantages as the embodiments presented herein. A person skilled in the art should also understand that such equivalent embodiments do not deviate from the inventive concept and scope of the present disclosure, and that they can make various changes, substitutions, and modifications here without deviating from the inventive concept and scope of the present disclosure. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 63 / 702,329

[0001]

Images