Hybrid transformer accelerator and method of operation

Abstract

The application discloses a kind of hybrid Transformer accelerator and running method, belong to artificial intelligence hardware acceleration technical field.The application realizes high energy efficiency Transformer acceleration by four big core innovations: dynamic sparse floating-point multiplication and addition mechanism, based on index and threshold self-comparison realizes bit-level sparse acceleration;Data multiplexing multiplication and addition circuit, pre-computing mantissa three modes and full-array sharing;SOT-MRAM and SRAM hybrid architecture, give consideration to static low power consumption and dynamic read-write speed;7T SRAM transposition array, transposition period is reduced from vector length related to only with bit width related.The application has extreme energy efficiency and computing speed performance, solves the contradiction that single memory cannot give consideration to static power consumption and read-write speed, significantly improves energy efficiency and transposition efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence hardware acceleration and in-memory computing technology, specifically to a hybrid Transformer accelerator and its operating method. Background Technology

[0002] The Transformer model has become the mainstream architecture in artificial intelligence fields such as natural language processing, computer vision, and speech recognition, demonstrating outstanding performance. The core attention mechanism of the Transformer involves a large number of matrix multiplication operations, leading to intensive data movement between memory and processing units, which in turn causes a severe "memory wall" bottleneck, resulting in extremely high energy consumption and latency.

[0003] Computing-in-Memory (CIM) technology integrates computing units within a memory array, supporting massively parallel multiply-accumulate operations and significantly reducing data movement, making it an effective solution to the memory wall problem and achieve energy-efficient matrix multiplication. However, existing CIM architectures have the following significant drawbacks when handling Transformer models:

[0004] First, there is a lack of native support for high-precision floating-point calculations, and this often results in a loss of precision. Most existing CIM architectures are limited to integer data formats and cannot natively support high-precision floating-point calculations. Existing floating-point calculation schemes typically rely on floating-point to integer conversions and off-array digital logic for simulation, which not only increases hardware overhead but also introduces precision loss due to quantization when processing complex datasets. The few CIM designs that attempt to directly support floating-point operations usually rely on complex preprocessing such as exponent alignment, introducing additional computational latency.

[0005] Second, existing sparsity strategies fail to utilize bit-level sparsity. Existing sparse computation methods mainly focus on algorithm-level modeling, such as space or token-level pruning, ignoring the inherent low-impact elements in matrix operations, such as near-zero floating-point numbers. In the CIM architecture, these low-impact elements are still read and accumulated, and the accumulation of values ​​of different magnitudes leads to frequent shift operations, resulting in unnecessary latency and energy consumption.

[0006] Third, a single type of memory cannot meet the differentiated requirements of static and dynamic data in Transformer. The Query, Key, and Value weight matrices in Transformer are fixed during inference, requiring the CIM array to have extremely low static power consumption. Spintronic memories such as SOT-MRAM are non-volatile and have near-zero static power consumption, making them suitable for storing weights, but their slow write speed cannot meet the frequent update requirements of dynamic matrices in the attention mechanism. While SRAM has fast read and write speeds, it cannot meet the low static power consumption requirements when storing static weights. Therefore, a CIM based on a single memory cannot efficiently support the complete computational flow of Transformer.

[0007] Fourth, the mismatch between row and column access in the CIM array leads to extremely high transpose write-back latency. The attention mechanism requires dynamically calculating the product of the transpose matrices of the query and key. CIM arrays typically perform read and write operations row-wise, while the transpose write-back of the dynamic key transpose matrix requires column-wise writing. This mismatch in access direction results in complex data rearrangement. In a conventional SRAM array, transposing and writing a row vector containing 256 elements requires 256 write cycles, severely limiting computational efficiency. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide a hybrid Transformer accelerator and its operation method. Through four core technologies, namely dynamic sparse floating-point multiply-accumulate, data multiplexing multiply-accumulate, hybrid storage architecture, and fast transpose array, it solves the problems of low energy efficiency, accuracy loss, inability of a single memory to balance power consumption and speed, and high transpose write-back latency in the existing CIM architecture when processing Transformers. It achieves Transformer hardware acceleration with high precision, low latency, and high energy efficiency.

[0009] To achieve the above objectives, the present invention provides the following technical solution:

[0010] This invention provides a hybrid Transformer accelerator, comprising:

[0011] Dynamic sparse floating-point multiply-accumulate module, data multiplexing multiply-accumulate module, SOT-MRAM and SRAM hybrid in-memory computing array, and 7TSRAM array supporting fast transpose write-back;

[0012] The dynamic sparse floating-point multiply-accumulate module is used to extract the exponent bits of the input floating-point number and the weight floating-point number and add them together in the floating-point matrix multiply-accumulate operation. The exponent sum is compared with a dynamically configured threshold. If the exponent sum is lower than the threshold, a skip signal is generated to block the reading of the mantissa of the corresponding weight and subsequent multiply-accumulate operations.

[0013] The data multiplexing multiply-accumulate module includes an input pre-calculation circuit and an array multiplexer. The input pre-calculation circuit generates three basic modes for the mantissa of the input floating-point number: the original value, the value shifted left by 1 bit, and the sum of the original value and the value shifted left. The multiplexer uses a multi-bit slice of the weighted mantissa as the selection signal, selects the corresponding basic mode as a partial product, and obtains the mantissa multiplication result through shifting and accumulation.

[0014] The SOT-MRAM and SRAM hybrid in-memory computing array includes a static weight storage array based on SOT-MRAM and a dynamic data storage array based on SRAM; the SOT-MRAM array is used to store the fixed WQ, WK, and WV weight matrices during inference, and the SRAM array is used to store the dynamically generated activation values ​​and intermediate matrices during inference.

[0015] The 7TSRAM array that supports fast transpose-write-back adopts a 7-transistor bit cell structure and is configured with vertical write word lines and horizontal read word lines. During the write phase, the vertical word lines are activated to write row vector data in parallel to different rows of memory cells in the column direction. The number of write cycles is only related to the data bit width. During the read phase, the horizontal read word lines are activated to read data in the row direction.

[0016] Furthermore, in the dynamic sparse floating-point multiply-accumulate module, the floating-point number adopts the FP16 format, which includes 1 sign bit, 5 exponent bits and 10 mantissa bits; the threshold is a dynamically configurable 5-bit parameter with a default value of 5.

[0017] Furthermore, in the dynamic sparse floating-point multiply-accumulate module, when the exponent sum is greater than or equal to the threshold, a valid enable signal is generated, the weight mantissa is read normally, and the multiplication and accumulation operation is performed; when the exponent sum is lower than the threshold, an invalid enable signal is generated, the corresponding product result is set to 0, and the mantissa reading and accumulation operation are skipped.

[0018] Furthermore, in the data multiplexing multiply-accumulate module, the weight mantissa is divided into consecutive 2-bit slices; the multiplexer is a 4-to-1 multiplexer, and the selection rules are: "00" selects 0, "01" selects the original value, "10" selects the value shifted left by 1 bit, and "11" selects the sum of the original value and the left-shifted value.

[0019] Furthermore, the three basic modes generated by the input pre-computation circuit are shared and multiplexed by the multiply-accumulate units of all columns in the array within the same calculation cycle.

[0020] Furthermore, the SOT-MRAM array possesses non-volatile and near-zero static power consumption characteristics, and can be powered off to enter a zero-power standby state during computation intervals; the SRAM array has high-speed read and write characteristics, supporting frequent updates of the dynamic matrix.

[0021] Furthermore, in the 7TSRAM array, for a row vector containing 256 FP16 elements, transpose writing only requires 16 cycles, regardless of the vector length; the 7TSRAM array is compatible with multiple data formats such as FP8, INT8, and FP32.

[0022] Furthermore, the bit cell of the 7T SRAM array includes: a storage node composed of two cross-coupled inverters, a vertical write path transistor, a horizontal read path transistor, and a read bit line discharge transistor; the write operation is controlled by the vertical word line to control the write path, and the read operation is controlled by the horizontal read word line to control the discharge transistor in conjunction with the sense amplifier. Compared with the standard 6T SRAM, the added transistor is used to separate the read and write paths, so that the write word line is vertical and the read word line is horizontal, thereby supporting column-oriented parallel writing and row-oriented reading.

[0023] Furthermore, the overall system architecture includes five in-memory computing blocks, an input buffer, a global buffer, and a top-level controller; three of the blocks are constructed using SOT-MRAM to store the static weights of Query, Key, and Value, respectively; one block is constructed using 7TSRAM that supports transposition to store the transpose matrix of the Key; and one block is constructed using conventional SRAM to store the dynamic activation value and the Value matrix.

[0024] A method for operating a hybrid Transformer accelerator includes the following steps:

[0025] S1. System initialization: Load the pre-trained Query, Key, and Value static weight matrices into three SOT-MRAM memory blocks respectively, utilize the non-volatile nature of SOT-MRAM for solidified storage, and turn off the power during the calculation interval to reduce static power consumption.

[0026] S2. Parallel generation of dynamic matrices: Input token data is extracted from the input buffer and broadcast in parallel to the SOT-MRAM blocks storing key and value weights. Each block performs pipelined multiply-accumulate operations.

[0027] S21. Read 16-bit input data, pre-calculate the mantissa, and generate three basic modes: original value, left-shifted value by 1 bit, and the sum of the original value and the left-shifted value.

[0028] S22. Read the weight exponent bits, add them to the input exponent bits to get the product exponent sum, and compare the exponent sum with the sparsity threshold.

[0029] S23. If the exponent sum is less than the threshold, it is determined to be a low-influence element, a skip signal is generated, and the result is directly set to 0; if the exponent sum is greater than or equal to the threshold, the weight mantissa is read, the pre-calculation mode is selected through a multiplexer, and the floating-point result is obtained through shifting and accumulation.

[0030] S3, Heterogeneous Storage and Transpose: The generated Value matrix is ​​written to a regular SRAM block; the generated Key matrix is ​​sent to a 7TSRAM block that supports transpose, the vertical write word line is activated, and the transpose is completed by writing in parallel column by column to obtain the transpose matrix of the Key.

[0031] S4. Attention mechanism pipeline execution: Input data is sent to the SOT-MRAM block storing Query weights to generate the Query matrix; the Query matrix is ​​multiplied and added with the transpose of the Key matrix to obtain the score matrix, which is then normalized by Softmax to obtain the attention weight matrix; the attention weight matrix is ​​multiplied and added with the Value matrix to obtain the final attention output, which is written back to the global buffer.

[0032] The present invention has the following advantages and beneficial effects:

[0033] 1. Extreme energy efficiency and computing speed: By combining data reuse and hybrid RAM architecture, the CIM macro implemented in this invention achieves an energy efficiency of up to 81.49 TFLOPS / W when performing FP16MAC operations, which is 2.45 to 5.0 times higher than the most advanced existing MRAMCIM macro design; at the same time, the access time is only 1.536 ns, which is 2.6 to 4.2 times faster.

[0034] 2. Balancing High Precision and High Computational Sparsity: The proposed dynamic sparse floating-point mechanism skips low-influence elements without additional preprocessing steps. With a threshold of 5, it skips 25.2% of useless operations while maintaining over 97.6% precision. In system-level tests of Fairseq and Helsinki-NLP models on the WMT14 dataset, the overall accuracy loss is only 0.17% and 0.03%, respectively, with a system energy efficiency of 68.77 TFLOPS / W and energy consumption as low as 3.6 µJ per token. Compared to existing Transformer accelerators, this represents a 4.33x improvement in energy efficiency and a 2.39x improvement in token-level cost.

[0035] 3. Perfectly matches the memory access characteristics of Transformer: The hybrid RAM architecture leverages the advantages of each material. Fixed weights are stored in SOT-MRAM to achieve static zero power consumption, while dynamic data is placed in SRAM to ensure computing throughput, thus resolving the contradiction between static power consumption and read / write speed that a single memory cannot balance.

[0036] 4. Completely solves the transpose write-back bottleneck: The proposed 7T transpose SRAM array reduces the transpose write cycle, which was originally linearly related to the vector length (e.g., 256 cycles for a 256-dimensional vector), to only be related to the data bit width; FP16 requires only 16 cycles, improving transpose efficiency by 93.75%. Furthermore, this write mode is highly scalable and can easily adapt to different data formats such as FP8, INT8, or FP32. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the dynamic sparse floating-point in-memory computation multiplication and addition method of the present invention;

[0038] Figure 2 This is a schematic diagram of the data multiplexing multiply-accumulate circuit structure of the present invention;

[0039] Figure 3 This is a diagram of the hybrid CIM macro architecture based on SOT-MRAM / SRAM of the present invention;

[0040] Figure 4 This is the workflow logic diagram of the Dynamic Sparse and Data Reuse CIM of this invention;

[0041] Figure 5 This is a diagram of the 7TSRAM fast transpose array and its bit cell structure according to the present invention.

[0042] Figure 6 The following is a system-level architecture and pipeline timing diagram of the HRAMTran accelerator of this invention, wherein (a) is a top-level structure block diagram of the system, (b) is a workflow logic diagram, and (c) is a pipeline timing analysis diagram of query matrix processing.

[0043] Explanation of reference numerals in the attached figures:

[0044] IN - Input data; W - Weight data; Eth - Sparse threshold; EN - Enable signal; IN1 - Original mantissa value; IN2 - Mantissa shifted left by 1 bit; IN3 - Sum of IN1 and IN2; RWL - Read word line; WL - Write word line; RBL - Read bit line; BL - Bit line; Q - Query matrix; K - Key matrix; V - Value matrix; KT - Key transpose matrix; A - Score matrix; A' - Attention weight matrix; Att - Attention output. Detailed Implementation

[0045] The present invention will be further described below with reference to the embodiments. It should be noted that these are merely examples and descriptions of the inventive concept. Those skilled in the art can make various modifications or additions to the specific embodiments described or use similar methods to replace them, as long as they do not deviate from the inventive concept or exceed the scope defined in the claims, they should all be considered to fall within the protection scope of the present invention.

[0046] The following describes specific embodiments and appendices. Figure 1 - Appendix Figure 6 The present invention will be described in further detail below. Those skilled in the art will readily understand other advantages and effects of the present invention from the content disclosed in this specification.

[0047] Example: Hybrid RAMTransformer Accelerator (HRAMTran)

[0048] like Figure 6 As shown in (a), the hybrid Transformer accelerator system of this embodiment mainly consists of five in-memory computing blocks Block0 to Block4, a 4Kb input buffer, a 4Kb global buffer, and a top-level controller.

[0049] Block 0, Block 1, and Block 2 are constructed using spin-orbit torque magnetic random access memory (SOT-MRAM) with near-zero static power consumption to store static weights; Block 3 is constructed using 7TSRAM with fast transpose capability to store dynamic transpose matrices; and Block 4 is constructed using conventional SRAM to handle frequently updated dynamic activation values.

[0050] The specific execution workflow of this accelerator is as follows, see below. Figure 4 and Figure 6 (b):

[0051] Step 1: System Initialization and Static Weight Preloading

[0052] Before performing inference computation, the system first loads the pre-trained invariant weight matrices WQ, WK, and WV into Block 0, Block 1, and Block 2 respectively via the controller. Because SOT-MRAM is non-volatile, the weight data can be permanently stored after being written. During computation intervals, the power to these three blocks can be turned off to enter a zero-power standby state, thereby significantly reducing static energy consumption during inference.

[0053] Step 2: Parallel generation and sparsification calculation of dynamic matrices K and V

[0054] After computation begins, the top-level controller retrieves input token data from the input buffer and broadcasts it in parallel to Block 1 (WK) and Block 2 (WV). Within each CIM block, pipelined multiply-accumulate operations are initiated:

[0055] a. Data Reading and Preprocessing: Read the 16-bit input and pre-generate three mantissa modes: original value IN1, left-shifted by 1 bit IN2, and the sum of IN1 and IN2 IN3, such as... Figure 2 As shown.

[0056] b. Dynamic Sparse Decision: Read the weight exponent bits and add them to the input exponent bits to obtain the product exponent sum. Then, compare the exponent sum with a set sparsity threshold Eth, for example, 5, as follows: Figure 1 As shown.

[0057] c. Conditional skip execution: If the exponent sum is less than Eth, the system determines it as a low-influence element, generates an EN=0 signal, directly skips the mantissa reading of the weight and subsequent multiplication steps, and sets the result to 0.

[0058] d. Multiplexing multiplication and addition calculation: If EN=1, the weight mantissa is read, and the pre-calculation mode is selected by using a 2-bit slice of the weight mantissa through a 4-to-1 multiplexer. After shifting and accumulation, the floating-point results of the K matrix and V matrix are obtained.

[0059] Step 3: Heterogeneous Storage and Transposition

[0060] The calculated V matrix is ​​directly written into Block 4, which is constructed from SRAM. Simultaneously, the generated K matrix is ​​sent to Block 3, which has transpose and write-back functionality.

[0061] During this process: the row vector of the K matrix containing 256 16-bit floating-point elements activates the vertical write word line of Block 3. Utilizing the column-oriented write characteristic of the 7T-bit cells, the system only requires 16 write cycles to write the same bit of all 256 elements column-wise at once, achieving a fast K-to-KT transpose. This significantly reduces the latency of transpose operations in traditional CIM architectures. Figure 5 As shown, a traditional SRAM array requires 256 cycles to complete the same transpose, while this invention reduces the number of cycles by 93.75%.

[0062] Step 4: Pipeline execution of the attention mechanism in the query path

[0063] After the KT and V matrices are ready, the input data is fed back into Block0 and combined with the stored WQ to generate the Q matrix. Subsequently, the system enters a continuous attention mechanism processing phase:

[0064] a. Score Calculation: The Q matrix is ​​sent as input to Block3, and floating-point multiplication and addition are performed with KT in it to generate the score matrix A.

[0065] b. Normalization process: Matrix A is fed into a dedicated Softmax module to calculate the attention weight matrix A'.

[0066] c. Weighted summation: Send A' to Block4, perform matrix multiplication with matrix V to obtain the final attention output, and write it back to the global buffer.

[0067] like Figure 6 As shown in (c), the present invention adopts the proposed pipeline architecture to realize the overlapping execution of multiple token calculations, which significantly improves the computing throughput compared with the serial execution under the traditional pipeline.

[0068] Through tape-out testing and system-level simulation verification, the CIM macro in this embodiment achieves an energy efficiency of 81.49 TFLOPS / W and an access time of only 1.536 ns when performing FP16MAC operations. In the WMT14 translation task test, the system energy efficiency reaches 68.77 TFLOPS / W, and the energy consumption for processing a single token is only 3.6 µJ. All indicators are significantly better than existing technologies.

[0069] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A hybrid Transformer accelerator, characterized in that, include: Dynamic sparse floating-point multiply-accumulate module, data multiplexing multiply-accumulate module, SOT-MRAM and SRAM hybrid in-memory computing array, and 7TSRAM array supporting fast transpose writeback; The dynamic sparse floating-point multiply-accumulate module is used to extract the exponent bits of the input floating-point number and the weight floating-point number and add them together in the floating-point matrix multiply-accumulate operation. The exponent sum is compared with a dynamically configured threshold. If the exponent sum is lower than the threshold, a skip signal is generated to block the reading of the mantissa of the corresponding weight and subsequent multiply-accumulate operations. The data multiplexing multiply-accumulate module includes an input pre-calculation circuit and an array multiplexer. The input pre-calculation circuit generates three basic modes for the mantissa of the input floating-point number: the original value, the value shifted left by 1 bit, and the sum of the original value and the value shifted left. The multiplexer uses a multi-bit slice of the weighted mantissa as the selection signal, selects the corresponding basic mode as a partial product, and obtains the mantissa multiplication result through shifting and accumulation. The SOT-MRAM and SRAM hybrid in-memory computing array includes a static weight storage array based on SOT-MRAM and a dynamic data storage array based on SRAM; the SOT-MRAM array is used to store the fixed WQ, WK, and WV weight matrices during inference, and the SRAM array is used to store the dynamically generated activation values ​​and intermediate matrices during inference. The 7TSRAM array that supports fast transpose writeback adopts a 7-transistor bit cell structure and is configured with vertical write word lines and horizontal read word lines. During the write phase, the vertical word line is activated, and the row vector data is written in parallel to the storage cells of different rows in the column direction. The number of write cycles is only related to the data bit width. During the reading phase, the horizontal read word line is activated, and data is read in the line direction.

2. The hybrid Transformer accelerator according to claim 1, characterized in that, In the dynamic sparse floating-point multiply-accumulate module, the floating-point number adopts the FP16 format, which includes 1 sign bit, 5 exponent bits and 10 mantissa bits; the threshold is a dynamically configurable 5-bit parameter with a default value of 5.

3. The hybrid Transformer accelerator according to claim 1, characterized in that, In the dynamic sparse floating-point multiply-accumulate module, when the exponent sum is greater than or equal to the threshold, a valid enable signal is generated, the weight mantissa is read normally, and the multiplication and accumulation operation is performed; when the exponent sum is lower than the threshold, an invalid enable signal is generated, the corresponding product result is set to 0, and the mantissa reading and accumulation operation are skipped.

4. The hybrid Transformer accelerator according to claim 1, characterized in that, In the data multiplexing multiply-accumulate module, the weight mantissa is divided into consecutive 2-bit slices; the multiplexer is a 4-to-1 multiplexer, and the selection rules are: "00" selects 0, "01" selects the original value, "10" selects the value shifted left by 1 bit, and "11" selects the sum of the original value and the left-shifted value.

5. The hybrid Transformer accelerator according to claim 1, characterized in that, The three basic modes generated by the input pre-computation circuit are shared and multiplexed by the multiply-accumulate units of all columns in the array within the same calculation cycle.

6. The hybrid Transformer accelerator according to claim 1, characterized in that, The SOT-MRAM array is non-volatile and has near-zero static power consumption. During computation intervals, the power can be turned off to enter a zero-power standby state. The SRAM array has high-speed read and write capabilities and supports frequent updates of the dynamic matrix.

7. The hybrid Transformer accelerator according to claim 1, characterized in that, In the 7TSRAM array, for a row vector containing 256 FP16 elements, transpose writing only requires 16 cycles, regardless of the vector length; the 7TSRAM array is compatible with multiple data formats such as FP8, INT8, and FP32.

8. The hybrid Transformer accelerator according to claim 1, characterized in that, The bit cell of the 7TSRAM array includes: a storage node consisting of two cross-coupled inverters, a vertical write path transistor, a horizontal read path transistor, and a read bit line discharge transistor; the write operation is controlled by the vertical word line to control the write path, and the read operation is implemented by the horizontal read word line to control the discharge transistor in conjunction with the read amplifier.

9. The hybrid Transformer accelerator according to claim 1, characterized in that, The overall system architecture includes five in-memory computing blocks, an input buffer, a global buffer, and a top-level controller; three of these blocks are constructed using SOT-MRAM to store static weights for Query, Key, and Value, respectively. One block is constructed using 7TSRAM that supports transposition to store the transpose matrix of the Key; another block is constructed using regular SRAM to store the dynamic activation value and the Value matrix.

10. A method for operating a hybrid Transformer accelerator based on any one of claims 1-9, characterized in that, Includes the following steps: S1. System initialization: Load the pre-trained Query, Key, and Value static weight matrices into three SOT-MRAM memory blocks respectively, utilize the non-volatile nature of SOT-MRAM for solidified storage, and turn off the power during the calculation interval to reduce static power consumption. S2. Parallel generation of dynamic matrices: Input token data is extracted from the input buffer and broadcast in parallel to the SOT-MRAM blocks storing key and value weights. Each block performs pipelined multiply-accumulate operations. S21. Read 16-bit input data, pre-calculate the mantissa, and generate three basic modes: original value, left-shifted value by 1 bit, and the sum of the original value and the left-shifted value. S22. Read the weight exponent bits, add them to the input exponent bits to get the product exponent sum, and compare the exponent sum with the sparsity threshold. S23. If the exponent sum is less than the threshold, it is determined to be a low-influence element, a skip signal is generated, and the result is directly set to 0; if the exponent sum is greater than or equal to the threshold, the weight mantissa is read, the pre-calculation mode is selected through a multiplexer, and the floating-point result is obtained through shifting and accumulation. S3, Heterogeneous Storage and Transpose: The generated Value matrix is ​​written into a regular SRAM block; The generated Key matrix is ​​fed into a 7TSRAM block that supports transposition, the vertical write word line is activated, and the transposition is completed by writing in parallel column by column, thus obtaining the transpose matrix of the Key. S4. Attention mechanism pipeline execution: Input data is sent to the SOT-MRAM block storing Query weights to generate the Query matrix; The Query matrix is ​​multiplied and added with the transpose of the Key matrix to obtain the score matrix, which is then normalized by Softmax to obtain the attention weight matrix; The attention weight matrix and the value matrix are multiplied and added to obtain the final attention output, which is then written back to the global buffer.

Images