ReRAM write delay elimination method and system based on Transform model reasoning

Abstract

The invention provides a ReRAM write delay elimination method and system based on Transform model reasoning, and the method is based on a storage and calculation integrated architecture system of a ReRAM, analyzes the relation between the KV increment generation characteristics in a Transform model and ReRAM write KV, and proposes a delay write KV idea to eliminate the storage delay on a reasoning key path; meanwhile, an SMAC multiply-add unit based on a digital circuit is designed and realized, write delay on a reasoning key path can be effectively eliminated under the condition that the model reasoning process is not influenced, and the system calculation performance is improved. According to the method, the overall calculation efficiency of the system is remarkably improved, and a feasible solution is provided for large model reasoning acceleration under the storage and calculation integrated architecture.

Description

Technical Field

[0001] This invention belongs to the field of computer science and artificial intelligence, and in particular relates to a ReRAM write latency elimination method and system based on Transformer model inference. Background Technology

[0002] The Transformer model, with its powerful generation and understanding capabilities, has become the mainstream architecture in many fields such as natural language processing and computer vision. The core mechanism of the Transformer model is self-attention, used to determine the strength of dependencies between input labels (attention score). However, the computational complexity of the attention module increases quadratically with the length of the input sequence. This scalability introduces a severe data movement bottleneck into traditional von Neumann architectures (such as GPUs), resulting in high inference latency. Unlike the von Neumann architecture, Processing-in-Memory (PIM) architecture integrates memory and processor, eliminating the need for weight movement during inference and significantly reducing system energy consumption and computational latency. PIM-based processing technology, using resistive random access memory (ReRAM), stands out due to its high density, non-volatility, fast read speed, and low energy consumption. This technology has achieved efficient in-situ computation in the analog domain, effectively eliminating the large-scale data transfer overhead involved in matrix-vector multiplication (MVM), a core Transformer operation. Therefore, exploring ReRAM-based architectures is a valuable research direction for accelerating Transformer workloads.

[0003] While many previous works have investigated ReRAM-based Transformer accelerators by focusing on mitigating constraints such as durability, capacity, and non-idealities, they often overlook a key dynamic characteristic of the attention mechanism: during autoregressive inference, keys (K) and values ​​(V) are newly generated activations, not static weights. Therefore, any ReRAM-based attention computation first requires converting these dynamic KV values ​​to conductance and writing them into the ReRAM. This fundamental KV write operation introduces a significant latency bottleneck for Transformer inference. KV writes are located on critical inference paths because attention computation can only begin after the KV of the current token has been fully written to the cross-switch. The total number of writes is very high, scaling with the number of attention layers and the length of the inference sequence. Given the relatively large write latency of ReRAM (approximately ~100 ns), these frequent KV write operations introduce extremely high latency, severely degrading inference performance. Therefore, designing optimizations to KV write latency to achieve high-performance Transformer inference is a challenge. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a ReRAM write latency elimination method and system based on Transformer model inference.

[0005] The objective of this invention is achieved through the following technical solution: This invention provides a ReRAM write latency elimination method based on Transformer model inference, employing a hybrid hardware system composed of parallel analog and digital circuits, and includes the following steps: Step 1: Obtain the structural information and input token information of each attention layer in the Transformer model as state information; Step 2: Based on the state information in Step 1, the query vector generated by the current token at each layer is transmitted to the simulation circuit, and attention is calculated with the key matrix and value matrix of all historical tokens pre-stored in the simulation circuit to obtain the first attention output; Step 3: Based on the state information in Step 1, simultaneously input the query vector, key vector, and value vector generated by the current token at each layer into the digital circuit for attention calculation to obtain the second attention output; Step 4: Send the output results from Steps 2 and 3 to the next inference module of the Transformer model, and simultaneously start KV delayed writing to write the KV of the current token into the ReRAM storing KV; the next inference module refers to starting the inference of the Feed-Forward Neural Network (FFN) module in the Transformer model after completing the attention calculation.

[0006] Furthermore, step two is implemented through the following sub-steps: (1) The query vector generated by the current token in each layer is sent into ReRAM and simulated domain matrix-vector multiplication operation is performed with the key matrix of the previous token to obtain the first attention calculation score; (2) The first attention calculation score is transmitted to the value matrix of all historical tokens stored in the analog circuit and simulated matrix-vector multiplication is performed to obtain the first attention output result.

[0007] Furthermore, step three is implemented through the following sub-steps: (1) Determine the scaling factor SF: in, , , and It is the scaling factor for ReRAM simulation calculations. and The inputs and weights are projected onto the range of resistance values ​​that ReRAM can represent, while The output of the ReRAM is converted from the analog domain to the digital domain by controlling the sampling range of the ADC. This is also the next simulation operation. The next simulation operation refers to the simulation operation in the next inference module (such as FFN).

[0008] (2) The query vector and key vector generated by the current token at each layer are transmitted to the digital circuit for digital domain vector multiplication to obtain the second attention score; (3) Input the second attention score and the value vector of the current token into the digital circuit to perform digital domain multiplication and addition operations to obtain the second attention output result.

[0009] Furthermore, step four is implemented through the following sub-steps: (1) After completing the attention calculations described in steps two and three, continue the reasoning process; (2) After the reasoning process enters the non-attention calculation module, the key vector of the current token is written into the ReRAM that stores the key vectors of all previous tokens and merged with the key vectors to facilitate the smooth progress of subsequent reasoning. (3) After the reasoning process enters the non-attention calculation module, the value vector of the current token is written into the ReRAM that stores the value vectors of all previous tokens and merged with the value vectors to facilitate the smooth progress of subsequent reasoning.

[0010] The present invention also provides a hybrid hardware system for implementing the above-described ReRAM write latency elimination method, the system comprising: The analog circuit module, including a ReRAM array, a digital-to-analog converter, an analog-to-digital converter, and a shift adder, is used to perform analog domain attention calculations based on the historical key-value matrix and supports delayed writing of the key-value vector; The digital circuit module includes a multiply-accumulator unit with a programmable scaling factor for performing digital domain attention computation based on the current token; The control and caching unit module is used to distribute input data to the analog computing module and the digital computing module, receive output data, and coordinate the parallel execution of the two modules with the delayed write operation.

[0011] Furthermore, the analog circuit module includes: a ReRAM array for storing model parameters in conductance form; a digital-to-analog converter for converting input data into voltage values ​​applied to the ReRAM array; an analog-to-digital converter for converting the analog current signal output by the ReRAM array back into a digital signal; and a shift adder for further processing the output of the analog-to-digital converter.

[0012] Furthermore, the digital circuit module includes: a multiply-accumulator with a scaling factor, two multipliers, one adder, and four registers, capable of performing multiply-accumulate operations in the digital domain; wherein, the registers include two input registers, one output register, and one scaling factor register; when the multiply-accumulator unit with the scaling factor performs the operation, its two input registers respectively store the query vector and key vector of the current token or the intermediate operation result, the scaling factor register stores the scaling factor used to calibrate the output between the analog domain and the digital domain, and the output register stores the final calculation result adjusted by the scaling factor.

[0013] The beneficial effects of this invention are as follows: This invention realizes an efficient large-model inference architecture based on ReRAM in-memory computing technology. Inspired by the characteristics of key-value increment generation in the Transformer model, it proposes the idea of ​​delayed key-value writing. Specifically, it designs and implements a digital circuit-based SMAC multiplier-accumulator integrated into the existing ReRAM analog circuit, realizing a hybrid architecture system that can perform attention calculation first and then realize key-value writing to ReRAM. Therefore, without affecting the model inference process, it can effectively eliminate the write delay on the critical path of inference and improve the system's computing performance. Attached Figure Description

[0014] Figure 1 This is a block diagram of a hybrid hardware system design, where (a) is an analog circuit module, including a ReRAM array, a digital-to-analog converter, an analog-to-digital converter, and a shift adder; (b) is a digital circuit module, including a multiply-adder with a scaling factor, two multipliers, an adder, and four registers, including two input registers, one output register, and one scaling factor register; the coordination between the two modules is handled by a global controller, which shares the cache within the computing unit.

[0015] Figure 2 It is a schematic diagram of the Transformer inference process and delayed KV write, including three sequentially executed modules: QKV Gen, MHA, and FFN. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0018] Figure 1 The diagram shows the hybrid hardware system design of this invention. The system adopts a hierarchical structure, integrating computing units into multi-level structures such as Tile and PE. Each level contains input / output buffers for temporary storage of input and output data. The entire system is controlled and managed by a global controller, which controls data flow, computation, etc., to ensure correct system operation. Each computing unit includes an analog circuit module and a digital circuit module. (a) is the analog circuit module, which includes a ReRAM array, a digital-to-analog converter, an analog-to-digital converter, and a shift adder. (b) is the digital circuit module, which includes a multiply-adder with a scaling factor, two multipliers, one adder, and four registers, including two input registers, one output register, and one scaling factor register. The coordination between the two modules is handled by the global controller, which shares the buffers within the computing unit.

[0019] This invention relates to a ReRAM write latency elimination method based on Transformer model inference, comprising the following steps: Step 1: Obtain the structural information and input token information of each attention layer in the Transformer model as state information, as shown in Table 1, which contains a total of 8 state information items.

[0020] Table 1. Correspondence between Status Information Symbols Step 2: Based on the state information in Step 1, generate tokens for each layer. Transmitted into ReRAM and the previously tokened 、 The process for attention calculation is divided into the following sub-steps.

[0021] (1) Generate tokens for each layer of the current token Transmitted into ReRAM and the previously tokened Used for attention score The calculation is as follows: (2) Transmitted into ReRAM and the previously tokened The output of the attention layer is calculated using the following formula: ReRAM structure as follows Figure 1 As shown in (a), As input, 、 As weights, analog domain matrix-vector multiplication is performed through the collaboration of DAC, ReRAM, and ADC.

[0022] Step 3: Based on the state information in Step 1, generate tokens for each layer. The data is transmitted into SMAC to realize the current token. and Attention calculation is divided into the following sub-steps.

[0023] (1) Determine the scaling factor SF: (2) Generate tokens for each layer of the current token And generated with the current token Transmitted into SMAC for attention score The calculation formula is as follows: (3) The data transmitted into SMAC and the current token The output of the attention layer is calculated using the following formula: SMAC structure as follows Figure 1 As shown in (b), As input , , As input It completes digital field vector multiplication operations through the coordinated operation of registers, multipliers, and adders.

[0024] Step 4: Combine the output results from Steps 2 and 3 The token is then sent to the next inference module (Feedforward Neural Network module FFN) of the Transformer model, and a key-value lazy write is initiated to store the current token. Write to storage 、 In the ReRAM, we obtain 、 ,like Figure 2 As shown, the specific steps are as follows: (1) After completing the attention calculations described in steps two and three, continue the reasoning process; (2) After the inference process enters the non-attention calculation module, the current token's All tokens before being written to storage In ReRAM, with Merging This facilitates smooth subsequent reasoning; (3) After the inference process enters the non-attention calculation module, the current token's All tokens before being written to storage In ReRAM, with Merging This facilitates smooth subsequent reasoning.

[0025] Figure 2 This describes the inference and delayed key-value (KV) writing process of the Transformer model, mainly consisting of three parts: QKVGen, MHA, and FFN, which are executed sequentially. One inference process includes: 1) The input token enters the QKV Gen module, generating the corresponding query (q), key (k), and value (v); 2) qkv enters the MHA module for attention calculation, as described in steps two and three, ultimately obtaining the module's output y; 3) y enters the FFN model for forward propagation network calculation, outputting a new token.

[0026] The ReRAM write latency elimination method of the present invention is implemented using a hybrid computing system composed of parallel analog and digital circuits, the system comprising: The analog circuit module, comprising a ReRAM array, a digital-to-analog converter (DAC), an analog-to-digital converter (ADC), and a shift adder, is used to perform analog domain attention calculation based on a historical key-value matrix and supports delayed writing of key-value vectors. The analog circuit module includes: a ReRAM array for storing model parameters in conductance form; a DAC for converting input data into voltage values ​​applied to the ReRAM array; an ADC for converting the analog current signal output from the ReRAM array back into a digital signal; and a shift adder for further processing the output of the ADC.

[0027] A digital circuit module includes a multiply-accumulator unit with a programmable scaling factor for performing digital domain attention calculations based on the current token. The digital circuit module comprises: a multiply-accumulator with a scaling factor, two multipliers, one adder, and four registers, capable of performing digital domain multiplication and addition operations. The registers include two input registers, one output register, and one scaling factor register. When the multiply-accumulator unit with the scaling factor performs the operation, its two input registers store the query vector and key vector of the current token, or intermediate operation results, respectively. The scaling factor register stores the scaling factor used to calibrate the analog domain and digital domain outputs, and the output register stores the final calculation result adjusted by the scaling factor.

[0028] The control and caching unit module is used to distribute input data to the analog computing module and the digital computing module, receive output data, and coordinate the parallel execution of the two modules with the delayed write operation. The effects of the present invention are further illustrated below with reference to a specific embodiment; the experimental results are shown in Table 2.

[0029] Example 1: (1) ReRAM accelerator simulator: MNSIM; (2) Model: OPT-125M standard code; (3) Dataset: WikiText2.

[0030] Test results: This experiment simulated the process of inferring a token using the OPT-125M model and compared the existing method with the method of this invention under the same experimental conditions. The existing method (Naïve) uses direct key-value (KV) writes, while the method of this invention uses delayed KV writes to eliminate write latency on critical inference paths and accelerate the inference process. This experiment measured the performance of the Transformer model OPT-125M inferring a token. Using a controlled variable approach, while maintaining a consistent experimental environment, the experiment tested the latency reduction capability of the method of this invention compared to the existing method. The MNSIM simulator was used as the simulation platform, and the system latency, area, and energy consumption indicators for both methods were output. Latency refers to the end-to-end time for the OPT-125M model to infer and output a token, calculated by the MNSIM simulation platform; a smaller latency indicates better system performance. Area refers to the total area overhead of all hardware resources used to deploy the OPT-125M model, calculated by the MNSIM simulation platform. Energy consumption refers to the total energy consumption during the OPT-125M model's inference and output of a token, calculated by multiplying power consumption by latency; power consumption and latency were calculated by the MNSIM simulation platform. Because the method of this invention adds an SMAC module to the existing ReRAM architecture, the area and power consumption will be slightly improved compared to existing methods.

[0031] The experimental results are shown in Table 2.

[0032] Table 2 Experimental test results of the present invention Compared to object A, which uses the existing direct key-value (Naïve) method, our proposed method B, which implements delayed key-value writing, achieves a significant improvement in latency performance. Specifically, object B exhibits the best latency metric, completing the inference task for a single token in just 0.57 milliseconds, representing only 78% of the inference latency of existing methods. Furthermore, method B does not show a significant difference in area and energy consumption compared to existing method A; the area increases by only 1.03 times, and the energy consumption increases by 1.0006 times, which are negligible. This demonstrates that our method can significantly improve Transformer inference performance with similar overhead.

[0033] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A ReRAM write latency elimination method based on Transformer model inference, characterized in that, The method employs a hybrid hardware system composed of parallel analog and digital circuits, including the following steps: Step 1: Obtain the structural information and input token information of each attention layer in the Transformer model as state information; Step 2: Based on the state information in Step 1, the query vector generated by the current token at each layer is transmitted to the simulation circuit, and attention is calculated with the key matrix and value matrix of all historical tokens pre-stored in the simulation circuit to obtain the first attention output; Step 3: Based on the state information in Step 1, simultaneously input the query vector, key vector, and value vector generated by the current token at each layer into the digital circuit for attention calculation to obtain the second attention output; Step 4: Transmit the output results from Steps 2 and 3 to the next inference module of the Transformer model, and simultaneously initiate delayed writing of the key vector and value vector to write the key vector and value vector of the current token into the storage array of the analog circuit. The inference module is a feedforward neural network module.

2. The ReRAM write latency elimination method according to claim 1, characterized in that, Step two is achieved through the following sub-steps: (1) The query vector generated by the current token in each layer is sent into ReRAM and simulated domain matrix-vector multiplication operation is performed with the key matrix of the previous token to obtain the first attention calculation score; (2) The first attention calculation score is transmitted to the value matrix of all historical tokens stored in the analog circuit and simulated matrix-vector multiplication is performed to obtain the first attention output result.

3. The ReRAM write latency elimination method according to claim 1, characterized in that, Step three is achieved through the following sub-steps: (1) Determine the scaling factor for calibration based on the physical characteristics of the analog circuit; (2) The query vector and key vector generated by the current token at each layer are transmitted to the digital circuit for digital domain vector multiplication to obtain the second attention score; (3) Input the second attention score and the value vector of the current token into the digital circuit to perform digital domain multiplication and addition operations to obtain the second attention output result.

4. The ReRAM write latency elimination method according to claim 1, characterized in that, Step four is achieved through the following sub-steps: (1) After completing the attention calculations described in steps two and three, continue the reasoning process; (2) After the inference process enters the non-attention calculation module, the key vector of the current token is written into the ReRAM that stores the key vectors of all previous tokens and merged with the key vector; (3) After the inference process enters the non-attention calculation module, the value vector of the current token is written into the ReRAM that stores the value vectors of all previous tokens and merged with the value vector.

5. The ReRAM write latency elimination method according to claim 3, characterized in that, The scaling factor SF is calculated using the following formula: In the formula, , , and It is the scaling factor for the simulation calculation; and The range of resistance values ​​that a ReRAM array can represent when the inputs and weights are projected onto the analog circuit is limited. The output of the ReRAM is converted from the analog domain to the digital domain by controlling the sampling range of the analog-to-digital converter; This is also the next simulation operation. The next simulation operation refers to the simulation operation in the next inference module.

6. A hybrid hardware system for implementing the ReRAM write latency elimination method according to any one of claims 1-5, characterized in that, include: The analog circuit module, including a ReRAM array, a digital-to-analog converter, an analog-to-digital converter, and a shift adder, is used to perform analog domain attention calculations based on the historical key-value matrix and supports delayed writing of the key-value vector; The digital circuit module includes a multiply-accumulator unit with a programmable scaling factor for performing digital domain attention computation based on the current token; The control and caching unit module is used to distribute input data to the analog computing module and the digital computing module, receive output data, and coordinate the parallel execution of the two modules with the delayed write operation.

7. The hybrid hardware system according to claim 6, characterized in that, The analog circuit module includes: ReRAM array, used to store model parameters in the form of conductivity; Digital-to-analog converter: used to convert input data into voltage values ​​that are applied to the ReRAM array; Analog-to-digital converter: used to convert the analog current signal output by the ReRAM array back into a digital signal; Shift adder: Used for further processing of the output of the analog-to-digital converter.

8. The hybrid hardware system according to claim 6, characterized in that, The digital circuit module includes: a multiply-accumulator with a scaling factor, two multipliers, one adder, and four registers, capable of performing multiply-accumulate operations in the digital domain; wherein, the registers include two input registers, one output register, and one scaling factor register; when the multiply-accumulator unit with the scaling factor performs the operation, its two input registers store the query vector and key vector of the current token or the intermediate operation result, respectively, the scaling factor register stores the scaling factor used to calibrate the output between the analog domain and the digital domain, and the output register stores the final calculation result adjusted by the scaling factor.

Images