GPU-reconfigurable RRAM in-memory computing device collaborative reasoning acceleration method and system

Abstract

The invention provides a collaborative reasoning acceleration system and method for computing equipment in a GPU-reconfigurable RRAM, and the system comprises a GPU module which is used for generating a key matrix K and a value matrix V based on the token features of an input prompt sequence through a GPU in a pre-filling stage, and constructing an initial KV cache; after the pre-filling stage is finished, the KV cache is transmitted to the RRAM-PIM equipment in batches by taking a decoder block as a unit through the GPU; the processor is also used for generating a query vector q, a key vector k and a value vector v based on hidden layer features of the token generated at the previous moment through the GPU in the decoding stage, and writing the newly generated key vector k and value vector v increment into the KV cache; receiving an attention context output by the RRAM-PIM module, and generating a token at the next moment after the attention context is projected by a decoder and processed by a feedforward network; and the RRAM-PIM module is used for realizing persistent storage of the KV cache, executing Scoring operation and Context operation on the KV cache to generate an attention context based on the query vector q generated by the GPU module in multi-head attention processing in a decoding stage, and feeding back the attention context to the GPU module.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence acceleration and in-memory computing technology, specifically to a method and system for accelerating collaborative inference in GPU-reconfigurable RRAM in-memory computing devices; more specifically, to a method for accelerating key-value cache (KV cache) offloading and collaborative inference for large-scale Transformer-type language model inference processes. Background Technology

[0002] With the widespread deployment of large language models in dialogue systems, code generation, image generation, and other applications, the scale of model parameters and the length of context are continuously increasing, leading to a sharp increase in computational and storage overhead during the inference phase. In mainstream GPU-based inference architectures, multi-head attention mechanisms perform context matching and retrieval through query (Q), key (K), and value (V) matrices. To avoid redundant computation, the system continuously saves the K and V matrices of each layer and head throughout the inference process; these persistent key-value data are collectively referred to as key-value caches (KV caches). Existing research shows that KV caches account for 30% to 80% of the total GPU memory in various large model scenarios, and this figure increases linearly with sequence length, becoming one of the key bottlenecks limiting the inference capabilities of large models with long contexts. To alleviate memory pressure, industry and academia commonly adopt KV cache offloading schemes and stage-based inference schemes. However, it should be noted that both of these schemes only solve part of the problem. Offloading schemes mainly address memory usage issues, but the KV data offloaded to external storage is passively stored in most systems until it is retrieved and processed during task rescheduling. The phase-based approach primarily aims to improve GPU utilization, but it doesn't fundamentally eliminate the problems of low arithmetic intensity and poor key-value (KV) processing efficiency in the decoding phase. On the other hand, Processing-In-Memory (PIM) technology can perform matrix-vector operations and other operators within the storage array, combining storage functionality with some computational power. Existing solutions show that in bandwidth-sensitive or low-arithmetic-intensity scenarios, PIM has the potential to significantly reduce data transfer overhead and improve energy efficiency. However, current technologies lack a system and method that simultaneously addresses KV cache offloading and processing acceleration, providing large-capacity, non-volatile KV persistence for large models, accelerating low-arithmetic-intensity, batch-hard KV-related operations in the decoding phase, and maintaining good integration with existing GPU inference and offloading systems. Summary of the Invention

[0003] To address the shortcomings of existing technologies, the purpose of this invention is to provide a GPU-reconfigurable RRAM in-memory computing device collaborative inference acceleration system and method.

[0004] A GPU-reconfigurable RRAM in-memory computing device co-inference acceleration system according to the present invention includes: The GPU module is used to generate a key matrix K and a value matrix V based on the token features of the input prompt sequence during the pre-filling stage, and to build an initial KV cache. After the pre-filling stage, the GPU transmits the KV cache to the RRAM-PIM device in batches according to decoder blocks. At the same time, it is also used to generate a query vector q, a key vector k, and a value vector v based on the hidden layer features of the token generated in the previous time step during the decoding stage, and incrementally write the newly generated key vector k and value vector v into the KV cache. It receives the attention context output by the RRAM-PIM module, and generates the token for the next time step after decoding projection and feedforward network processing. The RRAM-PIM module is used to implement persistent storage of the KV cache. In the multi-head attention processing during the decoding stage, based on the query vector q generated by the GPU module, it performs Scoring and Context operations on the KV cache to generate an attention context, and feeds it back to the GPU module.

[0005] Preferably, after the pre-filling stage is completed, the step of transmitting the KV cache to the RRAM-PIM device in batches by decoder block via the GPU includes: The GPU splits the key matrix K and value matrix V of the current decoder block into block matrices according to the attention head; The K / V block matrix is ​​transferred in parallel to the shared buffer of the RRAM-PIM device via DMA. The RRAM subprocessor maps the received K / V matrix to the specified RRAM cross array according to the attention head, thus completing the persistent storage of the KV cache.

[0006] Preferably, the step of performing Scoring and Context operations on the KV cache to generate an attention context based on the query vector q generated by the GPU module includes: The GPU generates the query vector q for the current step and transmits the query vector q for the current step to the RRAM-PIM device; The RRAM-PIM device uses the K matrix stored in the RRAM cross array to perform Scoring calculations in memory and obtain the Scoring results. The Softmax normalization result is obtained by performing Softmax normalization on the Scoring operation result through the RRAM subprocessor; RRAM-PIM devices utilize the V matrix stored in the RRAM cross array to perform Context operations and generate attention contexts within memory; The Scoring operation includes: q×K Where T is the transpose; the Context operation includes: S×V; where S is the expression for q×K. The result of performing Softmax normalization.

[0007] Preferably, the pre-filling stage includes: After the GPU generates the K matrix of the current decoder block, it transmits the K matrix to the RRAM-PIM device. During the transmission, the GPU triggers two operations in parallel: generating the V matrix for the same decoder block; and controlling the RRAM-PIM device to start the Scoring operation on the transmitted K matrix. After the GPU generates the V matrix, it triggers two operations in parallel: triggering the V matrix transfer operation; and simultaneously controlling the RRAM-PIM device to start the Context operation on the transferred V matrix.

[0008] Preferably, the decoding stage includes: The batch query operation for transferring vector q and key vector k overlaps with the operation for generating value vector v. The transfer operation of value vector v and q×K Dot product operations are performed concurrently. The completed v vector is appended to the tail of the V matrix in RRAM in a pipeline manner, using a circular buffer for management.

[0009] Preferably, the system further includes: The inference batch is dynamically divided into two sub-batches: Sub-Batch A and Sub-Batch B; In the first time period: the GPU processes the pre-filling calculations for Sub-Batch A; In the second time period: the GPU processes the pre-filling calculations for Sub-Batch B, while the RRAM-PIM device processes the decoding calculations for Sub-Batch A.

[0010] Preferably, the step of dynamically dividing the inference batch into two sub-batches includes: Monitor the Q / K / V matrix size of recent workloads that meet preset requirements; Estimate the ratio of GPU to RRAM-PIM computational throughput based on matrix size; Sub-batch sizes are allocated according to the stated ratio.

[0011] A method for accelerating co-inference of GPU-reconfigurable RRAM in-memory computing devices according to the present invention includes: Step S1: In the pre-filling stage, the GPU generates a key matrix K and a value matrix V based on the token features of the input prompt sequence to build an initial KV cache; after the pre-filling stage is completed, the GPU transmits the KV cache to the RRAM-PIM device in batches according to decoder blocks. Step S2: During the decoding stage, the GPU generates a query vector q, a key vector k, and a value vector v based on the hidden layer features of the token generated in the previous time step, and incrementally writes the newly generated key vector k and value vector v into the KV cache. Step S3: In the multi-head attention processing during the decoding stage, based on the generated query vector q, the Scoring operation and Context operation are performed on the KV cache to generate an attention context; Step S4: The attention context is projected by the decoder and processed by the feedforward network to generate the token for the next time step.

[0012] Preferably, after the pre-filling stage is completed, the step of transmitting the KV cache to the RRAM-PIM device in batches by decoder block via the GPU includes: The GPU splits the key matrix K and value matrix V of the current decoder block into block matrices according to the attention head; The K / V block matrix is ​​transferred in parallel to the shared buffer of the RRAM-PIM device via DMA. The RRAM subprocessor maps the received K / V matrix to the specified RRAM cross array according to the attention head, thus completing the persistent storage of the KV cache; The step of performing Scoring and Context operations on the KV cache based on the generated query vector q to generate an attention context includes: The GPU generates the query vector q for the current step and transmits the query vector q for the current step to the RRAM-PIM device; The RRAM-PIM device uses the K matrix stored in the RRAM cross array to perform Scoring calculations in memory and obtain the Scoring results. The Softmax normalization result is obtained by performing Softmax normalization on the Scoring operation result through the RRAM subprocessor; RRAM-PIM devices utilize the V matrix stored in the RRAM cross array to perform Context operations and generate attention contexts within memory; The Scoring operation includes: q×K Where T is the transpose; the Context operation includes: S×V; where S is the expression for q×K. The result of performing Softmax normalization.

[0013] Preferably, the method further includes: After the GPU generates the K matrix of the current decoder block, it transmits the K matrix to the RRAM-PIM device. During the transmission, the GPU triggers two operations in parallel: generating the V matrix for the same decoder block; and controlling the RRAM-PIM device to start the Scoring operation on the transmitted K matrix. After the GPU generates the V matrix, it triggers two operations in parallel: triggering the transfer operation of the V matrix; and simultaneously controlling the RRAM-PIM device to start the Context operation on the transferred V matrix. The batch query operation for transferring vector q and key vector k overlaps with the operation for generating value vector v. The transfer operation of value vector v and q×K Dot product operations are performed concurrently. The completed v vector is appended to the tail of the V matrix in RRAM in a pipeline manner, using a circular buffer for management.

[0014] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention significantly reduces GPU memory usage by offloading the KV cache to high-capacity, non-volatile RRAM-PIM storage, supporting longer contexts and more concurrent requests; at the same time, it completes Scoring and Context operations inside the PIM array, avoiding frequent transfer of the full KV matrix between the GPU and external storage, thus achieving a balance between cache persistence and computational acceleration. 2. This invention overlaps sub-batch pipelines with data transmission, enabling the GPU and PIM to work simultaneously for most of the time, reducing the idle period in the decoding stage; when combined with a typical KV offloading system, it can achieve parallelization of the offloading process and KV processing without modifying its eviction and backfilling strategies, further shortening end-to-end latency. 3. Compared with solutions that only use GPUs, the present invention can achieve a token generation performance improvement of approximately 1.5 to 6.5 times and an energy efficiency improvement of approximately 8 to 10 times in long context scenarios. Attached Figure Description

[0015] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the inference system structure of the present invention.

[0016] Figure 2 This is a schematic diagram of the KV buffer transmission-computation overlap technology of the present invention. Detailed Implementation

[0017] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0018] Example 1 According to the present invention, a deep learning large model KV cache offloading and collaborative computing system based on a reconfigurable RRAM in-memory computing architecture is provided, such as... Figure 1 As shown, it includes: The GPU is used to perform all computational tasks in the pre-filling stage, as well as batch-processable QKV generation, projection, and feedforward network operations in the decoding stage. The reconfigurable RRAM-PIM device interconnects with the GPU via a high-speed bus and includes multiple independently addressable RRAM crossbar switch arrays and bonded subprocessor units. It is used to persistently store the KV cache generated during the pre-filling stage and perform multi-head attention calculations during the decoding stage, including scoring (softmax(Q×K)). Operations with context(S×V) are performed.

[0019] The collaborative scheduling module is used to implement the following operations: The KV cache generated in the pre-filling stage is offloaded to the RRAM-PIM device in units of decoder blocks; During the decoding phase, the Scoring operation (Softmax(Q×K)) is performed by the RRAM-PIM device. )) and Context operations (S×V); By using a sub-batch inference task pipeline and KV cache transfer-computation overlap technology, parallel collaborative acceleration of GPU and RRAM-PIM devices is achieved.

[0020] Specifically, the step of offloading the KV cache generated in the pre-filling stage to the RRAM-PIM device in units of decoder blocks includes: The GPU splits the key matrix (K) and value matrix (V) into block matrices according to the attention head; The block matrix is ​​transferred in parallel to the shared buffer of the RRAM-PIM device via DMA. The RRAM subprocessor maps the K / V matrix to a specified RRAM cross array for persistent storage.

[0021] The multi-head attention calculation during the decoding phase includes: Read the K matrix from the RRAM array and perform q×K operations in memory. Dot product operation; Softmax normalization is performed via a subprocessor; Read matrix V from the RRAM array and perform in-memory calculation of the Softmax result × V.

[0022] Specifically, the KV buffer transmission-computation overlap technique is implemented in the pre-filling stage as follows: The transfer operation of the key matrix (K) is performed concurrently with the generation operation of the value matrix (V) and the scoring operation. The transfer operation of the value matrix (V) overlaps with the Context operation; The KV matrix is ​​transmitted in decoder blocks, and matrices from different attention heads are stored in parallel by designated subprocessors.

[0023] Specifically, the KV buffer transmission-computation overlap technique is implemented in the decoding stage as follows: The batch query operation for transferring vectors (q) and key vectors (k) overlaps with the operation for generating value vectors (v); The transfer operation of the value vector (v) and q×K Dot product operations are performed concurrently. The completed v vector is appended to the tail of the V matrix in RRAM in a pipeline manner, using a circular buffer for management.

[0024] The sub-batch inference task pipeline includes: The inference batch is dynamically divided into two sub-batches: Sub-Batch A and Sub-Batch B; In the first time period: the GPU processes the pre-filling calculations for Sub-Batch A; In the second time period: the GPU processes the pre-filling calculations for Sub-Batch B, while RRAM-PIM processes the decoding calculations for Sub-Batch A.

[0025] The sub-batch division is dynamically determined through the following steps: Monitor the size of the Q / K / V matrix for recent workloads; Estimate the ratio of GPU to RRAM-PIM computational throughput based on matrix size; Sub-batch sizes are allocated according to the stated ratio.

[0026] Example 2 Example 2 is a preferred example of Example 1. This invention provides a high-performance offloading and collaborative inference acceleration method for large deep learning models based on a reconfigurable RRAM in-memory computing architecture. Addressing the problems of high KV cache memory consumption, large offloading overhead, and low GPU utilization during the decoding stage in existing large model inference methods, this invention leverages reconfigurable RRAM in-memory computing technology and KV cache offloading methods. It proposes a GPU-PIM heterogeneous inference acceleration method, which fully exploits the parallel potential between GPU and RRAM PIM devices through a pipelined collaborative acceleration design of prefill, decoding, and their sub-stage tasks, thereby improving end-to-end inference speed, throughput, and energy efficiency.

[0027] Specifically, the method for high-performance offloading and collaborative inference acceleration of deep learning large model KV cache based on reconfigurable RRAM in-memory computing architecture includes: The KV cache offloading and processing method based on RRAM PIM includes: offloading the KV cache generated in the pre-filling stage to the RRAM in-memory computing device, realizing efficient transfer of KV cache and high utilization of GPU and RRAM PIM device, and finally achieving collaborative acceleration of inference.

[0028] More specifically, the entire inference system consists of a GPU and a reconfigurable RRAM-PIM device. The GPU is responsible for all computations in the pre-filling phase and for batch-processable QKV generation, projection, and feedforward network operations in the decoding phase. After the pre-filling phase, the GPU offloads the KV cache to the RRAM-PIM device. The RRAM-PIM device is responsible for persisting the KV cache and for scoring the KV cache (softmax(Q×K)) during multi-head attention processing in the decoding phase. Operations with context(S×V) are performed.

[0029] A pipelined collaborative inference acceleration method for GPU and RRAM PIM devices includes: considering that KV cache offloading in the above manner means that the inference task is jointly completed by the GPU and RRAM in-memory computing devices, the utilization rate of the GPU and RRAM in-memory computing devices will decrease to a certain extent. To address this issue, this invention proposes a pipelined collaborative acceleration method for GPU and RRAM PIM devices, which mainly includes: 1) sub-batch inference task pipeline; 2) two key technologies: KV cache transfer and computation overlap. The sub-batch inference task pipeline comprises the following: The key to the sub-batch inference task pipeline is dividing an inference batch into two parts. These two batches are cross-scheduled to be executed on the GPU and RRAM PIM devices, thus achieving a batch-level inference task pipeline. The size of the sub-batch is dynamically estimated based on the matrix operation scale of the recent workload to determine the computing power of the GPU and PIM. This technique ensures high device utilization of the GPU and RRAM PIM devices within the same inference batch, which is beneficial for improving system throughput and optimizing inference speed.

[0030] The KV cache transfer-computation overlap involves the following: the KV cache size of large deep learning models is considerable; for general-purpose buses such as PCIe, the overhead of KV cache transfer between the GPU and RRAM PIM becomes significant, and it will have a substantial impact on the overall inference performance of the system. To solve this problem, this invention proposes a KV cache transfer-computation overlap technique, such as... Figure 2 As shown, the invention aims to mask the overhead of KV buffer transmission by using a "compute-while-transmit" approach. Specifically, the invention achieves KV buffer transmission-computation overlap in both the pre-filling and decoding stages.

[0031] During the pre-filling phase, the method overlaps the transfer of key (K) and value (V) matrices from the KV cache with GPU computational operations. Specifically, the transfer of the key (K) matrix is ​​overlapped with scoring operations and value (V) matrix generation operations. After the value (V) matrix is ​​generated, its transfer is overlapped with context operations. The KV matrices in the pre-filling phase are transferred and persisted by decoder blocks. Matrices from different attention heads are persisted in parallel by designated PIM subprocessors. Computational operations on other subprocessors are unaffected during the transfer of the KV matrix for each decoder block.

[0032] In the decoding phase, the method also overlaps the batch query (q), key (k), and value (v) vector transmission operations from the decoding phase. The batch q and k vector transmission operations are overlapped with the v vector generation operation, and the v vector transmission operation is also overlapped with... The dot product operation is performed concurrently. The vector v is appended to the matrix V in a pipelined manner for storage.

[0033] Through the above system structure, data mapping, and pipelined scheduling, this invention achieves joint acceleration of KV cache unloading and processing while keeping the existing GPU inference framework and unloading strategy basically unchanged.

[0034] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0035] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A GPU-reconfigurable RRAM in-memory computing device collaborative inference acceleration system, characterized in that, include: The GPU module is used to generate a key matrix K and a value matrix V based on the token features of the input prompt sequence during the pre-filling stage, and to build an initial KV cache. After the pre-filling stage is completed, the KV cache is transmitted to the RRAM-PIM device in batches by the GPU in units of decoder blocks; at the same time, it is also used to generate query vector q, key vector k and value vector v by the GPU based on the hidden layer features of the token generated in the previous time step during the decoding stage, and the newly generated key vector k and value vector v are incrementally written into the KV cache. The attention context output by the RRAM-PIM module is received, and after being projected by the decoder and processed by the feedforward network, the token for the next time step is generated. The RRAM-PIM module is used to implement persistent storage of the KV cache. In the multi-head attention processing during the decoding stage, based on the query vector q generated by the GPU module, it performs Scoring and Context operations on the KV cache to generate an attention context, and feeds it back to the GPU module.

2. The GPU-Reconfigurable RRAM In-Memory Computing Device Co-Inference Acceleration System according to claim 1, characterized in that, After the pre-filling phase is completed, the KV cache is transferred to the RRAM-PIM device in batches by decoder block via the GPU, including: The GPU splits the key matrix K and value matrix V of the current decoder block into block matrices according to the attention head; The K / V block matrix is ​​transferred in parallel to the shared buffer of the RRAM-PIM device via DMA. The RRAM subprocessor maps the received K / V matrix to the specified RRAM cross array according to the attention head, thus completing the persistent storage of the KV cache.

3. The GPU-Reconfigurable RRAM In-Memory Computing Device Co-Inference Acceleration System according to claim 1, characterized in that, The query vector q generated by the GPU module is used to perform Scoring and Context operations on the KV cache to generate an attention context, including: The GPU generates the query vector q for the current step and transmits the query vector q for the current step to the RRAM-PIM device; The RRAM-PIM device uses the K matrix stored in the RRAM cross array to perform Scoring calculations in memory and obtain the Scoring results. The Softmax normalization result is obtained by performing Softmax normalization on the Scoring operation result through the RRAM subprocessor; RRAM-PIM devices utilize the V matrix stored in the RRAM cross array to perform Context operations and generate attention contexts within memory; The Scoring operation includes: q×K Where T is the transpose; the Context operation includes: S×V; where S is the expression for q×K. The result of performing Softmax normalization.

4. The GPU-Reconfigurable RRAM In-Memory Computing Device Co-Inference Acceleration System according to claim 1, characterized in that, The pre-filling stage includes: After the GPU generates the K matrix of the current decoder block, it transmits the K matrix to the RRAM-PIM device. During the transmission, the GPU triggers two operations in parallel: generating the V matrix for the same decoder block; and controlling the RRAM-PIM device to start the Scoring operation on the transmitted K matrix. After the GPU generates the V matrix, it triggers two operations in parallel: triggering the V matrix transfer operation; and simultaneously controlling the RRAM-PIM device to start the Context operation on the transferred V matrix.

5. The GPU-Reconfigurable RRAM In-Memory Computing Device Co-Inference Acceleration System according to claim 1, characterized in that, The decoding stage includes: The batch query operation for transferring vector q and key vector k overlaps with the operation for generating value vector v. The transfer operation of value vector v and q×K Dot product operations are performed concurrently. The completed v vector is appended to the tail of the V matrix in RRAM in a pipeline manner, using a circular buffer for management.

6. The GPU-Reconfigurable RRAM In-Memory Computing Device Co-Inference Acceleration System according to claim 1, characterized in that, The system also includes: The inference batch is dynamically divided into two sub-batches: Sub-Batch A and Sub-Batch B; In the first time period: the GPU processes the pre-filling calculations for Sub-Batch A; In the second time period: the GPU processes the pre-filling calculations for Sub-Batch B, while the RRAM-PIM device processes the decoding calculations for Sub-Batch A.

7. The GPU-Reconfigurable RRAM In-Memory Computing Device Co-Inference Acceleration System according to claim 6, characterized in that, The dynamic division of the reasoning batch into two sub-batches includes: Monitor the Q / K / V matrix size of recent workloads that meet preset requirements; Estimate the ratio of GPU to RRAM-PIM computational throughput based on matrix size; Sub-batch sizes are allocated according to the stated ratio.

8. A method for accelerating co-inference of GPU-reconfigurable RRAM in-memory computing devices, characterized in that, include Step S1: In the pre-filling stage, the GPU generates a key matrix K and a value matrix V based on the token features of the input prompt sequence to build an initial KV cache; After the pre-filling phase is completed, the KV cache is transmitted to the RRAM-PIM device in batches by the GPU, in units of decoder blocks; Step S2: During the decoding stage, the GPU generates a query vector q, a key vector k, and a value vector v based on the hidden layer features of the token generated in the previous time step, and incrementally writes the newly generated key vector k and value vector v into the KV cache. Step S3: In the multi-head attention processing during the decoding stage, based on the generated query vector q, the Scoring operation and Context operation are performed on the KV cache to generate an attention context; Step S4: The attention context is projected by the decoder and processed by the feedforward network to generate the token for the next time step.

9. The GPU-Reconfigurable RRAM In-Memory Computing Device Co-Inference Acceleration Method according to claim 8, characterized in that, After the pre-filling phase is completed, the KV cache is transferred to the RRAM-PIM device in batches by decoder block via the GPU, including: The GPU splits the key matrix K and value matrix V of the current decoder block into block matrices according to the attention head; The K / V block matrix is ​​transferred in parallel to the shared buffer of the RRAM-PIM device via DMA. The RRAM subprocessor maps the received K / V matrix to the specified RRAM cross array according to the attention head, thus completing the persistent storage of the KV cache; The step of performing Scoring and Context operations on the KV cache based on the generated query vector q to generate an attention context includes: The GPU generates the query vector q for the current step and transmits the query vector q for the current step to the RRAM-PIM device; The RRAM-PIM device uses the K matrix stored in the RRAM cross array to perform Scoring calculations in memory and obtain the Scoring results. The Softmax normalization result is obtained by performing Softmax normalization on the Scoring operation result through the RRAM subprocessor; RRAM-PIM devices utilize the V matrix stored in the RRAM cross array to perform Context operations and generate attention contexts within memory; The Scoring operation includes: q×K Where T is the transpose; the Context operation includes: S×V; where S is the expression for q×K. The result of performing Softmax normalization.

10. The GPU-Reconfigurable RRAM In-Memory Computing Device Co-Inference Acceleration Method according to claim 8, characterized in that, The method further includes: After the GPU generates the K matrix of the current decoder block, it transmits the K matrix to the RRAM-PIM device. During the transmission, the GPU triggers two operations in parallel: generating the V matrix for the same decoder block; and controlling the RRAM-PIM device to start the Scoring operation on the transmitted K matrix. After the GPU generates the V matrix, it triggers two operations in parallel: triggering the transfer operation of the V matrix; and simultaneously controlling the RRAM-PIM device to start the Context operation on the transferred V matrix. The batch query operation for transferring vector q and key vector k overlaps with the operation for generating value vector v. The transfer operation of value vector v and q×K Dot product operations are performed concurrently. The completed v vector is appended to the tail of the V matrix in RRAM in a pipeline manner, using a circular buffer for management.

Images