GPU storage and calculation integrated pre-filling decoding separation large model reasoning acceleration method and system

Abstract

The invention provides a GPU storage and calculation integrated pre-filling decoding separation large model reasoning acceleration method and system, and the method comprises the steps: obtaining an original text, carrying out the operation of the original text through a GPU, and generating a corresponding key value cache and a query vector; based on a storage and calculation integrated memory, respectively reading the key vector, the numerical vector and the query vector from the key value cache for calculation, and generating a context vector; the method comprises the following steps: acquiring a plurality of request sequences, performing dynamic scheduling based on a preset time window or a request quantity threshold value, and sorting according to sequence lengths; carrying out zero-filling packaging on the input sequence by utilizing a GPU (Graphic Processing Unit) and executing pre-filling calculation to obtain an initial key value cache and a query vector; and respectively conveying the key value cache and the query vector to a storage area and a ready queue.

Description

A GPU-in-memory computing, pre-filling, decoding, and large-model inference acceleration method and system Technical Field

[0001] This invention relates to the field of artificial intelligence acceleration and in-memory computing technology, specifically to a GPU in-memory computing pre-filling decoding separation large model inference acceleration method and system. Background Technology

[0002] The inference process of generative large language models can generally be divided into two main stages: the prefill stage and the decoding stage. In the prefill stage, a large amount of user input is processed by the model's encoding layer or the first few layers of the network to generate corresponding text representations for subsequent generation computation in the decoding stage. Because this stage requires parallel processing of a large number of input samples, the operators are mainly large-scale matrix multiplications, making it computationally intensive overall. In the decoding stage, each output token needs to be recursively calculated based on the previous token and combined with intermediate states cached during the complete generation sequence (e.g., key-value cache). This stage involves smaller matrix sizes and lower parallelism per step, but requires long-term retention and continuous updating of cached data, thus exhibiting storage-intensive behavior.

[0003] To meet the memory requirements of the decoding stage, the industry typically deploys large language model inference systems using multiple GPUs or optimizes cache management strategies to alleviate memory pressure. For example, Chinese patent application CN202411224816.2 discloses a partially fixed caching method and system for large model inference. This method converts the input sequence into a large model input matrix during the pre-filling stage and dynamically determines whether to cache the K and V matrices during the decoding stage based on factors such as the device's concurrent execution capacity and loading time, thereby reducing memory load through optimized caching strategies. However, although such strategy optimizations or hardware stacking solutions alleviate memory capacity limitations to some extent, the computational resource utilization of systems based on traditional architectures (such as GPUs) remains low during the decoding stage due to its inherently low arithmetic strength and limited parallelism, making it difficult to achieve breakthrough improvements in overall performance and energy efficiency.

[0004] On the other hand, Processing-In-Memory (PIM) technology can directly execute operators such as matrix-vector operations within the memory array, providing both storage and computational capabilities. Existing DRAM-PIM or ReRAM-PIM solutions demonstrate that, in bandwidth-sensitive or low-arithmetic-intensity scenarios, PIM has the potential to significantly reduce data transfer overhead and improve system energy efficiency. However, existing PIM solutions still suffer from limitations such as limited array parallelism, excessively large area occupied by high-precision analog-to-digital converters (ADCs), limited overall available storage capacity, and limited ability to handle high-parallelism tasks. These limitations make it difficult to simultaneously cover the computationally intensive tasks of the large language model pre-filling stage and the storage-intensive tasks of the decoding stage. Therefore, current technology lacks a method and system for accelerating large model inference that can simultaneously address the load characteristics of the pre-filling and decoding stages within a unified system and fully improve the utilization of computational and storage resources. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this invention is to provide a GPU-in-memory computing pre-filling decoding separation large model inference acceleration method and system.

[0006] According to one aspect of the present invention, a GPU-in-memory computing pre-filling decoding separation large model inference acceleration method includes: step S1: obtaining the original text, performing calculations on the original text using the GPU to generate corresponding key-value cache and query vector; step S2: based on the in-memory computing memory, reading the key vector, numerical vector and query vector from the key-value cache respectively and performing calculations to generate a context vector.

[0007] Preferably, step S1 includes: sub-step S1.1: obtaining multiple request sequences; sub-step S1.2: dynamically scheduling multiple request sequences based on a preset time window or request quantity threshold; sub-step S1.3: sorting multiple request sequences in descending order based on sequence length; sub-step S1.4: using the GPU to zero-padded the input sequence of the user request, generating a continuous tensor and performing pre-padded calculation to obtain an initial key-value cache and query vector; sub-step S1.5: sending the key-value cache to the storage area of ​​the in-memory computing memory and sending the query vector to the ready queue of the in-memory computing memory.

[0008] Preferably, sub-step S1.5 includes: sub-step S1.5.1: dividing the key-value cache into multiple fixed-size memory page units, and recording the topological mapping relationship between each inference request identifier and the physical storage address of the memory page through a logical block mapping table; sub-step S1.5.2: using a direct memory access controller, asynchronously pushing the generated key matrix and value matrix from the GPU to the shared buffer of the in-memory computing device in parallel; sub-step S1.5.3: parsing the logical topology of the key matrix and value matrix in the shared buffer, mapping the matrix data according to the attention head dimension and writing it into the specified in-memory computing memory array, thereby realizing the alignment and persistent storage of key-value data in the physical storage unit.

[0009] Preferably, step S2 includes: sub-step S2.1: inserting the query vectors corresponding to each request in the ready queue into the dynamic decoding slots according to priority scores; sub-step S2.2: reading the key matrix from the array, performing a dot product operation between the key matrix and the target query vector in the dynamic decoding slot to obtain a similarity score; sub-step S2.3: performing Softmax normalization on the similarity score to obtain dynamic attention weight coefficients; sub-step S2.4: reading the numerical matrix, multiplying the dynamic attention weight coefficients with the numerical matrix to achieve weighted summation and generate a context vector.

[0010] Preferably, the formula for calculating the priority score in sub-step S2.1 is as follows:

[0011] Where t is the waiting time of the query vector corresponding to the request in the ready queue. This indicates the size of the key-value cache corresponding to the request. This represents a weighting factor indicating the request waiting time, used to adjust the degree to which the waiting time of a request in the queue affects its priority. This represents the weighting coefficient of the KV cache size factor, used to adjust the impact of the waiting time of a request in the queue on its priority. Its actual value is manually preset based on the optimization target.

[0012] Preferably, step S2.2 specifically involves: aggregating multiple query vectors in the dynamic decoding slot into a query matrix, reading the key matrix from the array, performing a dot product operation between the key matrix and the target query matrix, and obtaining a similarity score.

[0013] According to another aspect of the present invention, a GPU-in-memory computing pre-filled decoding separation large model inference acceleration system includes: module M1: acquiring the original text, performing calculations on the original text using the GPU, and generating corresponding key-value caches and query vectors; module M2: based on the in-memory computing memory, reading the key vector, numerical vector, and query vector from the key-value cache respectively, performing calculations, and generating a context vector.

[0014] Preferably, module M1 includes: submodule M1.1: acquiring multiple request sequences; submodule M1.2: dynamically scheduling multiple request sequences based on a preset time window or request quantity threshold; submodule M1.3: sorting multiple request sequences in descending order based on sequence length; submodule M1.4: using the GPU to zero-padded the input sequence of the user request, generating continuous tensors and performing pre-padded calculations to obtain an initial key-value cache and query vector; submodule M1.5: sending the key-value cache to the storage area of ​​the in-memory computing memory and sending the query vector to the ready queue of the in-memory computing memory.

[0015] Preferably, the submodule M1.5 includes: submodule M1.5.1: dividing the key-value cache into multiple fixed-size memory page units, and recording the topological mapping relationship between each inference request identifier and the physical storage address of the memory page through a logical block mapping table; submodule M1.5.2: using a direct memory access controller, asynchronously pushing the generated key matrix and value matrix from the GPU to the shared buffer of the in-memory computing device in parallel; submodule M1.5.3: parsing the logical topology of the key matrix and value matrix in the shared buffer, mapping the matrix data according to the attention head dimension and writing it into the specified in-memory computing memory array, thereby realizing the alignment and persistent storage of key-value data in the physical storage unit.

[0016] Preferably, module M2 includes: submodule M2.1: inserting the query vectors corresponding to each request in the ready queue into the dynamic decoding slots according to priority scores; submodule M2.2: reading the key matrix from the array, performing a dot product operation between the key matrix and the target query vector in the dynamic decoding slot to obtain a similarity score; submodule M2.3: performing Softmax normalization on the similarity score to obtain dynamic attention weight coefficients; submodule M2.4: reading the numerical matrix, multiplying the dynamic attention weight coefficients with the numerical matrix to achieve weighted summation and generate a context vector.

[0017] Compared with existing technologies, this invention has the following beneficial effects: By decoupling pre-filling and decoding tasks, this invention achieves optimal mapping between computationally intensive and memory-intensive tasks on GPUs and PIM devices. In the pre-filling stage, a large-batch packaging strategy maximizes the parallel computing potential of the GPU, significantly improving the system's effective throughput. In the decoding stage, the organic combination of continuous batch processing and priority scheduling not only ensures that the PIM device always operates under high load but also precisely addresses performance fluctuations under multi-user concurrency, significantly reducing tail latency for high-priority services.

[0018] At the storage architecture level, the paging management mechanism introduced in this invention completely breaks through the fragmentation bottleneck caused by traditional GPU memory allocation, fundamentally improving space utilization efficiency. Combined with the persistent characteristics of non-volatile ReRAM, the KV cache can reside and be reused efficiently. This not only significantly reduces GPU memory usage by 30% to 80%, but also fundamentally alleviates the redundant GPU memory pressure during ultra-large-scale model inference. This heterogeneous hybrid architecture improves hardware resource utilization while building a low-latency, low-power, and highly scalable deep learning inference environment, providing a highly competitive technical path for the large-scale industrial deployment of large models. Attached Figure Description

[0019] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 is a flowchart of a GPU in-memory computing pre-filled decoding separation large model inference acceleration method. Detailed Implementation

[0020] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0021] Example 1: A GPU-in-memory computing pre-filling and decoding separation method for accelerating large model inference. This method uses the collaborative work of heterogeneous computing architectures to offload the computationally intensive pre-filling stage of large model inference to the GPU, while offloading the storage-intensive decoding stage to the PIM device, thereby achieving load balancing and energy efficiency optimization of computing resources.

[0022] Step S1: Obtain the original text, perform calculations on the original text using the GPU, and generate the corresponding key-value cache and query vector.

[0023] Based on the above scheme, the powerful parallel computing capabilities of the GPU are used to handle the high-load task of the prefill stage. In this stage, the input is a complete prompt. The GPU can use its numerous CUDA cores to compute the key-value matrix of all tokens in the input sequence in parallel and generate the query vector of the first token, establishing the initial state for the subsequent generation process.

[0024] It is understood that step S1 further includes the following sub-steps: Sub-step S1.1: Obtain multiple request sequences; Sub-step S1.2: Dynamically schedule multiple request sequences based on a preset time window or request quantity threshold; Sub-step S1.3: Sort multiple request sequences in descending order based on sequence length; Sub-step S1.4: Use the GPU to zero-padded the input sequence of the user request, generate continuous tensors, and perform pre-padded calculations to obtain the initial key-value cache and query vector; Sub-step S1.5: Send the key-value cache to the storage area of ​​the in-memory computing memory, and send the query vector to the ready queue of the in-memory computing memory.

[0025] Based on the above scheme, this part of the logic constructs an efficient batching scheduling mechanism. By setting time or quantity thresholds, the system can aggregate concurrent requests from multiple users. In particular, sorting by sequence length in descending order and performing zero-padding packing can effectively reduce computational bubbles caused by sequences of varying lengths, maximizing the GPU's computing power utilization during the pre-padding stage.

[0026] Furthermore, the above sub-step S1.5 specifically includes the following sub-steps: Sub-step S1.5.1: Divide the key-value cache into multiple fixed-size memory page units, and record the topological mapping relationship between each inference request identifier and the physical storage address of the memory page through a logical block mapping table; Sub-step S1.5.2: Using a direct memory access controller, push the generated key matrix and value matrix asynchronously from the GPU to the shared buffer of the in-memory computing device in parallel; Sub-step S1.5.3: Parse the logical topology of the key matrix and value matrix in the shared buffer, map the matrix data according to the attention head dimension, and write it into the specified in-memory computing memory array, thereby realizing the alignment and persistent storage of key-value data in physical storage units.

[0027] Based on the above solution, this step solves the challenges of "data transfer" and "storage management" from the GPU to the in-memory computing device. Through paging and logical block tables, the memory fragmentation problem caused by KV cache growth as the sequence increases is effectively eliminated (i.e., hardware adaptation of the PagedAttention concept). Simultaneously, asynchronous transfer using DMA achieves overlap between computation and communication, avoiding the CPU's data copying overhead.

[0028] Step S2: Based on the in-memory computing memory, read the key vector, numerical vector and query vector from the key-value cache respectively, perform calculations, and generate the context vector.

[0029] Based on the above scheme, once the decoding stage begins, the computation mode shifts to an autoregressive mode that generates tokens one by one. At this point, the massive KV cache is persistently stored in in-memory compute arrays such as ReRAM. The system no longer needs to frequently move weights or KV data from video memory; instead, it directly utilizes the "in-situ computation" characteristic of the in-memory compute array to perform vector and matrix operations within the storage unit, greatly reducing the latency and power consumption caused by the memory wall.

[0030] Understandably, step S2 also includes the following sub-steps: Sub-step S2.1: Insert the query vectors corresponding to each request in the ready queue into the dynamic decoding slots according to their priority scores; Sub-step S2.2: Read the key matrix from the array, and perform a dot product operation between the key matrix and the target query vector in the dynamic decoding slot to obtain a similarity score; Sub-step S2.3: Perform Softmax normalization on the similarity score to obtain the dynamic attention weight coefficients; Sub-step S2.4: Read the numerical matrix, multiply the dynamic attention weight coefficients with the numerical matrix to achieve a weighted summation, and generate a context vector.

[0031] Based on the above scheme, this is a complete mapping of the standard attention mechanism onto an in-memory computing architecture. Concurrency control of multiple tasks is achieved through slot management, followed by the sequential execution of $Q \times K^T$ (dot product) and $Score \times V$ (weighted summation) operations. This design ensures that the entire attention computation process is completed in a closed loop within the in-memory computing device.

[0032] Furthermore, the formula for calculating the priority score in sub-step S2.1 is as follows:

[0033] Where t is the waiting time of the query vector corresponding to the request in the ready queue. This indicates the size of the key-value cache corresponding to the request. This represents a weighting factor indicating the request waiting time, used to adjust the degree to which the waiting time of a request in the queue affects its priority. This represents the weighting coefficient of the KV cache size factor, used to adjust the impact of the waiting time of a request in the queue on its priority. Its actual value is manually preset based on the optimization target.

[0034] Furthermore, regarding the above sub-step S2.2, in order to improve array utilization, the specific execution method is as follows: Optimized S2.2: Aggregate multiple query vectors in the dynamic decoding slot into a query matrix, read the key matrix from the array, and perform a dot product operation between the key matrix and the target query matrix to obtain a similarity score.

[0035] Based on the above scheme, this represents a deep optimization targeting the physical characteristics of in-memory arrays. Since the query vector generated by a single decoding request is typically small, direct computation would leave most rows and columns of the in-memory array idle. By aggregating vectors from multiple slots into a matrix, the inefficient matrix-vector multiplication (GEMV) is transformed into the efficient generalized matrix multiplication (GEMM), thereby filling the computation pipeline of the ReRAM array and significantly improving inference throughput.

[0036] Example 2: This example proposes a method for accelerating deep learning large model inference based on a hybrid architecture of GPU and Digital In-Memory Computing (ReRAM-PIM). The core logic of this method is to use a "pre-filling-decoding separation" strategy, according to the load characteristics of different stages of large model inference, to schedule computationally intensive pre-filling tasks and storage-intensive decoding tasks to GPU and PIM devices respectively, and to optimize overall performance with differentiated batch processing and scheduling strategies.

[0037] First, in the prefill phase, the method employs a dynamic large-batch processing strategy on the GPU. The system establishes a prefill queue to receive user requests and initiates batch processing based on a dual trigger mechanism: computation is triggered when the time window reaches a set threshold (e.g., 50ms) or the number of requests in the queue reaches the minimum batch requirement (e.g., 8). To address the parallel efficiency issues caused by variable-length input sequences, this phase uses a zero-padding technique. After arranging the sequences in descending order of length, padding is used to physically concatenate the variable-length sequences into continuous tensors, combined with a masking mechanism to control effective attention computation. This approach maximizes the parallel efficiency of the GPU computing units, quickly completing input processing and generating an initial key-value cache (KVCache).

[0038] Secondly, upon entering the decoding phase, the computation task is transferred to the ReRAM-PIM device. This phase employs a continuous batch processing strategy combined with fine-grained priority scheduling. The PIM device maintains a dynamic decoding slot pool containing a fixed number (e.g., 32) and a ready queue. The system performs fine-grained resource scheduling; once any request completes a single-step token generation, computational resources are immediately released without waiting for the entire sequence to finish. The scheduler calculates the score of each request based on a preset priority formula (considering both waiting time and input / output sequence length), dynamically selecting the highest priority request to insert into an empty slot. This achieves wait-free reuse of computational resources, effectively reducing tail latency for high-priority requests.

[0039] Finally, to address the high memory usage and fragmentation issues during the decoding phase, this method implements fine-grained resource management and computation aggregation strategies at the PIM level. For storage, a key-value (KV) cache paging management mechanism is employed, dividing the KV cache into fixed-size (e.g., 4KB) memory pages. A logical block table records the mapping relationship between requests and physical pages, supporting non-contiguous storage and efficiently reclaiming physical pages upon request completion, significantly reducing memory fragmentation. For computation, for small-scale matrix multiplication (GEMM) common in the decoding phase, the system performs cross-request task aggregation, combining the computational operations of multiple independent requests into optimally sized batch tasks that conform to the physical characteristics of the ReRAM array. This reduces the startup latency of fine-grained computation and fully leverages the parallel performance of the in-memory computing array.

[0040] This embodiment can be viewed as a specific implementation scenario of Embodiment 1.

[0041] The present invention also provides a GPU in-memory computing pre-filled decoding separation large model inference acceleration system. The GPU in-memory computing pre-filled decoding separation large model inference acceleration system can be implemented by executing the process steps of the GPU in-memory computing pre-filled decoding separation large model inference acceleration method. That is, those skilled in the art can understand the GPU in-memory computing pre-filled decoding separation large model inference acceleration method as a preferred embodiment of the GPU in-memory computing pre-filled decoding separation large model inference acceleration system.

[0042] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0043] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A GPU-in-memory computing pre-filling decoding separation method for accelerating large model inference, characterized in that, include: Step S1: Obtain the original text, perform calculations on the original text using the GPU, and generate the corresponding key-value cache and query vector; Step S2: Based on the in-memory computing memory, read the key vector, value vector and query vector from the key-value cache respectively, perform calculations, and generate the context vector.

2. The method according to claim 1, characterized in that, Step S1 includes: Sub-step S1.1: Obtaining multiple request sequences; Sub-step S1.2: Dynamically scheduling multiple request sequences based on a preset time window or request quantity threshold; Sub-step S1.3: Sort multiple request sequences in descending order based on sequence length; Sub-step S1.4: Using the GPU to zero-padded the input sequence of the user request, generating a continuous tensor and performing pre-padded calculation to obtain the initial key-value cache and query vector; Sub-step S1.5: Sending the key-value cache to the storage area of ​​the in-memory computing memory and sending the query vector to the ready queue of the in-memory computing memory.

3. The method according to claim 2, characterized in that, Sub-step S1.5 includes: Sub-step S1.5.1: Dividing the key-value cache into multiple fixed-size memory page units, and recording the topological mapping relationship between each inference request identifier and the physical storage address of the memory page through a logical block mapping table; Sub-step S1.5.2: Using a direct memory access controller, asynchronously pushing the generated key matrix and value matrix from the GPU to the shared buffer of the in-memory computing device in parallel; Sub-step S1.5.3: Parsing the logical topology of the key matrix and value matrix in the shared buffer, mapping the matrix data according to the attention head dimension, and writing it into the specified in-memory computing memory array.

4. The method according to claim 3, characterized in that, Step S2 includes: Sub-step S2.1: Inserting the query vectors corresponding to each request in the ready queue into the dynamic decoding slots according to priority scores; Sub-step S2.2: Reading the key matrix from the array, performing a dot product operation between the key matrix and the target query vector in the dynamic decoding slot to obtain a similarity score; Sub-step S2.3: Performing Softmax normalization on the similarity score to obtain dynamic attention weight coefficients; Sub-step S2.4: Reading the numerical matrix, multiplying the dynamic attention weight coefficients with the numerical matrix to achieve weighted summation and generate a context vector.

5. The method according to claim 4, characterized in that, The formula for calculating the priority score in sub-step S2.1 is as follows: Where t is the waiting time of the query vector corresponding to the request in the ready queue. This indicates the size of the key-value cache corresponding to the request. This represents a weighting factor indicating the request waiting time, used to adjust the degree to which the waiting time of a request in the queue affects its priority. This represents the weighting factor of the key-value cache size factor, used to adjust the degree to which the waiting time of a request in the queue affects its priority.

6. The method according to claim 5, characterized in that, Step S2.2 specifically involves: aggregating multiple query vectors in the dynamic decoding slots into a query matrix, reading the key matrix from the array, performing a dot product operation between the key matrix and the target query matrix, and obtaining a similarity score.

7. A GPU-in-memory computing pre-filled decoding separation large model inference acceleration system, characterized in that, It includes: Module M1: acquires the original text, performs calculations on the original text using the GPU, and generates corresponding key-value cache and query vector; Module M2: based on in-memory computing memory, reads the key vector, numerical vector and query vector from the key-value cache respectively, performs calculations, and generates context vector.

8. The system according to claim 7, characterized in that, The module M1 includes: submodule M1.1: acquiring multiple request sequences; submodule M1.2: dynamically scheduling multiple request sequences based on a preset time window or request quantity threshold; submodule M1.3: sorting multiple request sequences in descending order based on sequence length; submodule M1.4: using the GPU to zero-padded the input sequence of the user requests, generating continuous tensors and performing pre-padded calculations to obtain initial key-value cache and query vector; submodule M1.5: sending the key-value cache to the storage area of ​​the in-memory computing memory and sending the query vector to the ready queue of the in-memory computing memory.

9. The system according to claim 8, characterized in that, Submodule M1.5 includes: Submodule M1.5.1: Dividing the key-value cache into multiple fixed-size memory page units, and recording the topological mapping relationship between each inference request identifier and the physical storage address of the memory page through a logical block mapping table; Submodule M1.5.2: Using a direct memory access controller, asynchronously pushing the generated key matrix and value matrix from the GPU to the shared buffer of the in-memory computing device in parallel; Submodule M1.5.3: Parsing the logical topology of the key matrix and value matrix in the shared buffer, mapping the matrix data according to the attention head dimension and writing it into the specified in-memory computing memory array.

10. The system according to claim 9, characterized in that, The module M2 includes: submodule M2.1: inserting the query vectors corresponding to each request in the ready queue into the dynamic decoding slots according to priority scores; submodule M2.2: reading the key matrix from the array, performing a dot product operation between the key matrix and the target query vector in the dynamic decoding slot to obtain a similarity score; submodule M2.3: performing Softmax normalization on the similarity score to obtain dynamic attention weight coefficients; submodule M2.4: reading the numerical matrix, multiplying the dynamic attention weight coefficients with the numerical matrix to achieve weighted summation and generate a context vector.

Images