Attention decoding acceleration apparatus and method

Abstract

The application provides an attention decoding acceleration device and method, the device comprising: a RISC-V vector processing unit, an exponential operation hardware module, a matrix vector multiplication operation hardware module and a storage subsystem; the exponential operation hardware module is used for calculating the exponential value required by the normalized exponential function; the matrix vector multiplication operation hardware module is used for performing the matrix vector multiplication of the query vector and the key data block and the matrix vector multiplication of the weight vector and the value data block; the RISC-V vector processing unit is used for maintaining the online normalized exponential function state, and updating the online normalized exponential function state and generating the decoding output by scheduling the operation hardware module and the matrix vector multiplication operation hardware module to process the key buffer and the value buffer in the storage subsystem in a block streaming manner. The application realizes the efficient and low-delay attention decoding through the special exponential and GEMV hardware cooperating with the RISC-V vector unit in a block streaming processing manner.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to an attention decoding acceleration device and method. Background Technology

[0002] Inference in a Large Language Model (LLM) based on the Transformer architecture is divided into two stages: prefilling and decoding. In the decoding stage, a new token is generated each time, and the computation is represented by General Matrix-Vector Multiplication (GEMV). This involves multiplying the current query vector with the historical key cache's query-key matrix vector (QK GEMV) and multiplying the attention weights with the value cache's probability-value matrix vector (PV GEMV), rather than the batch matrix multiplication (GEMM) of the prefilling stage. As the context length increases, this approach gradually exhibits characteristics of limited memory access and significant instruction scheduling overhead.

[0003] In related technologies, attention implementation based on general-purpose RISC-V (Reduced Instruction Set Computer V) vector extensions relies entirely on instruction combinations to complete operations such as GEMV and Softmax, resulting in problems such as long instruction sequences, low reduction efficiency, and frequent data movement between registers and memory. Existing GPUs (Graphics Processing Units) and NPUs (Neural Processing Units) are mostly designed with GEMM as their core objective. Directly using GEMM acceleration units may lead to insufficient array utilization, excessive data movement, and low power efficiency. Furthermore, in related technologies, dequantization operations are usually independent of multiply-accumulate calculations; the complete precision data after dequantization needs to be written back to memory, increasing intermediate data movement and memory bandwidth consumption.

[0004] Therefore, an attention decoding acceleration scheme that balances computational efficiency and memory access efficiency is needed to solve at least one of the aforementioned problems. Summary of the Invention

[0005] To overcome at least one of the above problems, the present invention proposes an attention decoding acceleration device and method.

[0006] To achieve the above objectives, the present invention adopts the following technical solution.

[0007] In a first aspect, the present invention provides an attention decoding acceleration device, comprising:

[0008] RISC-V vector processing unit, exponentiation hardware module, matrix-vector multiplication hardware module, and storage subsystem;

[0009] in,

[0010] The exponent calculation hardware module is used to calculate the exponent value required for the normalized exponent function;

[0011] The matrix-vector multiplication hardware module is used to perform matrix-vector multiplication of the query vector and the key data block, as well as matrix-vector multiplication of the weight vector and the value data block.

[0012] The RISC-V vector processing unit is used to maintain the state of the online normalized exponential function, and to process the key cache and value cache in the storage subsystem in a block-stream manner by scheduling the exponential operation hardware module and the matrix-vector multiplication operation hardware module, so as to update the state of the online normalized exponential function and generate the decoded output.

[0013] In some optional embodiments, the RISC-V vector processing unit is configured to process the key cache and value cache in a block-stream manner, including:

[0014] The matrix-vector multiplication hardware module is scheduled to generate a score block based on the query vector and the current key data block.

[0015] The maximum value of the score block is reduced, and the global maximum value in the state of the online normalized exponential function is updated;

[0016] The exponential calculation hardware module is scheduled to calculate the scaling factor and exponential weight;

[0017] Based on the scaling factor and the exponential weight, update the exponent sum in the state of the online normalized exponential function;

[0018] The matrix-vector multiplication hardware module is scheduled to update the output accumulation vector in the online normalized exponential function state based on the exponential weight, the current value data block, and the scaling factor.

[0019] After all data blocks have been processed, the output accumulation vector is normalized to obtain the attention output.

[0020] In some optional embodiments, the online normalized exponential function state includes a global maximum value, an exponent sum, and an output accumulation vector;

[0021] The RISC-V vector processing unit is configured as follows:

[0022] The reduced maximum value of the score block is compared with the current global maximum value, and the larger one is taken as the new global maximum value;

[0023] The old exponent sum is multiplied by the scaling factor and then added to the sum of the exponent weights of the current data block to update the exponent sum;

[0024] The old output accumulation vector is multiplied by the scaling factor, and then the weighted sum of each value in the current value data block and the exponential weight is added to update the output accumulation vector.

[0025] In some optional embodiments, the exponentiation hardware module is used for:

[0026] Calculate the exponential value of the difference between the old global maximum and the new global maximum, and use it as the scaling factor;

[0027] Calculate the exponential value of the difference between each score in the score block and the new global maximum value, and use it as the exponential weight.

[0028] In some optional embodiments, the exponentiation hardware module includes:

[0029] The range reduction module is used to decompose the input into a first part and a second part;

[0030] The core of the exponential approximation calculation is used to calculate the exponential approximation value of the second part;

[0031] The scaling and normalization module is used to combine the exponential approximation with the scaling factor generated based on the first part to obtain the final exponential result.

[0032] In some optional embodiments, the matrix-vector multiplication hardware module includes a multiplication-addition reduction module and a quantization unpacking and dequantization fusion module;

[0033] The multiply-add reduction module is used to perform dot product reduction of each key vector in the key data block with the query vector to generate a score block, and to perform weighted accumulation of each value vector in the value data block with the corresponding exponential weight to update the output accumulation vector.

[0034] The quantization unpacking and dequantization fusion module is used to perform unpacking and dequantization operations in real time when reading quantized key data or value data from the storage subsystem, and directly send the dequantization result to the multiply-accumulate reduction module for calculation.

[0035] In some optional embodiments, the key cache and value cache are organized according to one of the following two data layouts:

[0036] The first layout stores information sequentially by attention header, block, word within block, and channel, so that the complete channel data of the same word is arranged continuously.

[0037] The second layout stores data sequentially by attention header, block, channel segment, and word element within the block, so that data of multiple consecutive word elements within the same channel segment are arranged continuously.

[0038] In some optional embodiments, the RISC-V vector processing unit is further configured to, in grouped query attention or multi-query attention scenarios, handle the same key-value cache block shared by multiple query headers:

[0039] The key data block is loaded only once, and the matrix-vector multiplication hardware module is scheduled multiple times to perform query-key matrix-vector multiplication for each of the multiple query heads;

[0040] The value data block is loaded only once, and the matrix-vector multiplication hardware module is scheduled multiple times to perform probability-value matrix-vector multiplication on the multiple query heads respectively.

[0041] In some optional embodiments, the apparatus further includes address management logic for performing the conversion of the logical address of the paging key-value cache to a physical address, and providing the parsed physical address to the matrix-vector multiplication hardware module.

[0042] Secondly, the present invention provides an attention decoding acceleration method, comprising:

[0043] Divide the key cache and value cache into multiple data blocks respectively;

[0044] Initialize the state of the online normalized exponential function;

[0045] The plurality of data blocks are sequentially processed in a block-based streaming manner to update the state of the online normalized exponential function, including:

[0046] The scheduling matrix-vector multiplication hardware module generates score blocks based on the query vector and the current key data block;

[0047] The maximum value of the score block is reduced, and the global maximum value in the state of the online normalized exponential function is updated;

[0048] Schedule the exponential calculation hardware module to calculate the scaling factor and exponential weight;

[0049] Based on the scaling factor and the exponential weight, update the exponent sum in the state of the online normalized exponential function;

[0050] The matrix-vector multiplication hardware module is scheduled to update the output accumulation vector in the online normalized exponential function state based on the exponential weight, the current value data block, and the scaling factor.

[0051] After all data blocks have been processed, the output accumulation vector is normalized to obtain the decoded output.

[0052] In some optional embodiments, the online normalized exponential function state includes a global maximum value, an exponent sum, and an output accumulation vector;

[0053] The initialization of the online normalized exponential function state includes: setting the global maximum value to negative infinity, the exponent sum to zero, and the output accumulation vector to zero.

[0054] In some optional embodiments, updating the global maximum value in the state of the online normalized exponential function includes: comparing the reduced maximum value of the score block with the current global maximum value, and taking the larger one as the new global maximum value;

[0055] The step of updating the sum of exponents in the state of the online normalized exponential function includes: multiplying the old sum of exponents by the scaling factor and then adding the sum of the exponent weights of the current data block.

[0056] The step of updating the output accumulation vector in the state of the online normalized exponential function includes: multiplying the old output accumulation vector by the scaling factor, and then adding the weighted sum of each value in the current value data block and the exponential weight.

[0057] In some optional embodiments, the scaling factor is an exponential value of the difference between the old global maximum and the new global maximum;

[0058] The index weight is the index value of the difference between each score in the score block and the new global maximum value.

[0059] In some optional embodiments, the scheduling exponent calculation hardware module calculates the scaling factor and exponent weights, including:

[0060] The input for calculating the exponent value is decomposed into a first part and a second part;

[0061] Calculate the exponential approximation of the second part;

[0062] The final exponential result is obtained by multiplying the approximate exponential value by the scaling factor generated based on the first part.

[0063] In some optional embodiments, when reading key data or value data in quantized format, unpacking and dequantization operations are performed, and the dequantization results are directly used for matrix-vector multiplication operations.

[0064] In some optional embodiments, the key cache and value cache are stored according to one of the following data layouts:

[0065] The first layout stores information sequentially by attention header, block, word within block, and channel, so that the complete channel data of the same word is arranged continuously.

[0066] The second layout stores data sequentially by attention header, block, channel segment, and word element within the block, so that data of multiple consecutive words within the same channel segment are arranged continuously.

[0067] In some optional embodiments, in grouped query attention or multi-query attention scenarios, the method further includes:

[0068] For the same key-value cache block shared by multiple query heads, the key data block is loaded only once, and the matrix-vector multiplication hardware module is scheduled multiple times to perform query-key matrix-vector multiplication for each of the multiple query heads.

[0069] The value data block is loaded only once, and the matrix-vector multiplication hardware module is scheduled multiple times to perform probability-value matrix-vector multiplication on the multiple query heads respectively.

[0070] In some optional embodiments, it also includes:

[0071] Perform the translation of the logical address to the physical address of the pagination key-value cache;

[0072] The converted physical address is provided to the matrix-vector multiplication hardware module.

[0073] The technical solution of the present invention has the following advantages or beneficial effects:

[0074] (1) This invention avoids the problems of insufficient resource utilization and excessive data handling in GEMM hardware when processing word elements by using dedicated GEMV hardware to process QK GEMV and PV GEMV; by using a dedicated hardware pipeline for exponent calculation to replace the existing implementation method of multiple instruction combinations based on general RISC-V vector extension, the latency and instruction scheduling overhead of Softmax exponent calculation are significantly reduced; by combining block streaming processing and online Softmax state maintenance, the score blocks are directly used for PV GEMV after the exponent weight is calculated by the exponent calculation hardware module, which effectively reduces the memory access pressure in the long context decoding stage.

[0075] (2) After the exponential weight is calculated by the exponential operation hardware module, the score block of the present invention is directly provided to the matrix-vector multiplication operation hardware module for updating the output accumulation vector. In the block streaming process, it is not necessary to write the complete attention probability matrix back to the storage subsystem and then read it for use, thereby improving the execution efficiency and overall performance of the acceleration device.

[0076] (3) This invention defines the state of the online normalized exponential function as three fixed-size variables and limits the incremental update method of each variable when processing each data block. The three variables carry the Softmax intermediate results of all historical data blocks without retaining any original scores or attention weight data of processed data blocks, thereby reducing the storage overhead of intermediate data and significantly reducing the storage requirements in long context scenarios.

[0077] (4) The present invention loads the key data block only once and schedules the matrix-vector multiplication operation hardware module multiple times to perform query-key matrix-vector multiplication for the multiple query heads; loads the value data block only once and schedules the matrix-vector multiplication operation hardware module multiple times to perform probability-value matrix-vector multiplication for the multiple query heads; thereby enabling the same key-value cache block to be transmitted only once between the storage subsystem and the matrix-vector multiplication operation hardware module, thus improving the performance and efficiency of the acceleration device. Attached Figure Description

[0078] Figure 1 This is a schematic diagram of the overall structure of the RISC-V attention decoding acceleration device according to an embodiment of the present invention;

[0079] Figure 2 This is a schematic diagram illustrating the process of block-based streaming attention decoding in collaboration between the exponential operation hardware module and the matrix-vector multiplication operation hardware module according to an embodiment of the present invention.

[0080] Figure 3 This is a schematic diagram of the structure of the exponentiation hardware module according to an embodiment of the present invention;

[0081] Figure 4 This is a schematic diagram of the matrix-vector multiplication hardware module according to an embodiment of the present invention;

[0082] Figure 5 This is a schematic diagram illustrating the process of reusing Key / Value Cache in a GQA / MQA scenario according to an embodiment of the present invention. Detailed Implementation

[0083] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0084] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0085] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0086] To overcome at least one of the technical problems described in the background section, a first aspect of the present invention provides an attention decoding acceleration device, comprising: a RISC-V vector processing unit, an exponentiation operation hardware module, a matrix-vector multiplication operation hardware module, and a storage subsystem; wherein, the exponentiation operation hardware module is used to calculate the exponent value required for the normalized exponent function; the matrix-vector multiplication operation hardware module is used to perform matrix-vector multiplication of a query vector with a key data block, and matrix-vector multiplication of a weight vector with a value data block; the RISC-V vector processing unit is used to maintain the online normalized exponent function state, and by scheduling the exponentiation operation hardware module and the matrix-vector multiplication operation hardware module, to process the key cache and value cache in the storage subsystem in a block-stream manner, so as to update the online normalized exponent function state and generate a decoding output.

[0087] like Figure 1 The diagram shows the overall structure of a RISC-V attention decoding acceleration device. The main body includes a RISC-V vector processing unit, an exponentiation operation hardware module, a matrix-vector multiplication operation hardware module, and a storage subsystem. The storage subsystem is used to store query vectors. Key Cache Value Cache Including quantization parameters and other data. The exponent calculation hardware module is specifically responsible for calculating the exponent value required for the normalized exponential function (Softmax), including the scaling factor required in online Softmax calculation. and the exponential weight of the current data block This module can calculate exponential functions using methods such as look-up table (LUT), polynomial approximation, piecewise linear approximation, or a combination of look-up table and polynomial methods, and can output FP32, FP16, BF16, etc., results according to the model accuracy requirements. The matrix-vector multiplication hardware module is specifically responsible for performing two types of general matrix-vector multiplication (GEMV) operations: one is lookup vector... Key Block The query-key matrix-vector multiplication (QK GEMV) yields the score block. Second, the weight vector (composed of exponential weights) Composition) and Value Block The probability-value matrix-vector multiplication (PV GEMV) is used to update the output accumulator vector. Since the decoding phase typically processes a single token per thread or execution context, the query side is a vector, while the historical KV cache side is a matrix of multiple tokens, using dedicated GEMV hardware is more in line with actual computational needs than using hardware designed for General Matrix-Matrix Multiplication (GEMM).

[0088] The RISC-V Vector Processing Unit (RVV) is the core of the entire device's control and scheduling. It is responsible for maintaining the Online Softmax State and, through scheduling the exponential operation hardware module and the matrix-vector multiplication operation hardware module, processes the key and value caches in the storage subsystem in a block-stream manner. Block-stream processing refers to dividing K and V into multiple data blocks along the token dimension, processing each block sequentially, incrementally updating the online Softmax state during processing, and generating the final decoded output after all blocks have been processed. In some embodiments, the collaborative control interface between the RISC-V Vector Processing Unit and the two dedicated hardware modules can be implemented using RISC-V custom instructions. The RISC-V main processor (which can be integrated with the RISC-V Vector Processing Unit in the same processor core) is used to perform overall scheduling of model inference, maintenance of the paged key-value cache block table, logical address to physical address translation, exception handling, and task synchronization management tasks. For example, the RISC-V main processor passes the resolved physical address to the RISC-V vector processing unit, which then provides it to the matrix-vector multiplication hardware module. This allows the dedicated hardware module to always process data blocks with contiguous physical addresses, without needing to be aware of complex paging management logic. After completing their computation tasks, the matrix-vector multiplication hardware module and the exponentiation hardware module send completion status feedback to the RISC-V vector processing unit via status signals.

[0089] The collaborative control interface between the RISC-V vector processing unit and the two dedicated hardware modules of this invention can be implemented through RISC-V custom instructions. These custom instructions are used to launch the exponential operation hardware module or the matrix-vector multiplication hardware module, and the parameters passed include input address, data type, block size, head dimension, quantization parameter address, effective length, output address, and status flags. The RISC-V main processor (which can be integrated with the RISC-V vector processing unit in the same processor core) is used to perform overall scheduling of model inference, maintenance of the paged key-value cache block table, logical address to physical address conversion, exception handling, and task synchronization management tasks.

[0090] The embodiments of this invention delegate the most time-consuming exponential and matrix-vector multiplication operations in attention decoding to dedicated hardware, while retaining process scheduling, state updates, and special scenario handling on the RISC-V vector processing unit side, thus balancing performance, flexibility, and implementation complexity. Specifically, for the computational characteristics of word units and mini-batches in the decoding stage, dedicated GEMV hardware is used to process QK GEMV and PV GEMV. Compared with general-purpose accelerators designed with GEMM as the core, this significantly improves resource utilization and energy efficiency, avoiding the problems of insufficient resource utilization and excessive data handling in GEMM hardware when processing word units. By using an independent exponential operation hardware module to replace the existing implementation method of multiple instruction combinations based on general-purpose RISC-V vector extensions with a dedicated hardware pipeline, the latency and instruction scheduling overhead of Softmax exponential calculation are significantly reduced. Combined with block-based streaming processing and online Softmax state maintenance, the score blocks are directly used for PV GEMV after the exponential weights are calculated by the exponential operation hardware module, effectively reducing the memory access pressure in the long context decoding stage.

[0091] In some optional embodiments, the RISC-V vector processing unit is configured to process the key cache and value cache in a block-stream manner, including: scheduling the matrix-vector multiplication hardware module to generate a score block based on the query vector and the current key data block; performing maximum value reduction on the score block and updating the global maximum value in the online normalized exponential function state; scheduling the exponential operation hardware module to calculate the scaling factor and exponential weight; updating the exponential sum in the online normalized exponential function state based on the scaling factor and the exponential weight; scheduling the matrix-vector multiplication hardware module to update the output accumulation vector in the online normalized exponential function state based on the exponential weight, the current value data block, and the scaling factor; and normalizing the output accumulation vector after all data blocks have been processed to obtain the attention output.

[0092] like Figure 2 The diagram illustrates the workflow of block-based streaming attention decoding, where the exponential operation hardware module and the matrix-vector multiplication operation hardware module work together. A detailed description is provided using an example where the query vector in the current decoding step is q with dimension D; the historical context length is N; the key cache is K; and the value cache is V, where both K and V are N-row, D-column real number matrices. Specifically, attention decoding can be represented as:

[0093]

[0094] in, For word index, ; Indicates the query vector q and the first... Key vectors corresponding to each word The dot product result.

[0095]

[0096] in, For the first Attention weights for each word element, The variable to be summed has a range of values. The denominator represents the sum of the exponential scores of all N tokens.

[0097]

[0098] in, For the first The value vectors corresponding to each word, and o is the output vector obtained by weighting the value vectors of all words according to the attention weights.

[0099] K and V are divided into multiple data blocks according to the lexical dimension, with each data block containing B lexical units. In practice, B can be determined based on the vector register length, cache size, and data type of the RISC-V vector processing unit.

[0100] In practice, the first step is for the RISC-V vector processing unit to schedule the matrix-vector multiplication hardware module to execute QKGEMV. The multiply-accumulate and reduce module within the matrix-vector multiplication hardware module receives q and the current key data block. After completing the dot product reduction calculation, the result is written back to the score buffer:

[0101]

[0102] The second step involves the RISC-V vector processing unit processing the score blocks. Perform maximum value reduction to obtain the maximum value of the current data block. and update the global maximum value. :

[0103]

[0104] Where m is the global maximum value in the current online normalized exponential function state.

[0105] The third step involves the RISC-V vector processing unit scheduling the exponentiation hardware module to calculate the scaling factor. and index weight :

[0106]

[0107]

[0108] in For the term index within the current data block, For the current data block, the first The score of each word element, These are the corresponding index weights. Because... and The input to the exponential function is always a non-positive number, thus avoiding the problem of numerical overflow.

[0109] Fourth step: Update the exponential sum based on the scaling factor and exponential weights:

[0110]

[0111] in The sum of the exponents in the current online normalized exponential function state, This represents the sum of the exponential weights of all tokens within the current data block.

[0112] Step 5: The matrix-vector multiplication hardware module is scheduled to execute PV GEMV. The matrix-vector multiplication hardware module receives... Current value data block Calculate the sum of the old output vector o:

[0113]

[0114] in, For the current data block, the first The value vector corresponding to each word element This represents the result of summing the value vectors of all words within the current data block according to their corresponding exponential weights. The updated result is used as the new output accumulation vector.

[0115] After all data blocks have been processed, normalization is performed to obtain the final attention output:

[0116]

[0117] In this embodiment, an exponential operation hardware module and a matrix-vector multiplication hardware module are added to the RISC-V architecture, and their collaborative relationship with the RISC-V vector processing unit is defined. The decoding stage typically involves single-token processing, with core linear computations being QK GEMV and PV GEMV. Using the matrix-vector multiplication hardware module is more beneficial for improving array utilization and energy efficiency than using GEMM hardware. The matrix-vector multiplication hardware module accelerates score generation and value weighted accumulation through query caching, weight broadcasting, multiply-accumulate arrays, and reduction trees. Combined with online Softmax state updates, the score blocks, after obtaining exponential weights from the exponential operation hardware module, are directly used for PV GEMV without requiring complete write-back and reading of the attention probability array.

[0118] In some embodiments, the RISC-V vector processing unit can use vector load instructions to load query vectors, key data, and value data; use vector multiplication, extended precision multiplication, or vector fused multiply-add instructions to perform dot products and accumulations; and use vector reduction summation and reduction maximum instructions to perform summation and maximum value operations.

[0119] In some embodiments, the software and hardware collaboration process is as follows:

[0120] Step 1: The RISC-V main processor generates block scheduling information based on the current sequence, head number, context length, and KV Cache layout, and configures Query, Key block, Value block, quantization parameters, and output buffer address, etc.

[0121] Step two, the RISC-V / RVV or main processor initiates the matrix-vector multiplication hardware module to execute QK GEMV, obtaining the current block's... .

[0122] Step 3, RISC-V / RVV pair Perform maximum value reduction to obtain and update .

[0123] Step four, RISC-V / RVV will as well as Provided to the exponentiation hardware module, which then calculates the exponentiation. and .

[0124] Step 5, RISC-V / RVV update and will , The value block address and output accumulation address are provided to the matrix-vector multiplication hardware module.

[0125] Step six: The matrix-vector multiplication hardware module performs PV GEMV to obtain... .

[0126] Step 7, RISC-V / RVV Update , , Then determine if there are any unprocessed blocks. If so, continue to step two; otherwise, execute... And write it back to the output.

[0127] This embodiment of the invention achieves a reasonable division of labor between software and hardware tasks by clearly defining the scheduling relationship between the RISC-V vector processing unit and two dedicated hardware modules in each step. Specifically, computationally intensive matrix-vector multiplication and exponential operations are efficiently executed by dedicated hardware, while flow control and state updates are flexibly handled by the RISC-V vector processing unit. After the score block obtains its exponential weight through the exponential operation hardware module, it is directly used for PV GEMV updates, eliminating the need to write back and read the complete attention probability matrix between data blocks, thus reducing the storage and transportation overhead of intermediate data.

[0128] In some optional embodiments, the online normalized exponential function state includes a global maximum value, an exponential sum, and an output accumulation vector; the RISC-V vector processing unit is configured to: compare the reduced maximum value of the score block with the current global maximum value, and take the larger one as the new global maximum value; multiply the old exponential sum by the scaling factor, and add the sum of the exponential weights of the current data block to update the exponential sum; multiply the old output accumulation vector by the scaling factor, and add the weighted sum of each value in the current value data block and the exponential weights to update the output accumulation vector.

[0129] The state of the online normalized exponential function is determined by the global maximum value. index and and output accumulation vector Composition. Among them, This represents the maximum score among all processed data blocks. This is the sum of the exponential weights of all processed data blocks (as the denominator for the final normalization). This is a vector of dimension D (recording the result of an exponentially weighted sum of the value vectors of all processed data blocks). Before block-based streaming processing begins, the three variables are initialized to... , , .

[0130] Taking the current data block as an example, the update operations for the three variables are as follows:

[0131] Update global maximum value: If the maximum value of the current data block does not exceed the global maximum value, then scaling factor No scaling adjustment is required for the old state.

[0132] Update index and: ,in This is a scaling compensation for the old exponent and sum. It is the sum of the exponential weights of all tokens within the current data block.

[0133] Update the output accumulation vector: ,in It is a scaling compensation for the old output accumulation vector. For the current data block, the first The value vector corresponding to each word element This represents the value vector of each word. Multiply by the corresponding scalar weight Then sum by elements.

[0134] The purpose of the above incremental update formula is to: when the global maximum value increases due to new data blocks, adjust the scaling factor accordingly. The exponential sum and output accumulation vector in the old state are uniformly scaled to compensate for the impact of changes in the maximum value on the Softmax probability distribution, ensuring that any number of data blocks are processed successfully. It equals the output of the standard Softmax attention calculation.

[0135] In this embodiment, the online normalized exponential function state is explicitly defined as three fixed-size variables, and the incremental update method for each variable when processing each data block is limited. Specifically, it is achieved through maximum value comparison and scaling factor compensation. The three variables carry the Softmax intermediate results of all historical data blocks without retaining the original scores or attention weights of any processed data blocks, thus reducing the storage overhead of intermediate data and significantly reducing storage requirements in long context scenarios.

[0136] In some optional embodiments, the exponentiation hardware module is used to: calculate the exponent value of the difference between the old global maximum value and the new global maximum value, as the scaling factor; and calculate the exponent value of the difference between each score in the score block and the new global maximum value, as the exponent weight.

[0137] See Figure 3 In the embodiment shown, the exponent calculation hardware module is mainly responsible for two types of exponent calculations.

[0138] The first category is calculating the scaling factor. The input is the old global maximum value. With the new global maximum value The difference, output .because , , The range of is (0, 1). hour, This indicates that the current data block has not produced a larger score, and there is no need to scale the old exponent and the old output accumulation vector. hour, The scaling factor compresses and compensates for the old state to adapt to the change in the Softmax probability distribution caused by the update of the maximum value. As a scalar, each data block only needs to be computed once.

[0139] The second category involves calculating index weights. The input is the scores of each item in the current score block. With the new global maximum value The difference, output .because The maximum score for all processed tokens. This ensures that the input to the exponential function is always a non-positive number. It is a vector whose length is equal to the number of valid tokens in the current data block. Each element is used for subsequent exponentiation, updates, and PV GEMV weighted accumulation.

[0140] In practice, in long-context decoding scenarios, the score The numerical range may be large, and directly performing exponential operations on the original scores can easily lead to numerical overflow. By using... As input to the exponential function, the input value is always a non-positive number, and the output of the exponential function is always within the range of (0,1], fundamentally avoiding the floating-point overflow problem. This design enables online Softmax calculations to maintain numerical stability without sacrificing mathematical precision.

[0141] In some optional embodiments, the exponentiation hardware module includes: a range reduction module for decomposing the input into a first part and a second part; an exponent approximation calculation core for calculating the exponent approximation value of the second part; and a scaling and normalization module for combining the exponent approximation value with a scaling factor generated based on the first part to obtain the final exponentiation result.

[0142] See Figure 3 In the embodiment shown, the exponent calculation hardware module adopts a three-stage pipeline structure of "range reduction - approximation calculation - scaling and merging", which includes a range reduction module, an exponent approximation calculation core, and a scaling and normalization module.

[0143] The range reduction module is used to reduce the input The decomposition is divided into a first part and a second part, using the following mathematical decomposition:

[0144]

[0145] in It is an integer (i.e., the "first part"). For the remainder falling into the smaller range (i.e., the "second part"), the following conditions must be met: This factorization will decompose any real number. The exponential calculation is transformed into a small range The exponent is calculated as the product of a power of 2. For subsequent generation Scaling factor The range of values ​​for is strictly limited to Internally, it facilitates efficient calculation through methods such as table lookup or polynomial approximation. The purpose of this module is to compress the input from a potentially large dynamic range to a very small interval, reducing the complexity of subsequent approximation calculations while ensuring sufficient computational accuracy with limited hardware resources.

[0146] The core of the exponential approximation calculation is used to calculate the exponential approximation value of the second part. .because Limited to a narrow scope, this module can be implemented using methods such as LUT look-up table, polynomial approximation, piecewise linear approximation, or a combination of LUT and polynomial correction. The specific implementation method can be flexibly selected according to the model's accuracy requirements.

[0147] The scaling and normalization module approximates the core of the exponential calculation. With scaling factor generated based on the first part Combining these, we obtain the final index result:

[0148]

[0149] During attention decoding, the RISC-V vector processing unit inputs the scaling factor. and the input of exponential weights The results are passed to the exponential calculation hardware module. Since the input data for both types of calculations are independent and parallelizable, this module can process multiple inputs simultaneously or in a pipelined manner, and write the calculation results directly back to the corresponding α buffer and β buffer for subsequent online normalization exponential function state updates.

[0150] In some optional embodiments, the exponentiation hardware module further includes the following modules:

[0151] (1) RISC-V main processor / RVV input generation module: generates two types of exponential inputs based on the online Softmax state. Input is ; Input is .

[0152] (2) Command and configuration interface module: Receives startup commands and configuration parameters (input / output data types, vector length, mask, input address, output address, precision mode, and exception handling strategy, etc.) from RISC-V / RVV, and distinguishes whether the current task is scalar. Computation, or vector? calculate.

[0153] (3) Input buffer and format conversion module: Receives exponential input in scalar or vector form, converts FP32, FP16, BF16 or fixed-point format into the internal format used by the exponential calculation core, and handles tail mask, valid bits and input range restrictions.

[0154] (4) Output buffer and status feedback module: The calculated data will be used for output buffering and status feedback. scalar or The vector is written back to the specified buffer, and status signals such as done and exception are returned to the command interface.

[0155] (5) Exception handling: Handling exceptions such as underflow, overflow, saturation, and rounding.

[0156] In some optional embodiments, the matrix-vector multiplication hardware module includes a multiply-add reduction module and a quantization unpacking and dequantization fusion module. The multiply-add reduction module performs dot product reduction of each key vector in the key data block with the query vector to generate a score block, and performs weighted accumulation of each value vector in the value data block with its corresponding exponential weight to update the output accumulation vector. The quantization unpacking and dequantization fusion module performs unpacking and dequantization operations in real time when reading quantized key data or value data from the storage subsystem, and directly sends the dequantization result to the multiply-add reduction module for computation.

[0157] In some embodiments, the matrix-vector multiplication hardware module includes two core sub-modules: a multiply-addition reduction module and a quantization depacketization and inverse quantization fusion module. The multiply-addition reduction module is the core computational unit, supporting both QK GEMV and PV GEMV operating modes. In QK mode, this module receives the query vector q and the current key data block. The system performs dot product reduction on each key vector and query vector within the key data block. For each token in the data block, the hardware computes the products of D key elements and their corresponding query elements in parallel, then accumulates them level by level to obtain a scalar score. This process ultimately generates the score block for the current data block. In PV mode, this module receives the exponential weights. Scaling factor α, old output accumulation vector o, and current value data block The weighted summation of each value vector in the execution value data block with its corresponding exponent weight is performed. For each word... , its value vector Multiply by weight Then, the weighted result of all words is summed element by element, and the old output cumulative vector is scaled and updated by combining the scaling factor α.

[0158] The dequantization and dequantization fusion module supports efficient processing of low-bit key-value caches (KVCaches) such as INT4, INT8, and FP8. In large language model inference, low-bit quantization formats are typically used to reduce the storage overhead of KVCaches. When reading quantized key or value data from the storage subsystem, this module simultaneously reads the corresponding scaling factor and zero-point quantization parameters, performing dequantization, dequantization, and type conversion in real time on the input side. The dequantization result is not written back to the storage subsystem but is directly fed into the multiply-accumulate reduction module for computation, thus achieving data path fusion between dequantization and multiply-accumulate calculations.

[0159] In some optional implementations, the FP8 KV Cache also needs to synchronously read the corresponding scale (dynamic scaling factor) during reading. The matrix-vector multiplication hardware module performs FP8 unpacking and type conversion on the input side, and the dequantization result is directly sent to the multiply-accumulate array. Compared with INT8 / INT4, the FP8 format can avoid the problem of excessive loss of quantization precision, but it requires additional scaling factor management.

[0160] Taking INT8 quantization as an example, the quantized data satisfies:

[0161]

[0162]

[0163] in , Quantized key and value data stored in INT8 format. , This is the quantization scaling factor. , To quantize the zero point, , This is the dequantized floating-point data. When reading the quantization key block, the quantization depacketization and dequantization fusion modules synchronously read the corresponding... and After dequantization, the FP16 / BF16 format data is directly sent to the QK GEMV path of the multiply-accumulate reduction module. Similarly, when reading the quantized value block, the dequantization result is directly sent to the PV GEMV path. Throughout the process, the dequantized, complete precision data is not written back through the storage subsystem, but enters the computation path directly in a pipelined manner, eliminating intermediate buffering steps.

[0164] like Figure 4 As shown, the matrix-vector multiplication hardware module also includes the following sub-modules:

[0165] (1) RISC-V main processor / RVV command queue module: generates GEMV tasks (QK mode or PV mode), and is responsible for block scheduling, effective length control, tail block processing, paged KV Cache address resolution and task synchronization.

[0166] (2) GEMV command and configuration interface: Receives GEMV task configuration, including D (head dimension), B (number of tokens in the block), valid_len (actual number of valid tokens in the current block), dtype, as well as Query, Key, Value, β, output buffer, and quantization parameter address. Selects QK mode or PV mode according to the configuration.

[0167] (3) Address and Data Reading Interface: Receives the parsed physical block address and reads input data such as Q, K, V, β, and α. In the paged KV Cache scenario, the complex logical address to physical address translation is completed by the RISC-V main processor or runtime; the GEMV hardware only receives the physical address of the current block. Invalid tokens are excluded from calculation based on valid_len and tail mask.

[0168] (4) Result Write-back Module: Writes the calculation results back to the specified buffer. In QK mode, the score buffer is written back for RVV to perform subsequent maximum value reduction and exp input generation; in PV mode, the output accumulator is written back as the new value of the online Softmax state o. At the same time, the completion status, error status and count status are returned.

[0169] In some optional embodiments, the key cache and value cache are organized according to one of the following two data layouts: a first layout, which stores data in the order of attention head, block, word within block, and channel, so that the complete channel data of the same word is arranged continuously; and a second layout, which stores data in the order of attention head, block, channel fragment, and word within block, so that the data of multiple consecutive words within the same channel fragment are arranged continuously.

[0170] This embodiment provides two optional data layout methods to optimize the memory access efficiency of different types of GEMV operations.

[0171] First layout (token-first layout): The dimension order is attention head, block, tokens in block → channel, specifically represented as follows:

[0172]

[0173] In this layout, the complete channel data of the same word is arranged contiguously in storage. That is, the D channel data of the same word constitute a contiguous address space in physical storage. The advantages of this layout are: the complete vector of each word is stored contiguously, conforming to the conventional KV cache representation, simplifying software management, and requiring only the appending of a contiguous data segment when writing a new word. The contiguous arrangement of channel data for the same word facilitates reading all channel data of that word at once, making full use of storage bandwidth.

[0174] The second layout (channel shard priority layout): The dimension order is attention head, block, channel shard, and token in block, as shown below:

[0175]

[0176] In this layout, data from multiple consecutive terms within the same channel shard are stored contiguously. That is, for a fixed channel shard, the data of B terms within that shard constitutes a contiguous address space in physical storage. The advantage of this layout is that it is more suitable for QK GEMV computation. In QK GEMV, the same query vector channel needs to be multiplied element-wise with the key vectors of multiple terms within the data block. By contiguously organizing the data of multiple terms within the same channel shard, the matrix-vector multiplication hardware module can burst-read the data from the same channel shard of multiple terms at once. Combined with the parallel channels of the multiply-accumulate array, it can simultaneously calculate the partial dot product results of multiple terms on that channel shard, significantly improving memory access merging efficiency and multiply-accumulate array utilization.

[0177] The meanings of the above dimensions are as follows: head is the attention head index, which represents the KV head index in Grouped-Query Attention (GQA) or Multi-Query Attention (MQA) scenarios; block is the block index divided along the token / context dimension, which usually corresponds to a logical or physical data block in a continuous token or paged KV cache; token_in_block is the token offset index within the data block, which usually ranges from 0 to B-1; channel is the feature channel index, which is the dimension index within the head dimension, which usually ranges from 0 to D-1; channel_tile is the channel tile index, which represents the tile number after the channel dimension is tiled according to the hardware vector width, multiply-accumulate array width, or memory access granularity.

[0178] Because QK and PV have different memory access directions—QK focuses more on the parallel dot product of multiple tokens on the same channel, while PV focuses more on the weighted summation of the value vector of each token by channel—the Key Cache can preferentially adopt the second layout, while the Value Cache preferentially adopts the first layout. Both layouts can be used interchangeably in the same system depending on the model structure and hardware configuration.

[0179] In some optional embodiments, the RISC-V vector processing unit is further configured to, in grouped query attention or multi-query attention scenarios, for the same key-value cache block shared by multiple query heads: load the key data block only once, and schedule the matrix-vector multiplication operation hardware module multiple times to perform query-key matrix-vector multiplication for each of the multiple query heads; load the value data block only once, and schedule the matrix-vector multiplication operation hardware module multiple times to perform probability-value matrix-vector multiplication for each of the multiple query heads.

[0180] In standard Multi-Head Attention (MHA), each query head has its own independent key and value headers, meaning the number of query heads is equal to the number of KV heads. In GQA, multiple query heads are divided into groups, with query heads within the same group sharing the same KV header set. In MQA, all query heads share a unique KV header set. For example, with 8 query heads: the number of KV heads is 8 in MHA configuration, 4 in GQA configuration (each pair of query heads shares one KV header set), and only 1 in MQA configuration (all 8 query heads share one KV header set). While this sharing mechanism significantly reduces the storage overhead of the KV cache, it introduces a new scheduling problem during the decoding phase. The same set of KV data needs to be used sequentially to serve the computation of multiple query heads within the same group. If KV data is loaded separately for each query head in the conventional way, the same data will be read repeatedly, significantly increasing the storage bandwidth pressure.

[0181] Figure 5 The solution of the illustrated embodiment fully leverages the flexibility of the matrix-vector multiplication hardware module, which can be scheduled multiple times and accept different query vector inputs each time. The RISC-V vector processing unit independently maintains its own online normalized exponential function state for each query header that shares the same set of KV headers. , where h is the Query header number.

[0182] For the same key-value cache block shared by multiple query headers, the processing flow is divided into two stages:

[0183] QK multiplexing phase: The shared key data block is loaded only once from the storage subsystem into the input buffer of the matrix-vector multiplication hardware module. Then, the RISC-V vector processing unit schedules the matrix-vector multiplication hardware module multiple times, using the query vectors of each query header sequentially or in parallel. Perform QK GEMV with the loaded key data blocks to obtain the score blocks of each query header. Subsequently, the RISC-V vector processing unit performs maximum value reduction on the score blocks of each query header and updates their respective global maximum values. The scheduling exponential operation hardware module calculates its respective scaling factor. and index weight Update their respective indices and .

[0184] PV multiplexing phase: The shared value data block is loaded only once from the storage subsystem into the input buffer of the matrix-vector multiplication hardware module. Then, the RISC-V vector processing unit schedules the matrix-vector multiplication hardware module multiple times, using the values ​​calculated in the previous phase from each query header. and Combined with the loaded value data blocks, perform PV GEMV to update their respective output accumulation vectors. .

[0185] Through the aforementioned "load once, schedule multiple times" approach, the same key-value cache block is transferred only once between the storage subsystem and the matrix-vector multiplication hardware module, thus serving the computation of all query headers within the same group. The online normalized exponential function state of each query header is maintained independently, without interference, and ultimately each generates its own attention output independently.

[0186] This embodiment is also applicable to paged key-value cache scenarios. Under paging management, the RISC-V vector processing unit or address management logic first resolves the logical data block address into a physical address, and then executes the above-mentioned multiplexing process. Paging management and GQA / MQA multiplexing mechanisms are independent of each other and can be used in combination.

[0187] In a paged KV cache scenario, block table parsing and the translation of logical addresses to physical addresses are handled by the RISC-V main processor, runtime system, or independent address management logic. The matrix-vector multiplication hardware module receives the parsed physical key block address, value block address, and valid length, avoiding the need to embed complex paging management within the matrix-vector multiplication hardware module. In scenarios where GQA / MQA is used in conjunction with a paged KV cache:

[0188] 1. The RISC-V main processor first resolves the current logical block into a physical block address based on the block table;

[0189] 2. The RISC-V / RVV scheduling matrix-vector multiplication hardware module executes QKGEMV on multiple query heads respectively, reusing the loaded physical key blocks;

[0190] 3. Maintain the Softmax state independently for each query head. , , );

[0191] 4. The scheduling matrix-vector multiplication hardware module performs PV GEMV on multiple query heads, reusing the loaded physical value blocks.

[0192] The two mechanisms are independent of each other and can be used in combination to further improve storage bandwidth efficiency in long context scenarios.

[0193] As can be seen from the above description, in long-context decoding scenarios, KV Cache reading is the main storage bandwidth bottleneck. This solution effectively alleviates this problem by reusing loading results. Furthermore, since the reusing logic is implemented in software by the RISC-V vector processing unit using a scheduling mechanism, there is no need to add additional multi-query header management logic to the matrix-vector multiplication hardware module, maintaining the simplicity and efficiency of dedicated hardware. Complex scheduling, paged KV Cache, GQA / MQA, tail block, and output normalization are all handled in software by the RISC-V vector processing unit, resulting in clear hardware boundaries and controllable implementation costs. Through "load once, schedule multiple times," the same key-value cache block only needs to be transmitted once to serve all query headers in the same group, significantly reducing redundant KV Cache reads.

[0194] In some optional embodiments, the apparatus further includes address management logic for performing the conversion of the logical address of the paging key-value cache to a physical address, and providing the parsed physical address to the matrix-vector multiplication hardware module.

[0195] In large language model inference, the key-value (KV) cache needs to be dynamically managed in memory. As the context length increases, the storage requirements of the KV cache increase linearly. Pre-allocating a contiguous physical storage space for each sequence would lead to severe memory fragmentation and low utilization. Therefore, mainstream large language model inference frameworks generally adopt a paging management mechanism. Under this mechanism, logically continuous word sequences may be scattered across multiple non-contiguous physical pages. If the matrix-vector multiplication hardware module directly perceives and processes this paging logic, the hardware design becomes extremely complex. The hardware needs to maintain page tables, handle cross-page access, and respond to page boundary interruptions. This not only increases hardware area and power consumption but also tightly couples the hardware with the paging strategy of the specific inference framework, reducing versatility and scalability.

[0196] To this end, some embodiments of the present invention assign complex, irregular address management tasks to software or independent logic processing, allowing dedicated hardware to handle only regular computational tasks. Specifically, the apparatus of the present invention further includes address management logic. The address management logic is responsible for maintaining page tables, recording the physical key data block address and value data block address corresponding to each logical data block, as well as the number of valid tokens in that data block.

[0197] During attention decoding, the address management logic queries the page table based on the currently processed logical data block number to resolve the logical address into a physical address. After resolution, the physical key data block address, physical value data block address, and effective length are provided to the matrix-vector multiplication hardware module. The matrix-vector multiplication hardware module only receives these resolved physical address parameters and directly initiates continuous data reading and calculation according to the physical address, without needing to be aware of or understand the underlying paging logic.

[0198] When cross-page access is required, the address management logic splits the terms at the page boundary into two logical data blocks for separate processing, with each logical data block corresponding to an independent physical page. In this way, the matrix-vector multiplication hardware module always processes data blocks with contiguous physical addresses, significantly simplifying the hardware design. Simultaneously, the software side can flexibly adjust the paging strategy based on model characteristics and memory conditions without modifying the hardware.

[0199] A second aspect of the present invention provides an attention decoding acceleration method, which can be used in the apparatus of any of the first aspects of the present invention. In some embodiments, the method includes: dividing a key cache and a value cache into multiple data blocks; initializing an online normalized exponential function state; sequentially performing block-based streaming processing on the multiple data blocks to update the online normalized exponential function state, including: scheduling a matrix-vector multiplication hardware module to generate a score block based on a query vector and the current key data block; performing maximum value reduction on the score block and updating the global maximum value in the online normalized exponential function state; scheduling an exponential operation hardware module to calculate a scaling factor and an exponential weight; updating the exponential sum in the online normalized exponential function state based on the scaling factor and the exponential weight; scheduling the matrix-vector multiplication hardware module to update the output accumulation vector in the online normalized exponential function state based on the exponential weight, the current value data block, and the scaling factor; and normalizing the output accumulation vector after all data blocks have been processed to obtain a decoded output.

[0200] The method shown in this embodiment mainly comprises four stages. The first stage is data partitioning: the key cache K and value cache V are divided into multiple data blocks along the lexical dimension, with each data block containing B lexical units. The second stage is state initialization: the online normalized exponential function state is set to its initial value. The third stage is block-based streaming processing: five sub-steps are executed sequentially for each data block. Specifically, the matrix-vector multiplication hardware module is scheduled to perform QK GEMV to obtain a score block; the score block undergoes maximum value reduction and the global maximum value is updated; the exponential operation hardware module is scheduled to calculate the scaling factor and exponential weights, and the exponential sum is updated; the matrix-vector multiplication hardware module is scheduled to perform PV GEMV to update the output accumulated vector. Data dependencies exist between the sub-steps: the score block is the input for maximum value reduction and exponential weight calculation; the scaling factor and exponential weights are the inputs for exponential sum updates and PV GEMV. The fourth stage is output normalization: after all data blocks have been processed, the output accumulated vector is normalized to obtain the final decoded output.

[0201] This method can be executed on a RISC-V vector processing unit, where data block partitioning and state initialization are performed once at the start of the process, the five sub-steps of block streaming processing are executed cyclically for each data block, and normalization is performed once after all data blocks have been processed.

[0202] Compared to standard attention computation methods that process all lexical units at once, the acceleration method shown in this embodiment distributes computational and storage overhead across the processing of individual data blocks, making the data size processed in a single run manageable. By scheduling dedicated hardware to perform computationally intensive matrix-vector multiplication and exponential operations, the general-purpose processor is freed from heavy computational tasks, allowing it to focus on process control and state management.

[0203] In some optional embodiments, the online normalized exponential function state includes a global maximum value, an exponent sum, and an output accumulation vector; the initialization of the online normalized exponential function state includes setting the global maximum value to negative infinity, the exponent sum to zero, and the output accumulation vector to zero.

[0204] This embodiment clarifies the triplet composition of the online normalized exponential function state and its initial value setting. The online normalized exponential function state includes three variables: the global maximum value, the global maximum value, and the global maximum value. Used to track the maximum score, index, and... of processed data blocks. The sum of the exponential weights of the processed data blocks is used to accumulate the total weights (as the denominator for the final normalization), outputting the accumulated vector. Used to accumulate the value vector weighted result of processed data blocks (as the numerator of the final normalization). The initial value is set to... , , .

[0205] This initialization value ensures the correct processing of the first data block: when hour, This causes the state update of the first data block to degenerate into directly assigning the result of the current data block, without relying on any historical data.

[0206] In some optional embodiments, the method further includes: updating the global maximum value in the online normalized exponential function state, comprising: comparing the reduced maximum value of the score block with the current global maximum value, and taking the larger one as the new global maximum value; updating the exponent sum in the online normalized exponential function state, comprising: multiplying the old exponent sum by the scaling factor, and then adding the sum of the exponent weights of the current data block; updating the output accumulation vector in the online normalized exponential function state, comprising: multiplying the old output accumulation vector by the scaling factor, and then adding the weighted sum of each value in the current value data block and the exponent weights.

[0207] This embodiment further describes the specific operation methods for updating the three state variables. This includes: updating the global maximum value using a comparison operation, i.e. This determines the subsequent scaling factor. The value. If the current data block does not produce a higher score. 1. No scaling is required for the old state.

[0208] The update of the exponent sum uses a two-step scaling-accumulation operation, that is, first multiplying the old exponent sum by... Scaling is applied, plus the sum of the index weights of each term in the current data block. .

[0209] The update of the output accumulated vector also adopts the scaling-accumulation mode, that is, the old output accumulated vector is first multiplied by... Scaling is applied, plus a weighted sum of the value vectors of the current data block and their corresponding exponential weights. .

[0210] All three update operations can be executed efficiently on the vector processing unit using vector instructions, without needing to access the original data of the processed data block.

[0211] In some optional embodiments, the method further includes: the scaling factor being the exponential value of the difference between the old global maximum and the new global maximum; and the exponential weight being the exponential value of the difference between each score in the score block and the new global maximum. The scaling factor is defined as follows: .because The range of α is limited to (0,1]. When the global maximum value remains unchanged, α=1, indicating that the old state does not need to be scaled; when the global maximum value increases, α<1, and the old state is compressed proportionally to compensate for the probability distribution shift.

[0212] Exponential weight is defined as .because The maximum score for all processed tokens. The input to the exponential function is always a non-positive number. The range of values ​​is limited to (0,1], thus ensuring that the values ​​will not overflow. Both exponent calculations can be performed by calling the exponent operation hardware module.

[0213] In some optional embodiments, the scheduling exponent calculation hardware module in the method calculates the scaling factor and exponent weight, including: decomposing the input of the exponent value to be calculated into a first part and a second part; calculating the exponent approximation of the second part; and multiplying the exponent approximation by the scaling factor generated based on the first part to obtain the final exponent result.

[0214] In some optional embodiments, the method further includes: performing unpacking and dequantization operations when reading quantized key data or value data, and directly using the dequantization result for matrix-vector multiplication. When the key cache or value cache is stored in a low-bit quantization format such as INT8 or INT4, the unpacking and dequantization operations are completed synchronously during the data reading stage. The dequantized data is not transferred through external memory and is directly sent to the matrix-vector multiplication operation path to participate in subsequent multiplication and addition calculations. This method embeds the dequantization operation into the data path between data reading and calculation, making the entire process from reading quantized data from the storage subsystem to completing the matrix-vector multiplication operation a continuous pipeline.

[0215] In some optional embodiments, the method further includes storing the key cache and value cache according to one of the following data layouts: a first layout stores data in the order of attention head, block, terminology within the block, and channel, so that complete channel data of the same terminology are arranged continuously; a second layout stores data in the order of attention head, block, channel fragment, and terminology within the block, so that data of multiple consecutive terminology within the same channel fragment are arranged continuously. This embodiment provides two optional data storage layouts for key cache and value cache. The first layout (terminology-first layout) organizes data in the order of attention head, data block, terminology within the data block, and channel, with complete channel data of the same terminology arranged continuously in physical storage. This layout is beneficial for scenarios in PV GEMV computation where value vectors are read by terminology. The second layout (channel fragment-first layout) organizes data in the order of attention head, data block, channel fragment, and terminology within the data block, with data of multiple terminology within the same channel fragment arranged continuously in physical storage. This layout is beneficial for scenarios in QK GEMV computation where data of the same channel is read along the terminology direction. In practical applications, the two layouts can be flexibly selected or used in combination depending on the type of computation. The meaning of each dimension and a detailed analysis of the two layouts have been explained in the device section and will not be repeated here.

[0216] In some optional embodiments, in grouped query attention or multi-query attention scenarios, the method further includes: for the same key-value cache block shared by multiple query heads, loading the key data block only once, and scheduling the matrix-vector multiplication operation hardware module multiple times to perform query-key matrix-vector multiplication for each of the multiple query heads; loading the value data block only once, and scheduling the matrix-vector multiplication operation hardware module multiple times to perform probability-value matrix-vector multiplication for each of the multiple query heads. In practice, when multiple query heads share the same KV cache, during the processing of the shared data block, the key data block only needs to be loaded from the storage subsystem once, and then for each query head sharing the data block, the matrix-vector multiplication operation hardware module is scheduled to perform QK GEMV, and each query head uses its own query vector to obtain its own score block. Similarly, the value data block is also loaded only once, and then the matrix-vector multiplication operation hardware module is scheduled to perform PV GEMV for each query head. The online normalized exponential function state of each query head is maintained independently and does not interfere with each other. This method optimizes the KV Cache loading operation from "loading multiple times per query header" to "loading once per KV data block", reducing the number of loading times from the number of query headers to 1.

[0217] In some optional embodiments, the method further includes: performing a logical address to physical address conversion of the paging key-value cache; and providing the converted physical address to the matrix-vector multiplication hardware module. When the KV Cache uses paging management, there is a mapping relationship between the logical address and physical address of the data block. Before performing the matrix-vector multiplication operation, this method first completes the logical address to physical address conversion and provides the parsed physical address as a parameter to the matrix-vector multiplication hardware module. The matrix-vector multiplication hardware module directly accesses the data according to the physical address, without needing to be aware of the mapping relationship between the logical address and the physical address. The address conversion operation can be completed on the software side by the RISC-V main processor, runtime system, or independent address management logic, and the conversion granularity corresponds to the data block size.

[0218] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art, after considering the specification and practicing the technical solutions disclosed in this application, will readily conceive of other embodiments of this application. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary technical means in the art not disclosed in this application. The specification and embodiments are considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0219] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An attention decoding acceleration device, characterized in that, include: RISC-V vector processing unit, exponentiation hardware module, matrix-vector multiplication hardware module, and storage subsystem; in, The exponent calculation hardware module is used to calculate the exponent value required for the normalized exponent function; The matrix-vector multiplication hardware module is used to perform matrix-vector multiplication of the query vector and the key data block, as well as matrix-vector multiplication of the weight vector and the value data block. The RISC-V vector processing unit is used to maintain the state of the online normalized exponential function, and to process the key cache and value cache in the storage subsystem in a block-stream manner by scheduling the exponential operation hardware module and the matrix-vector multiplication operation hardware module, so as to update the state of the online normalized exponential function and generate the decoded output.

2. The apparatus according to claim 1, characterized in that, The RISC-V vector processing unit is configured to process the key cache and value cache in a block-stream manner, including: The matrix-vector multiplication hardware module is scheduled to generate a score block based on the query vector and the current key data block. The maximum value of the score block is reduced, and the global maximum value in the state of the online normalized exponential function is updated; The exponential calculation hardware module is scheduled to calculate the scaling factor and exponential weight; Based on the scaling factor and the exponential weight, update the exponent sum in the state of the online normalized exponential function; The matrix-vector multiplication hardware module is scheduled to update the output accumulation vector in the online normalized exponential function state based on the exponential weight, the current value data block, and the scaling factor. After all data blocks have been processed, the output accumulation vector is normalized to obtain the attention output.

3. The apparatus according to claim 2, characterized in that, The state of the online normalized exponential function includes the global maximum value, the exponent, and the output accumulation vector; The RISC-V vector processing unit is configured as follows: The reduced maximum value of the score block is compared with the current global maximum value, and the larger one is taken as the new global maximum value; The old exponent sum is multiplied by the scaling factor and then added to the sum of the exponent weights of the current data block to update the exponent sum; The old output accumulation vector is multiplied by the scaling factor, and then the weighted sum of each value in the current value data block and the exponential weight is added to update the output accumulation vector.

4. The apparatus according to claim 3, characterized in that, The exponentiation hardware module is used for: Calculate the exponential value of the difference between the old global maximum and the new global maximum, and use it as the scaling factor; Calculate the exponential value of the difference between each score in the score block and the new global maximum value, and use it as the exponential weight.

5. The apparatus according to claim 1, characterized in that, The exponentiation hardware module includes: The range reduction module is used to decompose the input into a first part and a second part; The core of the exponential approximation calculation is used to calculate the exponential approximation value of the second part; The scaling and normalization module is used to combine the exponential approximation with the scaling factor generated based on the first part to obtain the final exponential result.

6. The apparatus according to claim 1, characterized in that, The matrix-vector multiplication hardware module includes a multiplication-addition reduction module and a quantization decompression and inverse quantization fusion module. The multiply-add reduction module is used to perform dot product reduction of each key vector in the key data block with the query vector to generate a score block, and to perform weighted accumulation of each value vector in the value data block with the corresponding exponential weight to update the output accumulation vector. The quantization unpacking and dequantization fusion module is used to perform unpacking and dequantization operations in real time when reading quantized key data or value data from the storage subsystem, and directly send the dequantization result to the multiply-accumulate reduction module for calculation.

7. The apparatus according to claim 1, characterized in that, The key cache and value cache are organized according to one of the following two data layouts: The first layout stores information sequentially by attention header, block, word within block, and channel, so that the complete channel data of the same word is arranged continuously. The second layout stores data sequentially by attention header, block, channel segment, and word element within the block, so that data of multiple consecutive word elements within the same channel segment are arranged continuously.

8. The apparatus according to claim 1, characterized in that, The RISC-V vector processing unit is further configured to, in grouped query attention or multi-query attention scenarios, handle the same key-value cache block shared by multiple query headers: The key data block is loaded only once, and the matrix-vector multiplication hardware module is scheduled multiple times to perform query-key matrix-vector multiplication for each of the multiple query heads; The value data block is loaded only once, and the matrix-vector multiplication hardware module is scheduled multiple times to perform probability-value matrix-vector multiplication on the multiple query heads respectively.

9. The apparatus according to claim 1, characterized in that, The device further includes address management logic for performing the conversion of the logical address to the physical address of the paging key-value cache, and providing the parsed physical address to the matrix-vector multiplication hardware module.

10. A method for accelerating attention decoding, characterized in that, include: Divide the key cache and value cache into multiple data blocks respectively; Initialize the state of the online normalized exponential function; The plurality of data blocks are sequentially processed in a block-based streaming manner to update the state of the online normalized exponential function, including: The scheduling matrix-vector multiplication hardware module generates score blocks based on the query vector and the current key data block; The maximum value of the score block is reduced, and the global maximum value in the state of the online normalized exponential function is updated; Schedule the exponential calculation hardware module to calculate the scaling factor and exponential weight; Based on the scaling factor and the exponential weight, update the exponent sum in the state of the online normalized exponential function; The matrix-vector multiplication hardware module is scheduled to update the output accumulation vector in the online normalized exponential function state based on the exponential weight, the current value data block, and the scaling factor. After all data blocks have been processed, the output accumulation vector is normalized to obtain the decoded output.

11. The method according to claim 10, characterized in that, The state of the online normalized exponential function includes the global maximum value, the exponent, and the output accumulation vector; The initialization of the online normalized exponential function state includes: setting the global maximum value to negative infinity, the exponent sum to zero, and the output accumulation vector to zero.

12. The method according to claim 11, characterized in that, The step of updating the global maximum value in the state of the online normalized exponential function includes: comparing the reduced maximum value of the score block with the current global maximum value, and taking the larger one as the new global maximum value; The step of updating the sum of exponents in the state of the online normalized exponential function includes: multiplying the old sum of exponents by the scaling factor and then adding the sum of the exponent weights of the current data block. The step of updating the output accumulation vector in the state of the online normalized exponential function includes: multiplying the old output accumulation vector by the scaling factor, and then adding the weighted sum of each value in the current value data block and the exponential weight.

13. The method according to claim 12, characterized in that, The scaling factor is an exponential value of the difference between the old global maximum value and the new global maximum value; The index weight is the index value of the difference between each score in the score block and the new global maximum value.

14. The method according to claim 10, characterized in that, The scheduling exponent calculation hardware module calculates the scaling factor and exponent weights, including: The input for calculating the exponent value is decomposed into a first part and a second part; Calculate the exponential approximation of the second part; The final exponential result is obtained by multiplying the approximate exponential value by the scaling factor generated based on the first part.

15. The method according to claim 10, characterized in that, Also includes: When reading key or value data in quantized format, unpacking and dequantization operations are performed, and the dequantization results are directly used for matrix-vector multiplication operations.

16. The method according to claim 10, characterized in that, Also includes: The key cache and value cache are stored according to one of the following data layouts: The first layout stores information sequentially by attention header, block, word within block, and channel, so that the complete channel data of the same word is arranged continuously. The second layout stores data sequentially by attention header, block, channel segment, and word element within the block, so that data of multiple consecutive words within the same channel segment are arranged continuously.

17. The method according to claim 10, wherein, In scenarios involving grouped query attention or multi-query attention, the method further includes: For the same key-value cache block shared by multiple query heads, the key data block is loaded only once, and the matrix-vector multiplication hardware module is scheduled multiple times to perform query-key matrix-vector multiplication for each of the multiple query heads. The value data block is loaded only once, and the matrix-vector multiplication hardware module is scheduled multiple times to perform probability-value matrix-vector multiplication on the multiple query heads respectively.

18. The method according to claim 10, characterized in that, Also includes: Perform the translation of the logical address to the physical address of the pagination key-value cache; The converted physical address is provided to the matrix-vector multiplication hardware module.

Images