A hardware accelerator for realizing attention weighted summation and its operation method

Abstract

The present invention relates to the technical field of integrated circuit chips and discloses a hardware accelerator for implementing weighted summation of attention. The hardware accelerator comprises a control module, a dynamic random access memory, a ping-pong static random access memory, a data buffer, an input first-in-first-out queue module, an output first-in-first-out queue module, a processing unit array, an exp module, an addition tree module, and a log2 module. The hardware accelerator created by the present invention can realize efficient parallel computing and reduce memory access delay by designing a special circuit structure, thereby improving the calculation speed of the weighted summation of attention.

Description

Technical Field

[0001] The present invention relates to the technical field of integrated circuit chips, and in particular to a hardware accelerator for implementing attention weighted summation and an operating method thereof. Background Art

[0002] The Transformer architecture initially achieved great success in natural language processing and has been widely used in image recognition tasks in recent years, demonstrating its strong performance and potential. The attention mechanism plays a core role in the Transformer, and weighted attention summation is a key step in its implementation. Existing attention mechanism implementations typically use software running on general-purpose processors. However, due to processor architectural limitations and instruction execution order constraints, they struggle to fully utilize parallel computing resources. Furthermore, frequent memory accesses can lead to significant latency, reducing computational efficiency. Summary of the Invention

[0003] In order to solve the above technical problems, the present invention provides a hardware accelerator for realizing attention weighted summation. By designing a special circuit structure, efficient parallel computing can be achieved, memory access delay can be reduced, and the calculation speed of attention weighted summation can be greatly improved.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] A hardware accelerator for implementing attention weighted summation, comprising:

[0006] Control module: used to control the reading and writing of dynamic random access memory, the reading and writing of ping-pong static random access memory, and the data input source of the data buffer;

[0007] Dynamic random access memory: used to store the matrix to be calculated and the results of the attention weighted summation;

[0008] Ping-Pong SRAM: includes Ping SRAM and Pong SRAM, capable of time-sharing read and write. The input sources of Ping-Pong SRAM include dynamic random access memory read, data buffer output, and log2 module output. The output of Ping-Pong SRAM is sent to dynamic random access memory for writing, sent to the data buffer, or sent to the input first-in-first-out queue module.

[0009] Data buffer: contains multiple register groups, used to rearrange the data stored in the ping-pong static random access memory, or to splice the output results of the output first-in first-out queue module;

[0010] Input FIFO queue module: used to receive data from the ping-pong static random access memory and send it to the processing unit array;

[0011] Output FIFO module: used to receive data from the processing unit array and send it to the ping-pong static random access memory, data buffer or exp module;

[0012] Processing unit array: includes multiple processing units, each of which can receive one data from the input first-in first-out queue module in each cycle, and the processing unit in the rightmost column is an output processing unit, which can output one data to the output first-in first-out queue module in each cycle;

[0013] exp module: used to calculate the e-exponential result of the output data of the output FIFO queue module, and synchronously send the e-exponential result to the addition tree module and the log2 module;

[0014] Addition tree module: used to accumulate the e exponential results output by the exp module and send the accumulated sum to the log2 module;

[0015] log2 module: used to perform base-2 logarithm operations on the e-exponential results output by the exp module, and to perform base-2 logarithm operations on the accumulated sum of the e-exponential results output by the addition tree module, and send the results back to the ping-pong static random access memory.

[0016] Furthermore, the processing unit array is used to calculate the formula The max(x) operation in x i -max(x) operation, and Subtraction operations, and shift and accumulation operations in shift-and-accumulate;

[0017] Each processing unit supports addition, subtraction, multiplication, shift and comparison operations;

[0018] Each processing unit is equipped with an instruction static random access memory for storing instruction data. After the processing unit starts working, it reads instructions one by one to guide the data fetching, sending and calculation process.

[0019] Each processing unit is equipped with a data static random access memory for pre-storing calculation data in batches. In the shift-accumulation operation of the attention weight matrix and the value matrix, the attention weight matrix is ​​first pre-stored in the data static random access memory, and then the corresponding calculation data is read out in each cycle and shift-accumulated with the value matrix sent by the input first-in-first-out queue module;

[0020] Each processing unit is configured with two local registers for storing quantization factor data or temporary accumulated data required during the calculation process;

[0021] Each processing unit supports left and right interconnection, transmitting one data to the left or right processing unit in each cycle;

[0022] The processing unit can receive data sent by the input first-in first-out queue module at the same time, and can also receive data at intervals according to calculation needs.

[0023] Furthermore, the exp module is used to calculate the formula The e exponential operation in .

[0024] Furthermore, the addition tree module is used to calculate the formula The accumulation operation of n numbers is performed, the calculation results output by the exp module are accumulated, and the results are sent to the log2 module.

[0025] Furthermore, the log2 module utilizes the characteristics of binary numbers to approximately implement log2 operations through shift and addition operations, and can respectively calculate the output data of the exp module and the output data of the addition tree module.

[0026] A method for operating a hardware accelerator for implementing weighted attention summation, for performing weighted attention summation on a Transformer architecture, wherein the operating method comprises the following steps:

[0027] S1, the control module reads data from the dynamic random access memory and writes it into the ping-pong static random access memory in the ping-pong static random access memory; the ping-pong static random access memory reads data and writes it into the data static random access memory of the processing unit;

[0028] S2, the control module reads the instruction of the processing unit array from the dynamic random access memory and writes it into the pong static random access memory in the ping-pong static random access memory; the pong static random access memory reads the data and writes it into the instruction static random access memory of the processing unit;

[0029] S3, the processing units of the processing unit array start processing and output the processing results to the output first-in first-out queue module;

[0030] S4, the exp module receives the operation result output by the output FIFO queue module, and simultaneously calculates the e index of the data in a parallel processing manner, and outputs the e index result to the addition tree module and the log2 module;

[0031] S5, the addition tree module completes the accumulation of the e-index results in multiple times and sends the accumulated sum to the log2 module;

[0032] S6, the log2 module first performs a log2 operation on the input e exponential result and writes the calculation result into the ping static random access memory; then, the log2 operation is performed on the accumulated sum input from the addition tree module, and the log2 operation result is copied and filled and then written into the pong static random access memory;

[0033] S7, the control module writes the calculation result in the ping static random access memory into the data static random access memory of the processing unit array;

[0034] S8, the control module sends the calculation result in the Pong SRAM to the processing unit array through the input FIFO module. Each processing unit reads the data pre-stored in the local data SRAM, subtracts it from the data sent by the input FIFO module, obtains the difference, and writes it back to the local SRAM.

[0035] S9, after all differences are calculated, the attention weight results of the first m rows of the scaled attention score matrix are stored in the processing unit array;

[0036] S10, the control module reads the value matrix from the dynamic random access memory and writes it into the static random access memory; since the matrix data is arranged in the dynamic random access memory in the order of rows first and columns later, the data read out is the first s columns of the V matrix;

[0037] S11, the control module sends the first s columns of V matrix data in the ping static random access memory to the control module for rearrangement, and the rearranged data is the transposed s columns of V matrix data, and the rearranged data is written into the pong static random access memory;

[0038] S12: Send the rearranged s columns of V matrix data in the pong static random access memory to the processing unit array through the input first-in-first-out queue module; each row of processing units simultaneously receives one column of V matrix data and performs shift accumulation to obtain the first m rows of the first column of the attention output matrix, and writes it into the output first-in-first-out queue module through the rightmost processing unit;

[0039] S13, the output FIFO queue module sends the first m rows of data received successively into the data buffer for rearrangement, and writes the rearranged first m rows of data into the Pong SRAM;

[0040] S14, the control module writes the first m rows of data in the pong static random access memory back to the dynamic random access memory, completing the calculation and storage of the first m rows of data in the first s columns of the attention output matrix;

[0041] S15, by repeating steps S10 to S14, the calculation and storage of the first m rows of data of all columns of the attention output matrix are completed;

[0042] S16, by repeating steps S1 to S15, the calculation and storage of all data of the attention output matrix are completed.

[0043] Specifically, step S3 includes the following steps:

[0044] S31: Use the arithmetic logic unit to find the maximum value in the data and write it to the local register;

[0045] S32: Then, the maximum value is subtracted from each data using the arithmetic logic unit, and the data is sent to the output first-in-first-out queue module;

[0046] S33: The output FIFO queue module only receives the output of the rightmost processing unit, so the calculation results of other column processing units need to be passed to the left through the processing unit to the right of the column processing unit until they are written into the output FIFO queue module.

[0047] Compared with the prior art, the beneficial technical effects of the present invention are:

[0048] (1) By designing a data buffer, the data arrangement problem of the multiplied matrix and the data arrangement problem of the calculation result in the matrix multiplication operation is solved.

[0049] To ensure consistency in data arrangement between layers, data of the same type must be stored in the same arrangement during network inference. Matrix data is generally arranged in row order first, then in column order. When performing matrix multiplication, the multiplying matrix and the multiplied matrix are arranged in the same manner, and the matrix multiplication operation is the dot product of the row vectors of the multiplied matrix and the column vectors of the multiplied matrix. When continuously reading the multiplied matrix data, the read data is the row vectors of the matrix, which makes it difficult to deploy it in the PE array for parallel operation with the row vectors of the multiplication matrix.

[0050] To increase computational parallelism and efficiency, the hardware accelerator of the present invention adds a data buffer for transposing column vectors into row vectors. This process is accomplished using a ping-pong static random access memory (SRAM) and a data buffer. The multiplied matrix is ​​first stored in the ping-pong SRAM, then fed into the data buffer in a pre-set order. The rearranged data in the data buffer is then written to the pong SRAM for storage. The transposed multiplied matrix stored in the pong SRAM can be directly fed into the PE array for parallel computation with the row vectors of the multiplication matrix.

[0051] Similarly, for calculation results output by the PE array in column order, the calculation results are converted to row order by outputting the results to the data buffer for rearrangement, so as to be directly written into the memory.

[0052] (2) By designing a ping-pong static random access memory module, the calculation and loading processes of different group block data can be executed in parallel, hiding the loading time of data from the memory and improving the calculation speed.

[0053] The ping-pong static random access memory (SRAM) module is used to preload the data to be calculated. This accelerator sets the data read from memory to be written to the ping-pong SRAM, and the rearranged data to be written to the pong SRAM. After the calculation is completed, the result is sent to the data buffer for rearrangement and written back to the pong SRAM, without overwriting the data stored in the ping-pong SRAM. Therefore, while performing matrix multiplication on the current data, the next set of matrix data to be calculated can be pre-fetched in advance, thus minimizing the time it takes to read data and reducing the latency of the entire hardware accelerator in executing the attention-weighted summation.

[0054] (3) By using four groups of 1×4 PE arrays, parallel operations with a parallelism of 4 are achieved; by designing data transmission paths between each group of PEs, fast maximum and sum operations are achieved.

[0055] Using a 4-row 1×4 PE array, the four row vectors of the input matrix are computed in parallel, accelerating the attention-weighted summation operation. By designing a data transmission path between the four PEs in each row, the maximum value of the row vector can be quickly found by comparing data between the four PEs. The vector inner product operation can also be performed by accumulating the results of the four PEs to quickly calculate the final sum.

[0056] By performing log2 quantization on the softmax output and converting the exp formula, an efficient and simple softmax calculation architecture is implemented. Compared with hardware implementations without quantization, this further reduces storage overhead and computational complexity.

[0057] (4) As a hardware accelerator designed and implemented based on Softmax shift accumulation to realize attention weighted summation, the present invention can effectively improve the computational efficiency of attention weighted summation based on Softmax shift accumulation. By designing the data flow path and data rearrangement, the data transmission delay and data loading delay are further reduced, thereby improving the computational efficiency of the accelerator. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 FIG1 is a diagram showing the overall architecture of a hardware accelerator in an embodiment of the present invention;

[0059] Figure 2 This is a diagram of the internal architecture of a single processing unit in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0061] 1. First, let’s introduce the self-attention calculation process in the Transformer architecture:

[0062] 1. Calculate the query (Query, Q), key (Key, K) and value (Value, V) matrix: Usually, Q, K, V are obtained by linear transformation of the input. Assume that the input is X, and the weight matrix W is used. Q 、W K 、W V Perform linear transformation to obtain Q = XW Q , K=XW K 、V=XW V .

[0063] 2. Calculate the attention score: For each pair of query and key, calculate the dot product and get an attention score matrix AttentionScore = QK T , where each element represents the similarity between a query and a key.

[0064] 3. Scaling attention score: In order to prevent the dot product result from being too large, the attention score is scaled to obtain the scaled attention score where d k is the dimension of the key.

[0065] 4. Apply the Softmax function to obtain the attention weights AttentionWeights = softmax(ScaledAttentionScore).

[0066] 5. Calculate the attention output AttentionOutput = AttentionWeightsV.

[0067] Steps 4 and 5 are collectively referred to as the weighted summation of attention, a core operation of the attention mechanism in the Transformer architecture. They primarily consist of two operations: a softmax operation on a single matrix and a matrix multiplication of two matrices.

[0068] The accelerator included in this invention performs log2 quantization on the output of the softmax operation. This method comes from the paper "FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer". The principle is described as follows:

[0069] 1. The original softmax calculation formula is: The molecule is The denominator is The entire formula calculates the relative probability of each element. i represents the i-th element in the input vector, n represents the number of elements in the input vector; σ(x) i It represents the relative probability value of the i-th element in the input vector among all elements. Its value is between 0 and 1, and the sum of all output elements is 1.

[0070] 2. Perform log2 quantization on the output of softmax, and the operation is converted to: Convert division operations into log2 operations and subtraction operations.

[0071] The quantized softmax result can be represented using an integer with fewer bits, reducing storage overhead. The log2 operation can be implemented in hardware using shift circuits, further reducing computational complexity compared to the original division operation in softmax. Correspondingly, the matrix multiplication between the attention score matrix and the value matrix used to calculate the attention output is converted from a multiply-accumulate operation to a shift-accumulate operation, further reducing computational complexity.

[0072] When x in the softmax operation formula i When the value is large, directly calculate This may result in values ​​that are too large, and overflow may occur in hardware implementations. Therefore, we can effectively avoid overflow by first finding the maximum number max(x) in the input vector and then subtracting the maximum number from each number in the input vector.

[0073] because Therefore, the calculation formula is modified to The hardware accelerator included in the present invention uses this formula to calculate the output result of softmax.

[0074] 2. Next, we will introduce the framework of the hardware accelerator. The hardware accelerator mainly includes the following modules:

[0075] Control Unit: The accelerator's control module is mainly used to control the reading and writing of DRAM, the reading and writing of Ping-Pong SRAM, and the data input source of the data buffer.

[0076] Dynamic random access memory (DRAM): used to store matrices to be calculated and the calculation results of the attention weighted sum.

[0077] Ping-Pong SRAM: It contains two SRAMs, namely Ping SRAM and Pong SRAM, which can be read and written in time-sharing mode. The input can come from DRAM read, Data buffer output and log2 module output, and the output can be sent to DRAM write, Data buffer input and Input FIFO.

[0078] The explanations of Ping-Pong SRAM, Ping SRAM, and Pong SRAM are as follows:

[0079] Ping-Pong Static Random Access Memory (Ping-Pong SRAM) is a data buffer structure consisting of two SRAM memory areas that can be read and written alternately. This structure enables continuous data processing, eliminating latency and improving data processing efficiency and system throughput. "Ping SRAM" and "Pong SRAM" refer to the two memory areas in the ping-pong buffer. For example, one memory area is called "Ping SRAM" and the other is called "Pong SRAM."

[0080] Data buffer: Contains 4 rows and 16 columns of registers, which can be used to convert 16 groups of 4×8-bit input data into 4 groups of 128-bit data for output. It is used to rearrange the data stored in the Ping-Pong SRAM or splice the output results of the Output FIFO.

[0081] Input FIFO: receives data from the ping-pong SRAM and sends it to the 4×4 processing unit array.

[0082] Output FIFO: receives data from the 4×4 processing unit array and sends it to the ping-pong SRAM, data buffer, or exp module.

[0083] 4×4 Processing Element Array (PE Array): Contains 16 processing elements (PEs). Each PE can receive one data item from the input FIFO above in each cycle. The four PEs in the rightmost column are output PEs and can output one data item to the output FIFO in each cycle.

[0084] exp module: used to calculate the e-exponential result of the Output FIFO output data and synchronously send the calculation result to the Adder tree and log2 modules.

[0085] Adder tree module: used to accumulate the e-exponential results output by the exp module and send the accumulated sum to the log2 module.

[0086] log2 module: used to perform base-2 logarithm operations on the e-exponential results output by the exp module, and to perform base-2 logarithm operations on the accumulated sum of the e-exponential results output by the Adder tree module, and send the results back to the ping-pong SRAM.

[0087] 3. Next, we will further introduce the calculation module:

[0088] 4×4 processing unit array (PE Array): used to calculate the formula max(x), x in i The subtraction operation of -max(x) and two log2 results, as well as the shift operation and accumulation operation in the shift-accumulate operation, are the most important operation modules in the accelerator.

[0089] Each processing element (PE) supports addition, subtraction, multiplication, shift and comparison operations.

[0090] Each PE is equipped with an instruction SRAM to store instruction data. After the PE starts working, it reads instructions one by one to guide its data fetching, sending and calculation processes.

[0091] Each PE is configured with a data SRAM for pre-storing calculation data in batches. In the shift-and-accumulate operation of two matrices, the AttentionWeights shift matrix is ​​first pre-stored in the data SRAM. Then, the corresponding calculation data is read out in each cycle and shift-and-accumulates with the V matrix sent in by the Input FIFO.

[0092] Each PE is configured with two local registers to store quantization factor data or temporary accumulated data required during the calculation process.

[0093] Each PE supports left and right interconnection and can transmit one data to its left or right PE every cycle.

[0094] The four PE lines can receive data from the Input FIFO simultaneously, or they can receive data at intervals according to computational requirements.

[0095] Each PE can perform an operation in each cycle, so the maximum parallelism of the operation is 16.

[0096] exp module: used to calculate formulas The e-exponential operation in . The quantization method used is derived from the paper "A High-Speed ​​and Low-Complexity Architecture for Softmax Function in Deep Learning". The principle is described as follows:

[0097] First, replace the base e with the base 2, and the exponential function is converted to

[0098] Then x i *log2e is split into integer u i and decimal v i ,u i +v i =x i *log2e

[0099] The formula can be further transformed into:

[0100]

[0101] At this point, the exponential operation is transformed into and a shift operation where The operation is implemented using a lookup table.

[0102] Adder tree module: used to calculate formula The cumulative operation of the number is performed, the calculation results output by the exp module are accumulated, and the results are sent to the log2 module.

[0103] log2 module: Utilizes the characteristics of binary numbers to approximate log2 through shift and addition operations. It can calculate the output data of the exp module and the output data of the Adder tree module respectively.

[0104] Example

[0105] Taking the SWIN Transformer architecture with 8-bit fixed-point quantization and 96 channels as an example, the specific operation of the above accelerator when performing attention weighted summation is described. When the number of channels is 96, the scaled attention score ScaledAttentionScore is a 96x96 matrix, and the value matrix V is 96×d k Since the 4×4 PE Array contains 4 rows of PE groups that can be operated in parallel, the softmax results of the first 4 rows of the scaled attention score ScaledAttentionScore vector can be calculated simultaneously. The length of each vector is 64. The specific process is as follows:

[0106] (1) The Control Unit reads 4x64 8-bit data from the DRAM (the bit width is 128 bits, and the 4x64 8-bit data are stored as 16 128-bit data in the DRAM) and writes them into the Ping SRAM in the Ping SRAM; the Ping SRAM reads the data and writes it into the data SRAM of the PE.

[0107] (2) The Control Unit reads the instructions of the 4×4 PE Array from the DRAM and writes them into the Pong Static Random Access Memory (Pong SRAM) in the Ping-Pong SRAM; the Pong SRAM reads the data and writes it into the instruction SRAM of the PE.

[0108] (3) The 4×4 PE Array starts working, and 4 rows of PE simultaneously calculate 4 1x64 vectors:

[0109] ①First, use the arithmetic logic unit (ALU) to find the maximum value among the 64 data and write it into the local register;

[0110] ②Then use ALU to subtract the maximum value from each of the 64 data and send them to the Output FIFO;

[0111] ③The Output FIFO only receives the output of the rightmost PE. Therefore, the calculation results of the first three columns of PEs need to be passed to the left through the PEs on their right until they are written into the Output FIFO.

[0112] ④ Each PE outputs a 32-bit data, including four 8-bit operation results. The four PEs simultaneously output 16 8-bit operation results.

[0113] (4) The exp module receives 16 8-bit operation results from the Output FIFO and simultaneously calculates the e exponents of the 16 8-bit data with a parallelism of 16, and outputs them to the Adder tree and log2 modules.

[0114] (5) The Adder tree module completes four groups of 64 32-bit e-exponent results in four times, accumulates them, and outputs four 64-bit cumulative sums to be sent to the log2 module.

[0115] (6) The log2 module first performs a log2 operation on the 16 32-bit e exponent results sent in and writes the calculation results into the ping SRAM; then it performs a log2 operation on the 4 64-bit cumulative sums sent in by the Adder tree module. The calculation results are 4 8-bit data. Each 8-bit data is copied and padded to 128-bit data, and the 4 128-bit calculation results are written into the pong SRAM.

[0116] (7) The Control Unit writes the calculation results in the Ping SRAM into the data SRAM of the 4×4 PE Array.

[0117] (8) The Control Unit sends the calculation results in the Pong SRAM to the 4×4 PE Array through the Input FIFO. Each PE reads the data pre-stored in the local data SRAM, subtracts it from the data sent from the Input FIFO, obtains the difference, and writes it back to the local SRAM.

[0118] (9) After all differences are calculated, the AttentionWeights results of the first four rows of the ScaledAttentionScore matrix are stored in the 4×4PE Array.

[0119] (10) The Control Unit reads the V matrix from the DRAM and writes it into the SRAM. Since the matrix data is arranged in the DRAM in the order of rows first and columns later, the data read out is the first 16 columns of the V matrix.

[0120] (11) The Control Unit sends the first 16 columns of V matrix data in the ping SRAM to the Control Unit for rearrangement. The rearranged data is the transposed 16 columns of V matrix data, and the rearranged data is written into the pong SRAM.

[0121] (12) The rearranged 16 columns of V matrix data in the pong SRAM are sent to the 4×4 PE Array through the Input FIFO; 4 rows of PE simultaneously receive 1 column of V matrix data and perform shift accumulation to obtain the first 4 results of the first column of the AttentionOutput matrix, and write them into the Output FIFO through the rightmost PE; while executing the calculation of the first 16 columns (1 to 16 columns) of the first group of V matrices, the last 16 columns of the second group of V matrix data (17 to 32 columns) can be synchronously read from the DRAM and written into the ping SRAM (step (10)). After the calculation and write-back of the first group of data are completed, the rearrangement operation is performed (step (11)).

[0122] (13) Output FIFO sends the 16 groups of 4x8 bits received successively to the Data buffer for rearrangement. The rearranged data is 4 128-bit data and is written into the Pong SRAM.

[0123] (14) The Control Unit writes the four results in the Pong SRAM back to the DRAM. At this point, the calculation and storage process of the first four rows of the first 16 columns of the AttentionOutput matrix is ​​completed.

[0124] (15) The calculation and storage of the first four rows of all columns of the AttentionOutput matrix are completed by repeating steps (10) to (14); step (10) of the next set of data will be carried out in parallel with step (12) of the current set.

[0125] (16) Complete the calculation and storage of the AttentionOutput matrix by repeating steps (1) to (15).

[0126] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. It is intended that all variations within the meaning and range of equivalents of the claims be embraced herein, and any reference signs in the claims should not be construed as limiting the claims to which they relate.

[0127] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A method for operating a hardware accelerator for implementing weighted attention summation for a Transformer architecture, characterized in that: The operation method comprises the following steps: S1, the control module reads data from the dynamic random access memory and writes it into the ping-pong static random access memory in the ping-pong static random access memory; the ping-pong static random access memory reads data and writes it into the data static random access memory of the processing unit; S2, the control module reads the instruction of the processing unit array from the dynamic random access memory and writes it into the pong static random access memory in the ping-pong static random access memory; the pong static random access memory reads the data and writes it into the instruction static random access memory of the processing unit; S3, the processing unit of the processing unit array starts processing and outputs the processing result to the output first-in first-out queue module; S4, the exp module receives the operation result output by the output FIFO queue module, and simultaneously calculates the e index of the data in a parallel processing manner, and outputs the e index result to the addition tree module and the log2 module; S5, the addition tree module completes the accumulation of the e-index results in multiple times and sends the accumulated sum to the log2 module; S6, the log2 module first performs a log2 operation on the input e exponential result and writes the calculation result into the ping static random access memory; then, the log2 operation is performed on the accumulated sum input from the addition tree module, and the log2 operation result is copied and filled and then written into the pong static random access memory; S7, the control module writes the calculation result in the ping static random access memory into the data static random access memory of the processing unit array; S8, the control module sends the calculation result in the Pong SRAM to the processing unit array through the input FIFO module. Each processing unit reads the data pre-stored in the local data SRAM, subtracts it from the data sent by the input FIFO module, obtains the difference, and writes it back to the local SRAM. S9, after all differences are calculated, the attention weight results of the first m rows of the scaled attention score matrix are stored in the processing unit array; S10, the control module reads the value matrix from the dynamic random access memory and writes it into the static random access memory; since the matrix data is arranged in the dynamic random access memory in the order of row first and column later, the read data is The first s columns of the matrix; S11, the control module sends the first s columns of V matrix data in the ping static random access memory to the control module for rearrangement, and the rearranged data is the transposed s columns of V matrix data, and the rearranged data is written into the pong static random access memory; S12: Send the rearranged s columns of V matrix data in the pong static random access memory to the processing unit array through the input first-in-first-out queue module; each row of processing units simultaneously receives one column of V matrix data and performs shift accumulation to obtain the first m rows of the first column of the attention output matrix, and writes it into the output first-in-first-out queue module through the rightmost processing unit; S13, the output FIFO queue module sends the first m rows of data received successively into the data buffer for rearrangement, and writes the rearranged first m rows of data into the Pong SRAM; S14, the control module writes the first m rows of data in the pong static random access memory back to the dynamic random access memory, completing the calculation and storage of the first m rows of data in the first s columns of the attention output matrix; S15, by repeating steps S10 to S14, the calculation and storage of the first m rows of data of all columns of the attention output matrix are completed; S16, by repeating steps S1 to S15, the calculation and storage of all data of the attention output matrix are completed.

2. The method for operating a hardware accelerator for implementing weighted summation of attention according to claim 1, characterized in that: Step S3 includes the following steps: S31: Use the arithmetic logic unit to find the maximum value in the data and write it to the local register; S32: Then, the maximum value is subtracted from each data using the arithmetic logic unit, and the data is sent to the output first-in-first-out queue module; S33: The output FIFO queue module only receives the output of the rightmost processing unit, so the calculation results of other column processing units need to be passed to the left through the processing unit to the right of the column processing unit until they are written into the output FIFO queue module.

3. A hardware accelerator for implementing the operation method according to claim 1 or 2, characterized in that: include: Control module: used to control the reading and writing of dynamic random access memory, the reading and writing of ping-pong static random access memory, and the data input source of the data buffer; Dynamic random access memory: used to store the matrix to be calculated and the results of the attention weighted summation; Ping-Pong SRAM: includes Ping SRAM and Pong SRAM, capable of time-sharing read and write. The input sources of Ping-Pong SRAM include dynamic random access memory read, data buffer output, and log2 module output. The output of Ping-Pong SRAM is sent to dynamic random access memory for writing, sent to the data buffer, or sent to the input first-in-first-out queue module. Data buffer: contains multiple register groups, used to rearrange the data stored in the ping-pong static random access memory, or to splice the output results of the output first-in first-out queue module; Input FIFO queue module: used to receive data from the ping-pong static random access memory and send it to the processing unit array; Output FIFO module: used to receive data from the processing unit array and send it to the ping-pong static random access memory, data buffer or exp module; Processing unit array: includes multiple processing units, each of which can receive one data from the input first-in first-out queue module in each cycle, and the processing unit in the rightmost column is an output processing unit, which can output one data to the output first-in first-out queue module in each cycle; exp module: used to calculate the e-exponential result of the output data of the output FIFO queue module, and synchronously send the e-exponential result to the addition tree module and the log2 module; Addition tree module: used to accumulate the e exponential results output by the exp module and send the accumulated sum to the log2 module; log2 module: used to perform base-2 logarithm operations on the e-exponential results output by the exp module, and to perform base-2 logarithm operations on the accumulated sum of the e-exponential results output by the addition tree module, and send the results back to the ping-pong static random access memory.

4. The hardware accelerator according to claim 3, wherein: The processing unit array is used to calculate the formula in Operations, Operations, and Subtraction operations, and shift and accumulation operations in shift-and-accumulate; Each processing unit supports addition, subtraction, multiplication, shift and comparison operations; Each processing unit is equipped with an instruction static random access memory for storing instruction data. After the processing unit starts working, it reads instructions one by one to guide the data fetching, sending and calculation process. Each processing unit is equipped with a data static random access memory for pre-storing calculation data in batches. In the shift-accumulation operation of the attention weight matrix and the value matrix, the attention weight matrix is ​​first pre-stored in the data static random access memory, and then the corresponding calculation data is read out in each cycle and shift-accumulated with the value matrix sent by the input first-in-first-out queue module; Each processing unit is configured with two local registers for storing the quantization factor data or temporary accumulated data required during the calculation process; Each processing unit supports left and right interconnection, transmitting one data to the left or right processing unit in each cycle; The processing unit can receive data sent by the input first-in first-out queue module at the same time, and can also receive data at intervals according to calculation needs.

5. The hardware accelerator according to claim 3, wherein: The exp module is used to calculate the formula The e exponential operation in .

6. The hardware accelerator according to claim 3, characterized in that The addition tree module is used to calculate the formula The accumulation operation of n numbers is performed, the calculation results output by the exp module are accumulated, and the results are sent to the log2 module.

7. The hardware accelerator according to claim 3, characterized in that The log2 module utilizes the characteristics of binary numbers and approximately implements log2 operations through shift and addition operations, and can respectively calculate the output data of the exp module and the output data of the addition tree module.

Images