Hardware accelerator for realizing attention weighted summation and operation method thereof

Abstract

The invention relates to the technical field of integrated circuit chips, and discloses a hardware accelerator for realizing attention weighted summation, which comprises a control module, a dynamic random access memory, a ping-pong static random access memory, a data buffer, an input first-in first-out queue module, an output first-in first-out queue module, a processing unit array, an exp module, an adder tree module and a log2 module, according to the hardware accelerator, by designing a special circuit structure, efficient parallel computing can be achieved, memory access delay is reduced, and therefore the computing speed of attention weighted summation is increased.

Description

Technical Field

[0001] The present invention relates to the technical field of integrated circuit chips, and in particular to a hardware accelerator for realizing attention weighted summation and an operation method thereof. Background Art

[0002] The Transformer architecture initially achieved great success in the field of natural language processing. In recent years, it has also been widely used in image recognition tasks and has demonstrated strong performance and potential. Among them, the attention mechanism plays a core role in the Transformer, and the weighted sum of attention is one of the key steps to implement the attention mechanism. Existing attention mechanism implementation methods usually use software to run on general-purpose processors. Due to the architectural limitations of the processor and the constraints of the instruction execution order, it is difficult to fully utilize parallel computing resources. At the same time, frequent memory accesses will also cause large delays and reduce computing efficiency. Summary of the invention

[0003] In order to solve the above technical problems, the present invention provides a hardware accelerator for realizing attention weighted summation. By designing a special circuit structure, efficient parallel computing can be achieved and memory access delay can be reduced, thereby greatly improving the calculation speed of attention weighted summation.

[0004] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0005] A hardware accelerator for implementing attention weighted summation, comprising:

[0006] Control module: used to control the reading and writing of dynamic random access memory, the reading and writing of ping-pong static random access memory and the data input source of the data buffer;

[0007] Dynamic random access memory: used to store the matrix to be calculated and the result of the weighted sum of attention;

[0008] Ping-Pong SRAM: includes Ping SRAM and Pong SRAM, which can perform time-sharing reading and writing. The input sources of Ping-Pong SRAM include dynamic random access memory reading, data buffer output and log2 module output. The output of Ping-Pong SRAM is sent to dynamic random access memory for writing, sent to data buffer or sent to input first-in-first-out queue module.

[0009] Data buffer: contains multiple register groups, used to rearrange the data stored in the ping-pong static random access memory, or to splice the output results of the output first-in first-out queue module;

[0010] Input FIFO queue module: used to receive data from the ping-pong static random access memory and send it to the processing unit array;

[0011] Output FIFO module: used to receive data from the processing unit array and send it to the ping-pong static random access memory, data buffer or exp module;

[0012] Processing unit array: comprising a plurality of processing units, wherein each processing unit can receive one data from the input first-in first-out queue module in each cycle, and the processing unit in the rightmost column is an output processing unit, which can output one data to the output first-in first-out queue module in each cycle;

[0013] exp module: used to calculate the e-exponential result of the output data of the output FIFO queue module, and synchronously send the e-exponential result to the addition tree module and the log2 module;

[0014] Addition tree module: used to accumulate the e exponential results output by the exp module and send the accumulated sum to the log2 module;

[0015] log2 module: used to perform logarithmic operation with base 2 on the e-exponential result output by the exp module, and to perform logarithmic operation with base 2 on the accumulated sum of the e-exponential result output by the addition tree module, and send the result back to the ping-pong static random access memory.

[0016] Furthermore, the processing unit array is used to calculate the formula The max(x) operation in x i -max(x) operation, and Subtraction operations, and shift and accumulation operations in shift-and-accumulate;

[0017] Each processing unit supports addition, subtraction, multiplication, shift and comparison operations;

[0018] Each processing unit is equipped with an instruction static random access memory for storing instruction data. After the processing unit starts working, it reads instructions one by one to guide the data fetching, sending and calculation process.

[0019] Each processing unit is equipped with a data static random access memory for pre-storing calculation data in batches. In the shift-accumulation operation of the attention weight matrix and the value matrix, the attention weight matrix is ​​first pre-stored in the data static random access memory, and then the corresponding calculation data is read out in each cycle and shift-accumulated with the value matrix sent by the input first-in-first-out queue module;

[0020] Two local registers are configured in each processing unit to store the quantization factor data or temporary accumulated data required in the calculation process;

[0021] Each processing unit supports left-right interconnection, transferring one data to the left or right processing unit in each cycle;

[0022] The processing unit can simultaneously receive the data sent by the input first-in first-out queue module, and can also receive the data at intervals according to the calculation requirements.

[0023] Furthermore, the exp module is used to calculate the formula The e exponential operation in .

[0024] Furthermore, the addition tree module is used to calculate the formula The accumulation operation of n numbers is performed, the calculation results output by the exp module are accumulated, and the results are sent to the log2 module.

[0025] Furthermore, the log2 module utilizes the characteristics of binary numbers to approximately implement the log2 operation through shift and addition operations, and can respectively calculate the output data of the exp module and the output data of the addition tree module.

[0026] A method for operating a hardware accelerator for implementing weighted summation of attention, for performing weighted summation of attention on a Transformer architecture, characterized in that the operating method comprises the following steps:

[0027] S1, the control module reads data from the dynamic random access memory and writes it into the ping-pong static random access memory in the ping-pong static random access memory; the ping-pong static random access memory reads data and writes it into the data static random access memory of the processing unit;

[0028] S2, the control module reads the instruction of the processing unit array from the dynamic random access memory and writes it into the pong static random access memory in the ping-pong static random access memory; the pong static random access memory reads the data and writes it into the instruction static random access memory of the processing unit;

[0029] S3, the processing units of the processing unit array start processing and output the processing results to the output first-in first-out queue module;

[0030] S4, the exp module receives the operation result output by the output FIFO queue module, and simultaneously calculates the e index of the data in a parallel processing manner, and outputs the e index result to the addition tree module and the log2 module;

[0031] S5, the addition tree module completes the accumulation of the e-index results in multiple times, and sends the accumulated sum to the log2 module;

[0032] S6, the log2 module first performs a log2 operation on the input e-exponent result, and writes the calculation result into the ping static random access memory; then, the log2 operation is performed on the accumulated sum input into the addition tree module, and the log2 operation result is copied and filled and then written into the pong static random access memory;

[0033] S7, the control module writes the calculation result in the ping static random access memory into the data static random access memory of the processing unit array;

[0034] S8, the control module sends the calculation result in the pong static random access memory to the processing unit array through the input first-in first-out queue module, and each processing unit reads the data pre-stored in the local data static random access memory, subtracts the data sent by the input first-in first-out queue module, obtains the difference, and writes it back to the local static random access memory;

[0035] S9, after all differences are calculated, the attention weight results of the first m rows of the scaled attention score matrix are stored in the processing unit array;

[0036] S10, the control module reads the value matrix from the dynamic random access memory and writes it into the static random access memory; since the matrix data is arranged in the dynamic random access memory in the order of rows first and columns later, the read data is the first s columns of the V matrix;

[0037] S11, the control module sends the V matrix data of the first s columns in the ping static random access memory to the control module for rearrangement, and the rearranged data is the transposed s columns of V matrix data, and the rearranged data is written into the pong static random access memory;

[0038] S12, sending the rearranged s columns of V matrix data in the pong static random access memory to the processing unit array through the input first-in first-out queue module; each row of processing units simultaneously receives 1 column of V matrix data, and performs shift accumulation to obtain the first m rows of the first column of the attention output matrix, and writes them into the output first-in first-out queue module through the rightmost processing unit;

[0039] S13, the output FIFO queue module sends the first m rows of data received successively to the data buffer for rearrangement, and writes the rearranged first m rows of data into the Pong SRAM;

[0040] S14, the control module writes the first m rows of data in the pong static random access memory back to the dynamic random access memory, completing the calculation and storage of the first m rows of data in the first s columns of the attention output matrix;

[0041] S15, by repeating steps S10 to S14, the calculation and storage of the first m rows of data of all columns of the attention output matrix are completed;

[0042] S16, by repeating steps S1 to S15, the calculation and storage of all data of the attention output matrix are completed.

[0043] Specifically, step S3 includes the following steps:

[0044] S31: Use the arithmetic logic unit to find the maximum value in the data and write it into the local register;

[0045] S32: Then use the arithmetic logic unit to subtract the maximum value from the data and send the data to the output first-in first-out queue module;

[0046] S33: The output FIFO queue module only receives the output of the rightmost processing unit, so the calculation results of other column processing units need to be transferred to the left through the processing units on the right side of the column processing unit until they are written into the output FIFO queue module.

[0047] Compared with the prior art, the beneficial technical effects of the present invention are:

[0048] (1) By designing a data buffer, the data arrangement problem of the multiplied matrix and the data arrangement problem of the calculation result in the matrix multiplication operation is solved.

[0049] In order to ensure the consistency of data arrangement between layers, during the network inference process, data of the same type needs to be stored in the same arrangement. Matrix data is generally arranged in row order first and then in column order. When performing matrix multiplication operations, the multiplying matrix and the multiplied matrix are arranged in the same way, and the matrix multiplication operation is the dot product operation of the row vector of the multiplying matrix and the column vector of the multiplied matrix. When continuously reading the multiplied matrix data, the read data is the row vector of the matrix, which is difficult to deploy in the PE array for parallel operation with the row vector of the multiplication matrix.

[0050] In order to increase the parallelism and efficiency of operations, the hardware accelerator of the present invention adds a data buffer for transposing the column vector into a row vector. This process is implemented through the ping-pong static random access memory and the data buffer. The multiplied matrix is ​​first stored in the ping-pong static random access memory, and then sent to the data buffer in a pre-set order, and the data rearranged in the data buffer is written into the pong static random access memory for storage. The transposed multiplied matrix stored in the pong static random access memory can be directly sent to the PE array to perform parallel operations with the row vector of the multiplication matrix.

[0051] Similarly, for calculation results output by the PE array in column order, the calculation results are converted to row order by outputting the results to the data buffer for reordering, so as to be directly written into the memory.

[0052] (2) By designing a ping-pong static random access memory module, the calculation and loading processes of different group block data can be executed in parallel, hiding the loading time of data from the memory and improving the calculation speed.

[0053] By using the ping-pong static random access memory module, the preloading process of the data to be calculated is realized. This accelerator sets each set of data read from the memory to be written into the ping static random access memory, and the rearranged data is written into the pong static random access memory. After the calculation is completed, the calculation result enters the data buffer for rearrangement and is written back to the pong static random access memory, and the data stored in the ping static random access memory will not be overwritten. Therefore, when the current data performs a matrix multiplication operation, the next set of matrix data to be calculated can be pre-fetched in advance to achieve hiding the time of reading data and reduce the delay of the entire hardware accelerator to perform attention weighted summation.

[0054] (3) By using four groups of 1×4 PE arrays, parallel operations with a parallelism of 4 are achieved; by designing data transmission paths between each group of PEs, fast maximum and sum operations are achieved.

[0055] Using a 4-row 1×4 PE array, the 4 row vectors of the input matrix are operated in parallel to achieve the acceleration effect of the attention weighted sum operation. By designing the same data transmission path between the 4 PEs in each row, in the operation of finding the maximum value of the row vector, the data comparison between the 4 PEs can be completed to quickly find the maximum value; in the operation of calculating the vector inner product, the accumulation operation of the calculation results of the 4 PEs can be completed to quickly find the final accumulated sum.

[0056] By adopting the log2 quantization method of the softmax output and converting the exp formula, an efficient and simple softmax calculation architecture is implemented, which further reduces the storage overhead and computational complexity compared to the hardware implementation method without quantization.

[0057] (4) As a hardware accelerator designed and implemented based on Softmax shift accumulation to realize attention weighted summation, the present invention can effectively improve the computational efficiency of attention weighted summation based on Softmax shift accumulation. By designing the data flow path and data rearrangement, the data transmission delay and data loading delay are further reduced, thereby improving the computational efficiency of the accelerator. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a general architecture diagram of a hardware accelerator in an embodiment of the present invention;

[0059] Figure 2 4 is an internal architecture diagram of a single processing unit in an embodiment of the present invention. DETAILED DESCRIPTION

[0060] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0061] 1. First, the self-attention calculation process in the Transformer architecture is introduced:

[0062] 1. Calculate the query (Query, Q), key (Key, K) and value (Value, V) matrices: Usually, Q, K, V are obtained by linearly transforming the input. Assume that the input is X, and the weight matrix W is used. Q , W K , W V Perform a linear transformation to obtain Q = XW Q , K = XW K 、V=XW V .

[0063] 2. Calculate the attention score: For each pair of query and key, calculate the dot product and get an attention score matrix AttentionScore = QK T , where each element represents the similarity of a query to a key.

[0064] 3. Scaling the attention score: In order to prevent the dot product result from being too large, the attention score is scaled to obtain the scaled attention score where d k is the dimension of the key.

[0065] 4. Apply the Softmax function to obtain the attention weights AttentionWeights = softmax(ScaledAttentionScore).

[0066] 5. Calculate the attention output AttentionOutput = AttentionWeightsV.

[0067] Steps 4 and 5 are collectively referred to as the weighted sum operation of attention, which is one of the core operations of the attention mechanism in the Transformer architecture. It mainly includes two operations: the softmax operation of a single matrix and the matrix multiplication operation of two matrices.

[0068] The accelerator included in the present invention performs log2 quantization on the output of the softmax operation. This method comes from the paper "FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer". The principle is described as follows:

[0069] 1. The original softmax calculation formula is: The molecule is The denominator is The entire formula calculates the relative probability of each element. i represents the i-th element in the input vector, n represents the number of elements in the input vector; σ(x) i It represents the relative probability value of the i-th element in the input vector among all elements. Its value is between 0 and 1, and the sum of all output elements is 1.

[0070] 2. Perform log2 quantization on the output of softmax, and the operation is converted to: Convert division operations into log2 operations and subtraction operations.

[0071] The quantized softmax result can be represented by an integer with fewer bits, thus reducing storage overhead. The log2 operation can be implemented in hardware using a shift circuit, which can further reduce the computational complexity compared to the original division operation in softmax. Correspondingly, the matrix multiplication of the attention score matrix and the value matrix for calculating the attention output is also converted from a multiplication-accumulation operation to a shift-accumulation operation, further reducing the computational complexity.

[0072] When x in the softmax operation formula i When the value is large, directly calculate This may cause the value to be too large, and overflow may occur in hardware implementation. Therefore, we first find the maximum number max(x) in the input vector, and then subtract the maximum number from each number in the input vector to effectively avoid the overflow problem.

[0073] because Therefore, the calculation formula is modified to The hardware accelerator included in the present invention uses this formula to calculate the output result of softmax.

[0074] 2. Next, we will introduce the framework of the hardware accelerator. The hardware accelerator mainly includes the following modules:

[0075] Control Unit: The control unit of the accelerator is mainly used to control the reading and writing of DRAM, the reading and writing of ping-pong SRAM, and the data input source of the data buffer.

[0076] Dynamic random access memory (DRAM): used to store the matrix to be calculated and the calculation results of the attention weighted sum.

[0077] Ping-Pong Static Random Access Memory (Ping-Pong SRAM): It contains two SRAMs, namely Ping-Pong Static Random Access Memory (Ping SRAM) and Pong Static Random Access Memory (Pong SRAM), which can be read and written in time-sharing mode. The input can come from DRAM read, Databuffer output and log2 module output, and the output can be sent to DRAM write, Data buffer input and Input FIFO.

[0078] The explanations of Ping-Pong SRAM, Ping SRAM, and Pong SRAM are as follows:

[0079] Ping-Pong Static Random Access Memory (Ping-Pong SRAM): is a data buffer structure consisting of two SRAM storage areas that can perform read and write operations alternately. This structure can realize continuous data processing, avoid waiting time during data processing, and improve data processing efficiency and system throughput. "Ping SRAM" and "Pong SRAM" correspond to the two storage areas in the Ping-Pong buffer. For example, one of the storage areas is called "Ping SRAM" and the other is called "Pong SRAM".

[0080] Data buffer: Contains 4 rows and 16 columns of registers, which can be used to convert 16 groups of 4×8-bit input data into 4 groups of 128-bit data for output, and is used to rearrange the data stored in the ping-pong SRAM or splice the output results of the Output FIFO.

[0081] Input FIFO: It is used to receive data from the ping-pong SRAM and send it to the 4×4 processing unit array.

[0082] Output FIFO: It is used to receive data from the 4×4 processing unit array and send it to the ping-pong SRAM, Data buffer or exp module.

[0083] 4×4 Processing Element Array (PE Array): Contains 16 processing elements (PEs), each of which can receive 1 data from the Input FIFO above in each cycle. The 4 PEs in the rightmost column are output PEs and can output 1 data to the Output FIFO in each cycle.

[0084] exp module: used to calculate the e-exponential result of the Output FIFO output data, and synchronously send the calculation result to the Adder tree and log2 modules.

[0085] Adder tree module: used to accumulate the e exponential results output by the exp module and send the accumulated sum to the log2 module.

[0086] log2 module: used to perform logarithm operation with base 2 on the e-exponential result output by the exp module, and to perform logarithm operation with base 2 on the accumulated sum of the e-exponential results output by the Adder tree module, and send the result back to the ping-pong SRAM.

[0087] 3. Next, we will further introduce the calculation modules:

[0088] 4×4 Processing Unit Array (PE Array): used to calculate the formula max(x), x i The subtraction operation of -max(x) and two log2 results, as well as the shift operation and accumulation operation in the shift-accumulate operation, are the most important operation modules in the accelerator.

[0089] Each processing element (PE) supports addition, subtraction, multiplication, shift and comparison operations.

[0090] Each PE is configured with an instruction SRAM for storing instruction data. After the PE starts working, it reads instructions one by one to guide its data fetching, sending and calculation processes.

[0091] A data SRAM is configured in each PE to pre-store calculation data in batches. In the shift-and-add operation of the two matrices, the AttentionWeights shift matrix is ​​first pre-stored in the data SRAM, and then the corresponding calculation data is read out in each cycle and shift-and-adds with the V matrix sent in by the Input FIFO.

[0092] Two local registers are configured in each PE to store the quantization factor data or temporary accumulated data required during the calculation process.

[0093] Each PE supports left-right interconnection and can transmit one data to the PE on its left or right every cycle.

[0094] The four PE lines can receive data from the Input FIFO simultaneously, or they can receive data at intervals according to computational requirements.

[0095] Each PE can perform operations in each cycle, so the maximum parallelism of operations is 16.

[0096] exp module: used to calculate formulas The e exponential operation in . The quantization method used comes from the paper "A High-Speed ​​and Low-Complexity Architecture for Softmax Function in Deep Learning". The principle is described as follows:

[0097] First, replace the base e with the base 2, and the exponential function is converted to

[0098] Then x i *log2e is split into integer u i and the decimal v i ,u i +v i =x i *log2e

[0099] The formula can be further transformed into:

[0100]

[0101] At this point, the exponential operation is transformed into and a shift operation where The operation is implemented using a lookup table.

[0102] Adder tree module: used to calculate formula The calculation results output by the exp module are accumulated and sent to the log2 module.

[0103] log2 module: Utilizes the characteristics of binary numbers and approximates log2 through shift and addition operations. It can calculate the output data of the exp module and the output data of the Adder tree module respectively.

[0104] Example

[0105] Taking the SWIN Transformer architecture with 8-bit fixed-point quantization and 96 channels as an example, the specific operation of the above accelerator when performing attention weighted summation is described. When the number of channels is 96, the scaled attention score ScaledAttentionScore is a 96x96 matrix, and the value matrix V is 96×d k Since the 4×4 PE Array contains 4 rows of PE groups that can be operated in parallel, the softmax results of the first 4 rows of vectors of the scaled attention score ScaledAttentionScore can be calculated simultaneously. The length of each vector is 64. The specific process is as follows:

[0106] (1) The Control Unit reads 4x64 8-bit data from the DRAM (the bit width is 128 bits, and the 4x64 8-bit data are stored as 16 128-bit data in the DRAM) and writes them into the Ping Static Random Access Memory (Ping SRAM) in the Ping-Pong SRAM; the Ping SRAM reads the data and writes them into the Data SRAM of the PE.

[0107] (2) The Control Unit reads the instructions of the 4×4 PE Array from the DRAM and writes them into the pong static random access memory (Pong SRAM) in the Ping-Pong SRAM; the Pong SRAM reads the data and writes them into the instruction SRAM of the PE.

[0108] (3) The 4×4 PE Array starts working, and 4 rows of PE simultaneously calculate 4 1x64 vectors:

[0109] ①First, use the arithmetic logic unit (ALU) to find the maximum value among the 64 data and write it into the local register;

[0110] ②Then use ALU to subtract the maximum value from the 64 data and send them to the Output FIFO;

[0111] ③The Output FIFO only receives the output of the rightmost PE, so the calculation results of the first three columns of PE need to be passed to the left through the right PE until they are written into the Output FIFO;

[0112] ④ Each PE outputs a 32-bit data, including 4 8-bit operation results, and 4 PEs output 16 8-bit operation results simultaneously.

[0113] (4) The exp module receives the 16 8-bit operation results input by the Output FIFO, and simultaneously calculates the e-exponent of the 16 8-bit data with a parallelism of 16, and outputs them to the Adder tree and log2 modules.

[0114] (5) The Adder tree module completes four groups of 64 32-bit e-exponent results in four times, accumulates them, and outputs four 64-bit cumulative sums to be sent to the log2 module.

[0115] (6) The log2 module first performs a log2 operation on the 16 32-bit e exponent results sent in, and writes the calculation results into the ping SRAM; then the log2 operation is performed on the 4 64-bit cumulative sums sent in by the Adder tree module. The calculation results are 4 8-bit data. Each 8-bit data is copied and filled with 128-bit data, and the 4 128-bit calculation results are written into the pong SRAM.

[0116] (7) The Control Unit writes the calculation results in the Ping SRAM into the data SRAM of the 4×4 PE Array.

[0117] (8) The Control Unit sends the calculation results in the Pong SRAM to the 4×4 PE Array through the Input FIFO. Each PE reads the data pre-stored in the local data SRAM, subtracts it from the data sent by the Input FIFO, obtains the difference, and writes it back to the local SRAM.

[0118] (9) After all differences are calculated, the AttentionWeights results of the first 4 rows of the ScaledAttentionScore matrix are stored in the 4×4 PE Array.

[0119] (10) The Control Unit reads the V matrix from the DRAM and writes it into the SRAM. Since the matrix data is arranged in the DRAM in the order of rows first and columns later, the data read out are the first 16 columns of the V matrix.

[0120] (11) The Control Unit sends the first 16 columns of V matrix data in the ping-pong SRAM to the Control Unit for rearrangement. The rearranged data is the transposed 16 columns of V matrix data. The rearranged data is written into the pong SRAM.

[0121] (12) The rearranged 16 columns of V matrix data in the pong SRAM are sent to the 4×4 PE Array through the Input FIFO; 4 rows of PE simultaneously receive 1 column of V matrix data, and perform shift accumulation to obtain the first 4 results of the first column of the AttentionOutput matrix, and write them into the Output FIFO through the rightmost PE; while executing the calculation of the first 16 columns (1 to 16 columns) of the V matrix of the first group, the last 16 columns of data (17 to 32 columns) of the second group of V matrix can be synchronously read from the DRAM and written into the ping SRAM (step (10)), and after waiting for the calculation and write-back of the first group of data to be completed, the rearrangement operation is performed (step (11)).

[0122] (13) The Output FIFO sends the 16 groups of 4x8 bits received successively to the Data buffer for rearrangement. The rearranged data is 4 128-bit data, and the rearranged data is written into the Pong SRAM.

[0123] (14) The Control Unit writes the four results in the Pong SRAM back to the DRAM. At this point, the first four rows of the first 16 columns of the AttentionOutput matrix complete their calculation and storage process.

[0124] (15) The calculation and storage of the first 4 rows of all columns of the AttentionOutput matrix are completed by repeating steps (10) to (14); step (10) of the next set of data will be carried out in parallel with step (12) of the current set.

[0125] (16) Complete the calculation and storage of the AttentionOutput matrix by repeating steps (1) to (15).

[0126] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0127] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A hardware accelerator for implementing attention weighted summation, characterized in that: include: Control module: used to control the reading and writing of dynamic random access memory, the reading and writing of ping-pong static random access memory and the data input source of the data buffer; Dynamic random access memory: used to store the matrix to be calculated and the result of the weighted sum of attention; Ping-Pong SRAM: includes Ping SRAM and Pong SRAM, which can perform time-sharing reading and writing. The input sources of Ping-Pong SRAM include dynamic random access memory reading, data buffer output and log2 module output. The output of Ping-Pong SRAM is sent to dynamic random access memory for writing, sent to data buffer or sent to input first-in-first-out queue module. Data buffer: contains multiple register groups, used to rearrange the data stored in the ping-pong static random access memory, or to splice the output results of the output first-in first-out queue module; Input FIFO queue module: used to receive data from the ping-pong static random access memory and send it to the processing unit array; Output FIFO module: used to receive data from the processing unit array and send it to the ping-pong static random access memory, data buffer or exp module; Processing unit array: comprising a plurality of processing units, wherein each processing unit can receive one data from the input first-in first-out queue module in each cycle, and the processing unit in the rightmost column is an output processing unit, which can output one data to the output first-in first-out queue module in each cycle; exp module: used to calculate the e-exponential result of the output data of the output FIFO queue module, and synchronously send the e-exponential result to the addition tree module and the log2 module; Addition tree module: used to accumulate the e exponential results output by the exp module and send the accumulated sum to the log2 module; log2 module: used to perform logarithmic operation with base 2 on the e-exponential result output by the exp module, and to perform logarithmic operation with base 2 on the accumulated sum of the e-exponential result output by the addition tree module, and send the result back to the ping-pong static random access memory.

2. The hardware accelerator for implementing attention weighted summation according to claim 1, characterized in that: The processing unit array is used to calculate the formula The max(x) operation in x i -max(x) operation, and Subtraction operations, and shift and accumulation operations in shift-and-accumulate; Each processing unit supports addition, subtraction, multiplication, shift and comparison operations; Each processing unit is equipped with an instruction static random access memory for storing instruction data. After the processing unit starts working, it reads instructions one by one to guide the data fetching, sending and calculation process. Each processing unit is equipped with a data static random access memory for pre-storing calculation data in batches. In the shift-accumulation operation of the attention weight matrix and the value matrix, the attention weight matrix is ​​first pre-stored in the data static random access memory, and then the corresponding calculation data is read out in each cycle and shift-accumulated with the value matrix sent by the input first-in-first-out queue module; Two local registers are configured in each processing unit to store the quantization factor data or temporary accumulated data required in the calculation process; Each processing unit supports left-right interconnection, transferring one data to the left or right processing unit in each cycle; The processing unit can simultaneously receive the data sent by the input first-in first-out queue module, and can also receive the data at intervals according to the calculation requirements.

3. The hardware accelerator for implementing attention weighted summation according to claim 1, characterized in that: The exp module is used to calculate the formula The e exponential operation in .

4. The hardware accelerator for implementing attention weighted summation according to claim 1, characterized in that: The addition tree module is used to calculate the formula The accumulation operation of n numbers is performed, the calculation results output by the exp module are accumulated, and the results are sent to the log2 module.

5. The hardware accelerator for implementing attention weighted summation according to claim 1, characterized in that: The log2 module utilizes the characteristics of binary numbers to approximately implement the log2 operation through shift and addition operations, and can respectively calculate the output data of the exp module and the output data of the addition tree module.

6. A method for operating a hardware accelerator for implementing weighted summation of attention as claimed in any one of claims 1 to 5, for performing weighted summation of attention on a Transformer architecture, characterized in that: The operation method comprises the following steps: S1, the control module reads data from the dynamic random access memory and writes it into the ping-pong static random access memory in the ping-pong static random access memory; the ping-pong static random access memory reads data and writes it into the data static random access memory of the processing unit; S2, the control module reads the instruction of the processing unit array from the dynamic random access memory and writes it into the pong static random access memory in the ping-pong static random access memory; the pong static random access memory reads the data and writes it into the instruction static random access memory of the processing unit; S3, the processing units of the processing unit array start processing and output the processing results to the output first-in first-out queue module; S4, the exp module receives the operation result output by the output FIFO queue module, and simultaneously calculates the e index of the data in a parallel processing manner, and outputs the e index result to the addition tree module and the log2 module; S5, the addition tree module completes the accumulation of the e-index results in multiple times, and sends the accumulated sum to the log2 module; S6, the log2 module first performs a log2 operation on the input e-exponent result, and writes the calculation result into the ping static random access memory; then, the log2 operation is performed on the accumulated sum input into the addition tree module, and the log2 operation result is copied and filled and then written into the pong static random access memory; S7, the control module writes the calculation result in the ping static random access memory into the data static random access memory of the processing unit array; S8, the control module sends the calculation result in the pong static random access memory to the processing unit array through the input first-in first-out queue module, and each processing unit reads the data pre-stored in the local data static random access memory, subtracts the data sent by the input first-in first-out queue module, obtains the difference, and writes it back to the local static random access memory; S9, after all differences are calculated, the attention weight results of the first m rows of the scaled attention score matrix are stored in the processing unit array; S10, the control module reads the value matrix from the dynamic random access memory and writes it into the static random access memory; since the matrix data is arranged in the dynamic random access memory in the order of rows first and columns later, the read data is the first s columns of the V matrix; S11, the control module sends the V matrix data of the first s columns in the ping static random access memory to the control module for rearrangement, and the rearranged data is the transposed s columns of V matrix data, and the rearranged data is written into the pong static random access memory; S12, sending the rearranged s columns of V matrix data in the pong static random access memory to the processing unit array through the input first-in first-out queue module; each row of processing units simultaneously receives 1 column of V matrix data, and performs shift accumulation to obtain the first m rows of the first column of the attention output matrix, and writes them into the output first-in first-out queue module through the rightmost processing unit; S13, the output FIFO queue module sends the first m rows of data received successively to the data buffer for rearrangement, and writes the rearranged first m rows of data into the Pong SRAM; S14, the control module writes the first m rows of data in the pong static random access memory back to the dynamic random access memory, completing the calculation and storage of the first m rows of data in the first s columns of the attention output matrix; S15, by repeating steps S10 to S14, the calculation and storage of the first m rows of data of all columns of the attention output matrix are completed; S16, by repeating steps S1 to S15, the calculation and storage of all data of the attention output matrix are completed.

7. The method for operating a hardware accelerator for implementing attention weighted summation according to claim 6, characterized in that: Step S3 includes the following steps: S31: Use the arithmetic logic unit to find the maximum value in the data and write it into the local register; S32: Then use the arithmetic logic unit to subtract the maximum value from the data and send the data to the output first-in first-out queue module; S33: The output FIFO queue module only receives the output of the rightmost processing unit, so the calculation results of other column processing units need to be transferred to the left through the processing units on the right side of the column processing unit until they are written into the output FIFO queue module.

Images