An accelerator supporting attention mechanism

Abstract

The application discloses an accelerator supporting attention mechanism, which is used for implementing matrix multiplication operation, scaling operation or nonlinear activation processing involved in the attention mechanism, wherein a weight cache state machine module is used for controlling the flow of weight data from a system bus to a weight cache unit; a thread cache state machine module is used for controlling the flow of thread data from the system bus to a thread cache unit; an attention tensor processing unit state machine module is used for controlling the loading and flow of the thread data and the weight data from different directions of a two-dimensional systolic array, the thread data is controlled to be transmitted to a next processing unit every cycle, and the weight data is broadcast to processing units in a corresponding direction; and the calculation result of the two-dimensional systolic array is transmitted to different branches according to whether nonlinear activation processing needs to be performed. The application accelerates the attention mechanism with variable sequence length by data padding of a matrix to fit the data flow of the systolic array.

Description

Technical Field

[0001] This invention relates to the field of integrated circuit design technology, and more specifically, to an accelerator that supports an attention mechanism. Background Technology

[0002] Attention mechanisms involve many computationally intensive operators, such as matrix multiplication, thus requiring hardware acceleration. Currently, mainstream attention mechanism accelerators speed up sparse attention mechanisms or standard attention mechanisms, but they do not optimize for the variability of sequence length.

[0003] In processor design, to improve memory access speed, the system data width is typically multiple bytes. The drawback of this is that it's difficult to start storing or writing data from the middle of these multiple bytes; the same drawback exists for on-chip SRAM input / output ports. Therefore, to mitigate this drawback, data should ideally be stored in a neat, orderly manner, meaning the size of the data block should preferably be an integer multiple of the system data width.

[0004] In the attention mechanism, the sequence length is a variable. The attention score during the calculation process is the sequence length multiplied by the sequence length. If it is not handled properly, it may lead to the aforementioned difficulties in memory access and on-chip data flow. Therefore, certain strategies are needed to optimize the impact of the sequence length variable on the hardware accelerator.

[0005] In existing technologies, the target of acceleration is not the standard attention mechanism, thus limiting its applicability in practical applications. Furthermore, there are currently no specific optimizations for the variability of sequence length in attention mechanisms, and using coarse-grained padding significantly reduces performance or increases power consumption. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide an accelerator that supports attention mechanisms, used to implement matrix multiplication, scaling, or nonlinear activation processing involved in attention mechanisms. This accelerator includes a thread cache state machine module, a weight cache state machine module, an attention tensor processing unit state machine module, a weight cache unit, a thread cache unit, a floating-point coprocessor, a temporary cache unit, a post-processing unit, a submission cache unit, and a two-dimensional systolic array. This two-dimensional systolic array contains multiple processing units connected in a systolic manner; wherein:

[0007] The weight cache state machine module is used to control the flow of weight data from the system bus to the weight cache unit;

[0008] The thread cache state machine module is used to control the flow of thread data from the system bus to the thread cache unit;

[0009] The attention tensor processing unit state machine module is used to control the loading and flow of thread data and weight data from different directions of the two-dimensional pulsating array, wherein the thread data is controlled to be passed to the next processing unit in each cycle, and the weight data is broadcast to each processing unit in the corresponding direction.

[0010] The calculation result of the two-dimensional pulsating array is passed to different branches depending on whether nonlinear activation processing is required. If required, it is passed to the temporary cache unit and then to the floating-point coprocessor for calculation. If not required, it is passed to the post-processing unit, which performs requantization operation and then passes it to the submission cache unit.

[0011] Compared with the prior art, the advantages of the present invention are that it realizes an accelerator that supports the attention mechanism, uses a fine-grained padding method to reduce the impact of the arbitrariness of the sequence length in the attention mechanism on the hardware data flow, and improves the computational efficiency and application scope of the attention mechanism.

[0012] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0013] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0014] Figure 1 This is a schematic diagram of a fine-grained filling method for an algorithm layer according to an embodiment of the present invention;

[0015] Figure 2 This is a schematic diagram of the operating network of the attention mechanism according to an embodiment of the present invention;

[0016] Figure 3 This is a schematic diagram illustrating different filling methods for attention mechanisms according to an embodiment of the present invention;

[0017] Figure 4 This is an architecture diagram of a hardware accelerator according to an embodiment of the present invention;

[0018] In the attached diagram, FPU stands for Floating-point Unit; float for Floating-point; fixed for Fixed-point; PPU for Post-processing Unit; ActF for Activation Function; Padding for Padding; unpadding for Unpadding; leader for Leader; worker for Worker; FSM for FPU (Floating-point Unit State Machine); FSM for CB (Commit Buffer State Machine); FPB for Floating-point Buffer; and FSM for ATU (Attention Tensor Processing Unit State Machine). Detailed Implementation

[0019] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0020] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0021] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0022] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0023] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0024] Attention mechanisms can be applied to various types of neural network models, such as the Transformer model and the BERT model. Attention can be described as mapping a query and a set of key-value pairs to an output, where the query Q, key K, value V, and output are all vectors. To address the variability of sequence length in attention mechanisms, this invention aims to minimize the impact of sequence length variability on accelerator performance and area. It proposes a fine-grained padding method for the algorithm layer and designs a corresponding hardware architecture layer.

[0025] Figure 1 This is a schematic diagram illustrating the fine-grained padding method at the algorithm layer. Light gray areas represent padding with zeros, white areas represent dirty data, and dark gray areas represent valid data. Figure 1 As can be seen, by padding the matrix Q of the attention mechanism with rows that are multiples of 0 to 32, the size of each matrix can be a multiple of 32, thus avoiding the drawbacks caused by the multi-byte data width. Moreover, the impact of padding matrix Q with 0s, i.e., dirty data, has no effect on the valid data and does not participate in the calculation of the valid data.

[0026] In short, the computation module of the designed accelerator architecture is broadly divided into two parts: a fixed-point computation core for matrix multiplication and a floating-point computation core, which acts as a coprocessor for the fixed-point core and is used to compute activation functions, such as the SoftMax function. The core of the fixed-point core is a systolic array with a size of 32 x 32; the core of the floating-point core consists of 16 sets of exponentiation units, accumulation units, and division units.

[0027] In general, the attention mechanism includes the MatMul operation, used to perform the multiplication of two matrices Q and K; scaling; dequantization; the SoftMax function; and quantization. See [link to relevant documentation]. Figure 2 As shown. Considering that the attention mechanism's computation process involves the SoftMax function, which is a non-linear function and difficult to quantize, a floating-point kernel is chosen for processing, while the matrix multiplication part uses conventional fixed-point computation. The fine-grained padding method of the algorithm layer and the corresponding accelerator architecture will be described in detail below.

[0028] 1) Fine-grained filling of the algorithm layer

[0029] To minimize the impact on hardware performance, in one embodiment, zero-padding is performed only in the scaling dot product attention mechanism, and the key matrix for padding is the attention score matrix.

[0030] Figure 3 Three schemes are shown that can fill the size of the attention score matrix to a multiple of 32. Figure 3 It can be seen that padding matrix K (K-Padding) makes the vector length of the attention score matrix S greater than the vector length of matrix V, violating the premise of matrix multiplication. While padding both matrix Q and matrix K (QK-Padding) can circumvent this drawback, it introduces significant additional computational power, and dirty data participates in the calculation of valid data, requiring additional control logic in the hardware implementation. Padding matrix Q (Q-Padding) ensures that dirty data is only stored in the last few rows of the vector throughout the attention mechanism's computation process, having no impact on the valid data. After obtaining the final result, the dirty data is not written back to memory.

[0031] 2) Accelerator hardware architecture design

[0032] In one embodiment, the accelerator's hardware architecture needs to support a scaled dot-multiplication attention mechanism with zero-padding operations. This can be broken down into three operators: matrix multiplication, scaling, and the SoftMax activation function. For example, if the attention mechanism being targeted is the original Transformer or BERT network, the scaling operation is always divided by 8, which is simply a right shift of three bits.

[0033] Specifically, in combination Figure 4 As shown, the provided accelerator architecture supporting the attention mechanism generally includes a weight cache unit (WB), a thread cache unit (TB), a systolic array, a floating-point coprocessor, a temporary cache unit, a post-processing unit, and a commit cache unit. Furthermore, a corresponding state machine is designed for each unit to control the processing state and computation process. For example, the thread cache state machine (FSM-TB) controls the flow of thread data from the system bus to the thread cache unit, and the weight cache state machine (FSM-WB) controls the flow of weight data from the system bus to the weight cache unit. The attention tensor processing unit state machine controls the loading and flow of thread data and weight data from the corresponding cache units to the systolic array.

[0034] Still combined Figure 4 As shown, the systolic array is a two-dimensional systolic array containing multiple processing units (PEs). PEs in the same column share a finite state machine. Each processing unit is responsible for data computation. Weight data is input from the top of the systolic array and propagates from top to bottom, while thread data is input from the left side of the systolic array and propagates from left to right. Each processing unit performs calculations on the received data and weights, and the results are output from the right side of the systolic array.

[0035] For example, when multiplying matrices Q and K, Q is placed in the thread cache unit, and K is placed in the weight cache unit. Similarly, when multiplying matrices S and V, S is placed in the thread cache unit, and V is placed in the weight cache unit. It should be understood that in this paper, "thread cache" and "weight cache" are merely name identifiers to represent different caches, as many operators in a neural network operate on both input and weight data. Furthermore, the fine-grained padding scheme can be implemented using FSM-TB, which is controlled by the thread cache state machine.

[0036] For example, matrix multiplication is performed using a systolic array of fixed-point computing cores. Data originates from memory, passes through the DRAM controller and PHY to the AXI4 module, and then, under the control of the thread cache state machine and weight cache state machine, reaches the thread cache and weight cache, before being input into the systolic array. Weights are propagated in a broadcast manner, meaning that a weight buffer transmits weight data to every processing unit in the same column. Thread data is passed one cell to the right per cycle, continuing to the right until it reaches the processing units in the last column. The systolic array consists of 32 by 32 processing units. Processing units in the same column share a finite state machine, and the state machine transition logic is executed only by the processing unit leader. The execution of the processing unit is controlled by the state machine signals sent by the processing unit leader. The computation result of the systolic array leads to different branches depending on whether the current operator has an activation function (ActF). If an activation function is present, the result is sent to the stage buffer and then to the floating-point coprocessor (FPU) for processing. If no activation function is present, the result is sent to the post-processing unit (PPU), where requantization, zero-point addition, and truncation to 8 bits are performed before the result is sent to the commit buffer. The zero-filling operation is controlled by the thread cache state machine, and the discarding of dirty data is controlled by the commit buffer state machine (FSM-CB).

[0037] The SoftMax activation function is calculated by the floating-point coprocessor (FPU). To balance processor speed and chip area, the floating-point core has 16 arithmetic units, allowing the fixed-point and floating-point cores to achieve similar processing speeds and operate like a large pipeline. Data originates from the fixed-point core's temporary buffer, is converted to 32-bit floating-point numbers by a fixed-to-float module, and then processed by the exponentiation unit (EXP), accumulation unit (ACC), and division unit (DIV) to obtain the 32-line SoftMax activation function result. This result is then written to the commit buffer via a floating-point to fixed-point module. This module includes quantization scaling and zero-point parameters and converts division by the scaling parameter to multiplication by its reciprocal, thus saving area by transforming the required divider into a multiplier. Finally, the data is written back to memory.

[0038] To further verify the effectiveness of the invention, simulation experiments were conducted. Synopsys' VCS was used for simulation, and Xilinx's VIVADO was used for simulation, synthesis, placement, and routing.

[0039] 1) Simulation of the process of multiplying matrix Q and matrix K

[0040] In the simulation experiments, the matrix multiplication results of 128x64 matrices were verified, as well as the matrix multiplication results of 111x64 matrices with zero-padding. During this process, for matrix Q whose row number is not a multiple of 32, the number of row vectors in matrix Q was padded to a multiple of 32 before being input into the systolic array for calculation. The simulation results show that for matrix multiplication requiring zero padding, the calculation process proceeds normally, and there is a certain speedup effect compared to the case where the number of rows is a multiple of 32.

[0041] 2) Simulation of multiplication of matrices S and V

[0042] The matrix multiplication results of 128x128 and 64x128 matrices, as well as the matrix multiplication results of 128x111 and 64x111 matrices with zero-padding, were verified. In this process, matrix S had already been padded with multiples of 32 in the previous step. Simulation results show that for matrix multiplication with zero-padding, the calculation proceeds normally, and there is a certain speedup effect compared to the case of coarse-grained padding with multiples of 32.

[0043] 3) Hardware resources and performance

[0044] By using Vivado for placement and routing, the hardware resource usage information of this invention is obtained, as shown in Table 1.

[0045] Table 1: Hardware Resource Usage

[0046]

[0047] The hardware performance of this invention was obtained by counting the number of simulation cycles, as shown in Table 2.

[0048] Table 2: Hardware Performance

[0049]

[0050] As can be seen, the matrix multiplication and scaling dot product attention mechanism of this invention has a speed-up effect compared to CPU and GPU platforms.

[0051] In summary, this invention designs a fine-grained padding scheme for the algorithm layer of the attention mechanism, and provides a hardware architecture with the fine-grained padding scheme for the attention mechanism, as well as a data flow model of the hardware architecture. It can flexibly handle different sequence lengths of different samples in the attention mechanism, and has little impact on hardware performance. It also supports standard attention mechanisms and has a wider range of applications.

[0052] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0053] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0054] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0055] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, etc., and conventional procedural programming languages ​​such as "C" or similar languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0056] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0057] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0058] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0059] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0060] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. An accelerator supporting attention mechanisms, used to implement matrix multiplication, scaling, or nonlinear activation processing involved in attention mechanisms, the accelerator comprising a thread cache state machine module, a weight cache state machine module, an attention tensor processing unit state machine module, a weight cache unit, a thread cache unit, a floating-point coprocessor, a temporary cache unit, a post-processing unit, a submission cache unit, and a two-dimensional systolic array, the two-dimensional systolic array containing multiple processing units connected in a systolic manner; wherein: The weight cache state machine module is used to control the flow of weight data from the system bus to the weight cache unit; The thread cache state machine module is used to control the flow of thread data from the system bus to the thread cache unit; The attention tensor processing unit state machine module is used to control the loading and flow of thread data and weight data into the two-dimensional pulsating array from different directions. The thread data is controlled to be passed to the next processing unit in each cycle, and the weight data is broadcast to each processing unit in the corresponding direction. The calculation results of the two-dimensional pulsating array are passed to different branches depending on whether nonlinear activation processing is required. If so, they are passed to the temporary cache unit and then to the floating-point coprocessor for calculation. If not needed, the data is passed to the post-processing unit, which performs a requantization operation, and then passes it to the submission cache unit.

2. The accelerator according to claim 1, characterized in that, The nonlinear activation processing is performed using the SoftMax activation function. The floating-point coprocessor includes a fixed-point to floating-point module, a floating-point to fixed-point module, a floating-point cache unit, an exponentiation unit, an accumulation unit, and a division unit; wherein: The fixed-point to floating-point module converts the data from the temporary cache unit into floating-point numbers. Then, the data is calculated by the exponentiation unit, the accumulation unit, and the division unit to obtain the result of the SoftMax activation function. The result is then written to the commit cache unit through the floating-point to fixed-point module. This floating-point to fixed-point module has quantization scaling parameters and zero-point parameters. By converting division by the scaling parameters into multiplication by the reciprocal of the scaling parameters, the required divider is converted into a multiplier. Finally, the data is written back to memory.

3. The accelerator according to claim 1, characterized in that, The two-dimensional pulsating array contains 32*32 processing units.

4. The accelerator according to claim 3, characterized in that, The floating-point coprocessor's floating-point core has 16 arithmetic units.

5. The accelerator according to claim 1, characterized in that, For the two-dimensional pulsating array, the processing units in the same column share a finite state machine. The transition logic of the finite state machine is executed only by the processing unit leader. The processing unit executor is controlled by the state machine signal sent by the processing unit leader, and each processing unit processes the data flowing through it simultaneously and in parallel.

6. The accelerator according to claim 3, characterized in that, The thread data loaded into the two-dimensional pulsating array is thread data after fine-grained zero-filling operation.

7. The accelerator according to claim 6, characterized in that, Fine-grained zero-padding is performed on matrix Q in the attention mechanism to make the size of the attention score matrix a multiple of 32. The zero-padding operation is controlled by the thread cache state machine module, and the discarding of dirty data is controlled by the commit cache state machine.

8. The accelerator according to claim 7, characterized in that, The zero-padding operation is performed only in the scaling dot product attention mechanism.

9. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that, When the processor executes the computer program, it implements the accelerator as described in any one of claims 1 to 8.

Images