Accelerator supporting attention mechanism

Abstract

The invention discloses an accelerator supporting an attention mechanism, which is used for realizing matrix multiplication operation, scaling operation or nonlinear activation processing related to the attention mechanism, and is characterized in that a weight cache state machine module is used for controlling weight data to flow from a system bus to a weight cache unit; the thread cache state machine module is used for controlling thread data to flow from the system bus to the thread cache unit; the attention tensor processing unit state machine module is used for controlling loading and flowing of thread data and weight data from the two-dimensional systolic array in different directions, the thread data is controlled to be transmitted to the next processing unit in each period, and the weight data is spread to the processing units in the corresponding directions in a broadcast mode; the calculation result of the two-dimensional systolic array is transmitted to different branches according to whether nonlinear activation processing needs to be executed or not. According to the method, data filling is carried out on the matrix so that the matrix can be matched with the systolic array data stream, and then the attention mechanism with the variable sequence length is accelerated through hardware.

Description

Technical Field

[0001] The present invention relates to the technical field of integrated circuit design, and more specifically, to an accelerator supporting an attention mechanism. Background Art

[0002] The attention mechanism has many computationally intensive operators, such as matrix multiplication, so there is a certain demand for hardware acceleration. At present, the mainstream attention mechanism accelerator accelerates the sparse attention mechanism or the standard attention mechanism, but does not optimize the variability of sequence length.

[0003] In processor design, in order to increase memory access speed, the system data bit width is usually multiple bytes. The disadvantage of this data is that it is difficult to store or write data from the middle of these multiple bytes; this disadvantage also exists for the input and output ports of on-chip SRAM. Therefore, in order to alleviate this disadvantage, data is best stored in a neat form, that is, the size of the data block is preferably an integer multiple of the system data bit width.

[0004] In the attention mechanism, sequence length is a variable. The size of the attention score in the calculation process is the multiplication of the sequence length by the sequence length. If it is not processed, it is likely to cause the above-mentioned difficulty in memory access and difficulty in on-chip data flow. Therefore, certain strategies are needed to optimize the impact of the sequence length variable on the hardware accelerator.

[0005] In the existing technology, the target of acceleration is not the standard attention mechanism, which has limited applicability in practical applications. In addition, there is currently no special optimization for the variability of sequence length in the attention mechanism, and the use of coarse-grained padding will significantly reduce performance or increase power consumption. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide an accelerator supporting the attention mechanism, which is used to implement the matrix multiplication operation, scaling operation or nonlinear activation processing involved in the attention mechanism. The accelerator includes a thread cache state machine module, a weight cache state machine module, an attention tensor processing unit state machine module, a weight cache unit, a thread cache unit, a floating point coprocessor, a temporary cache unit, a post-processing unit, a submission cache unit and a two-dimensional systolic array, wherein the two-dimensional systolic array includes a plurality of processing units connected in a systolic form; wherein:

[0007] The weight cache state machine module is used to control the flow of weight data from the system bus to the weight cache unit;

[0008] The thread cache state machine module is used to control the flow of thread data from the system bus to the thread cache unit;

[0009] The attention tensor processing unit state machine module is used to control the loading and flow of thread data and weight data from the two-dimensional systolic array in different directions, wherein the thread data is controlled to be passed to the next processing unit in each cycle, and the weight data is propagated to each processing unit in the corresponding direction in a broadcast form;

[0010] The calculation result of the two-dimensional systolic array is transmitted to different branches according to whether nonlinear activation processing needs to be performed. If necessary, it is transmitted to the temporary cache unit and then transmitted to the floating-point coprocessor for calculation; if not, it is transmitted to the post-processing unit, and the post-processing unit performs a requantization operation and then transmitted to the submission cache unit.

[0011] Compared with the prior art, the advantage of the present invention is that it realizes an accelerator that supports the attention mechanism, uses a fine-grained filling method to reduce the impact of the arbitrariness of the sequence length in the attention mechanism on the hardware data flow, and improves the computational efficiency and application scope of the attention mechanism.

[0012] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0014] Figure 1 is a schematic diagram of a fine-grained filling method of an algorithm layer according to an embodiment of the present invention;

[0015] Figure 2 is a schematic diagram of an operation network of an attention mechanism according to an embodiment of the present invention;

[0016] Figure 3 is a schematic diagram of different filling methods for the attention mechanism according to an embodiment of the present invention;

[0017] Figure 4 is an architectural diagram of a hardware accelerator according to an embodiment of the present invention;

[0018] In the accompanying figure, FPU-floating point unit; float-floating point; fixed-fixed point; PPU-post-processing unit; ActF-activation function; Padding-padding; unpadding-the reverse operation of padding; leader-leader; worker-executor; FSM-FPU-floating point unit state machine; FSM-CB-commit cache state machine; FPB-floating point cache; FSM-ATU-attention tensor processing unit state machine. DETAILED DESCRIPTION

[0019] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0020] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0021] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0022] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0023] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0024] The attention mechanism can be applied to various types of neural network models, such as the Transformer model and the BERT model. Attention can be described as mapping a query and a set of key-value pairs to an output, where the query Q (Query), key K (Key), value V (Value) and output are all vectors. Aiming at the variability of sequence length in the attention mechanism, the present invention aims to minimize the impact of sequence length variability on accelerator performance and area, proposes a fine-grained filling method at the algorithm layer, and designs the corresponding hardware architecture layer.

[0025] Figure 1 This is a diagram of the fine-grained filling method of the algorithm layer, where the light gray area represents the filling 0, the white area represents the dirty data, and the dark gray part represents the valid data. Figure 1 It can be seen that by filling the attention mechanism matrix Q with rows that are multiples of 0 to 32, the size of each matrix can be an integer multiple of 32, so the disadvantages of the multi-byte data width mentioned above can be avoided. Moreover, the impact of filling the matrix Q with 0, that is, dirty data, has no effect on the valid data throughout the process and does not participate in the calculation of the valid data.

[0026] In short, the computing module of the designed accelerator architecture is roughly divided into two parts. One part is the fixed-point computing core, which is used to calculate matrix multiplication. The other part is the floating-point computing core, which is used as a coprocessor of the fixed-point computing core to calculate activation functions such as the SoftMax function. The core of the fixed-point computing core is a systolic array with a size of 32 times 32; the core of the floating-point computing core is 16 groups of exponential computing units, accumulation computing units, and division computing units.

[0027] In general, the attention mechanism includes the MatMul operation, which is used to perform the multiplication of two matrices Q and K; Scale; dequantize, SoftMax function and quantize, see Figure 2 As shown. Considering that the calculation process of the attention mechanism contains the SoftMax function, which is a nonlinear function and difficult to quantize, we choose to use floating-point computing kernels for processing, and the matrix multiplication part uses conventional fixed-point calculations. The fine-grained filling method of the algorithm layer and the corresponding accelerator architecture will be described in detail below.

[0028] 1) Fine-grained filling at the algorithm layer

[0029] In order to achieve the principle of minimizing the impact on hardware performance, in one embodiment, only the zero padding operation is performed in the scaled point product attention mechanism, and the key matrix to be filled is the attention score matrix.

[0030] Figure 3 Three solutions are shown to fill the size of the attention score matrix to a multiple of 32. Figure 3 It can be seen that padding matrix K (K-Padding) makes the vector length of attention score matrix S greater than the vector length of matrix V, violating the premise of matrix multiplication. Although padding matrix Q and matrix K (QK-Padding) at the same time can bypass the above shortcomings, it will cause significant additional computing power consumption, and dirty data is involved in the calculation of valid data, and additional control logic is required in hardware implementation. When padding matrix Q (Q-Padding), dirty data is only stored in the last few rows of vectors during the entire calculation process of the attention mechanism, which has no effect on valid data. After obtaining the final result, the dirty data can be left unwritten back to the memory.

[0031] 2) Accelerator hardware architecture design

[0032] In one embodiment, the hardware architecture of the accelerator needs to support the scaling dot multiplication attention mechanism with padding 0 operation, which is decomposed into operators, that is, the need to calculate matrix multiplication, scaling, and softmax activation function. For example, if the attention mechanism is the original transformer and BERT network, the scaling operation is always divided by 8, that is, right shift by three bits to solve it.

[0033] Specifically, combined Figure 4 As shown, the accelerator architecture supporting the attention mechanism provided as a whole includes a weight cache unit (WB), a thread cache unit (TB), a systolic array, a floating-point coprocessor, a temporary cache unit, a post-processing unit, a submission cache unit, etc. In addition, a corresponding state machine is designed for each unit to control the processing state and the operation process. For example, the thread cache state machine (FSM-TB) is used to control the flow of thread data from the system bus to the thread cache unit, and the weight cache state machine (FSM-WB) is used to control the flow of weight data from the system bus to the weight cache unit. The attention tensor processing unit state machine is used to control the loading and flow of thread data and weight data from the corresponding cache unit to the systolic array.

[0034] Still combined Figure 4 As shown, the systolic array is a two-dimensional systolic array, which contains multiple processing elements (PEs). The processing elements in the same column share a finite state machine. Each processing element is responsible for data calculation. The weight data is input from the top of the systolic array and propagates from top to bottom. The thread data is input from the left side of the systolic array and propagates from left to right. Each processing element calculates the received data and weight, and the result is output from the right side of the systolic array.

[0035] For example, when calculating the multiplication of matrix Q and matrix K, Q is placed in the thread cache unit and K is placed in the weight cache unit. When calculating the multiplication of matrix S and matrix V, S is placed in the thread cache unit and V is placed in the weight cache unit. It should be understood that in this article, "thread cache" and "weight cache" are only used to represent the name identifiers of different caches, because many operators of neural networks operate on input data and weight data. In addition, the fine-grained filling scheme can be implemented by FSM-TB, that is, it is controlled by the thread cache state machine.

[0036] For example, matrix multiplication is performed through the systolic array of the fixed-point computing core. The data starts from the memory, passes through the DRAM controller and PHY to the AXI4 module, and then reaches the thread cache and weight cache under the control of the thread cache state machine and the weight cache state machine, and then inputs the data to the systolic array. The weight is propagated in a broadcast form, that is, a weight buffer transmits weight data to each processing unit in the same column, and the thread data is transmitted one grid to the right in one cycle, and continues to pass to the right until it reaches the last column of processing units. The systolic array consists of 32 times 32 processing units. The processing units in the same column share a finite state machine. The jump logic of the state machine is only executed by the processing unit leader, and the processing unit executor is controlled by the state machine signal sent by the processing unit leader. The calculation results of the systolic array are sent to different branches depending on whether the current operator has an activation function (ActF). If there is an activation function, it is sent to the temporary buffer unit (Stage Buffer), and then to the floating point coprocessor (FPU coprocessor) for calculation; if there is no activation function, it is sent to the post-processing unit (PPU), which performs requantization, adds quantization zeros and truncates to 8 bits, and then sends it to the commit buffer unit (Commit Buffer). The filling 0 operation is controlled by the thread cache state machine, and the discarding of dirty data is controlled by the commit cache state machine (FSM-CB).

[0037] The SoftMax activation function is calculated by the floating-point coprocessor (FPU coprocessor). In order to consider both the processor speed and chip area, the arithmetic unit of the floating-point core is set to 16, so that the fixed-point core and the floating-point core can achieve similar processing speeds, so the two can work like a large pipeline. The data comes from the temporary buffer of the fixed-point core, and is converted to 32-bit floating-point numbers through the fixed-point to floating-point module (Fixed to Float), and then successively passes through the exponential operation unit (EXP), the accumulation operation unit (ACC), and the division operation (DIV) for calculation to obtain the result of the SoftMax activation function of the 32 lines, and then is written to the submission buffer through the floating-point to fixed-point module (Float to Fixed). The floating-point to fixed-point module has quantized scaling parameters and zero-point parameters, and by converting the division by the scaling parameter into the inverse of the multiplication by the scaling parameter, the required divider is converted into a multiplier to save area, and finally the data is written back to the memory.

[0038] To further verify the effect of the present invention, simulation experiments were conducted, using VCS of Synopsys Company for simulation, and VIVADO of XILINX for simulation, synthesis, and layout and routing.

[0039] 1) Simulation of the process of multiplying matrix Q and matrix K

[0040] In the simulation experiment, the matrix multiplication results of 128x64 matrix and 128x64 matrix, as well as the matrix multiplication results of 111x64 matrix and 111x64 matrix with padding 0 were verified. In this process, when the number of rows of matrix Q is not an integer multiple of 32, the number of row vectors of matrix Q is padded to an integer multiple of 32, and then input into the systolic array for calculation. From the simulation results, it can be seen that for matrix multiplication that needs to be padded with 0, the calculation process proceeds normally, and there is a certain acceleration effect compared to the case of an integer multiple of 32.

[0041] 2) Simulation of multiplication of matrix S and matrix V

[0042] The results of matrix multiplication of 128x128 matrix and 64x128 matrix, as well as the results of matrix multiplication of 128x111 matrix and 64x111 matrix with padding 0 were verified respectively. In this process, the number of rows of matrix S has been padded to multiples of 32 in the previous step. From the simulation results, it can be seen that for the matrix multiplication filled with 0, the calculation process is normal, and there is a certain acceleration effect compared to the case of coarse-grained padding to multiples of 32.

[0043] 3) Hardware resources and performance

[0044] By using Vivado to perform layout and routing, the hardware resource occupancy information of the present invention is obtained, see Table 1.

[0045] Table 1: Hardware resource usage

[0046]

[0047] The hardware performance of the present invention is obtained by simulating the number of cycles, see Table 2.

[0048] Table 2: Hardware performance

[0049]

[0050] It can be seen that the matrix multiplication and scaling point multiplication attention mechanism of the present invention have an acceleration effect compared with the CPU and GPU platforms.

[0051] In summary, the present invention designs a fine-grained filling scheme for the algorithm layer of the attention mechanism, and provides a hardware architecture with a fine-grained filling scheme for the attention mechanism, as well as a data flow model of the hardware architecture, which can flexibly handle different sequence lengths of different samples in the attention mechanism with little impact on hardware performance, and supports the standard attention mechanism, with a wider range of applications.

[0052] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0053] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0054] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0055] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0056] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0057] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0058] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0059] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0060] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. An accelerator supporting an attention mechanism, for implementing matrix multiplication operations, scaling operations or nonlinear activation processing involved in the attention mechanism, the accelerator comprising a thread cache state machine module, a weight cache state machine module, an attention tensor processing unit state machine module, a weight cache unit, a thread cache unit, a floating point coprocessor, a temporary cache unit, a post-processing unit, a commit cache unit and a two-dimensional systolic array, wherein the two-dimensional systolic array comprises a plurality of processing units connected in a systolic form; wherein: The weight cache state machine module is used to control the flow of weight data from the system bus to the weight cache unit; The thread cache state machine module is used to control the flow of thread data from the system bus to the thread cache unit; The attention tensor processing unit state machine module is used to control the loading and flow of thread data and weight data from the two-dimensional systolic array in different directions, wherein the thread data is controlled to be passed to the next processing unit in each cycle, and the weight data is propagated to each processing unit in the corresponding direction in a broadcast form; The calculation result of the two-dimensional systolic array is transmitted to different branches according to whether nonlinear activation processing needs to be performed. If necessary, it is transmitted to the temporary cache unit and then transmitted to the floating-point coprocessor for calculation; If not needed, it is transferred to the post-processing unit, which performs a re-quantization operation and then transfers it to the submission buffer unit.

2. The accelerator according to claim 1, characterized in that The nonlinear activation process is a SoftMax activation function operation, and the floating-point coprocessor includes a fixed-point to floating-point module, a floating-point to fixed-point module, a floating-point cache unit, an exponential operation unit, an accumulation operation unit, and a division operation unit; wherein: The fixed-point to floating-point module converts the data from the temporary cache unit into floating-point numbers, and then calculates through the exponential operation unit, the accumulation operation unit and the division operation unit to obtain the result of the SoftMax activation function, and then writes it to the submission buffer unit through the floating-point to fixed-point module. The floating-point to fixed-point module has quantized scaling parameters and zero point parameters, and converts the required divider into a multiplier by converting the division by the scaling parameter into the multiplication by the inverse of the scaling parameter, and finally the data is written back to the memory.

3. The accelerator according to claim 1, characterized in that: The two-dimensional systolic array includes 32*32 processing units.

4. The accelerator according to claim 3, characterized in that: The number of arithmetic units of the floating-point core of the floating-point coprocessor is set to 16.

5. The accelerator according to claim 1, characterized in that: For the two-dimensional systolic array, processing units in the same column share a finite state machine, the jump logic of the finite state machine is only executed by the processing unit leader, the processing unit executors are controlled by the state machine signals sent by the processing unit leader, and each processing unit processes the data flowing through it simultaneously and in parallel.

6. The accelerator according to claim 3, characterized in that: The thread data loaded into the two-dimensional systolic array is thread data after a fine-grained zero-filling operation.

7. The accelerator according to claim 6, characterized in that A fine-grained zero-filling operation is performed on the matrix Q in the attention mechanism to fill the size of the attention score matrix to an integer multiple of 32. The zero-filling operation is controlled by the thread cache state machine module, and the discarding of dirty data is controlled by the commit cache state machine.

8. The accelerator according to claim 7, characterized in that The zero padding operation is only performed in the scaled dot product attention mechanism.

9. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the accelerator according to any one of claims 1 to 8 is implemented.

Images