Attention mechanism-supporting accelerator

Abstract

Disclosed is an attention mechanism-supporting accelerator, used for implementing a matrix multiplication operation, a scaling operation, or nonlinear activation processing involved in an attention mechanism. A weight cache state machine module is used for controlling the flow of weight data from a system bus to a weight cache unit. A thread cache state machine module is used for controlling the flow of thread data from the system bus to a thread cache unit. An attention tensor processing unit state machine module is used for controlling the loading and flow of the thread data and the weight data from different directions into a two-dimensional systolic array. The thread data is controlled to be delivered to the next processing unit in each period, and the weight data is propagated to each processing unit in a corresponding direction in the form of broadcasting. A computation result of the two-dimensional systolic array is delivered to different branches on the basis of whether nonlinear activation processing needs to be executed. According to the present invention, data padding is performed on a matrix so that the matrix is aligned with the systolic array data flow, thereby accelerating an attention mechanism having a variable sequence length by using hardware.

Description

An accelerator that supports attention mechanisms Technical Field

[0001] The present invention relates to the technical field of integrated circuit design, and more particularly, to an accelerator supporting an attention mechanism. Background Art

[0002] Because the attention mechanism contains many computationally intensive operators, such as matrix multiplication, it requires hardware acceleration. Currently, mainstream attention mechanism accelerators accelerate sparse attention mechanisms or standard attention mechanisms, but do not optimize for sequence length variability.

[0003] In processor design, to increase memory access speed, the system data bit width is often multiple bytes. However, this type of data has the disadvantage of making it difficult to store or write data starting from the middle of these bytes. This disadvantage also applies to the input and output ports of on-chip SRAM. Therefore, to mitigate this disadvantage, data is best stored in a neat format, meaning that the data block size should be an integer multiple of the system data bit width.

[0004] In the attention mechanism, sequence length is a variable. The size of the attention score in the calculation process is the sequence length multiplied by the sequence length. If left unaddressed, it is likely to lead to the aforementioned difficulty in memory access and difficulty in on-chip data flow. Therefore, a certain strategy is needed to optimize the impact of the sequence length variable on the hardware accelerator.

[0005] Existing technologies don't target standard attention mechanisms for acceleration, limiting their applicability in practical applications. Furthermore, there's currently no dedicated optimization for sequence length variability in attention mechanisms, and using coarse-grained padding can significantly degrade performance or increase power consumption. Technical issues

[0006] Existing technologies don't target standard attention mechanisms for acceleration, limiting their applicability in practical applications. Furthermore, there's currently no dedicated optimization for sequence length variability in attention mechanisms, and using coarse-grained padding can significantly degrade performance or increase power consumption. Technical Solutions

[0007] The present invention aims to overcome the shortcomings of the above-mentioned prior art and provide an accelerator that supports the attention mechanism, which is used to implement matrix multiplication, scaling, or nonlinear activation processing involved in the attention mechanism. The accelerator includes a thread cache state machine module, a weight cache state machine module, an attention tensor processing unit state machine module, a weight cache unit, a thread cache unit, a floating-point coprocessor, a temporary cache unit, a post-processing unit, a commit cache unit, and a two-dimensional systolic array. The two-dimensional systolic array includes multiple processing units connected in a systolic manner; wherein:

[0008] The weight cache state machine module is used to control the flow of weight data from the system bus to the weight cache unit;

[0009] The thread cache state machine module is used to control the flow of thread data from the system bus to the thread cache unit;

[0010] The attention tensor processing unit state machine module is used to control the loading and flow of thread data and weight data from the two-dimensional systolic array in different directions, wherein the thread data is controlled to be passed to the next processing unit in each cycle, and the weight data is propagated to each processing unit in the corresponding direction in a broadcast form;

[0011] The calculation result of the two-dimensional systolic array is transmitted to different branches according to whether nonlinear activation processing needs to be performed. If necessary, the result is transmitted to the temporary cache unit and then transmitted to the floating-point coprocessor for calculation; if not, the result is transmitted to the post-processing unit, which performs a requantization operation and then transmitted to the commit cache unit. Beneficial effects

[0012] Compared with the existing technology, the advantage of the present invention is that it realizes an accelerator that supports the attention mechanism, uses a fine-grained filling method to reduce the impact of the arbitrariness of sequence length in the attention mechanism on the hardware data flow, and improves the computational efficiency and application scope of the attention mechanism.

[0013] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0015] FIG1 is a schematic diagram of a fine-grained filling method of an algorithm layer according to an embodiment of the present invention;

[0016] FIG2 is a schematic diagram of an operating network of an attention mechanism according to an embodiment of the present invention;

[0017] FIG3 is a schematic diagram of different filling methods for the attention mechanism according to one embodiment of the present invention;

[0018] FIG4 is an architectural diagram of a hardware accelerator according to an embodiment of the present invention;

[0019] In the accompanying figure, FPU-floating point unit; float-floating point; fixed-fixed point; PPU-post-processing unit; ActF-activation function; Padding-padding; unpadding-reverse operation of padding; leader-leader; worker-executor; FSM-FPU-floating point unit state machine; FSM-CB-commit buffer state machine; FPB-floating point buffer; FSM-ATU-attention tensor processing unit state machine. Best Mode for Carrying Out the Invention

[0020] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0021] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0022] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0023] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0024] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0025] The attention mechanism can be applied to various neural network models, such as the Transformer model and the BERT model. Attention can be described as mapping a query and a set of key-value pairs to an output, where the query Q (Query), key K (Key), value V (Value), and output are all vectors. To address the variability of sequence length in the attention mechanism, this paper proposes a fine-grained padding method at the algorithm level and designs a corresponding hardware architecture level, aiming to minimize the impact of sequence length variability on accelerator performance and area.

[0026] Figure 1 illustrates the fine-grained padding scheme at the algorithm level, where light gray areas represent zero padding, white areas represent dirty data, and dark gray areas represent valid data. As shown in Figure 1, by padding the attention mechanism's matrix Q with rows from 0 to multiples of 32, each matrix size is a multiple of 32, thus avoiding the aforementioned drawbacks of multi-byte data widths. Furthermore, the effect of padding matrix Q with zeros, i.e., dirty data, has no effect on valid data and does not participate in valid data calculations.

[0027] In short, the computing module of the designed accelerator architecture is broadly divided into two parts: one is the fixed-point core, which performs matrix multiplication. The other is the floating-point core, which serves as a coprocessor for the fixed-point core and calculates activation functions such as the SoftMax function. The core of the fixed-point core is a systolic array with a size of 32 by 32; the core of the floating-point core is 16 groups of exponential units, accumulation units, and division units.

[0028] Generally speaking, the attention mechanism includes the MatMul operation, which multiplies two matrices Q and K; scaling; dequantization; the SoftMax function; and quantization, as shown in Figure 2. Given that the attention mechanism's computation involves the SoftMax function, a nonlinear function that is difficult to quantize, floating-point kernels are used for processing, while conventional fixed-point calculations are used for the matrix multiplication. The following sections describe the fine-grained padding method for the algorithm layer and the corresponding accelerator architecture.

[0029] 1) Fine-grained filling at the algorithm layer

[0030] In order to achieve the principle of minimizing the impact on hardware performance, in one embodiment, the zero padding operation is only performed in the scaled dot product attention mechanism, and the key matrix for padding is the attention score matrix.

[0031] Figure 3 shows three methods for padding the size of the attention score matrix to a multiple of 32. As shown in Figure 3, padding the matrix K (K-Padding) causes the vector length of the attention score matrix S to be larger than the vector length of the matrix V, violating the prerequisite for matrix multiplication. While padding both the matrix Q and the matrix K (QK-Padding) can circumvent the aforementioned drawbacks, it incurs significant additional computational power consumption. Furthermore, dirty data is incorporated into the calculation of valid data, requiring additional control logic for hardware implementation. When padding the matrix Q (Q-Padding), dirty data is stored only in the last few rows of the attention mechanism's computation, leaving it unaffected by the valid data. After obtaining the final result, the dirty data is simply not written back to memory.

[0032] 2) Accelerator hardware architecture design

[0033] In one embodiment, the accelerator's hardware architecture needs to support a scaled dot-product attention mechanism with zero padding. This is broken down into three operators: matrix multiplication, scaling, and the softmax activation function. For example, if the attention mechanism being used is the original Transformer or BERT network, the scaling operation is typically divided by 8, meaning a right shift of three bits.

[0034] Specifically, as shown in Figure 4, the provided accelerator architecture supporting the attention mechanism generally includes a weight buffer unit (WB), a thread buffer unit (TB), a systolic array, a floating-point coprocessor, a temporary buffer unit, a post-processing unit, a commit buffer unit, and the like. Furthermore, a corresponding state machine is designed for each unit to control the processing state and computational process. For example, the thread buffer state machine (FSM-TB) is used to control the flow of thread data from the system bus to the thread buffer unit, and the weight buffer state machine (FSM-WB) is used to control the flow of weight data from the system bus to the weight buffer unit. The attention tensor processing unit state machine is used to control the loading and flow of thread data and weight data from the corresponding buffer units to the systolic array.

[0035] Still referring to Figure 4, a systolic array is a two-dimensional array consisting of multiple processing elements (PEs). PEs in the same column share a finite state machine. Each PE is responsible for data computation. Weight data is input from the top of the systolic array and propagates from top to bottom. Thread data is input from the left side of the systolic array and propagates from left to right. Each PE performs computations on the received data and weights, and the results are output from the right side of the systolic array.

[0036] For example, when calculating the multiplication of matrix Q and matrix K, Q is placed in the thread cache unit and K is placed in the weight cache unit. When calculating the multiplication of matrix S and matrix V, S is placed in the thread cache unit and V is placed in the weight cache unit. It should be understood that in this article, "thread cache" and "weight cache" are merely names used to identify different caches. This is because many operators in neural networks operate on input data and weight data. In addition, the fine-grained filling scheme can be implemented by FSM-TB, that is, it is controlled by the thread cache state machine.

[0037] For example, matrix multiplication is performed using the systolic array of fixed-point compute cores. Data originates from memory, passes through the DRAM controller and PHY, and reaches the AXI4 module. Then, under the control of the thread cache state machine and weight cache state machine, it reaches the thread cache and weight cache, and is then input into the systolic array. Weights are broadcast, with each weight buffer transmitting weight data to every processing element in the same column. Thread data is passed rightward one grid per cycle, continuing rightward until it reaches the last processing element in the column. The systolic array consists of 32 by 32 processing elements. Processing elements in the same column share a finite state machine. The state machine jump logic is executed only by the processing element leader, and the processing element executors are controlled by state machine signals sent by the processing element leader. The results of the systolic array calculations are routed to different branches depending on whether the current operator has an activation function (ActF). If an activation function is present, the results are sent to the stage buffer and then to the floating-point coprocessor (FPU) for computation. If no activation function is present, the results are routed to the post-processing unit (PPU), which performs requantization, adds zeros, truncates the result to 8 bits, and then sends the result to the commit buffer. Zero padding is controlled by the thread cache state machine, while dirty data discarding is controlled by the commit buffer state machine (FSM-CB).

[0038] The SoftMax activation function is calculated by the floating-point coprocessor (FPU). To balance processor speed and chip area, the floating-point core's arithmetic units are set to 16. This allows the fixed-point and floating-point cores to achieve similar processing speeds, allowing them to operate as a large pipeline. Data originates from the fixed-point core's temporary buffer and is converted to 32-bit floating-point numbers via the Fixed to Float conversion module. The data then passes through the Exponentiation Unit (EXP), Accumulation Unit (ACC), and Division (DIV) to calculate the SoftMax activation function for each of the 32 rows. The result is then written to the submission buffer via the Float to Fixed conversion module. This module incorporates quantized scaling and zero-point parameters. By converting division by the scaling parameter into multiplication by the inverse of the scaling parameter, the required dividers are converted into multipliers, saving area. Finally, the data is written back to memory.

[0039] To further verify the effect of the present invention, simulation experiments were conducted using Synopsys's VCS for simulation, and XILINX's VIVADO for simulation, synthesis, and layout and routing.

[0040] 1) Simulation of the process of multiplying matrix Q and matrix K

[0041] In simulation experiments, we verified the results of matrix multiplications of 128x64 matrices and 128x64 matrices, as well as the results of matrix multiplications of 111x64 matrices and 111x64 matrices with zero padding. In this process, when the number of rows in matrix Q is not an integer multiple of 32, the number of row vectors in matrix Q is padded to an integer multiple of 32 and then input into the systolic array for calculation. The simulation results show that for matrix multiplications requiring zero padding, the calculation process proceeds normally and is significantly accelerated compared to the case where the number of rows is an integer multiple of 32.

[0042] 2) Simulation of multiplication of matrix S and matrix V

[0043] We verified the results of matrix multiplications of a 128x128 matrix and a 64x128 matrix, as well as the results of matrix multiplications of a 128x111 matrix and a 64x111 matrix with zero padding. In this process, matrix S had its rows padded to multiples of 32 in the previous step. The simulation results show that the calculation process for zero-padded matrix multiplications proceeds normally and achieves a certain speedup compared to the coarse-grained matrix multiplication with padded rows to multiples of 32.

[0044] 3) Hardware resources and performance

[0045] By using Vivado to perform layout and routing, the hardware resource occupancy information of the present invention is obtained, see Table 1.

[0046] Table 1: Hardware resource usage

[0047]

[0048] The hardware performance of the present invention is obtained by simulating the number of cycles, see Table 2.

[0049] Table 2: Hardware performance

[0050]

[0051] It can be seen that the attention mechanism for calculating matrix multiplication and scaled point multiplication in the present invention has an acceleration effect compared to the CPU and GPU platforms.

[0052] In summary, the present invention designs a fine-grained filling scheme for the algorithm layer of the attention mechanism, and provides a hardware architecture with a fine-grained filling scheme for the attention mechanism, as well as a data flow model of the hardware architecture, which can flexibly handle different sequence lengths of different samples in the attention mechanism, with little impact on hardware performance, and supports the standard attention mechanism, with a wider range of applications.

[0053] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0054] A computer-readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure within a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0055] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0056] The computer program instructions for performing the operations of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, and conventional procedural programming languages ​​such as "C" or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, the state information of the computer-readable program instructions is used to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), so that the electronic circuit can execute the computer-readable program instructions, thereby implementing various aspects of the present invention.

[0057] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0058] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0059] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0060] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0061] While various embodiments of the present invention have been described above, the foregoing description is illustrative and non-exhaustive, and is not intended to limit the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the disclosed embodiments. The scope of the present invention is defined by the appended claims.

Claims

1. An accelerator supporting an attention mechanism, for implementing matrix multiplication operations, scaling operations or nonlinear activation processing involved in the attention mechanism, the accelerator comprising a thread cache state machine module, a weight cache state machine module, an attention tensor processing unit state machine module, a weight cache unit, a thread cache unit, a floating point coprocessor, a temporary cache unit, a post-processing unit, a commit cache unit and a two-dimensional systolic array, wherein the two-dimensional systolic array comprises a plurality of processing units connected in a systolic form; wherein: The weight cache state machine module is used to control the flow of weight data from the system bus to the weight cache unit; The thread cache state machine module is used to control the flow of thread data from the system bus to the thread cache unit; The attention tensor processing unit state machine module is used to control the loading and flow of thread data and weight data from the two-dimensional systolic array in different directions, wherein the thread data is controlled to be passed to the next processing unit in each cycle, and the weight data is propagated to each processing unit in the corresponding direction in a broadcast form; The calculation result of the two-dimensional systolic array is transmitted to different branches according to whether nonlinear activation processing needs to be performed. If necessary, it is transmitted to the temporary cache unit and then transmitted to the floating-point coprocessor for calculation; If not needed, it is transferred to the post-processing unit, which performs a re-quantization operation and then transfers it to the submission buffer unit.

2. The accelerator according to claim 1, characterized in that The nonlinear activation process is a SoftMax activation function operation, and the floating-point coprocessor includes a fixed-point to floating-point module, a floating-point to fixed-point module, a floating-point cache unit, an exponential operation unit, an accumulation operation unit, and a division operation unit; wherein: The fixed-point to floating-point module converts the data from the temporary cache unit into floating-point numbers, and then calculates through the exponential operation unit, the accumulation operation unit and the division operation unit to obtain the result of the SoftMax activation function, and then writes it to the submission buffer unit through the floating-point to fixed-point module. The floating-point to fixed-point module has quantized scaling parameters and zero point parameters, and converts the required divider into a multiplier by converting the division by the scaling parameter into the multiplication by the inverse of the scaling parameter, and finally the data is written back to the memory.

3. The accelerator according to claim 1, characterized in that: The two-dimensional systolic array includes 32*32 processing units.

4. The accelerator according to claim 3, characterized in that: The number of arithmetic units of the floating-point core of the floating-point coprocessor is set to 16.

5. The accelerator according to claim 1, characterized in that For the two-dimensional systolic array, processing units in the same column share a finite state machine, the jump logic of the finite state machine is only executed by the processing unit leader, the processing unit executors are controlled by the state machine signals sent by the processing unit leader, and each processing unit processes the data flowing through it simultaneously and in parallel.

6. [Corrected 09.01.2024 in accordance with Rule 26] An accelerator according to claim 3, characterized in that The thread data loaded into the two-dimensional systolic array is thread data after a fine-grained zero-filling operation.

7. The accelerator according to claim 6, characterized in that A fine-grained zero-filling operation is performed on the matrix Q in the attention mechanism to fill the size of the attention score matrix to an integer multiple of 32. The zero-filling operation is controlled by the thread cache state machine module, and the discarding of dirty data is controlled by the commit cache state machine.

8. The accelerator according to claim 7, characterized in that The zero padding operation is only performed in the scaled dot product attention mechanism.

9. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the accelerator according to any one of claims 1 to 8 is implemented.

Images