Attention mechanism-oriented bidirectional systolic array device

Abstract

The invention discloses an attention mechanism-oriented bidirectional systolic array device, which is used for attention mechanism operation in a neural network. The device comprises a weight buffer used for caching weight data; the excitation buffer is used for caching excitation data; the systolic array comprises N * M two-dimensional processing units connected in a systolic form, the excitation data is controlled to enter and flow towards the systolic array in a first direction, and the weight data is controlled to enter and flow towards the systolic array in a second direction; the SoftMax processing unit is used for receiving the output number of the systolic array in the first direction and executing SoftMax function processing according to a set activation function; and the result buffer is used for receiving a calculation result of the systolic array or the SoftMax processing unit. According to the method, short vectors and long vectors in an attention mechanism can be flexibly processed, and the overall calculation delay is reduced.

Description

Technical Field

[0001] The present invention relates to the field of integrated circuit technology, and more particularly to a bidirectional systolic array device oriented to an attention mechanism. Background Art

[0002] The attention mechanism requires hardware acceleration because it contains many computationally intensive operators, such as matrix multiplication. At the same time, certain edge applications, such as self-driving cars and edge cameras, also require local edge computing.

[0003] Mainstream edge matrix multiplication accelerators are often based on systolic array architectures because they offer both high parallelism and low power consumption. Currently, these mainstream systolic arrays share a common characteristic: their output direction is perpendicular to the input direction of the stimulus data, thereby maintaining row vector continuity in the output data. However, the length of the matrix row vectors corresponding to the matrix multiplication operator in the attention mechanism is typically on the order of tens. Using traditional systolic arrays results in a significant amount of additional time overhead due to the inherent structural limitations of the systolic arrays.

[0004] After analysis, the existing accelerators based on systolic arrays have the following main defects:

[0005] 1) The acceleration goals are relatively general, usually directly transplanted from traditional CNN accelerators, without specific optimization for the characteristics of the attention mechanism;

[0006] 2) Current accelerators that support the attention mechanism tend to focus on model pruning and SoftMax processing. Matrix multiplication acceleration, because it is relatively mature, has received less attention, and few solutions optimize it.

[0007] 3) There is currently no optimization solution for the multiplication of long and short vectors in the attention mechanism, and the existing solution cannot take into account the computational efficiency of long and short vectors. Summary of the Invention

[0008] The present invention aims to overcome the above-mentioned shortcomings of the prior art and provide a bidirectional systolic array device for attention mechanism operation in neural networks. The device comprises:

[0009] Weight buffer: used to cache weight data;

[0010] Stimulus buffer: used to cache stimulus data;

[0011] A systolic array comprising N*M two-dimensional processing units connected in a systolic manner, wherein the excitation data is controlled to be fed into and flowed into the systolic array in a first direction, and the weight data is controlled to be fed into and flowed into the systolic array in a second direction;

[0012] A SoftMax processing unit is configured to receive the output of the systolic array in the first direction and perform SoftMax function processing according to a set activation function;

[0013] Result buffer: used to receive the calculation results of the systolic array or the SoftMax processing unit.

[0014] In one embodiment, the systolic array is controlled to perform calculations according to the following timing:

[0015] Phase 1: performing an input to the systolic array and performing a scalar multiplication on the input data, wherein the input direction of the stimulus data is the same as the output direction of the systolic array;

[0016] Phase 2: When no new data enters, the last number that entered the systolic array unit is passed on until the calculation is completed;

[0017] Phase 3: When all the input numbers are calculated, the data in the systolic array unit starts to be transferred out and written into the result buffer.

[0018] In one embodiment, when performing multiplication operations on the query matrix Q and the key matrix K in the attention mechanism of the neural network, in the second and third stages, the data flow direction of the systolic array is the same as that in the first stage.

[0019] In one embodiment, when performing calculations on a linear layer of a multi-head attention mechanism in the neural network, in the third stage, the data flow direction of the systolic array is controlled to be perpendicular to that of the first stage so that the output of the systolic array is row vector continuous.

[0020] Compared with the existing technology, the advantage of the present invention is that, in view of the characteristics that the vector length of matrix multiplication in the attention mechanism is relatively small and the vector length of the generalized matrix multiplication represented by the linear layer is relatively large, the present invention designs a hardware architecture that performs calculations with the most suitable data flow according to different instructions, which can flexibly handle short vectors and long vectors in the attention mechanism and reduce the overall calculation delay.

[0021] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0023] Figure 1 This is a schematic diagram of the classic systolic array output method in the prior art;

[0024] Figure 2 It is a data flow diagram of a classic systolic array in the prior art;

[0025] Figure 3 1 is a schematic diagram of a method for outputting data of a bidirectional systolic array device for an attention mechanism according to an embodiment of the present invention;

[0026] Figure 4 Schematic diagram of the data flow of a bidirectional systolic array device for an attention mechanism according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0028] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0029] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0030] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0031] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0032] First, it should be noted that the attention-oriented bidirectional systolic array device (or architecture) provided by the present invention can be a processor designed for neural network computing, or it can be only part of a processor. The processor can be applied to fields such as text processing, speech recognition and processing, multilingual translation, image recognition, biometric recognition, and intelligent control. It can be used as an intelligent computing processor, a robot, a mobile device, and can also be used to build large-scale neural network computing supercomputers. The processor based on the present invention improves the universality of neural network computing and reduces operating power consumption.

[0033] The attention mechanism can be applied to various types of neural network models, such as Transformer networks or BERT networks. Attention can be described as mapping a query and a set of key-value pairs to an output, where the query matrix Q (Query), the key matrix K (Key), the value matrix V (Value) and the output are all vectors. The present invention calculates the matrix multiplication in the attention mechanism. After investigation and analysis, the vector length of its matrix multiplication is generally 64 or the same order of magnitude, which is different from the matrix multiplication performed after the convolution in CNN through the img2col operation (convolution is achieved through matrix multiplication). The vector length of the latter matrix multiplication is generally on the order of thousands, and sometimes even tens of thousands. The vector lengths of these two matrix multiplications are different in order of magnitude, so it is necessary to adopt different hardware architecture designs and different optimization schemes.

[0034] In addition, considering that the generalized matrix multiplication represented by the linear layer in the attention mechanism has a relatively long vector length, ranging from hundreds to thousands, the characteristics of these two vector lengths require the optimization solution to have a certain degree of compatibility and need to change as the input characteristics change.

[0035] In response to the above-mentioned characteristics of the attention mechanism, the present invention optimizes the classic systolic array hardware architecture to significantly improve the calculation speed of the systolic array when the vector length is low, thereby reducing the overall calculation latency, while being compatible with scenarios with long vector lengths such as linear layers.

[0036] Figure 1 This is an example of the architecture and data output method of a classic systolic array. To ensure row-vector continuity, a classic systolic array outputs data perpendicular to the stimulus input. This design simplifies subsequent data processing and improves the efficiency of writing data back to memory, facilitating the next call.

[0037] Figure 2 This is the data flow in the classic systolic array calculation process. Taking a 5x5 systolic array as an example, it is generally divided into the following three stages.

[0038] The first stage is to advance the number and perform scalar multiplication on the input data. Under the premise of advancing one number in one cycle, the number of clock cycles spent in this stage is the same as the vector length of the matrix multiplication.

[0039] In the second stage, no new data enters. The last number entering the systolic array is passed back until it is calculated in the systolic array. The number of clock cycles spent in this stage is the array width of the systolic array minus one.

[0040] The third stage is when all the numbers are calculated. The data in the systolic array begins to move out and is written to the result buffer. Then, as needed, it is sent to other units for calculation or written back to memory. The number of clock cycles spent in this stage is the number of rows in the systolic array.

[0041] Next, we analyze the characteristics of the attention mechanism. This paper mainly focuses on the mainstream Transformer and its variant networks. Here we take the original Transformer, its mainstream network BERT in natural language processing, and its mainstream network ViT in computer vision as examples. The results of the investigation are shown in Table 1 below.

[0042] Table 1: Model parameters of the Transformer network

[0043]

[0044] Table 1 shows that the dimension of each head (Dim / Heads), or the length of the row vector, in the attention mechanisms of these networks is a constant 64. Compared to the vector lengths in convolutional neural networks, which can reach tens of thousands, this is a very small value. If the attention mechanism is run on a classic 32x32 systolic array, the number of clock cycles required for one round of computation is 64 + 31 + 32 = 127. The 31 and 32 cycles are constants determined by the size of the systolic array. This shows that the vector length plays a smaller role than in convolutional neural network computations. In the design of accelerators for the attention mechanism, if the latter two factors can be reduced, the overall computational latency will be significantly reduced, as the attention mechanism is a core computational module in Transformer networks and is repeatedly calculated.

[0045] Figure 3It is a bidirectional systolic array device for the attention mechanism provided by the present invention, which is a new systolic array hardware architecture. In this article, the processor where this systolic array is located is named AtteNPU, which means an NPU that supports the Attention mechanism (specifically referring to the hardware architecture where the bidirectional systolic array of the present invention is located). In general, the device includes: a weight buffer for storing weight data; an excitation buffer for storing excitation data; a systolic array, which includes a plurality of processing units (PEs) connected in a systolic form, wherein the excitation data can be loaded and flowed to the systolic array from the row direction or the column direction, and the weight data can be loaded and flowed to the systolic array from the column direction or the row direction; a SoftMax processing unit: for receiving the output of the systolic array and performing SoftMax function processing according to the set activation function; a result buffer for receiving the calculation results of the systolic array or the SoftMax processing unit.

[0046] See also Figure 4 As shown, the characteristic of this new systolic array output is that for the multiplication of matrix Q and matrix K in the attention mechanism, the systolic array output direction is in the same direction as the stimulus data input direction, as shown in Figure 4 As shown in , the third stage and the second stage can also be in the same direction. The advantage of this is that the second and third stages of the three stages of the classic systolic array calculation process mentioned above can be combined, that is, after the PE calculation of the first column is completed, it can be shifted out of the PE array in the next cycle. Since this data shifting method is very fast, this article names it Flash-Shift (abbreviated as F-Shift, specifically referring to the new design of the present invention in which the data shifting direction is consistent with the excitation data input direction). When F-Shift is activated, for the multiplication of matrices Q and K in the attention mechanism, one round of data calculation only requires 64+31+1=96 cycles to complete, and the latency reduction ratio is as high as 24.41%, which is approximately 1 / 4 of the latency reduction, which can significantly improve processor performance.

[0047] When calculating the linear layer in the multi-head attention mechanism, the vector length is very large (see Table 1). In Bert_base, it is 768, and in Transformer_base, it is 512, which is an order of magnitude longer than the vector length in the single-head attention mechanism. Therefore, optimizing the second and third stages of the systolic array data flow does not bring much gain. Therefore, the data flow direction of stage 3 is kept perpendicular to that of stage 1 to ensure that the data output by the systolic array is a continuous row vector, which facilitates subsequent data processing.

[0048] To further verify the effectiveness of the present invention, experimental verification was conducted, for example, using Synopsys's VCS for simulation and XILINX's VIVADO for simulation, synthesis, and layout and routing. Other simulation platforms can also be used to demonstrate the feasibility of the present invention.

[0049] 1) Behavioral simulation results of matrix multiplication operator

[0050] Taking the matrix multiplication of a 128x64 matrix and a 128x64 matrix as an example, analysis of the proposed systolic array simulation waveform shows that the number of clock cycles required is: (16815-255) / 10 = 1656 clks. In contrast, analysis of the classic systolic array simulation waveform shows that the number of clock cycles required is: (22095-255) / 10 = 2184 clks.

[0051] Comparing the waveforms of a single systolic array calculation using the present invention with those of a classic systolic array reveals that the present invention significantly reduces computational latency. This comparison also demonstrates that combining the second and third stages of the classic systolic array into a single Flash-Shift stage can achieve significant acceleration when vector lengths are short and the first stage is not significant. Calculations show that the present invention saves (2184 - 1656) / 2184 = 24.18% of the matrix multiplication process, slightly less than the 24.41% savings achieved by the single systolic array calculation process analyzed above, as each systolic array calculation involves some control flow operations that also take a small amount of time.

[0052] 2) Behavioral simulation results of linear operators

[0053] Taking the results of a single round of linear layer operations on 128x768 matrices and 768x768 matrices as an example, analysis shows that the time consumption is mainly in the first stage of the systolic array, while the second and third stages account for very small proportions, resulting in low optimization benefits. In this case, the present invention adopts a solution in which stage three and stage one are perpendicular to each other to ensure that the row vectors of the data output by the systolic array are continuous, facilitating subsequent processing.

[0054] 3) Hardware resource consumption

[0055] By using Vivado to perform synthesis and layout and routing, the hardware resource occupancy of the present invention is obtained, see Table 2.

[0056] Table 2: Hardware resource usage

[0057]

[0058] As can be seen from Table 2, the hardware resource consumption is not large. The experiment selects DSP as the object of synthesis of each processing unit, so the 32x32 systolic array occupies a total of 1024 DSPs.

[0059] In summary, the present invention designs a hardware architecture of a bidirectional systolic array that supports the attention mechanism and a data flow model of this hardware architecture, which can flexibly handle the short vectors of matrix multiplication in the attention mechanism and the long vectors in the generalized matrix multiplication represented by the linear layer. This case-by-case optimization method reduces the overall computational latency.

[0060] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0061] Computer-readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device.Computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof.More specific examples (non-exhaustive list) of computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof.Computer-readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0062] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0063] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0064] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0065] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0066] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0067] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0068] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A bidirectional systolic array device for an attention mechanism, for use in attention mechanism operations in a neural network, comprising: Weight buffer: used to cache weight data; Stimulus buffer: used to cache stimulus data; A systolic array comprising N*M two-dimensional processing units connected in a systolic manner, wherein the excitation data is controlled to be fed into and flowed into the systolic array in a first direction, and the weight data is controlled to be fed into and flowed into the systolic array in a second direction; a SoftMax processing unit, configured to receive the output of the systolic array in the first direction and perform SoftMax function processing according to a set activation function; Result buffer: used to receive the calculation results of the systolic array or the SoftMax processing unit.

2. The device according to claim 1, characterized in that The systolic array is controlled to perform calculations according to the following timing: Phase 1: performing an input to the systolic array and performing a scalar multiplication on the input data, wherein the input direction of the stimulus data is the same as the output direction of the systolic array; Phase 2: When no new data enters, the last number that entered the systolic array unit is passed on until the calculation is completed; Phase 3: When all the input numbers are calculated, the data in the systolic array unit starts to be transferred out and written into the result buffer.

3. The device according to claim 2, characterized in that When performing multiplication operations on the query matrix Q and the key matrix K in the attention mechanism of the neural network, in the second and third stages, the data flow direction of the systolic array is the same as that in the first stage.

4. The device according to claim 2, characterized in that When performing calculations on a linear layer of a multi-head attention mechanism in the neural network, in the third stage, the data flow direction of the systolic array is controlled to be perpendicular to that of the first stage so that the output of the systolic array is row vector continuous.

5. The device according to claim 1, characterized in that N and M are the same or different integers.

6. The device according to claim 1, characterized in that The neural network is a Transformer network.

7. The device according to claim 1, characterized in that The systolic array comprises 32*32 two-dimensional processing units connected in a systolic manner.

Images