A dataflow design method compatible with multi-head self-attention and convolution computation

Abstract

This invention discloses a dataflow design method for Transformer models that is compatible with multi-head self-attention and convolution computation. First, this invention proposes an equivalent computational form for the multi-head self-attention module in the Transformer model based on the ternary matrix multiplication operator. Then, based on the data transmission format of a systolic array, this invention designs a dataflow for the ternary matrix multiplication operator through operator fusion. This dataflow includes vector × vector patterns and element × vector patterns, keeping intermediate computation results within the computation unit and completely avoiding data migration of intermediate results to off-chip storage and on-chip global cache. Finally, based on this dataflow, this invention proposes a computational strategy compatible with multi-head self-attention and convolution, along with corresponding mapping schemes, fully utilizing various data reuse opportunities to achieve high-energy-efficiency computation of multi-head self-attention and convolution operators.

Description

Technical Field

[0001] This invention belongs to the field of hardware acceleration technology for deep neural networks, and particularly relates to a data flow design method for Transformer models that is compatible with multi-head self-attention and convolution computation. Background Technology

[0002] The visual Transformer model is derived from the Transformer model in the field of natural language processing. Initially, the visual Transformer used an encoding module based on multi-head self-attention (MHSA) as its core structure. However, in the process of model evolution, convolutional modules have been proven to effectively improve the performance of the visual Transformer model. Therefore, the hybrid structure of multi-head self-attention and convolution has become a new design paradigm for the visual Transformer model.

[0003] Currently, the visual Transformer model has demonstrated performance comparable to traditional convolutional neural networks in various computer vision tasks. However, in practical applications, it not only incurs massive computational demands and high memory access overhead, but also involves diverse operators and complex topologies. Therefore, to meet the low latency and low power consumption requirements of edge computing scenarios, neural network accelerators are generally designed based on Application Specific Integrated Circuits (ASICs). The core of an ASIC accelerator is an array of Processing Elements (PEs) controlled by data flow. The role of data flow is to control the efficient flow and computation of data between PEs according to preset rules, maximizing data reuse opportunities to reduce memory access overhead and achieve large-scale parallel computing, thus making it easier to meet the deployment performance requirements of edge computing scenarios.

[0004] However, current research on data flow for visual Transformer models is insufficient, mainly for two reasons: (1) such as Figure 1As shown, the MHSA operator contains multiple linear layer computations and matrix multiplications, and a Softmax operator is inserted between the matrix multiplication operators. This diverse operator composition and complex topology make the MHSA operator have multiple data reuse opportunities and a huge operator fusion space. Therefore, the data flow design for the MHSA operator is quite difficult, and the existing data flow design has not yet fully explored its high energy efficiency potential. (2) Convolution and MHSA both play important roles in the visual Transformer model. Therefore, the visual Transformer accelerator needs to take into account the energy efficiency of convolution computation while processing MHSA with high energy efficiency. Otherwise, it will lead to a significant decline in the overall energy efficiency of the accelerator. However, since convolution and MHSA have very different data reuse opportunities, this poses a considerable challenge to designing a data flow that can take into account the energy efficiency of both. Therefore, the existing visual Transformer accelerator has not yet been able to take into account the high energy efficiency of convolution computation. Summary of the Invention

[0005] The purpose of this invention is to provide a dataflow design method that is compatible with multi-head self-attention and convolution computation, thereby enabling energy-efficient deployment of hybrid visual Transformer models based on multi-head self-attention and convolution on edge devices.

[0006] To achieve the above-mentioned objectives, the technical solution of the present invention is as follows:

[0007] A dataflow design method compatible with multi-head self-attention and convolution computation includes the following steps:

[0008] S100: Through matrix transformation and matrix operation laws, the original computational form of the multi-head self-attention operator is mathematically derived, resulting in an equivalent computational form based on the ternary matrix multiplication operator and the Softmax operator, comprising a total of 5 computational steps. The ternary matrix multiplication operator involves the consecutive multiplication of three matrices, with the general formula T = A × B × C. T This ternary matrix multiplication operator can cover all linear computations of multi-head self-attention;

[0009] Five computational steps for the equivalent computational form of the multi-head self-attention operator:

[0010] Step 1:

[0011] Step 2:

[0012] Step 3: (S out-1 ,S out-2 ,...,S out-h ) = (Softmax(S in-1 ),Softmax(S in-2),...,Softmax(S in-h ))

[0013] Step 4: V = X f W fV

[0014] Step 5: O = Concat(S) out-1 V1,S out-2 V2,...,S out-h V h W fO

[0015] Where X f W is a fusion vector of the input features and a column vector of all "1"s. fQ / fK / fV It is the fusion vector of Q / K / V weight vector and bias vector in the multi-head self-attention operator, d is the number of channels in each attention head; Sin and sout are intermediate results in the calculation process, h is the number of attention heads, U is the normalized key vector, V is the generally accepted value vector, and O is the generally accepted multi-head self-attention output feature.

[0016] S200: Proposes an operator fusion computation strategy for ternary matrix multiplication operators, and based on the fusion computation strategy, completes the computation of ternary matrix multiplication operators in two data flow modes: vector × vector and element × vector, through the data transmission form of pulsating array.

[0017] S300: Proposes a computational strategy that can be compatible with matrix multiplication and convolution operators through the data flow of ternary matrix multiplication operators, and realizes the mapping of multi-head self-attention operators and convolution operators to accelerators.

[0018] Preferably, the matrix transformation described in S100 includes the fusion of the bias vector and the weight vector, as well as the fusion of the column vector with all elements equal to 1 and the input feature vector.

[0019] Preferably, the equivalent computational forms of the multi-head self-attention operator described in S100 include the standard form of the Softmax operator, the standard form of the ternary matrix multiplication operator, and the variant form of the ternary matrix multiplication operator. The variant form of the ternary matrix multiplication operator includes ordinary matrix multiplication and ternary matrix multiplication after introducing channel splicing.

[0020] Preferably, one operator fusion strategy of the ternary matrix multiplication operator described in S200 is that after an element in the intermediate result is calculated, that element is immediately used to multiply the matrix C. T The operation is performed on a row of elements in the output T to generate a partial row of the output result T.

[0021] Preferably, the vector × vector data stream mode described in S200 refers to a pulsating array where, in each computation cycle, the computation unit can read the input data and complete one multiplication-accumulation operation. Then, it immediately passes the input data received in this cycle to the computation units to its right and below, respectively, so that the input data can be used by the adjacent computation units in the next cycle. The partial sum calculated in this cycle will remain in the computation unit and be continuously accumulated and updated. When this partial sum is accumulated into a complete result, the computation unit will store the element in its local cache for the computation unit to read and use in the element × vector mode.

[0022] Preferably, the element × vector data stream mode described in S200 refers to the pulsating transmission of the input matrix C between column computation units, while the pulsating transmission between row computation units is a partial sum of the output matrix T. Before starting the pulsating transmission of matrix C, each computation unit reads an element from the intermediate result M in its local cache, performs a multiplication operation after receiving the data in matrix C, and accumulates it with the partial sum transmitted from the computation unit to its left. Then, it transmits the newly received matrix C to the computation unit below it, while transmitting the calculated partial sum to the computation unit to its right. Finally, the rightmost column computation unit pulsatingly transmits a series of partial sums of T, which are then fed into a FIFO (First-in First-out) and continuously accumulated within the FIFO.

[0023] Preferably, a computational strategy and corresponding mapping scheme for multi-head self-attention operators and convolution operators that are compatible with ternary matrix multiplication operator data streams include:

[0024] S310: For steps 1 and 4 of the multi-head self-attention operator in the equivalent computation form, based on the storage characteristics of intermediate results in the local cache of the computation unit under the vector × vector data stream mode, the input data is pre-stored, and then the computation is completed through the element × vector mode. For step 5, the computation unit array executes iteratively in the vector × vector mode until the computation within all attention heads is completed. During this computation process, the computation unit stores the computation results within each attention head in the local cache of the computation unit according to the channel concatenation order. Then, the computation unit calculates the final output result of step 5 in the element × vector mode.

[0025] S320: For convolution, based on the storage format of each matrix vector in memory during the calculation of the multi-head self-attention operator, and according to the element × vector data flow pattern, the corresponding row vectors are extracted from the convolution kernel and input features according to the sliding rule of the convolution window and formed into a two-dimensional matrix, and then the calculation is completed according to the element × vector pattern.

[0026] The present invention also provides a Transformer accelerator, whose core computing module (mainly matrix multiplication calculation) applies the above-mentioned dataflow design method.

[0027] The dataflow design of this invention, which is compatible with multi-head self-attention and convolution computation, has the following advantages:

[0028] (1) This invention proposes an equivalent computational form for the multi-head self-attention operator, such as Figure 1 and Figure 4 The original computational form reduces the number of basic operators (9 in total, 5 types) to 5 in total, including only the ternary matrix multiplication operator and the Softmax operator. Furthermore, the multi-head self-attention operator in the equivalent computational form has a simpler topology, reducing the original 6 levels of intermediate results to 4 levels. Therefore, when designing dataflows based on operator fusion, the design space is significantly reduced. Moreover, except for the Softmax operator, the types of operators that can participate in fusion are identical, meaning they possess the same computational logic, which greatly reduces the complexity of dataflow design. Thus, the simple topology of this equivalent computational form reduces the difficulty of finding high-efficiency dataflows globally.

[0029] (2) This invention proposes a data flow design for the ternary matrix multiplication operator by combining a systolic array data transmission method with the operator fusion concept. This design includes vector × vector and element × vector modes, ensuring that intermediate results generated during the ternary matrix multiplication operation are always kept in the local cache of the PE, completely avoiding data migration to off-chip storage and on-chip global cache, thus significantly reducing memory access power consumption. Simultaneously, this invention ensures high throughput of the PE array through systolic transmission and fully utilizes the reuse opportunities of matrix row and column data in matrix multiplication operations.

[0030] (3) Based on the data flow of the ternary matrix multiplication operator, this invention proposes a computational strategy and mapping scheme for multi-head self-attention operators and convolution operators, which enables the unification of the storage format of vectors in the two operators in memory, avoids inefficient memory operations, and also enables the implementation of multi-head self-attention operators and convolution operators through the data flow of the ternary matrix multiplication operator, and makes full use of various data reuse opportunities in multi-head self-attention operators and convolution operators. Attached Figure Description

[0031] Figure 1 This represents the topological structure of the multi-head self-attention operator in its original computational form.

[0032] Figure 2 This refers to the process of fusing the bias vector and the weight matrix;

[0033] Figure 3This refers to the transformation process of the input matrix;

[0034] Figure 4 The topology of the multi-head self-attention operator in the equivalent computational form;

[0035] Figure 5 A fusion computation strategy for ternary matrix multiplication operators;

[0036] Figure 6 These are two modes of data flow for the ternary matrix multiplication operator;

[0037] Figure 7 This is a computational strategy for convolution that is compatible with the data stream of the ternary matrix multiplication operator. Detailed Implementation

[0038] The present invention will be further described below with reference to the embodiments.

[0039] This invention can be applied to the design of dedicated accelerators for deep neural networks, such as Transformer inference accelerator chips.

[0040] In the first aspect, the present invention proposes a hardware-friendly equivalent computational form of a multi-head self-attention operator, the main steps of which include matrix-vector transformation and derivation of the equivalent form.

[0041] Matrix-vector transformation: Transforming the matrices and vectors involved in the multi-head self-attention operator. The transformation process involves transforming various bias vectors B... Q B K B V and B O Each is integrated into the weight matrix W Q W K W V and W O , obtain W fQ W fK W fV and W fO , with B Q and W Q For example, the integration of, such as Figure 2 As shown, B Q It is a column vector of length C, with elements of length C. W Q It is a matrix of size [C, C], with elements inside it being... First, B Q Transpose it into a row vector, then add it to W along the row dimension. Q W was obtained after the tail. fQ That is, W fQ The elements in rows 1 to C are the original matrix W. Q The element in row (C+1) is the original vector B.Q Correspondingly, the input feature X is also transformed according to the requirements of the matrix multiplication rules, such as... Figure 3 As shown, X is a matrix of size [HW, C], with elements x inside. i,j Then, a unit column vector of length HW is constructed, and this vector is added to the end of X according to its column dimensions to obtain X. f , that is, X f The elements in columns 1 to C are the original matrix X, and the elements in column (C+1) are all 1.

[0042] Equivalent Form Derivation: Based on the transformed matrix vectors, and by applying the distributive and associative laws of matrix multiplication, the equivalent computational form of the multi-head self-attention operator based on the ternary matrix multiplication operator is derived, as shown in the following equation. The ternary matrix multiplication operator involves the consecutive multiplication of three matrices, and its general formula is T = A × B × C. T .

[0043]

[0044] The operator topology corresponding to the above equivalent computational form is as follows: Figure 4 In this form, S in-i It can be directly calculated using ternary matrix multiplication. Although O involves concatenation, it is essentially a ternary matrix multiplication with channel concatenation. Therefore, in hardware execution, it can be performed according to the ternary matrix multiplication calculation mode through a reasonable data scheduling strategy. As for generating the V vector and U... i The corresponding ordinary matrix multiplications are all sub-operators of ternary matrix multiplication, so when executed on hardware, they can necessarily adapt to the calculation mode of ternary matrix multiplication operators.

[0045] Secondly, combining the idea of ​​operator fusion, this invention proposes a fusion computation strategy for ternary matrix multiplication operators, such as... Figure 5 Once an element in the intermediate result M has been calculated, that element will be immediately used in conjunction with matrix C. T The algorithm performs operations on a row of elements in the array to generate a partial sum of a row in the output result T. This fusion computing strategy involves two computing modes: vector × vector and element × vector. Therefore, this invention proposes dataflow designs for each mode based on the data transmission format of the pulsating array.

[0046] Vector × Vector Calculation Mode: In this mode, the ternary matrix multiplication operation M = A × B is performed, such as... Figure 6As shown, using a pulsating array, in each computation cycle, the PE can read input matrices A and B, complete one multiplication-accumulation operation, and then immediately pass the input data received in this cycle to the PE to its right and the PE below it, so that the input data can be used by the adjacent PE in the next cycle. The partial sum psum-M calculated in this cycle will remain in the PE and be continuously accumulated and updated. When this partial sum is accumulated into a complete result, the PE will store the element in its local cache for the PE to read and use in element × vector mode.

[0047] Element × Vector Calculation Mode: This mode performs the ternary matrix multiplication operation T = M × C. T ,like Figure 6 As shown. At this point, the input matrix C is pulsated between column PEs, while the partial sum psum-T of the output matrix T is pulsated between row PEs. Before starting the pulsation transmission of matrix C, each PE reads an element from its local buffer of the intermediate result M, then performs a multiplication operation upon receiving the data from matrix C, and accumulates it with the partial sum from the PE to its left. It then transmits the newly received matrix C to the PE below it, and transmits the calculated partial sum to the PE to its right, allowing the right PE to complete the multiplication and accumulation operation in the next cycle. Finally, the rightmost column PE pulsates a series of partial sums of T, which are then fed into a FIFO (First-in First-out) and continuously accumulated. The main function of the FIFO is to temporarily store a series of partial sums generated by each row PE. In a new round of pulsation calculation, when the rightmost column PE outputs a new batch of partial sums, the adder extracts the corresponding old partial sums from the FIFO one by one and accumulates them with the newly arrived partial sums. After accumulation, the contents stored in the FIFO are updated promptly. After multiple rounds of accumulation and updates, the partial sums in the FIFO are accumulated into a complete output element T, which is then transmitted by the FIFO to the on-chip global buffer.

[0048] Thirdly, this invention proposes a computational strategy for multi-head self-attention and convolution, as well as a corresponding mapping scheme, based on the data flow designed for the ternary matrix multiplication operator.

[0049] Multi-head self-attention: In the equivalent computational form, the computation process of the multi-head self-attention operator can be clearly broken down into the following 5 basic operators, namely the basic computational steps.

[0050] Step 1:

[0051] Step 2:

[0052] Step 3: (Sout-1 ,S out-2 ,...,S out-h ) = (Softmax(S in-1 ),Softmax(S in-2 ),...,Softmax(S in-h ))

[0053] Step 4: V = X f W fV

[0054] Step 5: O = Concat(S) out-1 V1,S out-2 V2,...,S out-h V h W fO

[0055] In the above calculation steps, step 2 is the standard calculation form of the ternary matrix multiplication operator, which can be calculated using the data stream of the ternary matrix multiplication operator. Step 3 is the Softmax operator, which can be calculated using the dedicated Softmax calculation unit.

[0056] For steps 1 and 4, the present invention first stores a block of input data in the local cache of each PE according to the storage characteristics of various matrix vectors in the PE local cache under this mode. Then, it calculates a block of output results in steps 1 and 4 through the element × vector pattern in the ternary matrix multiplication operator data stream. Then, it pre-stores another block of input data in the local cache of the PE and calculates another block of output results in steps 1 and 4 again according to the element × vector pattern. This process is repeated until all output results of steps 1 and 4 are obtained.

[0057] For step 5, the PE array is executed iteratively according to the vector × vector pattern in the ternary matrix multiplication operator data stream until all attention heads have completed the S... out-i V i The calculation process involves the PE storing the results from each attention head in a local cache according to the channel concatenation order. Then, the PE calculates the final output of step 5 using an element-multiply vector pattern.

[0058] Convolution: This invention proposes a computational strategy for convolution that unifies the memory storage format of tensors in convolution and multi-head self-attention. For a convolution kernel of size (R, S, C), this invention uses (r, s) to represent the spatial coordinates of a vector of length C within the kernel. Figure 7M vectors with the same spatial coordinates are selected from M convolution kernels and formed into a matrix of size C×M, denoted as W(i), where i = (r-1)×R+s. Then, according to the sliding rule of the convolution window, EF rows of data are extracted from the input features, which is a matrix X(i) of size EF×C. The row index row_idx of the extracted input data is calculated by the following formula.

[0059]

[0060] Then, matrix multiplication X(i)×W(i) is calculated according to the element × vector pattern, thus obtaining the partial sum matrix O(i) of the convolution result O. This process is repeated RS times, and finally, the RS partial sum matrices are accumulated element by element to obtain the complete convolution output O.

[0061] The present invention also provides a Transformer accelerator, whose core computing module (mainly matrix multiplication calculation) applies the above-mentioned dataflow design method.

Claims

1. A dataflow design method compatible with multi-head self-attention and convolution computation, characterized in that, Includes the following steps: S100: Transforms the matrix vectors involved in the multi-head self-attention operator, including the fusion of the bias vector and the weight vector, and the fusion of the column vector with all elements of 1 with the input feature vector. S200: Based on the transformed matrix vectors, the equivalent computational forms of multi-head self-attention operators are obtained, including ternary matrix multiplication operators, binary matrix multiplication operators, and the Softmax operator. Among them, the ternary matrix multiplication operator is the continuous multiplication of three matrices, and the computational form is T = A × B × C. T ; S300: Performs calculations of ternary matrix multiplication operators and binary matrix multiplication operators using two data flow modes: vector × vector and element × vector, respectively. The vector × vector data stream mode refers to a process where, through a pulsating array, each computational unit can read input data A and B and perform a multiplication-accumulation operation within each computation cycle. Immediately afterward, the input data received in this cycle is passed to the computational units to the right and below, respectively, so that the input data can be used by adjacent computational units in the next cycle. The partial sum M = A × B calculated in this cycle will remain in the computational unit and be continuously accumulated and updated. When this partial sum is accumulated into a complete result, the computational unit will store the element in its local cache for the computational unit to read and use in the element × vector mode. The element-vector data stream mode refers to the pulsating transmission of the input matrix C between column computation units, while the pulsating transmission between row computation units is a partial sum of the output matrix T. Before starting the pulsating transmission of matrix C, each computation unit reads an element from the intermediate result M in its local buffer, performs a multiplication operation after receiving the data in matrix C, and accumulates it with the partial sum from the computation unit to its left. Then, it transmits the newly received matrix C to the computation unit below it, while transmitting the calculated partial sum to the computation unit to its right. Finally, the rightmost column computation unit pulsatingly transmits a series of partial sums of T, which are then fed into a FIFO and continuously accumulated within the FIFO. S400: The method of convolution is as follows: based on the storage format of each matrix vector in memory during the calculation of the multi-head self-attention operator, and according to the element × vector data flow pattern, the corresponding row vectors are extracted from the convolution kernel and input features according to the rules of the convolution window sliding and formed into a two-dimensional matrix. Then, the calculation of the convolution operator is completed according to the element × vector pattern. Five computational steps for the equivalent computational form of the multi-head self-attention operator: Step 1: Step 2: Step 3: (S out-1 ,S out-2 ,...,S out-h ) = (Softmax(S in-1 ),Softmax(S in-2 ),...,Softmax(S in-h )); Step 4: V = X f W fV ; Step 5: O = Concat(S) out-1 V1,S out-2 V2,...,S out-h V h W fO ; Where X f W is a fusion vector of the input features and the column vector of all "1"s. fQ W fK W fV These are the fusion vectors of the Q, K, and V weight vectors and bias vectors in the multi-head self-attention operator, respectively, where d is the number of channels in each attention head; S in S out This is an intermediate result in the computation process, where h is the number of attention heads, U is the normalized key vector, V is the value vector, and O is the multi-head self-attention output feature.

2. The dataflow design method compatible with multi-head self-attention and convolution computation according to claim 1, characterized in that, In the computation of the ternary matrix multiplication operator, once an element in the intermediate result has been calculated, that element is immediately used to multiply the matrix C. T The operation is performed on a row of elements in the output T to generate a partial row of the output result T.

3. The dataflow design method compatible with multi-head self-attention and convolution computation according to claim 1, characterized in that, For steps 1 and 4 of the multi-head self-attention operator in the equivalent computation form, the input data is pre-stored based on the storage characteristics of intermediate results in the local cache of the computation unit in the vector × vector data stream mode, and then the computation is completed through the element × vector mode.

4. The dataflow design method compatible with multi-head self-attention and convolution computation according to claim 1, characterized in that, For step 5, the computing unit array executes in a vector × vector pattern in a loop until the computation within all attention heads is completed; during this computation process, the computing unit stores the computation results within each attention head in the order of channel concatenation into the local cache of the computing unit. Then, the computing unit calculates the final output result of step 5 according to the element × vector pattern.

5. A dataflow design method compatible with multi-head self-attention and convolution computation according to claim 1, characterized in that, S400 specifically includes: For a convolutional kernel of size (R,S,C), (r,s) represents the spatial coordinates of a vector of length C within the kernel; M vectors with the same spatial coordinates are selected from M convolutional kernels and formed into a matrix of size C×M, denoted as W(i), where i = (r-1)×R+s; and rows EF of data are extracted from the input features according to the sliding rules of the convolutional window, i.e., a matrix X(i) of size EF×C. The row index row_idx of the extracted input data is calculated by the following formula: Then, matrix multiplication X(i)×W(i) is calculated according to the element × vector pattern, thus obtaining the partial sum matrix O(i) of the convolution result O. This process is repeated RS times, and finally, the RS partial sum matrices are accumulated element by element to obtain the complete convolution output O.

6. A Transformer accelerator, characterized in that, A dataflow design method compatible with multi-head self-attention and convolution computation, as described in any one of claims 1-5, is adopted.

Images