Data stream design method compatible with multi-head self-attention and convolution calculation

By deducing the equivalent calculation form of the multi-head self-attention operator and the fusion strategy of the ternary matrix multiplication operator, combined with the pulsating array data transmission, a data flow compatible with multi-head self-attention and convolution is designed, the visual Transformer model accelerator is solved. The high energy efficiency of multi-head self-attention and convolutional computing achieves efficient deployment of edge-end devices.

CN119003955AActive Publication Date: 2024-11-22SOUTHEAST UNIV

Patent Information

Application Number
CN202411017062.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-29
Publication Date
2024-11-22
Estimated Expiration
2044-07-29

AI Technical Summary

Technical Problem

The existing visual Transformer model accelerator is difficult to take into account the high energy efficiency of multi-head self-attention and convolutional computing, resulting in delay and power consumption problems in edge-end scenarios. The data flow design is high, and the high energy-efficiency potential of the operator is not fully explored. .

Method used

Through matrix deformation and matrix operation laws, the equivalent calculation form of the multi-head self-attention operator is derived, the ternary matrix multiplication operator is adopted and the operator fusion strategy is proposed, and the data transmission form of the pulsating array is combined to design vector × vector and elements. The two data flow modes of × vector realize the calculation of the ternary matrix multiplication operator, and propose a calculation strategy that is compatible with matrix multiplication and convolution operators, unifying the data flow design of the multi-head self-attention and convolution operators.

Benefits of technology

Simplifies the topological structure of the multi-head self-attention operator, reduces the energy consumption of storage and memory access of intermediate results, improves the efficiency of data flow design, realizes high-energy-efficient processing of multi-head self-attention and convolutional calculations, and reduces the Design complexity improves the performance of edge devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119003955A_ABST
    Figure CN119003955A_ABST
Patent Text Reader

Abstract

The invention discloses a data flow design method oriented to a Transform model and compatible with multi-head self-attention and convolution calculation. The method comprises the following steps: firstly, providing an equivalent calculation form based on a ternary matrix multiplication operator for a multi-head self-attention module in a Transform model; then, based on a data transmission form of a systolic array, a data stream of a ternary matrix multiplication operator is designed through operator fusion, the data stream comprises a vector * vector mode and an element * vector mode, and an intermediate calculation result is kept in a calculation unit; and data migration of an intermediate result to an off-chip memory and an on-chip global cache is completely avoided. And finally, based on the data stream, the invention provides a calculation strategy compatible with multi-head self-attention and convolution and a corresponding mapping scheme, various data multiplexing opportunities are fully utilized, and high-energy-efficiency calculation of a multi-head self-attention operator and a convolution operator is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of hardware acceleration of deep neural networks, and in particular to a data flow design method for a Transformer model that is compatible with multi-head self-attention and convolution calculations. Background Art

[0002] The visual Transformer model is migrated from the Transformer model in the field of natural language processing. Initially, the visual Transformer used an encoding module based on the Multi-Head Self-Attention (MHSA) mechanism as its core structure, but in the evolution of the model, the convolution module has been proven to effectively improve the performance of the visual Transformer model. Therefore, the hybrid structure of multi-head self-attention and convolution has become a new design paradigm for the visual Transformer model.

[0003] At present, the visual Transformer model has demonstrated performance comparable to that of traditional convolutional neural networks in various tasks in the field of computer vision. However, in practical applications, it not only has a huge amount of computation and high memory access overhead, but also has a multi-operator composition and complex topological structure. Therefore, in order to meet the requirements of low latency and low power consumption in edge scenarios, neural network accelerators are generally designed based on application-specific integrated circuits (ASICs). The core of the ASIC accelerator is a processing element (PE) array controlled by data flow, where the role of the data flow is to control the effective flow and calculation of data between PEs according to preset rules, and to make full use of data reuse opportunities as much as possible to reduce data memory access overhead and realize large-scale parallel computing, so as to make it easier to meet the deployment performance requirements of edge scenarios.

[0004] However, the current research on the data flow of the visual Transformer model is not sufficient. There are two main reasons: (1) Figure 1As shown in the figure, the MHSA operator contains multiple linear layer calculations and matrix multiplications, and a Softmax operator is inserted between the matrix multiplication operators. This multi-element operator composition and complex topological structure give the MHSA operator multiple data reuse opportunities and a huge operator fusion space. Therefore, the data flow design for the MHSA operator is difficult, and the existing data flow design has not yet fully tapped its high energy efficiency potential. (2) Both convolution and MHSA play an important role in the visual Transformer model, so the visual Transformer accelerator needs to take into account the energy efficiency of convolution calculations while processing MHSA with high energy efficiency, otherwise it will lead to a significant decline in the overall energy efficiency of the accelerator. However, since convolution and MHSA have very different data reuse opportunities, it poses a considerable challenge to design a data flow that can take into account the energy efficiency of both. Therefore, the existing visual Transformer accelerator has not yet been able to take into account the high energy efficiency of convolution calculations. Summary of the invention

[0005] The purpose of the present invention is to provide a data flow design method that is compatible with multi-head self-attention and convolution calculations, so as to achieve energy-efficient deployment of a hybrid visual Transformer model based on multi-head self-attention and convolution on edge devices.

[0006] In order to achieve the above-mentioned object of the invention, the technical solution of the present invention is as follows:

[0007] A data flow design method compatible with multi-head self-attention and convolution calculations includes the following steps:

[0008] S100: Through matrix transformation and matrix operation law, the original calculation form of the multi-head self-attention operator is mathematically derived, and an equivalent calculation form of a total of 5 calculation steps based on the ternary matrix multiplication operator and the Softmax operator is obtained. The ternary matrix multiplication operator is the continuous multiplication of three matrices, and the general formula is T = A × B × C T , this ternary matrix multiplication operator can cover all linear computations of multi-head self-attention;

[0009] The five calculation steps of the equivalent calculation form of the multi-head self-attention operator are:

[0010] Step 1:

[0011] Step 2:

[0012] Step 3: out-1 ,S out-2 ,...,S out-h )=(Softmax(S in-1 ),Softmax(S in-2),...,Softmax(S in-h ))

[0013] Step 4: V = X f W fV

[0014] Step 5: O = Concat(S out-1 V1,S out-2 V2,...,S out-h V h )W fO

[0015] Where X f is the fusion vector of the input feature and the full “1” column vector, W fQ / fK / fV It is the fusion vector of the Q / K / V weight vector and the bias vector in the multi-head self-attention operator, d is the number of channels in each attention head; Sin and sout are the intermediate results in the calculation process, h is the number of attention heads, U is the normalized key vector, V is the recognized value vector, and O is the recognized multi-head self-attention output feature.

[0016] S200: an operator fusion calculation strategy for a ternary matrix multiplication operator is proposed, and according to the fusion calculation strategy, the calculation of the ternary matrix multiplication operator is completed in two data flow modes, vector×vector and element×vector, through the data transmission form of the systolic array;

[0017] S300: Propose a computing strategy that can make matrix multiplication operators and convolution operators compatible through the data flow of ternary matrix multiplication operators, and realize the mapping of multi-head self-attention operators and convolution operators to accelerators.

[0018] Preferably, the matrix deformation described in S100 includes the fusion of the bias vector and the weight vector, and the fusion of the column vector whose elements are all 1 and the input feature vector.

[0019] Preferably, the equivalent calculation form of the multi-head self-attention operator described in S100 includes the standard form of the Softmax operator, the standard form of the ternary matrix multiplication operator, and the deformed form of the ternary matrix multiplication operator, wherein the deformed form of the ternary matrix multiplication operator includes ordinary matrix multiplication and ternary matrix multiplication after introducing channel splicing.

[0020] Preferably, an operator fusion strategy of the ternary matrix multiplication operator in S200 is that when an element of the intermediate result is calculated, the element is immediately used to combine with the matrix C T The operation is performed on a row of elements in , thereby generating a row of partial sums in the output result T.

[0021] Preferably, the vector × vector data flow mode described in S200 means that, through the form of a pulsating array, in each computing cycle, the computing unit can read the input data and complete a multiplication and accumulation operation, and then immediately pass the input data received in this cycle to the computing units on the right and below respectively, so that the input data can be used by the adjacent computing units in the next cycle, and the partial sum calculated in this cycle will remain in the computing unit and be continuously accumulated and updated. When this partial sum is accumulated into a complete result, the computing unit will store the element in its local cache for the computing unit to read and use in the element × vector mode.

[0022] Preferably, the element × vector data flow mode described in S200 means that the input matrix C is transmitted in a pulsating manner between the column computing units, while the partial sum of the output matrix T is transmitted in a pulsating manner between the row computing units. Before starting the pulsating transmission of the matrix C, each computing unit will read an element of the intermediate result M from the local cache, and then perform a multiplication operation after receiving the data in the matrix C, and accumulate it with the partial sum transmitted by the computing unit on the left, and then transmit the just received matrix C to the computing unit below, and transmit the partial sum obtained by the operation to the computing unit on the right. Finally, the computing unit in the rightmost column will pulsate a series of partial sums of T, which will be sent to the FIFO (First-in First-out) and continuously accumulated in the FIFO.

[0023] Preferably, a computing strategy of the multi-head self-attention operator and the convolution operator that is compatible with the ternary matrix multiplication operator data flow and a corresponding mapping scheme include:

[0024] S310: For steps 1 and 4 of the multi-head self-attention operator in the equivalent calculation form, the input data is pre-stored according to the storage characteristics of the intermediate results in the local cache of the computing unit in the vector × vector data flow mode, and then the calculation is completed through the element × vector mode. For step 5, the computing unit array is executed in a loop according to the vector × vector mode until the calculations in all attention heads are completed. During this calculation process, the computing unit stores the calculation results in each attention head in the local cache of the computing unit in the order of channel splicing. Then, the computing unit calculates the final output result of step 5 in the element × vector mode.

[0025] S320: For convolution, according to the storage format of each matrix vector in the memory during the calculation process of the multi-head self-attention operator, and according to the element × vector data flow mode, the corresponding row vectors are extracted from the convolution kernel and input features according to the convolution window sliding rule and form a two-dimensional matrix, and then the calculation is completed according to the element × vector mode.

[0026] The present invention also provides a Transformer accelerator, whose core computing module (mainly matrix multiplication calculation) applies the above-mentioned data flow design method.

[0027] The data flow design of the present invention that is compatible with multi-head self-attention and convolution calculation has the following advantages:

[0028] (1) The present invention proposes an equivalent calculation form for the multi-head self-attention operator, such as Figure 1 and Figure 4 , reducing the 9 basic operators in the original calculation form from 5 to 2, including only the ternary matrix multiplication operator and the Softmax operator. At the same time, the multi-head self-attention operator has a simpler topological structure in the equivalent calculation form, and the 6-level intermediate results in the original topological structure are reduced to 4 levels. Therefore, when designing data flows based on operator fusion, the design space of the data flow is significantly reduced, and except for the Softmax operator, the other types of operators that can participate in the fusion are the same, that is, they have the same calculation logic, which can greatly reduce the complexity of data flow design. Therefore, the simple topological structure of this equivalent calculation form reduces the difficulty of finding energy-efficient data flows globally.

[0029] (2) The present invention proposes a data flow design for the ternary matrix multiplication operator through the data transmission form of the systolic array and combined with the operator fusion concept, including vector × vector mode and element × vector mode, so that the intermediate results generated in the calculation process of the ternary matrix multiplication operator are always kept in the local cache of the PE, completely avoiding the data movement of the intermediate results to the off-chip storage and the on-chip global cache, and reducing a large amount of memory access energy consumption. At the same time, the present invention ensures the high throughput of the PE array through systolic transmission, and fully utilizes the reuse opportunities of the row and column data of the matrix in the matrix multiplication operation.

[0030] (3) The present invention proposes a computing strategy and mapping scheme for the multi-head self-attention operator and the convolution operator based on the data flow of the ternary matrix multiplication operator, so that the storage format of the vectors in the two operators in the memory can be unified, avoiding inefficient memory operations. It also makes it possible to realize the multi-head self-attention operator and the convolution operator through the data flow of the ternary matrix multiplication operator, and fully utilize the various data reuse opportunities in the multi-head self-attention operator and the convolution operator. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is the topological structure of the multi-head self-attention operator in the original calculation form;

[0032] Figure 2 It is the fusion process of bias vector and weight matrix;

[0033] Figure 3is the deformation process of the input matrix;

[0034] Figure 4 It is the topological structure of multi-head self-attention operator in equivalent computational form;

[0035] Figure 5 It is the fusion calculation strategy of the ternary matrix multiplication operator;

[0036] Figure 6 There are two modes of data flow for the ternary matrix multiplication operator;

[0037] Figure 7 A computational strategy for convolution that is compatible with the dataflow of the ternary matrix multiplication operator. DETAILED DESCRIPTION

[0038] The present invention will be further described below in conjunction with the embodiments.

[0039] The present invention can be applied to the field of deep neural network dedicated accelerator design, such as Transformer reasoning accelerator chip.

[0040] In the first aspect, the present invention proposes a hardware-friendly equivalent calculation form of a multi-head self-attention operator, the main steps of which include matrix-vector transformation and equivalent form derivation.

[0041] Matrix vector deformation: The matrix vector involved in the multi-head self-attention operator is deformed. The deformation process is: various bias vectors B Q , B K , B V and B O They are fused into the weight matrix W Q , W K , W V and W O , we get W fQ , W fK , W fV and W fO , with B Q and W Q For example, the fusion of Figure 2 As shown, B Q is a column vector of length C, with the elements inside being W Q is a matrix of size [C, C], with the elements inside being First, B Q Transpose to a row vector and then add it to W along the row dimension Q After the tail of W fQ , that is, W fQ The elements from the 1st row to the Cth row are the original matrix W Q , the element in the (C+1)th row is the original vector BQ . Accordingly, the input feature X is also transformed according to the needs of the matrix multiplication rule, such as Figure 3 As shown, X is a matrix of size [HW, C], and the internal elements are x i,j , then construct a unit column vector of length HW, and then add this vector to the tail of X in the column dimension to obtain X f , that is, X f The elements from the 1st to the Cth columns of are the original matrix X, and the elements in the (C+1)th column are all 1.

[0042] Derivation of equivalent form: Based on the deformed matrix vector, by applying the distributive law and associative law of matrix multiplication, the equivalent calculation form of the multi-head self-attention operator based on the ternary matrix multiplication operator is derived, as shown in the following formula, where the ternary matrix multiplication operator is the continuous multiplication of three matrices, and its general operation formula is T = A × B × C T .

[0043]

[0044] The operator topology corresponding to the above equivalent calculation form is as follows Figure 4 , in this form, S in-i It can be directly calculated by ternary matrix multiplication. Although O has undergone splicing operations, it is essentially the ternary matrix multiplication after channel splicing. Therefore, when executed on hardware, it can be executed according to the calculation mode of ternary matrix multiplication through a reasonable data scheduling strategy. As for generating V vector and U i The corresponding ordinary matrix multiplications, are all sub-operators of the ternary matrix multiplication, so they must be able to adapt to the calculation mode of the ternary matrix multiplication operator when executed on hardware.

[0045] Secondly, combining the idea of ​​operator fusion, the present invention proposes a fusion calculation strategy for ternary matrix multiplication operators, such as Figure 5 , when an element of the intermediate result M is calculated, it will be immediately used to compare with the matrix C T A row of elements in is operated to generate a row of partial sums in the output result T. The fusion calculation strategy involves two calculation modes, namely vector×vector and element×vector. Therefore, the present invention proposes data flow designs for them respectively based on the data transmission form of the systolic array.

[0046] Vector × vector calculation mode: In this mode, the ternary matrix multiplication operation M = A × B is completed, such as Figure 6As shown. Through the form of systolic array, in each calculation cycle, PE can read the input matrices A and B and complete a multiplication and accumulation operation, and then immediately pass the input data received in this cycle to the PE on the right and the PE below, respectively, so that the input data can be used by the adjacent PE in the next cycle, and the partial sum psum-M calculated in this cycle will always reside in the PE and be continuously accumulated and updated. When this partial sum is accumulated into a complete result, PE will store the element in its local cache for the PE to read and use in element × vector mode.

[0047] Element × vector calculation mode: In this mode, the ternary matrix multiplication operation T = M × C is completed. T ,like Figure 6 As shown. At this time, the input matrix C is transmitted in pulses between the column PEs, while the partial sum psum-T of the output matrix T is transmitted in pulses between the row PEs. Before starting to transmit the matrix C in pulses, each PE will read an element of the intermediate result M from its local cache, and then perform a multiplication operation after receiving the data in the matrix C, and accumulate it with the partial sum transmitted by the left PE. Then, the matrix C just received will be transmitted to the PE below, and the partial sum obtained by the operation will be transmitted to the PE on the right, so that the PE on the right can complete the multiplication and accumulation operation in the next cycle. Finally, the rightmost column of PE will pulse a series of T partial sums, which will be sent to the FIFO (First-in First-out) and continuously accumulated in the FIFO. The main function of FIFO is to temporarily store a series of partial sums generated by each row of PEs. Then, in a new round of pulse calculation, when the rightmost column of PE outputs a new batch of partial sums, the adder will extract the corresponding old partial sums in the FIFO one by one, and accumulate them with the newly arrived partial sums. After the accumulation is completed, the storage content in the FIFO is updated in time. After multiple rounds of accumulation and updating, the partial sums in the FIFO are accumulated into a complete output element T, and then the FIFO transmits the complete output element T to the on-chip global cache.

[0048] Thirdly, the present invention proposes a computational strategy for multi-head self-attention and convolution and a corresponding mapping scheme based on the data flow designed for the ternary matrix multiplication operator.

[0049] Multi-head self-attention: In the equivalent calculation form, the calculation process of the multi-head self-attention operator can be clearly divided into the following five basic operators, that is, basic calculation steps.

[0050] Step 1:

[0051] Step 2:

[0052] Step 3:out-1 ,S out-2 ,...,S out-h )=(Softmax(S in-1 ),Softmax(S in-2 ),...,Softmax(S in-h ))

[0053] Step 4: V = X f W fV

[0054] Step 5: O = Concat(S out-1 V1,S out-2 V2,...,S out-h V h )W fO

[0055] In the above calculation steps, step 2 is the standard calculation form of the ternary matrix multiplication operator, which can be calculated by applying the data flow of the ternary matrix multiplication operator, and step 3 is the Softmax operator, which can be calculated by applying the Softmax dedicated calculation unit.

[0056] For step 1 and step 4, the present invention first pre-stores a block of input data in the local cache of each PE according to the storage characteristics of various matrix vectors in the PE local cache under this mode, and then calculates a block of output results in step 1 and step 4 through the element × vector mode in the ternary matrix multiplication operator data stream, and then pre-stores another input data block in the local cache of the PE again, and calculates another block of output results in step 1 and step 4 again according to the element × vector mode, and repeats this cycle until all the output results of step 1 and step 4 are obtained.

[0057] For step 5, the PE array loops through the vector × vector pattern in the ternary matrix multiplication operator data flow until all S in the attention heads are completed. out-i V i During this calculation process, PE stores the calculation results in each attention head in the local cache in the order of channel splicing. Then, PE calculates the final output result of step 5 in element × vector mode.

[0058] Convolution: This paper proposes a computational strategy for convolution, which unifies the storage format of tensors in convolution and multi-head self-attention in memory. For a convolution kernel of size (R, S, C), this paper uses (r, s) to represent the spatial coordinates of a vector of length C in the convolution kernel. Figure 7, select M vectors with the same spatial coordinates from the M convolution kernels and form a matrix of size C×M, denoted as W(i), where i=(r-1)×R+s, and take out EF rows of data from the input features according to the sliding rule of the convolution window, that is, the matrix X(i) of size EF×C. The row index row_idx of the taken input data is calculated by the following formula.

[0059]

[0060] Then, the matrix multiplication X(i)×W(i) is calculated in element × vector mode to obtain the partial sum matrix O(i) of the convolution result O. This cycle is repeated RS times, and finally the RS partial sum matrices are accumulated element by element to obtain the complete convolution output O.

[0061] The present invention also provides a Transformer accelerator, whose core computing module (mainly matrix multiplication calculation) applies the above-mentioned data flow design method.

Claims

1. A data flow design method compatible with multi-head self-attention and convolution calculation, characterized in that: The following steps are involved: S100: Transform the matrix vectors involved in the multi-head self-attention operator, including the fusion of the bias vector and the weight vector, and the fusion of the column vector whose elements are all 1 and the input feature vector; S200: According to the deformed matrix vector, an equivalent calculation form of the multi-head self-attention operator is obtained, including a ternary matrix multiplication operator, a binary matrix multiplication operator and a Softmax operator, wherein the ternary matrix multiplication operator is a continuous multiplication of three matrices, and the calculation form is T = A × B × C T ; S300: completing the calculation of the ternary matrix multiplication operator and the binary matrix multiplication operator in two data flow modes: vector×vector and element×vector; The vector × vector data flow mode means that, in each computing cycle, the computing unit can read the input data A and B and complete a multiplication and accumulation operation in the form of a systolic array, and then immediately pass the input data received in this cycle to the computing unit to the right and below, respectively, so that the input data can be used by the adjacent computing unit in the next cycle, and the partial sum M = A × B calculated in this cycle will always reside in the computing unit and be continuously accumulated and updated. When this partial sum is accumulated into a complete result, the computing unit will store the element in its local cache for reading and use by the computing unit in the element × vector mode; The element × vector data flow mode means that the input matrix C is transmitted between the column computing units in a pulsating manner, while the partial sums of the output matrix T are transmitted between the row computing units in a pulsating manner; before starting to transmit the matrix C in a pulsating manner, each computing unit will read an element of the intermediate result M from the local cache, and then perform a multiplication operation after receiving the data in the matrix C, and accumulate it with the partial sums transmitted by the computing unit on the left, and then transmit the just received matrix C to the computing unit below, and transmit the partial sums obtained by the operation to the computing unit on the right; finally, the computing unit in the rightmost column will pulsate a series of partial sums of T, which will be sent to the FIFO and continuously accumulated in the FIFO; S400: The calculation method of convolution is: according to the storage format of each matrix vector in the memory during the calculation process of the multi-head self-attention operator, and according to the element × vector data flow mode, the corresponding row vectors are extracted from the convolution kernel and the input features according to the convolution window sliding rule and form a two-dimensional matrix, and then the calculation of the convolution operator is completed according to the element × vector mode.

2. According to claim 1, a data flow design method compatible with multi-head self-attention and convolution calculation is characterized in that: The five calculation steps of the equivalent calculation form of the multi-head self-attention operator are: Step 1: Step 2: Step 3: out-1 ,S out-2 ,...,S out-h )=(Softmax(S in-1 ),Softmax(S in-2 ),...,Softmax(S in-h )); Step 4: V = X f W fV ; Step 5: O = Concat(S out-1 V1,S out-2 V2,...,S out-h V h )W fO ; Where X f is the fusion vector of the input feature and the full "1" column vector, W fQ / fK / fV It is the fusion vector of the Q / K / V weight vector and the bias vector in the multi-head self-attention operator, d is the number of channels in each attention head; Sin and sout are the intermediate results in the calculation process, h is the number of attention heads, U is the normalized key vector, V is the value vector, and O is the multi-head self-attention output feature.

3. According to claim 1, a data flow design method compatible with multi-head self-attention and convolution calculation is characterized in that: When an element of the intermediate result is calculated during the calculation of the ternary matrix multiplication operator, it is immediately used to multiply the matrix C. T The operation is performed on a row of elements in , thereby generating a row of partial sums in the output result T.

4. According to claim 2, a data flow design method compatible with multi-head self-attention and convolution calculation is characterized in that: For steps 1 and 4 of the multi-head self-attention operator in the equivalent computational form, the input data is pre-stored based on the storage characteristics of the intermediate results in the local cache of the computing unit in the vector × vector data flow mode, and then the computation is completed through the element × vector mode.

5. According to claim 2, a data flow design method compatible with multi-head self-attention and convolution calculation is characterized in that: For step 5, the computation unit array is executed in a vector × vector mode until the computation in all attention heads is completed; during this computation process, the computation unit stores the computation results in each attention head in the order of channel splicing into the local cache of the computation unit; Then, the computing unit calculates in element × vector mode to obtain the final output result of step 5.

6. According to claim 1, a data flow design method compatible with multi-head self-attention and convolution calculation is characterized in that: S400 specifically includes: for a convolution kernel of size (R, S, C), using (r, s) to represent the spatial coordinates of a vector of length C in the convolution kernel; selecting M vectors with the same spatial coordinates from the M convolution kernels, and forming a matrix of size C×M, denoted as W(i), where i=(r-1)×R+s, and taking out EF rows of data from the input features according to the convolution window sliding rule, that is, a matrix X(i) of size EF×C, and the row index row_idx of the taken out input data is calculated by the following formula: Then, the matrix multiplication X(i)×W(i) is calculated in element × vector mode to obtain the partial sum matrix O(i) of the convolution result O. This cycle is repeated RS times, and finally the RS partial sum matrices are accumulated element by element to obtain the complete convolution output O.

7. A Transformer accelerator, characterized in that: A data flow design method compatible with multi-head self-attention and convolution calculations is adopted as described in any one of claims 1-6.

Citation Information

Patent Citations

  • Implementation system and method based on systolic array self-attention mechanism

    CN116187407A

  • Urban rail transit multi-step short-time passenger flow prediction method based on attention mechanism

    CN116227668A

  • Swin-Transform reasoning hardware accelerator

    CN117010460A

  • Model training method and device, electronic equipment and readable storage medium

    CN117591881A

  • Attention mechanism accelerator based on data stream

    CN117634568A

Cited By

  • Attention mechanism fusion method and device based on data stream architecture accelerator

    CN119940434A