Attention Mechanism Fusion Method and Device Based on Dataflow Architecture Accelerator

Abstract

This invention proposes an attention mechanism fusion method and apparatus based on a dataflow architecture accelerator, including a method for accelerating attention computation on a GPDPU accelerator. This method selects a fusion scheme based on the product of the dimension of the embedding in the attention and the length of the input sequence. For computations with smaller dimensions, all operations are fused into the same kernel function by pre-passing transposed data, thereby reducing the time for configuring instructions and memory access overhead. For computations with larger dimensions, the input data is divided into blocks and passed into the memory of the temporary data cache (SPM) for computation. The attention computation steps are fused into two highly reusable kernel functions to reduce the configuration time of instruction information.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence (AI) and high-performance computing (HPC) technologies, and particularly to an attention mechanism fusion method, apparatus, electronic device, computer-readable storage medium, and computer program product based on a dataflow architecture. Background Technology

[0002] Attention is one of the core mechanisms of modern deep learning models (especially in natural language processing, computer vision, and multimodal tasks). In recent years, accelerating attention computation in Transformer models has been an important area of ​​deep learning research, particularly in optimizing hardware efficiency and sequence length expansion.

[0003] For example, Flash Attention is a specially optimized attention implementation that relies on memory access efficiency and computational flow optimization. It is meticulously designed for GPU hardware, dividing attention computation into small blocks to avoid writing and reading the entire attention matrix from memory, and reducing memory access time waste through operator fusion. However, it is primarily designed for GPUs and is difficult to port to dataflow architecture chips. Sparse Attention reduces the computational complexity of traditional global attention through sparsity strategies, but its sparsity mode needs to be designed according to specific tasks and data, resulting in poor versatility and making it difficult to use on dataflow chip DPUs.

[0004] Currently, the embedding dimension of Attention in large models ranges from 64 to 12288. Dataflow chips need to be able to handle inference for large models of different scales. Configuring instruction information for each kernel function in a dataflow chip consumes a lot of time. Furthermore, some calculations in Attention require matrix transposition, which may require multiple data transfers, consuming considerable memory access time. At the same time, the softmax nonlinear calculation in Attention requires dedicated hardware support for acceleration. Although its computational cost is only a small fraction compared to matrix multiplication, it requires multiple transfers of all input values, which may result in huge overhead if not handled properly.

[0005] In view of this, there is an urgent need to provide a method that can handle attention computations of various scales, can reuse kernel functions as much as possible to reduce instruction information configuration time, can reduce unnecessary memory access overhead through fusion operations, and can provide a transposed data method with low memory access consumption to match the matrix multiplication and softmax computations in attention. Summary of the Invention

[0006] This invention is a GPDPU accelerator based on a dataflow architecture. The dataflow architecture processor GPDPU accelerator includes: main memory DRAM, data cache SPM, and PE array. In order to solve the above-mentioned technical problems, namely (1) the GPDPU accelerator needs to be able to handle large-scale model inference computations of different scales, (2) configuring instruction information for each kernel function in the GPDPU accelerator consumes a lot of time, and it is necessary to reduce the proportion of instruction information configuration time in attention computation, (3) some computations in attention require matrix transposition, which may require multiple data transmissions, which will consume a considerable amount of memory access time, and it is necessary to reduce this memory access time. Ming proposed a method to accelerate Attention computation on the GPDPU accelerator. This method selects a fusion scheme based on the product of the dimension of the embedding in Attention and the length of the input sequence. For computations with smaller dimensions (such as 768), all operations are fused into the same kernel function by pre-passing transposed data, thereby reducing the time for configuring instructions and memory access overhead. For computations with larger dimensions (such as 4096), the input data is divided into blocks and passed into the memory of the temporary data cache SPM for computation. The Attention computation steps are fused into two highly reusable kernel functions to reduce the configuration time of instruction information.

[0007] To address the shortcomings of existing technologies, such as Figure 6 As shown, this invention proposes an attention mechanism fusion method based on a dataflow architecture accelerator, which includes:

[0008] The initial step involves acquiring a dataflow architecture accelerator for performing attention computation, which includes main memory, cache, and processing units; when the product of the length of the input sequence matrix and the embedding dimension is below a threshold, the first fusion step is executed.

[0009] In the first fusion step, the input sequence matrix, its transpose, the transpose of weight matrix Wq, weight matrix Wk, and the transpose of weight matrix Wv are transferred from main memory to the cache. Matrix multiplication is then performed to obtain the transpose of query matrix Q, key matrix K, and value matrix V. The processing unit then performs matrix multiplication to multiply the transposes of key matrix K and query matrix Q to obtain the transpose of S. Bitwise division and column-wise softmax activation are then performed on the transpose of S to obtain the transpose of P. Finally, matrix multiplication is performed between the transpose of value matrix V and the transpose of P to obtain the transpose of O. The transpose of O is then transposed again and transferred back to main memory to obtain matrix O, which serves as the attention calculation result.

[0010] The attention mechanism fusion method based on the dataflow architecture accelerator, wherein the initial step includes performing a second fusion step when the product of the length of the input sequence matrix and the embedding dimension is greater than or equal to a threshold:

[0011] The first block-segmentation step involves dividing the input sequence into blocks according to its length dimension to obtain subsequences. The weight matrix W and the subsequences are then written into the cache. A matrix T of size equal to the length of the subsequence and the length of the input sequence is passed in. If the weight matrix W is used to calculate a K matrix or a V matrix, then matrix T is a zero matrix. If the query matrix Q and the query matrix S are used to calculate the query matrix Q and the query matrix S, then matrix T is the transpose of the K matrix.

[0012] The first calculation step involves passing the weight matrix W, the subsequence, and the matrix T to the cache. When calculating the K or V matrix, the passed T matrix is ​​a zero matrix; when calculating the Q matrix, the passed T matrix is ​​the transpose of the K matrix. Matrix multiplication is performed on the weight matrix W and the subsequence to obtain the Q, K, or V matrix. The K or V matrix is ​​then multiplied by the matrix T to obtain the S matrix, which is entirely zero. The resulting K or V matrix block and the resulting zero-valued S matrix are then exported to main memory. When calculating the Q matrix, the weight matrix W is multiplied by the subsequence... Multiplying the T matrix yields the S matrix block, and the resulting Q matrix block and the transposed S matrix block are then output to main memory. Since the input sequence length is n, this first calculation step will be repeated 3 × n / 64 times. The first n / 64 iterations are used to calculate the K matrix, the middle n / 64 iterations are used to calculate the V matrix, and the last n / 64 iterations are used to calculate the Q and S matrices. Only the non-zero S matrix obtained in the last n / 64 iterations is needed; the zero S matrices obtained in the first 2 × n / 64 iterations are discarded. This first calculation step is repeated until the main memory contains the complete transposed S matrix.

[0013] The second block-splitting step involves dividing the transposed complete S matrix into intermediate matrix blocks and passing them into the cache, along with the transposed matrix of V.

[0014] The second calculation step involves receiving the intermediate matrix block, performing bitwise division on it, and then performing column-wise softmax calculation to obtain the transpose matrix of P. This transpose matrix is ​​then multiplied by the transpose matrix of V to obtain a portion of the transpose matrix of O. This portion is then transposed again and sent to the main memory. This second calculation step is repeated until a complete O matrix is ​​obtained in the main memory, which serves as the attention calculation result.

[0015] The attention mechanism fusion method based on the dataflow architecture accelerator is described above, wherein the threshold is the capacity of the cache.

[0016] The attention mechanism fusion method based on a dataflow architecture accelerator is described above, wherein the dataflow architecture accelerator is a dataflow chip DPU.

[0017] like Figure 7 As shown, this invention also proposes an attention mechanism fusion device based on a dataflow architecture accelerator, which includes:

[0018] The initial module acquires a dataflow architecture accelerator for performing attention computation, which includes main memory, cache, and processing units; when the product of the length of the input sequence matrix and the embedding dimension is below a threshold, the first fusion module is executed.

[0019] The first fusion module transfers the input sequence matrix, its transpose, the transpose of the weight matrix Wq, the weight matrix Wk, and the transpose of the weight matrix Wv from main memory to the cache. It then performs matrix multiplication to obtain the transpose of the query matrix Q, the key matrix K, and the value matrix V. Next, the processing unit performs matrix multiplication to multiply the transposes of the key matrix K and the query matrix Q to obtain the transpose of S. The transpose of S is then subjected to bitwise division and column-wise softmax activation to obtain the transpose of P. Finally, the transpose of the value matrix V is multiplied by the transpose of P to obtain the transpose of O. The transpose of O is then transposed again and transferred back to main memory to obtain matrix O, which serves as the attention calculation result.

[0020] The attention mechanism fusion device based on the dataflow architecture accelerator, wherein the initial module includes executing the first fusion module when the product of the length of the input sequence matrix and the embedding dimension is greater than or equal to a threshold, i.e., when the storage space occupied by the first fusion module during computation does not exceed the cache capacity, and formulating it as follows: The input sequence length is x, the embedding dimension is d, the number of attention heads is h, the number of bytes per data item is b, the buffer capacity is C bytes, and the second fusion module is executed:

[0021] The first block-segmentation module divides the input sequence into blocks according to its length dimension to obtain subsequences, and writes the weight matrix W and the subsequences into the cache; it takes in a matrix T of size equal to the length of the subsequence and the length of the input sequence. If the weight matrix W is used to calculate the weight matrix W, which is a K matrix or a V matrix, then the matrix T is a 0 matrix; if it is used to calculate the query matrix Q and the S matrix, then the matrix T is the transpose of the K matrix.

[0022] The first calculation module inputs the weight matrix W, the subsequence, and the T matrix into the cache. When calculating the K or V matrix, the input T matrix is ​​a zero matrix; when calculating the Q matrix, the input T matrix is ​​the transpose of the K matrix. Matrix multiplication is performed on the weight matrix W and the subsequence to obtain the Q, K, or V matrix. The K or V matrix is ​​then multiplied by the T matrix to obtain the S matrix, which is entirely zero. The resulting K or V matrix block and the resulting zero-valued S matrix are then output to main memory. When calculating the Q matrix, the Q matrix is ​​multiplied by the T matrix... Multiplying the T matrix yields the S matrix block, and the resulting Q matrix block and the transposed S matrix block are then sent to main memory. Since the input sequence length is n, this first calculation module will be executed 3×n / 64 times. The first n / 64 iterations are used to calculate the K matrix, the middle n / 64 iterations are used to calculate the V matrix, and the last n / 64 iterations are used to calculate the Q and S matrices. Only the non-zero S matrix obtained in the last n / 64 iterations is needed; the zero S matrices obtained in the first 2×n / 64 iterations are discarded. This first calculation module is executed repeatedly until the transposed complete S matrix is ​​stored in main memory.

[0023] The second block module divides the transposed complete S matrix into intermediate matrix blocks and passes them into the cache, while also passing in the transposed matrix of V;

[0024] The second calculation module receives the intermediate matrix block, performs bitwise division on it, performs column-wise softmax calculation to obtain the transpose matrix of P, multiplies it with the transpose matrix of V to obtain a part of the transpose matrix of O, transposes it again and outputs it to the main memory. The second calculation module is executed repeatedly until the complete O matrix is ​​obtained in the main memory, which is used as the attention calculation result.

[0025] The attention mechanism fusion device based on the dataflow architecture accelerator is described above, wherein the threshold is the capacity of the cache; the dataflow architecture accelerator is a dataflow chip DPU.

[0026] This invention proposes an electronic device, including the aforementioned attention mechanism fusion device based on a dataflow architecture accelerator. The electronic device may be connected to an information display device, which is used to display the attention calculation result using user-set display parameters, attributes, or through an artificial intelligence model.

[0027] The present invention proposes a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the attention mechanism fusion method based on a dataflow architecture accelerator.

[0028] The present invention proposes a computer program product, including a computer program, wherein when the computer program is executed by a processor, it implements the steps of the attention mechanism fusion method based on a dataflow architecture accelerator.

[0029] As can be seen from the above solutions, the advantages of the present invention are:

[0030] For attention calculations with a small product of embedding dimension and input sequence length, the five-step attention calculation can be fused into one operator, maximizing the use of SPM's storage space and reducing memory access time and instruction configuration time. For attention calculations with a large product of embedding dimension and input sequence length, the input data is passed in in blocks, saving SPM's storage space. The five-step attention calculation can be fused into two operators, and the softmax calculation in the second operator can be accelerated by adjusting the arrangement of the output data of the first operator. Attached Figure Description

[0031] Figure 1 A schematic diagram of the overall structure of the GPDPU accelerator;

[0032] Figure 2 A schematic diagram of the standard implementation of Attention computation on GPDPU;

[0033] Figure 3 This is a schematic diagram illustrating how the first embodiment of the present invention merges five kernel functions into one kernel function;

[0034] Figure 4 This is a schematic diagram of the first kernel function of Scheme 2 of the present invention;

[0035] Figure 5 This is a schematic diagram of the second kernel function in Scheme 2 of the present invention;

[0036] Figure 6 This is a flowchart of the method of the present invention;

[0037] Figure 7 This is a block diagram of the device of the present invention;

[0038] Figure 8 This is a schematic diagram of the structure of the first electronic device of the present invention;

[0039] Figure 9 This is a schematic diagram of the application environment structure of the first electronic device of the present invention;

[0040] Figure 10 This is a schematic diagram of the structure of the second electronic device of the present invention.

[0041] Figure label:

[0042] A - First electronic device;

[0043] B-Attention mechanism fusion device based on dataflow architecture accelerator;

[0044] C-Data acquisition equipment;

[0045] D-Information display device;

[0046] 1000 - Second electronic device;

[0047] Ⅰ-Computational Unit;

[0048] II-ROM;

[0049] III-RAM;

[0050] N-bus;

[0051] V-Interface;

[0052] VI - Input Unit;

[0053] VII - Output Unit;

[0054] VIII - Storage medium;

[0055] IX - Communication Unit. Detailed Implementation

[0056] It should be noted that, in this invention, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0057] In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0058] The processor described in this invention is the control center of an electronic device. It can be a single processor or a collective term for multiple processing elements. For example, it can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of this invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0059] Alternatively, the processor can perform various functions of the electronic device by running or executing software programs stored in memory and by calling data stored in memory.

[0060] In a specific implementation, as one example, the processor may include one or more CPUs. Each of these processors may be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions). Electronic devices may include servers, desktop computers, laptops, smartphones, tablets, embedded computers, etc., where the embedded computer includes vehicles and robots, etc.

[0061] The memory is used to store the software program that executes the solution of the present invention, and the execution is controlled by the processor. For specific implementation methods, please refer to the above method embodiments, which will not be repeated here.

[0062] It should be noted that the structure of the electronic device shown in the accompanying drawings of this invention does not constitute a limitation thereof. The actual knowledge structure recognition device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0063] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0064] It should also be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0065] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0066] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0067] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0068] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0069] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0070] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] To achieve the above-mentioned technical effects, the present invention proposes the following key technical points:

[0072] Key point 1: The Attention fusion scheme is determined based on the product of the embedding dimension and the length of the input sequence; Technical effect: For smaller Attention calculations, the calculations can be fused to the maximum extent, making full use of the SPM's storage space and reducing the time for transmitting intermediate calculation results back and forth. For larger Attention calculations, five kernel functions are fused into two by reusing kernel functions.

[0073] Key point 2: By reusing kernel functions, the five kernel functions are merged, which reduces the proportion of instruction information configuration time and memory access time. Technical effect: Reduce the time spent on transferring intermediate computation data back and forth between main memory and SPM, minimize the time spent transferring instruction information, and greatly reduce the storage pressure on SPM by passing the input sequence in blocks.

[0074] Key point 3: Both schemes use transposed data transmission. The smaller model scheme uses transposed transmission because the matrix calculations for Attention and softmax calculations require matrix transposition. This is achieved by pre-passing in the transposed input sequence (e.g., the vector feature matrix obtained after encoding a text sequence, or the vector feature sequence obtained after image processing) and the weight matrix (the weight parameter matrix obtained during model training, W). q / W k / W v The required transpose matrix can be calculated directly without passing out intermediate results, which maximizes the degree of fusion. In the scheme for larger models, transpose transmission is used in two places. The first is also to make the kernel function fusion high, and the second is to make full use of the acceleration effect of SIMD parallelization in conjunction with the subsequent softmax calculation.

[0075] This invention accelerates the overall computation time of Attention on the DPU, primarily by reducing the time spent transferring data between main memory and SPM during Attention computation through operator fusion. The computation time itself is not reduced. During Attention computation, intermediate results need to be transposed. Conventionally, this requires transposing the result from SPM, transmitting it to main memory, and then transmitting it back to SPM. On the DPU, transposing the matrix involves calling a transfer function between main memory and SPM, which consumes significant time in data transmission. Furthermore, re-transmitting the instruction information to SPM also takes considerable time. The method described in this invention, which directly calculates the transpose of intermediate results using the transpose of the input sequence, avoids this time consumption.

[0076] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.

[0077] This invention accelerates attention mechanism computation and optimizes the GPDPU accelerator to address issues such as the need to handle attention mechanism computations of different scales, the long time required to transmit instruction information in the GPDPU accelerator, and the long memory access time required to transpose data.

[0078] To make the above features and effects of the present invention clearer and easier to understand, practical examples are given below, and detailed descriptions are provided in conjunction with the accompanying drawings.

[0079] like Figure 1 The diagram shows the overall structure of the GPDPU accelerator used in this invention. GPDPU is a coarse-grained dataflow accelerator that utilizes the dataflow concept and employs a flow-controlled execution method internally. Traditional control-flow execution accelerators drive execution with instructions, while this structure uses the data flow direction of the computational dataflow graph to drive execution. Compared to traditional structures, this structure is more suitable for accelerating scientific computing with high parallelism. The GPDPU core consists of six parts: main memory, a microcontroller, a 4x4 execution array (PE), an instruction cache (Cbuf), a data cache (SPM), and a transport network. Main memory stores configuration information, instructions, and data transmitted from the CPU; the instruction cache (Cbuf) caches the instructions to be executed by the execution units (PEs), transmitted from main memory; the data cache (SPM) caches the input data, weight data, calculated output data, and intermediate result data required by the execution units; the 4x4 PE array completes the entire computation process; and the transport network is responsible for data transfer between main memory and cache, and between cache and PE array. The execution array consists of 16 execution instances (PEs) interconnected by a mesh network. Each PE contains an instruction cache, register file, four-stage pipelined arithmetic unit, router, and an internal microcontroller. The microcontroller is used to configure the instruction information for each of the 16 PEs, control the execution order of the PEs, and control the return of the computation results to main memory after completion.

[0080] like Figure 2 The diagram shows a standard implementation of Attention computation on a GPDPU. In this standard implementation, for example... Figure 2As shown on the left, five kernel functions need to be called, in the following order: matrix multiplication 1 (the matrix sizes for multiplication are [input_length, embedding_size] and [embedding_size, embedding_size / h]), matrix multiplication 2 (the matrix sizes for multiplication are [input_length, embedding_size / h] and [embedding_size / h, input_length]), bitwise division, row-wise softmax, and matrix multiplication 3 (the matrix sizes for multiplication are [input_length, input_length] and [input_length, embedding_size / h]), where input_length is the length of the input sequence, i.e., the number of characters. The number, embedding_size is the dimension of each input character after being converted into a vector representation, and h indicates the use of the h-head attention mechanism. Each kernel function depends on the previous calculation results. The data dependency relationship determines that the 5 kernel functions need to be executed serially. After each kernel function is calculated, the calculation result needs to be transferred from SPM back to main memory, and then transferred from main memory to SPM for the calculation of the next kernel function. Therefore, memory access is one of the bottlenecks for accelerating attention calculation. If, under the premise of ensuring data dependency, intermediate results can be saved by using registers to save intermediate results to avoid memory access, or intermediate results can be stored back to the lower-consuming storage device SPM instead of main memory, the overhead of storing and accessing this part of intermediate result data can be saved or reduced. This takes advantage of the data storage characteristics of GPDPU, which is: main memory access overhead > SPM access overhead > inter-PE access overhead > PE internal register access overhead.

[0081] The implementation steps of this invention are described below. The input sequence is actually a matrix, the shape of which is the length of the input sequence × the embedding dimension. First, it is determined whether the product of the embedding dimension and the length of the input sequence is less than 1. If the conditions are met, then Option 1 will be used; otherwise, Option 2 will be used.

[0082] Option 1:

[0083] The above five kernel functions are merged into one, such as... Figure 3As shown, firstly, the input sequence matrix, the transpose of the input sequence, the transpose of the Wq weight matrix, the Wk weight matrix, and the transpose of the Wv weight matrix are transferred from main memory to SPM. The transposes of the query matrix (Q matrix), key matrix (K matrix), and value matrix (V matrix) are calculated by matrix multiplication by 1. Then, the transposes of K and Q are multiplied by 2 to obtain the transpose of S (similarity matrix). Next, the transpose of P is calculated by bitwise division and column-wise softmax activation function. Finally, the transpose of V and the transpose of P (attention weight matrix) are multiplied by 3 to obtain the transpose of O. Finally, the transpose of O (output matrix) is transposed again and transferred back to main memory to obtain the O matrix. The O matrix is ​​the output obtained by the standard Attention implementation. Therefore, the correct calculation result can be obtained through Scheme 1.

[0084] This approach requires only one kernel function, meaning it only needs to transfer instruction information from main memory to SPM once, which can greatly reduce the overall time required for Attention.

[0085] This scheme only transfers data from main memory to SPM once. The following calculation shows the difference in the amount of data transferred between this scheme and the standard implementation.

[0086] The standard implementation of Attention transmits a total amount of data per head:

[0087] 1) Input sequence: input_length * embedding_size

[0088] 2) Pass in the Wq, Wk, and Wv matrices: embedding_size * embedding_size / h * 3

[0089] 3) Output K, Q, V matrices: input_length * embedding_size / h * 3

[0090] 4) Input the Q matrix and the transpose of K: input_length * embedding_size / h * 2

[0091] 5) Output and then input S matrix: input_length * input_length * 2

[0092] 6) Output and then input the S' matrix: input_length * input_length * 2

[0093] 7) Output and then input the P matrix, and input the transpose of the V matrix: input_length*input_length*2+input_length*embedding_size / h

[0094] 8) Output O matrix: input_length * embedding_size / h

[0095] Total 6*input_length^2+(7 / h+1)*input_length*embedding_size+3 / h*embedding_size^2

[0096] Attention's first approach achieves the following total data transmission volume (per head):

[0097] 1) Input sequence and its transpose: input_length * embedding_size * 2

[0098] 2) Pass in the transpose of the weight matrix Wq, the weight matrix Wk, and the transpose of the weight matrix Wv: embedding_size * embedding_size / h * 3

[0099] 3) Output O matrix: input_length * embedding_size / h

[0100] Total(2+1 / h)*input_length*embedding_size+3 / h*embedding_size^2

[0101] It can be seen that compared with the standard implementation, this scheme can save the data transmission time of 6*input_length^2+(6 / h-1)input_length*embedding_size for each head.

[0102] This scheme eliminates the step of transmitting the S matrix back to main memory and then transposing it before passing it to the SPM by directly calculating the transpose of S, thus making it possible to integrate the calculation of Attention into a single operator on the GPDPU.

[0103] Option 2:

[0104] When the product of the embedding dimension and the length of the input sequence is less than 2^20, the above 5 kernel functions can be merged into 2. Since the matrix multiplication calculation in Attention requires a large amount of storage space at this time, for example, when input_length = embedding_size = 4096, the data type of the calculation is fp16 and it is a 32-head attention, the maximum amount of data that needs to be passed to SPM in the standard implementation is 35MB, which is much larger than the capacity of SPM. Therefore, the input sequence needs to be passed in in blocks.

[0105] 1) Kernel function 1

[0106] Kernel function 1 as follows Figure 4 As shown, where Figure 4 The left parts of the two images are different, indicating that they are calculating different subsequences of the input sequence.

[0107] Data Blocking and Input: The single-head weight matrix is ​​small enough that it is not divided into blocks. The input sequence is divided into blocks of 64 units along the input_length dimension, but not along the embedding_size dimension. The weight matrix W (Wq, Wk, or Wv matrix) and the block-sized input sequence are input into SPM. The size of the weight matrix is ​​[embedding_size, length_per_head], where length_per_head is typically 64 or 128. The size of the block-sized input sequence is [64, embedding_size]. In addition, a matrix T of size [length_per_head, input_length] is also required. Matrix T is actually used to obtain the transpose of matrix K of matrix S (and is also calculated by kernel function 1). If the kernel function is used to calculate matrix K or matrix V, the input matrix T is a zero matrix. If the kernel function is used to calculate matrix Q and matrix S, the input matrix T should be the transpose of matrix K.

[0108] In this design, the input sequence is divided into blocks of 64 units, a relatively small number chosen to account for the capacity limitations of the SPM. Based on the current mainstream large-scale models, the embedding_size and number of attention heads are also set to 128. This invention is not limited to this; for ease of computation in the DPU, it is optimally set to a multiple of 64. Let the embedding_size be d, the block size be x, the number of attention heads be h, the data type size in bytes be b, and the SPM capacity be C bytes, then the following conditions are met. That's all.

[0109] Calculation and Result Output: Matrix multiplication is performed on the weight matrix and the segmented input sequence, i.e., multiplying a matrix of size [64, embedding_size] with a matrix of size [embedding_size, length_per_head] to obtain a Q, K, or V matrix of size [64, length_per_head]. When the kernel function calculates a K or V matrix, the input T matrix is ​​a 0 matrix. Multiplying the K or V matrix with this 0 matrix yields a matrix of all zeros. The resulting K or V matrix block and the resulting 0 S matrix are then output to main memory. The Q matrix, like the K / V matrix, is obtained by matrix multiplication of the weight matrix and the segmented input sequence. When the kernel function calculates a Q matrix, the input T matrix is ​​the transpose of the K matrix. Multiplying the Q matrix with the transpose of the K matrix yields a partial S matrix. The resulting Q matrix block and the S matrix block are then output to main memory. Note that the S matrix block needs to be transposed and returned to main memory to fit the subsequent softmax calculation.

[0110] The ultimate goal of kernel function 1 is to compute the S matrix. It combines the two kernel functions from the standard Attention implementation, even when the input sequence must be segmented. First, kernel function 1 is called to compute the K matrix, with the input T matrix being a zero matrix. Each call to kernel function 1 yields a K matrix block of [64, length_per_head]. Calling kernel function 1 64 times (input_length / 64) yields the complete K matrix. After obtaining the complete K matrix, kernel function 1 is called again to compute the Q matrix, with the input T matrix being the transpose of the K matrix. Each call to kernel function 1 yields a Q matrix block of [64, length_per_head] and an S matrix of [64, input_length]. Calling kernel function 1 64 times (input_length / 64) yields the complete Q matrix and the complete S matrix. The matrix multiplication process involves continuous accumulation on the same S matrix. Each call to kernel function 1 computes a different subsequence of the input sequence. After computing all subsequences of the input sequence, the complete S matrix is ​​obtained.

[0111] 2) Kernel function 2

[0112] Kernel function 2, as shown Figure 5 As shown, where Figure 5 The left parts of the two images are different because the input is a different part of the ST matrix.

[0113] Data partitioning and input: After kernel function 1 calculates the transpose of the complete S matrix of size [input_length, input_length] multiple times, it is divided into matrix blocks of size [input_length, 256] and input into SPM. In addition, the transpose matrix of V needs to be input. Here, the transpose matrix of S is divided into 256 columns to facilitate the parallelization of softmax calculation using SIMD (Single Instruction Multiple Data).

[0114] Calculation and Result Output: First, perform bitwise division on the transpose matrix S. Then, perform column-wise softmax calculation to obtain the transpose matrix P. Multiply it by the transpose matrix V (pre-inputted), i.e., multiply a matrix of size [length_per_head, input_length] by a matrix of size [input_length, 256] to obtain a portion of the transpose matrix O of size [length_per_head, 256]. Finally, transpose and output this partial transpose matrix O to obtain a partial matrix of O.

[0115] Each call to kernel function 2 yields an O matrix block of [256, length_per_head]. Using kernel function 2 2 times (input_length / 256) results in the complete O matrix. The complete O matrix is ​​[input_length, length_per_head]. Each calculation yields the [256, length_per_head] portion; therefore, calculating input_length / 256 gives the complete O matrix.

[0116] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0117] like Figure 7 As shown, this invention also proposes an attention mechanism fusion device based on a dataflow architecture accelerator, which includes:

[0118] The initial module acquires a dataflow architecture accelerator for performing attention computation, which includes main memory, cache, and processing units; when the product of the length of the input sequence matrix and the embedding dimension is below a threshold, the first fusion module is executed.

[0119] The first fusion module transfers the input sequence matrix, its transpose, the transpose of the weight matrix Wq, the weight matrix Wk, and the transpose of the weight matrix Wv from main memory to the cache. It then performs matrix multiplication to obtain the transpose of the query matrix Q, the key matrix K, and the value matrix V. Next, the processing unit performs matrix multiplication to multiply the transposes of the key matrix K and the query matrix Q to obtain the transpose of S. The transpose of S is then subjected to bitwise division and column-wise softmax activation to obtain the transpose of P. Finally, the transpose of the value matrix V is multiplied by the transpose of P to obtain the transpose of O. The transpose of O is then transposed again and transferred back to main memory to obtain matrix O, which serves as the attention calculation result.

[0120] The attention mechanism fusion device based on the dataflow architecture accelerator, wherein the initial module includes a threshold (b×) that is applied when the product of the length of the input sequence matrix and the embedding dimension is greater than or equal to the threshold. Execute the second fusion module:

[0121] The first block-segmentation module divides the input sequence into blocks according to its length dimension to obtain subsequences, and writes the weight matrix W and the subsequences into the cache; it takes in a matrix T of size equal to the length of the subsequence and the length of the input sequence. If the weight matrix W is used to calculate the weight matrix W, which is a K matrix or a V matrix, then the matrix T is a 0 matrix; if it is used to calculate the query matrix Q and the S matrix, then the matrix T is the transpose of the K matrix.

[0122] The first calculation module inputs the weight matrix W, the subsequence, and the T matrix into the cache. When calculating the K or V matrix, the input T matrix is ​​a zero matrix; when calculating the Q matrix, the input T matrix is ​​the transpose of the K matrix. Matrix multiplication is performed on the weight matrix W and the subsequence to obtain the Q, K, or V matrix. The K or V matrix is ​​then multiplied by the T matrix to obtain the S matrix, which is entirely zero. The resulting K or V matrix block and the resulting zero-valued S matrix are then output to main memory. When calculating the Q matrix, the Q matrix is ​​multiplied by the T matrix... Multiplying the T matrix yields the S matrix block, and the resulting Q matrix block and the transposed S matrix block are then sent to main memory. Since the input sequence length is n, this first calculation module will be executed 3×n / 64 times. The first n / 64 iterations are used to calculate the K matrix, the middle n / 64 iterations are used to calculate the V matrix, and the last n / 64 iterations are used to calculate the Q and S matrices. Only the non-zero S matrix obtained in the last n / 64 iterations is needed; the zero S matrices obtained in the first 2×n / 64 iterations are discarded. This first calculation module is executed repeatedly until the transposed complete S matrix is ​​stored in main memory.

[0123] The second block module divides the transposed complete S matrix into intermediate matrix blocks and passes them into the cache, while also passing in the transposed matrix of V;

[0124] The second calculation module receives the intermediate matrix block, performs bitwise division on it, performs column-wise softmax calculation to obtain the transpose matrix of P, multiplies it with the transpose matrix of V to obtain a part of the transpose matrix of O, transposes it again and outputs it to the main memory. The second calculation module is executed repeatedly until the complete O matrix is ​​obtained in the main memory, which is used as the attention calculation result.

[0125] The attention mechanism fusion device based on the dataflow architecture accelerator is described above, wherein the threshold is the capacity of the cache; the dataflow architecture accelerator is a dataflow chip DPU.

[0126] like Figure 8 As shown, in another embodiment of the present invention, a first electronic device A is also proposed, including the aforementioned attention mechanism fusion device based on a data flow architecture accelerator.

[0127] like Figure 9 As shown, the first electronic device A can also be connected to the data acquisition device C and the information display device D through a wired or wireless information transmission scheme. The data acquisition device C is used to acquire the vector feature matrix obtained after encoding the text sequence, or the vector feature sequence obtained after processing the image. The information display device D is used to display the attention calculation results, text semantic recognition results, or image classification results obtained by the present invention.

[0128] The information display device D can process and organize the data output by the first electronic device A based on an information display mechanism to improve the readability of the data. This information display mechanism can be manually preset, for example, visualizing the data output by the first electronic device A. It can present the user with the specified key information based on user-defined display parameters and / or attributes, such as the data range and font, color, and scrolling options. Users can access this information more quickly without needing to navigate to secondary pages or scroll through pages, saving them time and effort. Alternatively, the information display mechanism can be an artificial intelligence (AI) display model that learns the user's key information interests based on past usage habits, such as viewing time, click count, and edit count, and automatically presents rich and necessary key information.

[0129] The present invention also provides a computer program product, the computer program product including a computer program that can be stored on a readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the attention mechanism fusion method based on the dataflow architecture accelerator provided by the above methods.

[0130] In another embodiment, the present invention also proposes a storage medium VIII for storing a computer program that executes the attention mechanism fusion method based on the dataflow architecture accelerator. It should be understood that the storage medium in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0131] Figure 10 A schematic block diagram of a second electronic device 1000 that can be used to implement embodiments of the present invention is shown. The second electronic device 1000 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The second electronic device 1000 can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein. The second electronic device 1000 may be the same as or different from the first electronic device A.

[0132] The second electronic device 1000 includes a computing unit I, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory II (ROM) or a computer program loaded from storage medium VIII into random access memory (RAM) III. The RAM III may also store various programs and data required for the operation of the device 1000. The computing unit I, ROM II, and RAM III are interconnected via bus IV. An input / output (I / O) interface V is also connected to bus IV.

[0133] Multiple components in the second electronic device 1000 are connected to I / O interface V, including: input unit VI, such as a keyboard, mouse, etc.; output unit VII, such as various types of displays, speakers, etc.; storage medium VIII, such as a disk, optical disk, etc.; and communication unit IX, such as a network card, modem, wireless transceiver, etc. Communication unit IX allows the second electronic device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0134] The computing unit I can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of computing unit I include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit I performs the various methods and processes described above, such as method steps S1-S2. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage medium VIII. In some embodiments, part or all of the computer program can be loaded and / or installed on device 1000 via ROM II and / or communication unit IX. When the computer program is loaded into RAM III and executed by computing unit I, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, computing unit I can be configured to perform methods by any other suitable means (e.g., by means of firmware).

[0135] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. For those skilled in the art, other modifications can be easily made. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and illustrations shown and described herein.

Claims

1. A fusion method for attention mechanisms based on a dataflow architecture accelerator, characterized in that, include: The initial step involves acquiring a dataflow architecture accelerator for performing attention computation, which includes main memory, cache, and processing units. When the product of the length of the input sequence matrix and the embedding dimension is less than a threshold The first fusion step is executed, which merges the five kernel function operations into a single kernel function by pre-inputting transposed data; when the product of the length of the input sequence matrix and the embedding dimension is greater than or equal to a certain threshold... The second fusion step is performed, in which the input data is divided into blocks and passed into the memory of the temporary data cache SPM for calculation. The five kernel functions in the Attention calculation step are fused into two kernel functions; where x is the length of the input sequence, d is the embedding dimension, h is the number of attention heads, b is the number of bytes occupied by a single data, and C is the number of bytes of the cache capacity. The first fusion step includes: transferring the input sequence matrix, the transpose of the input sequence, the transpose of the weight matrix Wq, the weight matrix Wk, and the transpose of the weight matrix Wv from main memory to the cache; obtaining the transpose of the query matrix Q, the key matrix K, and the value matrix V through matrix multiplication by 1; then multiplying the transpose of the key matrix K and the query matrix Q by 2 through the processing unit to obtain the transpose of S; sequentially performing bitwise division and column-wise softmax activation calculation on the transpose of S to obtain the transpose of P; finally, multiplying the transpose of the value matrix V and the transpose of P by 3 to obtain the transpose of O; and finally transposing the transpose of O back into main memory to obtain the O matrix, which is used as the attention calculation result.

2. The attention mechanism fusion method based on a dataflow architecture accelerator as described in claim 1, characterized in that, The second fusion step includes: The first block-segmentation step involves dividing the input sequence into blocks according to its length dimension to obtain subsequences. The weight matrix W and the subsequences are then written into the cache. A matrix T of size equal to the length of the subsequence and the length of the input sequence is passed in. If the weight matrix W is used to calculate a K matrix or a V matrix, then matrix T is a zero matrix. If the query matrix Q and the query matrix S are used to calculate the query matrix Q and the query matrix S, then matrix T is the transpose of the K matrix. The first calculation step involves inputting the weight matrix W, the subsequence, and the matrix T into the cache. When calculating the K or V matrix, the input T matrix is ​​a zero matrix; when calculating the Q matrix, the input T matrix is ​​the transpose of the K matrix. The weight matrix W and the subsequence are multiplied by 1 to obtain the Q, K, or V matrix. The K or V matrix is ​​multiplied by the matrix T to obtain the S matrix, which is all zeros. The resulting K or V matrix block and the resulting zero-valued S matrix are then output to main memory. When calculating the Q matrix, the Q matrix is ​​multiplied by 2 by the T matrix to obtain the S matrix block. The resulting Q matrix block and the transpose of the S matrix block are then output to main memory. If the length of the input sequence is n, this first calculation step will be repeated 3 × n / the length of the subsequence. The first n / the length of the subsequence are used to calculate the K matrix, the middle n / the length of the subsequence are used to calculate the V matrix, and the last n / the length of the subsequence are used to calculate the Q and S matrices. The second block-splitting step involves dividing the transposed complete S matrix into intermediate matrix blocks and passing them into the cache, along with the transposed matrix of V. The second calculation step involves receiving the intermediate matrix block, performing bitwise division on it, and then performing column-wise softmax calculation to obtain the transpose matrix of P. This transpose matrix is ​​then multiplied by 3 with the transpose matrix of V to obtain a portion of the transpose matrix of O. This portion is then transposed again and sent to the main memory. This second calculation step is repeated until a complete O matrix is ​​obtained in the main memory, which serves as the attention calculation result.

3. The attention mechanism fusion method based on a dataflow architecture accelerator as described in claim 1, characterized in that, This threshold represents the capacity of the cache.

4. The attention mechanism fusion method based on a dataflow architecture accelerator as described in claim 1, characterized in that, The dataflow architecture accelerator is the dataflow chip DPU.

5. An attention mechanism fusion device based on a dataflow architecture accelerator, characterized in that, include: The initial module acquires a dataflow architecture accelerator for performing attention computation, which includes main memory, cache, and processing units; When the product of the length of the input sequence matrix and the embedding dimension is less than a threshold The first fusion module is executed, which merges the five kernel function operations into a single kernel function by pre-passing transposed data; when the product of the length of the input sequence matrix and the embedding dimension is greater than or equal to a certain threshold... The second fusion module is executed, which divides the input data into blocks and passes them into the memory of the temporary data cache SPM for calculation. The five kernel functions in the Attention calculation step are fused into two kernel functions; where x is the length of the input sequence, d is the embedding dimension, h is the number of attention heads, b is the number of bytes occupied by a single data, and C is the number of bytes of the cache capacity. The first fusion module includes: transferring the input sequence matrix, the transpose of the input sequence, the transpose of the weight matrix Wq, the weight matrix Wk, and the transpose of the weight matrix Wv from main memory to the cache; obtaining the transpose of the query matrix Q, the key matrix K, and the value matrix V through matrix multiplication by 1; then multiplying the transpose of the key matrix K and the query matrix Q by 2 through the processing unit to obtain the transpose of S; sequentially performing bitwise division and column-wise softmax activation calculation on the transpose of S to obtain the transpose of P; finally, multiplying the transpose of the value matrix V and the transpose of P by 3 to obtain the transpose of O; and finally transposing the transpose of O back into main memory to obtain the O matrix, which is used as the attention calculation result.

6. The attention mechanism fusion device based on a dataflow architecture accelerator as described in claim 5, characterized in that, The second fusion module includes: The first block-segmentation module divides the input sequence into blocks according to its length dimension to obtain subsequences, and writes the weight matrix W and the subsequences into the cache; it takes in a matrix T of size equal to the length of the subsequence and the length of the input sequence. If the weight matrix W is used to calculate the weight matrix W, which is a K matrix or a V matrix, then the matrix T is a 0 matrix; if it is used to calculate the query matrix Q and the S matrix, then the matrix T is the transpose of the K matrix. The first calculation module inputs the weight matrix W, the subsequence, and the T matrix into the cache. When calculating the K or V matrix, the input T matrix is ​​a 0 matrix. When calculating the Q matrix, the input T matrix is ​​the transpose of the K matrix. The weight matrix W and the subsequence are multiplied by 1 to obtain the Q, K, or V matrix. The K or V matrix is ​​multiplied by the T matrix to obtain the S matrix, which is all zeros. The resulting K or V matrix block and the resulting S matrix are then sent to main memory. When calculating the Q matrix, the Q matrix is ​​multiplied by the T matrix by 2 to obtain the S matrix block. The resulting Q matrix block and the transposed S matrix block are then sent to main memory. If the length of the input sequence is n, the first calculation step will be repeated 3 × n / subsequence length times. The first n / subsequence length times are used to calculate the K matrix, the middle n / subsequence length times are used to calculate the V matrix, and the last n / subsequence length times are used to calculate the Q and S matrices. This first calculation step is repeated until the main memory contains the transposed complete S matrix. The second block module divides the transposed complete S matrix into intermediate matrix blocks and passes them into the cache, while also passing in the transposed matrix of V; The second calculation module receives the intermediate matrix block, performs bitwise division on it, performs column-wise softmax calculation to obtain the transpose matrix of P, multiplies it by 3 with the transpose matrix of V to obtain a part of the transpose matrix of O, transposes it again and outputs it to the main memory. The second calculation module is executed repeatedly until the complete O matrix is ​​obtained in the main memory, which is used as the attention calculation result.

7. The attention mechanism fusion device based on a dataflow architecture accelerator as described in claim 5, characterized in that, The threshold is the capacity of the cache; the dataflow architecture accelerator is the dataflow chip DPU.

8. An electronic device, characterized in that, The device includes the attention mechanism fusion apparatus based on a dataflow architecture accelerator as described in any one of claims 5-7, wherein the electronic device is connected to an information display device, which is used to display the attention calculation result using user-set display parameters, attributes, or through an artificial intelligence model.

9. A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the attention mechanism fusion method based on a dataflow architecture accelerator as described in any one of claims 1-4.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the attention mechanism fusion method based on the dataflow architecture accelerator as described in any one of claims 1-4.

Images