Deployment method of efficient Transform model reasoning accelerator based on Versa ACAP

Abstract

The invention discloses an efficient Transform model reasoning accelerator deployment method based on Versaal ACAP, and belongs to the field of software and hardware collaborative acceleration numerical calculation. Firstly, load analysis is carried out on the model, and meanwhile, according to hardware performance, the types and scales of operators needing to be borne by hardware of all parts are determined; then, hardware is equipped with a data transmitter, a data receiver, a nonlinear operator and other modules; then, the modules are reasonably deployed and connected in hardware, pipeline parallelization is completed, and calculation of a multi-head attention stage and a feedforward neural network stage is achieved; and finally, integrating the two stages to realize accelerated calculation of the whole encoder / decoder layer so as to accelerate the reasoning speed of the upper layer model. Experiments prove that the throughout of the model is effectively improved, the reasoning delay is reduced, meanwhile, the accelerator has huge advantages in the aspect of batch reasoning, and the accelerator has wide application prospects in the aspects of reducing the model reasoning cost and improving the model reasoning speed.

Description

Technical Field

[0001] The present invention relates to a deployment method of an efficient Transformer model reasoning accelerator based on Versal ACAP, and belongs to the field of software and hardware collaborative acceleration of numerical calculations. Background Art

[0002] Transformer is a neural network architecture based on the self-attention mechanism, which has achieved significant improvements in sequence-to-sequence tasks. Transformer was originally designed for machine translation tasks, but later it has been widely used and successful in the field of natural language processing. At the same time, Transformer has a place in the field of computer vision, and its performance is better than that of convolutional neural networks. It is precisely because of the emergence of Transformer that the attention mechanism has become mainstream, and subsequently many language or visual models based on Transformer (such as BERT, RoBERTa and ViT) have emerged, making the field of machine learning move forward at an unprecedented speed.

[0003] Although Transformer has achieved considerable advantages, this is closely related to the huge number of model parameters. Intensive computing and huge memory overhead hinder their deployment and popularization on devices. In order to speed up the reasoning speed of the model, GPU is currently the main hardware platform for accelerating Transformer. However, due to its own hardware limitations, high power consumption is difficult to avoid, and it is difficult to further customize the hardware structure. On other platforms (such as FPGA, ASIC), although they have finer design granularity, the design space that can be explored is larger, and generally have a longer design cycle and higher development costs.

[0004] Considering the increasing acceleration demand of Transformer, the accelerator effect implemented by traditional hardware platforms is not ideal. The present invention proposes a deployment method of an efficient Transformer model reasoning accelerator based on Versal ACAP. First, the model is load analyzed, and the type and scale of matrix multiplication that the AI ​​engine (AIE) needs to undertake are determined based on the hardware performance, and multiple matrix multiplication processing units are deployed in the hardware; then the AIE is equipped with a data transmitter, a data receiver, and hardware modules of nonlinear operators; then these modules are reasonably deployed and connected in the hardware to complete the pipeline parallelization and realize the calculation of the multi-head attention stage and the feedforward neural network stage; finally, the two stages are integrated to realize the accelerated calculation of the entire encoder / decoder layer, thereby accelerating the reasoning speed of the upper model. Summary of the invention

[0005] Different from the existing Transformer accelerator deployment method, this invention uses the characteristics of Versal ACAP to propose a deployment method for an efficient Transformer model inference accelerator based on Versal ACAP. First, the scale of matrix multiplication operators required for the operation of the Transformer model is extracted, and the type and scale of matrix multiplication that AIE needs to undertake are determined based on the AIE of Versal ACAP and its peripheral hardware performance. Secondly, design and actually deploy multiple matrix multiplication processing units (PUs) to the AIE array to bear the matrix multiplication load in the Transformer; then develop data transmitters and data receivers for PU services on the PL side of the Versal ACAP, and develop hardware modules for Softmax, Layernorm, GELU and matrix transposition operators to bear the remaining load except matrix multiplication; then use the already developed hardware modules to combine them into linear layer modules (LB) and attention modules (ATB), which respectively bear the calculation of a linear layer and dot product attention in the Transformer, and these modules can be executed in parallel; finally, connect multiple LBs and ATBs using internal data streams, and the connections between these LBs and ATBs can be reconstructed to form two stages: the multi-head attention stage and the feedforward neural network stage. The two stages serially complete the calculation of the multi-head attention and feedforward neural network of a Transformer, and finally realize the calculation of the entire encoder / decoder layer, thereby accelerating the reasoning speed of the upper model. This method can be divided into the following five steps:

[0006] (1) Transformer model load analysis

[0007] The research object of this invention is the Transformer model accelerator. Firstly, the scale of matrix multiplication operators required for the operation of the Transformer model is extracted. According to the AIE of Versal ACAP and its peripheral hardware performance, the type and scale of matrix multiplication that AIE needs to undertake are determined to ensure that AIE can be fully utilized.

[0008] (2) Deployment of matrix multiplication processing units in the AIE array

[0009] For the designed AIE matrix multiplication processing unit, the AIE core is developed using AIE Intrinsics to ensure that the pipeline can be formed smoothly inside the AIE, with 1 cycle as the iteration cycle of the iterative pipeline. After that, multiple AIE cores are connected through AIE Graph, and at the same time, the high reuse of PLIO port data is ensured to meet the data requirements of a large number of AIE cores with limited PLIO ports.

[0010] (3) Design PL data transmitter and receiver, and design nonlinear operator hardware module

[0011] On the PL side, multiple matrix multiplication processing units in the computing engine are equipped with dedicated data transmitters and receivers to ensure that the data exchange speed between the PL-side data engine and the computing engine does not become a performance bottleneck. Then, nonlinear operators are developed to ensure that the processing speed of the modules on the PL side does not become a performance bottleneck in the pipeline, thereby maximizing the performance of the hardware.

[0012] (4) Combining hardware modules into linear layer modules and attention modules

[0013] Based on the developed modules, they are combined into a variety of linear layer modules and attention modules. At the same time, they are connected using internal data streams to form a complete multi-head attention stage and feedforward neural network stage, preparing for the combination of the final accelerator architecture.

[0014] (5) Scheduling the multi-head attention stage and the feedforward neural network stage

[0015] Allocate hardware resources to the two stages so that each stage has all the computing resources and storage resources when executing. After scheduling is completed, the output of a Transformer encoder / decoder layer is finally obtained. After multiple calls on the HOST side, the reasoning acceleration of the entire model is completed.

[0016] Compared with the prior art, the present invention has the following obvious advantages and beneficial effects:

[0017] First, compared with the traditional hardware-implemented Transformer accelerator, the present invention makes full use of the hardware customizability of Versal ACAP, implements the Transformer inference accelerator deployment architecture, and greatly improves the availability of Versal ACAP.

[0018] Secondly, the accelerator developed based on the proposed Versal ACAP-based efficient Transformer model inference accelerator deployment method effectively improves the model throughput and reduces inference latency. At the same time, this accelerator has great advantages in batch inference.

[0019] Finally, the present invention makes full use of optimization methods such as pipeline parallel strategy, multi-channel data transmitter, shared on-chip cache and AIE manual layout strategy to improve the overall performance of the system and maximize the utilization of hardware resources.

[0020] Experiments have confirmed that when the present invention accelerates the ViT-Base model, under the conditions of AIE of 1.33GHZ and PL of 300MHZ frequency, the throughput of the multi-head attention stage is 40.24TOPS, the throughput of the feedforward neural network stage is 29.87TOPS, the overall system throughput is 30.28TOPS, the power consumption is 62.865W, and the energy efficiency is 481.67GOPS / W. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Design schematics for various matrix multiplication processing units.

[0022] Figure 2 Schematic diagram of the AIE array layout.

[0023] Figure 3 Detailed architecture diagram of the accelerator.

[0024] Figure 4 This is a schematic diagram of the top-level architecture of the accelerator. DETAILED DESCRIPTION

[0025] According to the above description, the following is a specific implementation process, but the scope of protection of this patent is not limited to this implementation process.

[0026] Step 1: Transformer model load analysis

[0027] Step 1.1: Extract the size of the matrix multiplications required to run the Transformer model

[0028] The configuration information of the Transformer model includes the number of heads (Head), embedding vector dimension (Embed_dim) and FFN hidden layer dimension (Dff), input sequence length (L), and number of layers (Layer). In the inference process, five types of matrix multiplications are involved: [L×Embed_dim×Embed_dim], [L×Embed_dim / Head×L], [L×L×Embed_dim / Head], [L×Embed_dim×Dff], and [L×Dff×Embed_dim].

[0029] Step 1.2: Determine the maximum PU size that AIE can support without sacrificing performance

[0030] There are 400 AIE cores on the VCK5000 board of the Versal ACAP architecture. Under the Int8 data type, the load borne by each AIE single core is [64×64×64]. The maximum matrix size that AIE can process at one time without losing performance is [256×256×256]. 64 AIE cores are organized in the PU, and the accumulation method of block matrix multiplication is used. The calculation time of the PU is the same as that of the AIE single core.

[0031] Step 1.3: Determine the type and size of PU

[0032] Various matrix multiplication processing unit designs are shown in the attached Figure 1 As shown, the accelerator contains three types of PUs:

[0033] Large PU, undertakes 256x256x256 scale matrix multiplication, consumes 64AIE cores, undertakes [L×Embed_dim×Embed_dim], [L×Embed_dim×Dff]

[0034] With three types of loads [L×Dff×Embed_dim]; 4 groups are deployed.

[0035] Standard PU, undertakes 128x256x128 scale matrix multiplication, consumes 16AIE cores, and bears [L×Embed_dim / Head×L] load; 4 groups are deployed.

[0036] Small PU, responsible for 64x64x256 scale matrix multiplication, consuming 4AIE cores,

[0037] Bears [L × L × Embed_dim / Head] load; deploys 8 groups.

[0038] Step 2: Deploy the multiplication processing unit in the AIE array

[0039] Step 2.1: Develop AIE single core based on AI Engine Intrinsics

[0040] Each AIE core uses the AIE Intrinsics instruction to implement 64x64x64 matrix multiplication operations in the AIE core using Gustavson's matrix multiplication method. The vector processor in the AIE core is used to implement the calculation, and an instruction pipeline with an iteration interval of 1 is formed internally. The matrix data is stored through a 4KB AIE window.

[0041] Step 2.2: Organize multiple AIE cores based on AIE Graph

[0042] AIE Graph is used to configure the connection between multiple AIE cores and storage space. The internal AIEwindow is used for data transmission between multiple AIE cores. The cascaded data stream is used to complete the data accumulation between multiple cores. The maximum cascade level is 4. PLIO is used to complete the data input and output of PU. Packet switching and broadcasting are used to improve the multiplexing rate of PLIO port data. The maximum number of packet switching paths is 4.

[0043] Large PU, 16 groups of cascaded cores with a depth of 4, each group of cascades outputs one result, AIE cascade data flow is used between cascaded cores, and the input and output matrices are stored in 4KB AIE window.

[0044] Standard PU, 4 groups of cascaded cores with a depth of 4, each group of cascaded cores outputs one result, AIE cascade data flow is used between cascaded cores, and the input and output matrices are stored in 4KB AIEwindow.

[0045] Small PU, 4 groups of single cores without cascade accumulation, each core outputs one result, and the input and output matrices are stored in a 4KB AIE window.

[0046] Step 2.3: Lay out multiple cores in a two-dimensional AIE array

[0047] The AIE array layout is shown in the attached Figure 2 As shown, the two-dimensional AIE array of VCK5000 is arranged in an 8x50 space, using location <kernel>The instruction arranges and restricts the position of each AIE in the entire AIE array. The 64 AIE cores of the large PU adopt a 4x16 layout, the 16 AIE cores of the standard PU adopt a 4x4 layout, and the 4 AIE cores of the small PU adopt a 1x4 layout.

[0048] Step 3: Design the PL data transmitter and receiver, and design the nonlinear operator hardware module.

[0049] Step 3.1: Data transmitter and receiver serving PU

[0050] The data transmitter and receiver use PLIO to complete the data exchange between PL and AIE. PU adopts the packet switching strategy on the PLIO port. One PLIO is responsible for the transmission of multiple matrices. In this step, N is the number of packet switching paths. When packet switching is adopted, N is usually 4.

[0051] The workflow of a single PLIO data transmitter is as follows:

[0052] 1. Read a 128-bit data packet from the on-chip memory and unpack it in 32-bit units.

[0053] 2. Shift the data backward by 32 bits, add a header to the first 32 bits, and save the last 32 bits.

[0054] 3. Send the 128-bit data packet with the header added to PLIO.

[0055] 4. Read the new 128-bit data packet, use the last 32 bits saved in step 2 as the next packet header, and repeat this step 255 times to complete the sending of a 64x64 matrix.

[0056] 5. Repeat steps 1, 2, 3, and 4 for a total of N-1 times, with the packet header ID increasing each time.

[0057] Complete the packet-switched transmission of N matrix data.

[0058] The workflow of a single PLIO data receiver is as follows:

[0059] 1. Receive a 128-bit data packet from AIE and decompress it in 32-bit units.

[0060] 2. Take the first 32 bits as the package ID, determine the storage location, and save the last 96 bits separately.

[0061] 3. Read the new 128-bit data packet, save the last 96 bits, combine the first 32 bits of this data packet with the last 96 bits saved in step 2 into a 128-bit data packet for storage, and repeat this step 255 times to complete the reception of a 64x64 matrix.

[0062] 4. Repeat steps 1, 2, 3, and 4 a total of N-1 times to complete the packet-switched transmission of N matrix data.

[0063] For different PUs, different data transmitter and receiver configurations are used. The data exchange strategies are as follows:

[0064] Large PU, 8-way input PLIO (packet switching number = 4), 16-way output PLIO (no packet switching), input matrix A [256x256] is divided into 4x4 matrix blocks, and 4 rows of matrix blocks are sent in parallel through 4 PLIOs; input matrix B [256x256] is divided into 4x4 matrix blocks, and 4 columns of matrix blocks are sent in parallel through 4 PLIOs; output matrix C [256x256]

[0065] It is divided into 4x4 matrix blocks and transmitted in parallel through 16 output PLIOs.

[0066] Standard PU, 4 input PLIOs (packet switching paths = 4), 1 output PLIO (packet switching paths = 4), input matrix A [128x256] is divided into 2x4 matrix blocks, and 2 row matrix blocks are sent in parallel through 2 PLIOs; input matrix B [256x128] is divided into 4x2 matrix blocks, and 2 column matrix blocks are sent in parallel through 2 PLIOs; output matrix C [128x128] is divided into 2x2 matrix blocks and transmitted through 1 output PLIO.

[0067] Small PU, 2 input PLIOs (broadcast, packet switching = 4), 1 output PLIO (packet switching = 4), input matrix A [64x64], input through 1 PLIO and broadcast to 4 cores; input matrix B [64x256] is divided into 1x4 matrix blocks and sent through 1 PLIO; output matrix C [64x256] is divided into 1x4 matrix blocks,

[0068] Transmitted via 1 output PLIO.

[0069] When the data transmitter and receiver work, they are parallelized in units of PLIO, that is, all PLIOs can exchange data simultaneously at one moment.

[0070] Step 3.2: Hardware modules for Softmax, Layernorm, GELU and matrix transposition operators

[0071] Step 3.2.1: Softmax PL-side hardware module

[0072] The calculation performed by the Softmax module is shown in Formula 1:

[0073]

[0074] In the formula, x i is the i-th element of the input vector, x k is the kth element of the input vector, y i The i-th element of the output vector, M is the vector length, which corresponds to the input sequence length L of the Transformer in the dot product attention calculation.

[0075] The Softmax module uses 64 parallelism, that is, it operates on 64 rows of data at a time, and the input and output data are packed and unpacked in units of 64x64 matrix blocks.

[0076] Step 3.2.2: Layernorm PL-side hardware module

[0077] The calculation performed by the Layernorm module is shown in Formula 2:

[0078]

[0079] Where, L x is the input vector, L y is the output vector, L x , L y The vector dimension corresponds to the embedding vector dimension (Embed_dim) of Transformer. γ and β are fixed parameters of the model. γ and β are included in the parameters of the pre-trained model. Different models have different values ​​of γ and β. They need to be extracted from the parameters of the upper model to be accelerated. ∈ is a constant with a value of 1e-5.

[0080] The Layernorm module uses 64 degrees of parallelism, that is, it normalizes 64 rows of data at a time, and the input and output data are packed and unpacked in units of 64x64 matrix blocks.

[0081] Step 3.2.3: PL-side hardware module of GELU

[0082] The calculation performed by the GELU module is shown in Formula 3:

[0083]

[0084] In the formula, G x is a single element of the input, G y is a single element of the output.

[0085] The GELU operation is only valid for a single element, so it is embedded at the end of the data receiver and coupled with the data operation, and is not planned as a separate module.

[0086] Step 3.2.4: PL-side hardware module for matrix transposition

[0087] This module uses the standard implementation method of matrix transposition, but it needs to add data unpacking operation before that, disassemble the 128-bit data packet into 16 In8 data and then transpose it, and then pack the data again when outputting it, restoring it to a 128-bit data packet. The parallelism of the matrix transposition module is 4, parallelizing the transposition operation of 4 matrices.

[0088] Step 4: Combine the designed hardware modules into linear layer modules and attention modules.

[0089] Step 4.1: Design the linear layer module

[0090] The accelerator has four linear layer modules: QKV_LB, Proj_LB, FFN1_LB, and FFN2_LB. The detailed design is as follows:

[0091] QKV_LB: It consists of three hardware parts: data transmitter, three large PUs, and data receiver, which form a pipeline parallel.

[0092] ●Proj_LB: It consists of three hardware parts: data fusion, data transmitter, a large PU, and data receiver, which form a pipeline parallel. The data fusion is used to collect the calculation results of multiple attention modules.

[0093] ●FFN1_LB: It consists of four hardware parts: data transmitter, two large PUs, data receiver, and GELU module, which form a parallel pipeline.

[0094] FFN2_LB: It consists of three hardware parts: a data transmitter, two large PUs, and a data receiver, which form a pipeline parallel.

[0095] Iter PU iteration times of QKV_LB and Proj_LB Type1 Set to:

[0096]

[0097] FFN1_LB and FFN2_LB PU iteration times Iter Type2 Set to:

[0098]

[0099] In Equations 4 and 5, L, Embed_dim, and Dff correspond to the sequence length, embedding vector dimension, and hidden layer dimension of the Transformer, respectively; Iter is the required number of iterations.

[0100] Step 4.2: Design the attention module

[0101] Each attention module (ATB) in the accelerator is able to calculate single-head attention (dot product attention), with the input being the three QKV matrices from QKV_LB and the output being the attention result. ATB has two parts inside:

[0102] Part 1: It consists of a QKV storage buffer, a matrix transposition module, a data transmitter, a standard PU, and a data receiver, and forms a pipeline parallelism.

[0103] Part 2: It consists of a Softmax module, a data transmitter, two small PUs, and a data receiver, and forms a pipeline in parallel.

[0104] At the same time, part 1 and part 2 also form a parallel assembly line organization.

[0105] Four groups of such attention modules are deployed in the accelerator to parallelize the attention calculation. The number of iterations of all standard PUs in ATB is Iter Type3 Set to:

[0106]

[0107] Iter Iter of all small PUs in ATB Type4 Set to:

[0108]

[0109] In Equations 6 and 7, L, Embed_dim, and Head correspond to the sequence length, embedding vector dimension, and number of attention heads of Transformer, respectively; Iter is the required number of iterations.

[0110] Step 5: Multi-head attention stage and feedforward neural network stage, and schedule the two to complete the calculation of the encoder / decoder layer.

[0111] Step 5.1: Design the multi-head attention stage based on LB and ATB.

[0112] The detailed architecture of the accelerator is as follows Figure 3 As shown, in the multi-head attention stage, the above-mentioned multiple LBs and ATBs are combined to complete the calculation of multi-head attention. LBs and ATBs are all parallel modules, taking the BERT model as an example.

[0113] The operation process of the multi-head attention stage is as follows:

[0114] 1. First, QKV_LB calculates the QKV matrix and outputs three QKV matrices of size 256x64 to ATB in units of the data required by 4 attention heads.

[0115] 2.4 ATBs start to execute, receive the QKV matrix, use the transpose module to transpose K, and then input it into two small PUs to obtain a 256x256 matrix.

[0116] 3. The 256x256 matrix is ​​input into the Softmax module for processing and then sent to the standard PU. Finally, each ATB outputs a 256x64 matrix.

[0117] 4. Aggregate the calculation results of the 4 ATBs into a 256x256 matrix and send it to the large PU of Proj_LB for calculation to obtain one third of the final result.

[0118] 5. Repeat steps 2, 3, 4, and 5 three times to get the complete final output of multi-head attention, and finally pass it through the Layernorm module to get the final output result.

[0119] Step 5.2: Design the feedforward neural network stage based on LB.

[0120] The stage contains two LBs executing in parallel. The operation process of the feedforward neural network stage is as follows:

[0121] 1. Read the results of the multi-head attention stage and input them into the two large PUs of FFN1_LB for calculation. After the results are obtained, they are immediately input into FFN2_LB.

[0122] 2. After the two large PUs of FFN2_LB complete the calculation, they output the calculation results.

[0123] 3. Finally, pass the output of FFN2_LB through the Layernorm module to get the final result.

[0124] Step 5.3: Resource allocation and scheduling of the multi-head attention stage and the feedforward neural network stage

[0125] The two stages share the computing and storage resources on the chip and are executed serially. Each stage has all the computing and storage resources when executing. After the feedforward neural network stage is executed, the output of a Transformer encoder / decoder layer is obtained. After calling Layer on the HOST side, the reasoning of the entire model is completed. The top-level architecture diagram of the accelerator is as follows: Figure 4 shown.

[0126] The customized accelerator implements the Transformer encoder or decoder as a whole in the Versal ACAP architecture, so that a single call from the HOST can complete the execution of the Transformer encoder layer or decoder layer in hardware.< / kernel>

Claims

1. A method for deploying an efficient Transformer model inference accelerator based on Versal ACAP, characterized by: First, we extract the scale of matrix multiplication operators required for the Transformer model during operation. Based on the AIE of the Versal ACAP and the performance of its peripheral hardware, we determine the type and scale of matrix multiplication that the AIE needs to undertake. Secondly, we design and actually deploy multiple matrix multiplication processing units (PUs) to the AIE array to undertake the matrix multiplication load in the Transformer. Then, we implement the matrix multiplication operation in the Versal ACAP. The PL side of ACAP is developed as a data transmitter and data receiver for PU services. At the same time, hardware modules of Softmax, Layernorm, GELU and matrix transposition operators are developed to bear the remaining load except matrix multiplication. Then, the developed hardware modules are combined into linear layer modules (LB) and attention modules (ATB), which respectively bear the calculation of a linear layer and dot product attention in Transformer. These modules can be executed in parallel. Finally, multiple LBs and ATBs are connected using internal data streams. The connections between these LBs and ATBs can be reconstructed to form two stages: the multi-head attention stage and the feedforward neural network stage. The two stages serially complete the calculation of the multi-head attention and feedforward neural network of a Transformer, and finally realize the calculation of the entire encoder / decoder layer, thereby accelerating the inference speed of the upper-level model.

2. The method according to claim 1, characterized in that: Step 1: Transformer model load analysis Step 1.1: Extract the size of the matrix multiplications required to run the Transformer model The configuration information of the Transformer model includes the number of heads (Head), embedding vector dimension (Embed_dim) and FFN hidden layer dimension (Dff), input sequence length (L), and number of layers (Layer). In the inference process, five scales of matrix multiplication are involved: [L×Embed_dim×Embed_dim], [L×Embed_dim / Head×L], [L×L×Embed_dim / Head], [L×Embed_dim×Dff], and [L×Dff×Embed_dim]. Step 1.2: Determine the maximum PU size that AIE can support without sacrificing performance There are 400 AIE cores on the VCK5000 board of the Versal ACAP architecture. Under the Int8 data type, the load borne by each AIE single core is [64×64×64]. The maximum matrix size that AIE can process at one time without losing performance is [256×256×256]. 64 AIE cores are organized in the PU, and the accumulation method of block matrix multiplication is used. The calculation time of the PU is the same as that of the AIE single core. Step 1.3: Determine the type and size of PU The accelerator contains three types of PUs: 1) Large PU, undertakes 256x256x256 scale matrix multiplication, consumes 64AIE cores, Bear three types of loads: [L×Embed_dim×Embed_dim], [L×Embed_dim×Dff] and [L×Dff×Embed_dim]; deploy 4 groups; 2) Standard PU, undertakes 128x256x128 scale matrix multiplication, consumes 16AIE cores, Bear [L×Embed_dim / Head×L] load; deploy 4 groups; 3) Small PU, which performs 64x64x256 matrix multiplication, consumes 4AIE cores, and bears a load of [L×L×Embed_dim / Head]. Eight groups are deployed.

3. The method according to claim 1, characterized in that: Step 2: Deploy the matrix multiplication processing unit in the AIE array Step 2.1: Develop AIE single core based on AI Engine Intrinsics Each AIE core uses the AIE Intrinsics instruction to implement 64x64x64 matrix multiplication operations in the AIE core using Gustavson's matrix multiplication method. The vector processor in the AIE core is used to implement the calculation, and an instruction pipeline with an iteration interval of 1 is formed internally. The matrix data is stored through a 4KB AIE window. Step 2.2: Organize multiple AIE cores based on AIE Graph AIE Graph is used to configure the connection between multiple AIE cores and storage space. The internal AIEwindow is used for data transmission between multiple AIE cores. The cascaded data stream is used to complete the data accumulation between multiple cores. The maximum cascade level is 4. PLIO is used to complete the data input and output of PU. Packet switching and broadcasting are used to improve the multiplexing rate of PLIO port data. The maximum number of packet switching paths is 4. 1) Large PU, 16 groups of cascaded cores with a depth of 4, each group of cascades outputs one result, AIE cascade data flow is used between cascade cores, and 4KB AIEwindow storage is used for input and output matrices; 2) Standard PU, 4 groups of cascaded cores with a depth of 4, each group of cascaded cores outputs one result, AIE cascade data flow is used between cascaded cores, and the input and output matrices are stored using 4KB AIEwindow; 3) Small PU, 4 groups of single cores without cascade accumulation, each core outputs one result, The input and output matrices are stored using a 4KB AIE window; Step 2.3: Arrange multiple cores in a two-dimensional AIE array The two-dimensional AIE array of VCK5000 is arranged in 8x50 space, using location <kernel> The instruction arranges and restricts the position of each AIE in the entire AIE array. The 64 AIE cores of the large PU adopt a 4x16 layout, the 16 AIE cores of the standard PU adopt a 4x4 layout, and the 4 AIE cores of the small PU adopt a 1x4 layout.< / kernel> 4. The method according to claim 1, characterized in that: Step 3: Design the PL data transmitter and receiver, and design the nonlinear operator hardware module; Step 3.1: Data transmitter and receiver serving PU The data transmitter and receiver use PLIO to complete the data exchange between PL and AIE. PU adopts the packet switching strategy on the PLIO port. One PLIO is responsible for the transmission of multiple matrices. In this step, N is the number of packet switching paths. When the packet switching method is adopted, N is usually 4. The workflow of a single PLIO data transmitter is as follows: 1) Read a 128-bit data packet from the on-chip storage and unpack it in 32-bit units; 2) Shift the data backward by 32 bits, add a header to the first 32 bits, and save the last 32 bits; 3) Send the 128-bit data packet with the header added to PLIO; 4) Read the new 128-bit data packet, use the last 32 bits saved in step 2 as the next packet header, and repeat this step 255 times to complete the sending of a 64x64 matrix; 5) Repeat steps 1, 2, 3, and 4 for a total of N-1 times, with the packet header ID increasing each time, to complete the packet switching of N matrix data; The workflow of a single PLIO data receiver is as follows: 1) Receive a 128-bit data packet from AIE and unpack it in 32-bit units; 2) Take the first 32 bits as the packet ID, determine the storage location, and save the last 96 bits separately; 3) Read a new 128-bit data packet, save the last 96 bits, combine the first 32 bits of this data packet with the last 96 bits saved in step 2 into a 128-bit data packet for storage, and repeat this step 255 times to complete the reception of a 64x64 matrix; 4) Repeat steps 1, 2, 3, and 4 a total of N-1 times to complete the packet-switched transmission of N matrix data; For different PUs, different data transmitter and receiver configurations are used. The data exchange strategies are as follows: 1) Large PU, 8-way input PLIO (packet switching number = 4), 16-way output PLIO (no packet switching), input matrix A [256x256] is divided into 4x4 matrix blocks, and 4 rows of matrix blocks are sent in parallel through 4 PLIOs; input matrix B [256x256] is divided into 4x4 matrix blocks, and 4 columns of matrix blocks are sent in parallel through 4 PLIOs; output matrix C [256x256] is divided into 4x4 matrix blocks, and transmitted in parallel through 16 output PLIOs; 2) Standard PU, 4 input PLIOs (packet switching paths = 4), 1 output PLIO (packet switching paths = 4), input matrix A [128x256] is divided into 2x4 matrix blocks, and 2 row matrix blocks are sent in parallel through 2 PLIOs; input matrix B [256x128] is divided into 4x2 matrix blocks, and 2 column matrix blocks are sent in parallel through 2 PLIOs; output matrix C [128x128] is divided into 2x2 matrix blocks and transmitted through 1 output PLIO; 3) Small PU, 2 input PLIOs (broadcast, packet switching number = 4), 1 output PLIO (packet switching number = 4), input matrix A [64x64], input through 1 PLIO and broadcast to 4 cores; input matrix B [64x256] is divided into 1x4 matrix blocks and sent through 1 PLIO; output matrix C [64x256] is divided into 1x4 matrix blocks and transmitted through 1 output PLIO; When the data transmitter and receiver work, they are parallelized in PLIO units, that is, all PLIOs can exchange data at the same time; Step 3.2: Hardware modules for Softmax, Layernorm, GELU and matrix transpose operators Step 3.2.1: PL-side hardware module for Softmax The calculation performed by the Softmax module is shown in Formula 1: In the formula, x i is the i-th element of the input vector, x k is the kth element of the input vector, y i The i-th element of the output vector, M is the vector length, which corresponds to the input sequence length L of the Transformer in the dot product attention calculation; The Softmax module uses 64 degrees of parallelism, that is, it operates on 64 rows of data at a time, and the input and output data are packed and unpacked in units of 64x64 matrix blocks; Step 3.2.2: Layernorm PL-side hardware module The calculation performed by the Layernorm module is shown in Formula 2: Where, L x is the input vector, L y is the output vector, L x , L y The vector dimension corresponds to the embedding vector dimension (Embed_dim) of Transformer. γ and β are fixed parameters of the model. γ and β are included in the parameters of the pre-trained model. Different models have different values ​​of γ and β. They need to be extracted from the parameters of the upper model to be accelerated. ∈ is a constant of 1e-5. The Layernorm module uses 64 degrees of parallelism, that is, 64 rows of data are normalized at a time, and the input and output data are packed and unpacked in units of 64x64 matrix blocks; Step 3.2.3: PL-side hardware module of GELU The calculation performed by the GELU module is shown in Formula 3: In the formula, G x is a single element of the input, G y is a single element of the output; The GELU operation is only valid for a single element, so it is embedded at the end of the data receiver and coupled with the data operation, and is not planned as a separate module; Step 3.2.4: PL-side hardware module for matrix transposition This module uses the standard implementation method of matrix transposition, but it needs to add data unpacking operation before that. The 128-bit data packet is disassembled into 16 In8 data and then transposed. At the same time, the data is packaged again when output and restored to a 128-bit data packet. The parallelism of the matrix transposition module is 4, which parallelizes the transposition operation of 4 matrices.

5. The method according to claim 1, characterized in that: Step 4: Combine the designed hardware modules into a linear layer module and an attention module; Step 4.1: Design the linear layer module The accelerator has four linear layer modules: QKV_LB, Proj_LB, FFN1_LB, and FFN2_LB. The detailed design is as follows: 1) QKV_LB: It consists of three hardware parts: a data transmitter, three large PUs, and a data receiver, which form a pipeline parallel; 2) Proj_LB: It consists of three hardware parts: data fusion, data transmitter, a large PU, and data receiver, which form a pipeline parallel. The data fusion is used to collect the calculation results of multiple attention modules. 3) FFN1_LB: It consists of four hardware parts: data transmitter, two large PUs, data receiver, and GELU module, which form a pipeline parallel; 4) FFN2_LB: It consists of three hardware parts: a data transmitter, two large PUs, and a data receiver, which form a pipeline parallel; Iter PU iteration times of QKV_LB and Proj_LB Type1 Set to: FFN1_LB and FFN2_LB PU iteration times Iter Type2 Set to: In Equations 4 and 5, L, Embed_dim, and Dff correspond to the sequence length, embedding vector dimension, and hidden layer dimension of the Transformer, respectively; Iter is the number of required iterations; Step 4.2: Design the attention module Each attention module (ATB) in the accelerator is able to calculate single-head attention (dot product attention), with the input being the three QKV matrices from QKV_LB and the output being the attention result. ATB has two parts inside: 1) Part 1: consists of a QKV storage buffer, a matrix transposition module, a data transmitter, a standard PU, and a data receiver, and forms a pipeline parallelism; 2) Part 2: It consists of a Softmax module, a data transmitter, two small PUs, and a data receiver, and forms a pipeline in parallel; At the same time, part 1 and part 2 also form a parallel assembly line organization; Four groups of such attention modules are deployed in the accelerator to parallelize the calculation of attention; the number of iterations of all standard PUs in ATB is Iter Type3 Set to: Iter Iter of all small PUs in ATB Type4 Set to: In Equations 6 and 7, L, Embed_dim, and Head correspond to the sequence length, embedding vector dimension, and number of attention heads of Transformer, respectively; Iter is the required number of iterations.

6. The method according to claim 1, characterized in that: Step 5: Multi-head attention stage and feedforward neural network stage, and schedule the two to complete the calculation of the encoder / decoder layer; Step 5.1: Design a multi-head attention stage based on LB and ATB; In the multi-head attention stage, the above-mentioned LBs and ATBs are combined to complete the calculation of multi-head attention. LBs and ATBs are all parallel modules, taking the BERT model as an example; The operation process of the multi-head attention stage is as follows: 1) First, QKV_LB calculates the QKV matrix and outputs three QKV matrices of size 256x64 to ATB in units of the data required by 4 attention heads; 2) 4 ATBs start to execute, receive the QKV matrix, and use the transpose module to transpose K. After inputting into two small PUs, a 256x256 matrix is ​​obtained; 3) The 256x256 matrix is ​​input into the Softmax module for processing and then sent to the standard PU. Finally, each ATB outputs a 256x64 matrix; 4) Aggregate the calculation results of the four ATBs into a 256x256 matrix and send it to the large PU of Proj_LB for calculation to obtain one third of the final result; 5) Repeat steps 2, 3, 4, and 5 three times to get the complete final output of the multi-head attention, and finally pass it through the Layernorm module to get the final output result; Step 5.2: Design the feedforward neural network stage based on LB; The stage contains two LBs executing in parallel. The operation process of the feedforward neural network stage is as follows: 1) Read the results of the multi-head attention stage and input them into the two large PUs of FFN1_LB for calculation. After the results are obtained, they are immediately input into FFN2_LB; 2) After the two large PUs of FFN2_LB complete the calculation, they output the calculation results; 3) Finally, the output of FFN2_LB is passed through the Layernorm module to obtain the final result; Step 5.3: Resource allocation and scheduling of the multi-head attention stage and the feedforward neural network stage The two stages share the computing and storage resources on the chip and are executed serially. Each stage has all the computing and storage resources during execution. After the feedforward neural network stage is completed, the output of a Transformer encoder / decoder layer is obtained; after calling Layer times on the HOST side, the reasoning of the entire model is completed.

Images