Operation acceleration method and device of GPT model based on reconfigurable cluster and accelerator

Abstract

The invention discloses an operation acceleration method, device and accelerator for a GPT model based on a reconfigurable cluster, and the method comprises the steps: dividing the weight parameters of a target level of the GPT model to each board card before the GPT model is operated; constructing a matrix processing unit and a vector function unit as operation cores of all board cards; in the operation process of the GPT model, parameter fragmentation is executed according to the operation core of the target hierarchy, and operation of the target hierarchy is executed; and obtaining a target parameter through data synchronization between the board cards so as to complete the operation of the target hierarchy. It can be seen that the weight parameters of the target hierarchy of the GPT model can be divided to each board card, operation cores of different hierarchies can be constructed, operation is executed according to the fragmented parameters in the operation process, the target parameters are obtained through data synchronization to complete operation, computing resources of a reconfigurable cluster can be fully utilized, and the computing efficiency of the reconfigurable cluster is improved. And the operation efficiency of the GPT model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning, and in particular, to an operation acceleration method, device, and accelerator for a GPT model based on a reconfigurable cluster. Background Art

[0002] In recent years, large language models based on Transformer have had a profound impact on both academic research and daily life. Among them, the GPT series of models has become a research hotspot in the field of artificial intelligence. The GPT model can achieve excellent results in complex natural language processing tasks such as scientific writing, machine translation, and text classification. These advantages come from the architectural innovation brought by the self-attention mechanism on the one hand, and the large number of model parameters and training data on the other hand, both of which also bring huge computational and storage overheads to the deployment platform.

[0003] The main framework of the GPT model consists of a decoder from Transformer. Its inference process can be divided into two stages: prefill and decode. In the prefill stage, the model processes all input tokens from the user at once and obtains a single output token after the calculation is completed. The decode stage iteratively runs the entire model to generate all subsequent tokens, where each iteration takes the single output token from the previous iteration as the input for this iteration and generates a single output token. The model iteration continues until the end token is output. Among them, the main computational load in the prefill stage of the GPT model is matrix multiplication, and the main computational load in the decode stage of the GPT model is vector multiplication by matrix.

[0004] Nowadays, data centers mainly use GPU clusters to complete the deployment of the GPT model to provide services related to natural language processing tasks to a large number of users. However, the following potential problems will lead to low utilization of the GPU cluster. First, the decode stage of the GPT model needs to process individual input tokens sequentially and frequently. Second, in order to improve the service quality of the data center and meet the real-time requirements, the acceleration cluster does not process small-batch user inputs by waiting for more data, so the cluster usually sets the batch size for processing to 1. Generally speaking, the scenario of deploying the GPT model in the data center will generate a large demand for processing small batches of inputs, while the design concept of the GPU architecture aims to use data parallelism to process large batches of input data. Therefore, in this scenario, the GPU cluster will face the problem of insufficient computational intensity, which can be alleviated by customizing an acceleration framework for small-batch inputs on a reconfigurable board.

[0005] Reconfigurable boards represented by FPGAs have the advantages of short development cycles, low latency, and high energy efficiency, and are widely used in deploying various neural network accelerators. In the past, large language model accelerators based on reconfigurable boards were usually only deployed on a single board. Using these accelerators to host the entire GPT model will face serious storage constraints. Taking the GPT-3 model as an example, it has more than 175 billion parameters. However, advanced data center reconfigurable boards, such as the U280 board of Xilinx, only have 32GB of DDR and 8GB of HBM storage space. It is difficult for a single U280 board to complete the inference task of the GPT-3 model, and there is no standardized acceleration method for the GPT model based on a reconfigurable board cluster in the industry. For example, DFX published by the Korea Advanced Institute of Science and Technology in 2022 has problems in both the acceleration core design and the synchronization strategy, and cannot fully utilize the computing resources of the reconfigurable board cluster, resulting in low computing efficiency. Therefore, it is particularly important to provide an operation acceleration method for the GPT model based on a reconfigurable board cluster to improve the operation efficiency of the GPT model. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide an operation acceleration method for the GPT model based on a reconfigurable cluster and a GPT accelerator, which can improve the operation efficiency of the GPT model.

[0007] To solve the above technical problems, the first aspect of the present invention discloses an operation acceleration method for the GPT model based on a reconfigurable cluster, which is characterized in that the method includes:

[0008] Before the GPT model operation, divide the weight parameters of the target layer of the GPT model onto each board in the board cluster; at least two boards are included in the board cluster; the target layer includes a vector function layer and a matrix multiplication layer; the vector function layer includes a LayerNorm layer, and the matrix multiplication layer includes a QKV generation layer and all fully connected layers;

[0009] Construct a matrix processing unit and a vector function unit as the operation cores of all the boards; the operation cores are used to execute the operation of the GPT model; wherein, the matrix processing unit is used for matrix operations in the matrix multiplication layer, and the vector function unit is used for vector operations of the LayerNorm layer and activation functions; the operation types of the matrix processing unit include vector-matrix multiplication operations and matrix-matrix multiplication operations;

[0010] During the operation of the GPT model, for each of the target levels, when the operation core corresponding to the target level is the matrix processing unit, the input parameters and the weight parameters are sliced based on the operation type corresponding to the matrix processing unit of the target level, and the operation of the target level is performed based on the sliced input parameters and the sliced weight parameters; when the operation core corresponding to the target level is the vector function unit, the input parameters are sliced and the operation of the target level is performed based on the sliced input parameters.

[0011] For each of the target levels, target parameters are obtained through data synchronization operations between all the boards to complete the operation of the target level.

[0012] As an optional implementation manner, in the first aspect of the present invention, the dividing the weight parameters of the target level of the GPT model onto each board in the board cluster includes:

[0013] For the QKV generation layer, the weight parameters of the QKV generation layer are divided onto each board in the board cluster according to head averaging.

[0014] For the fully connected layer and the LayerNorm layer, the weight parameters of the fully connected layer and the LayerNorm layer are divided onto each board in the board cluster according to column averaging.

[0015] As an optional implementation manner, in the first aspect of the present invention, before slicing the input parameters and the weight parameters based on the operation type corresponding to the matrix processing unit of the target level, the method further includes:

[0016] Pack several rows of the input parameters into a group.

[0017] For the input parameters in different rows of the same group, they are input to the target level in a parallel manner.

[0018] For the input parameters in different groups, they are input to the target level using a pipelining algorithm.

[0019] As an optional implementation manner, in the first aspect of the present invention, the matrix processing unit is constructed by arranging several sub-processing units; each sub-processing unit includes a multiplier, an adder, and an accumulation register.

[0020] And, the slicing the input parameters based on the operation type corresponding to the matrix processing unit of the target level and slicing the weight parameters based on the weight slicing strategy includes:

[0021] When the operation type of the matrix processing unit is matrix multiplication, the input parameters in matrix form are sliced into a number of sub-input matrices, and the weight parameters in matrix form are sliced into a number of sub-weight matrices; wherein, the dimensions of the sub-input matrices and the sub-weight matrices are the same;

[0022] When the operation type of the matrix processing unit is vector-matrix multiplication, the input parameters in vector form are sliced into a number of sub-input vectors, and the weight parameters in matrix form are sliced into a number of sub-weight matrices; wherein, the length of the sub-input vectors is the same as the dimension of the sub-weight matrices.

[0023] As an optional implementation manner, in the first aspect of the present invention, the vector function unit is constructed by a summation module and a scaling module; each of the summation modules includes a number of accumulators and adders arranged in a binary tree form, and each accumulator includes an exponential operation unit, a square operation unit, an adder, and an accumulation register; the scaling module is used to perform multiplication, division, addition, and subtraction calculations between vectors;

[0024] And, when the operation core corresponding to the target level is the vector function unit, slicing the input parameters and performing the operation of the target level based on the sliced input parameters includes:

[0025] For each of the target levels, when the operation core corresponding to the target level is the vector function unit, the input parameters in matrix form of the target level are split into a number of target vectors row by row, and the operation of the target level is performed based on all the target vectors.

[0026] As an optional implementation manner, in the first aspect of the present invention, for each of the target levels, obtaining target parameters through data synchronization operations between all the boards to complete the operation of the target level includes:

[0027] For the LayerNorm layer in the vector function layer on each board, calculate the sum of each row of the input parameters according to the input parameters of each board to obtain the sum result of each row of the input parameters on each board; and determine the target parameters based on all the sum results, and synchronize the target parameters to all the other boards through broadcast communication; when each board receives the target parameters on all the other boards, determine the output parameters of the LayerNorm layer on this board based on all the sum results on this board and all the received target parameters;

[0028] For each matrix multiplication layer on each of the said boards, determine the input parameters and target parameters of each matrix multiplication layer based on the operation results before each matrix multiplication layer on the board, and synchronize the target parameters to all other boards through broadcast communication; when each board receives the target parameters from all other boards, determine the output parameters of the matrix multiplication layer on the board based on the input parameters of the matrix multiplication layer on the board and the received target parameters.

[0029] As an optional implementation manner, in the first aspect of the present invention, the method further includes:

[0030] Suspend the process for broadcast communication according to the completion time when the target parameters are synchronized to all other boards.

[0031] The second aspect of the present invention discloses an operation acceleration device for a GPT model based on a reconfigurable cluster, and the device includes:

[0032] A partitioning module, configured to partition the weight parameters of the target layer of the GPT model to each board in the board cluster before the operation of the GPT model; at least two boards are included in the board cluster; the target layer includes a vector function layer and a matrix multiplication layer; the vector function layer includes a LayerNorm layer, and the matrix multiplication layer includes a QKV generation layer and all fully connected layers;

[0033] A construction module, configured to construct a matrix processing unit and a vector function unit as the operation cores of all the boards; the operation cores are used to execute the operation of the GPT model; wherein, the matrix processing unit is used for matrix operations of the matrix multiplication layer, and the vector function unit is used for vector operations of the LayerNorm layer and activation functions; the operation types of the matrix processing unit include vector-matrix multiplication operations and matrix-matrix multiplication operations;

[0034] A sharding module, configured to, during the operation of the GPT model, for each target layer, when the operation core corresponding to the target layer is the matrix processing unit, shard the input parameters and the weight parameters based on the operation type corresponding to the matrix processing unit of the target layer, and execute the operation of the target layer based on the sharded input parameters and the sharded weight parameters; when the operation core corresponding to the target layer is the vector function unit, shard the input parameters and execute the operation of the target layer based on the sharded input parameters;

[0035] A synchronization module, configured to, for each target layer, obtain target parameters through data synchronization operations between all boards to complete the operation of the target layer.

[0036] As an alternative implementation, in the second aspect of the present invention, the partitioning module partitions the weight parameters of the target layer of the GPT model onto each board in the board cluster. The specific method includes:

[0037] For the QKV generation layer, the weight parameters of the QKV generation layer are partitioned onto each board in the board cluster according to head averaging;

[0038] For the fully connected layer and the LayerNorm layer, the weight parameters of the fully connected layer and the LayerNorm layer are partitioned onto each board in the board cluster according to column averaging.

[0039] As an alternative implementation, in the second aspect of the present invention, the device further includes:

[0040] A packing module, configured to pack several rows of the input parameters into a group before the sharding module shards the input parameters and the weight parameters according to the operation type corresponding to the matrix processing unit of the target layer;

[0041] An input module, configured to input different rows of the input parameters in the same group to the target layer in parallel; and input different groups of the input parameters to the target layer using a pipeline algorithm.

[0042] As an alternative implementation, in the second aspect of the present invention, the matrix processing unit is constructed by arranging several sub-processing units; each sub-processing unit includes a multiplier, an adder, and an accumulation register;

[0043] Moreover, the sharding module shards the input parameters according to the operation type corresponding to the matrix processing unit of the target layer, and shards the weight parameters according to the weight sharding strategy. The specific method includes:

[0044] When the operation type of the matrix processing unit is matrix multiplication, the input parameters in matrix form are sharded into several sub-input matrices, and the weight parameters in matrix form are sharded into several sub-weight matrices; wherein, the dimensions of the sub-input matrices and the sub-weight matrices are the same;

[0045] When the operation type of the matrix processing unit is vector multiplication, the input parameters in vector form are sharded into several sub-input vectors, and the weight parameters in matrix form are sharded into several sub-weight matrices; wherein, the length of the sub-input vectors is the same as the dimension of the sub-weight matrices.

[0046] As an alternative implementation, in the second aspect of the present invention, the vector function unit is constructed by a summation module and a scaling module; each of the summation modules includes a number of accumulators and adders arranged in a binary tree form, and each accumulator includes an exponentiation operator, a squaring operator, an adder, and an accumulation register; the scaling module is used to perform multiplication, division, addition, and subtraction calculations between vectors;

[0047] And, when the operation core corresponding to the target level is the vector function unit, the sharding module shards the input parameters and performs the operations at the target level based on the sharded input parameters. The specific method includes:

[0048] For each of the target levels, when the operation core corresponding to the target level is the vector function unit, the matrix-form input parameters at the target level are split into a number of target vectors row by row, and the operations at the target level are performed based on all the target vectors.

[0049] As an alternative implementation, in the second aspect of the present invention, the synchronization module includes:

[0050] A first determination sub-module, which is used for each LayerNorm layer in the vector function layer on each board card, calculates the sum of each row of the input parameters according to the input parameters of each board card to obtain the sum result of each row of the input parameters on each board card; and determines the target parameters based on all the sum results, and synchronizes the target parameters to all the other board cards through broadcast communication; when each board card receives the target parameters from all the other board cards, based on all the sum results on this board card and the received target parameters, determines the output parameters of the LayerNorm layer on this board card;

[0051] A second determination sub-module, which is used for each matrix multiplication layer on each board card, determines the input parameters and target parameters of each matrix multiplication layer based on the operation results before each matrix multiplication layer on this board card, and synchronizes the target parameters to all the other board cards through broadcast communication; when each board card receives the target parameters from all the other board cards, based on the input parameters of each matrix multiplication layer on this board card and the received target parameters, determines the output parameters of the matrix multiplication layer on this board card.

[0052] As an alternative implementation, in the second aspect of the present invention, the synchronization module further includes:

[0053] A suspension sub-module, which is used to suspend the process for broadcast communication according to the completion time when the target parameters are synchronized to all the other board cards.

[0054] The third aspect of the present invention discloses an accelerator for accelerating the operation of a GPT model based on a reconfigurable cluster, and the accelerator includes:

[0055] A memory storing executable program code;

[0056] A processor coupled to the memory;

[0057] The processor calls the executable program code stored in the memory and executes the operation acceleration method of the GPT model based on a reconfigurable cluster according to any one of claims 1-7.

[0058] The fourth aspect of the present invention discloses a computer-readable storage medium storing computer instructions, which are used to execute the operation acceleration method of the GPT model based on a reconfigurable cluster according to any one of claims 1-7 when the computer instructions are called.

[0059] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0060] Before the operation of the GPT model, the present invention divides the weight parameters of the target layer of the GPT model onto each board; constructs a matrix processing unit and a vector function unit as the operation cores of all boards; during the operation of the GPT model, parameter sharding is performed according to the operation cores of the target layer and the operation of the target layer is executed; the target parameters are obtained through data synchronization between the boards to complete the operation of the target layer. It can be seen that the present invention can divide the weight parameters of the target layer of the GPT model onto each board, construct operation cores of different layers, execute operations according to the sharded parameters during the operation, and obtain target parameters through data synchronization to complete the operation, which is beneficial to making full use of the computing resources of the reconfigurable cluster and improving the operation efficiency of the GPT model. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0062] Figure 1 is a flowchart of an operation acceleration method of a GPT model based on a reconfigurable cluster disclosed in an embodiment of the present invention;

[0063] Figure 2 is a GPT operation flowchart based on two boards of an operation acceleration method of a GPT model based on a reconfigurable cluster disclosed in an embodiment of the present invention;

[0064] Figure 3 It is the overall framework diagram of an operation acceleration method for a GPT model based on a reconfigurable cluster disclosed in an embodiment of the present invention;

[0065] Figure 4 It is the matrix multiplication matrix operation sharding strategy with a weight calculation storage ratio of d in an embodiment of the present invention;

[0066] Figure 5 It is the vector multiplication matrix operation sharding strategy in an embodiment of the present invention;

[0067] Figure 6 It is the LayerNorm layer synchronization strategy based on two boards in an embodiment of the present invention;

[0068] Figure 7 It is the fully connected layer synchronization strategy based on two boards in an embodiment of the present invention;

[0069] Figure 8 It is the structural schematic diagram of an operation acceleration device for a GPT model based on a reconfigurable cluster disclosed in an embodiment of the present invention;

[0070] Figure 9 It is the structural schematic diagram of an accelerator for operation acceleration of a GPT model based on a reconfigurable cluster disclosed in an embodiment of the present invention;

[0071] Figure 10 It is the structural schematic diagram of another operation acceleration device for a GPT model based on a reconfigurable cluster disclosed in an embodiment of the present invention;

[0072] Figure 11 It is the test example and configuration mode schematic diagram of the Aurora IP core;

[0073] Figure 12 It is the inference latency comparison diagram of DFX and the present invention for deploying the 345M GPT-2 model on a single U280 board and deploying the 774M GPT-2 model on two U280 boards;

[0074] Figure 13 It is the inference latency comparison diagram of the V100 GPU, DFX and the present invention for deploying the 345M GPT-2 model on a single board;

[0075] Figure 14 It is the inference latency comparison diagram of the V100 GPU, DFX and the present invention for deploying the 774M GPT-2 model on two boards. Detailed implementation manners

[0076] To enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative work fall within the protection scope of the present invention.

[0077] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, device, product or terminal comprising a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or terminals.

[0078] Referring to "embodiment" herein means that a specific feature, structure or characteristic described in connection with the embodiment may be included in at least one embodiment of the present invention. The phrase appears in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0079] The present invention discloses an operation acceleration method, device and accelerator for a GPT model based on a reconfigurable cluster, which can divide the weight parameters of the target layer of the GPT model onto each board, construct operation cores of different layers, perform operations according to the sharded parameters during the operation process, and obtain the target parameters through data synchronization to complete the operation, which is beneficial to making full use of the computing resources of the reconfigurable cluster and improving the operation efficiency of the GPT model. The following will be described in detail respectively.

[0080] Embodiment 1

[0081] Please refer to Figure 1 , Figure 1 , which is a schematic flowchart of an operation acceleration method for a GPT model based on a reconfigurable cluster disclosed in an embodiment of the present invention. Among them, Figure 1 The method shown can be applied to the operation scenario of a GPT model based on a reconfigurable cluster. The number of boards in the reconfigurable cluster can be two or more, and the embodiments of the present invention do not make limitations. As Figure 1 shown, taking two boards as an example, the operation acceleration method for the GPT model based on the reconfigurable cluster may include the following operations:

[0082] 101. Before the operation of the GPT model, divide the weight parameters of the target layer of the GPT model onto each board in the board cluster; the board cluster includes at least two boards; the target layer includes a vector function layer and a matrix multiplication layer; the vector function layer includes a LayerNorm layer, and the matrix multiplication layer includes a QKV generation layer and all fully connected layers;

[0083] In the embodiments of the present invention, it can be understood that the board cluster can be a reconfigurable board represented by an FPGA, and the board cluster can have multiple boards jointly deploy a large language model to accelerate model operations.

[0084] 102. Construct a matrix processing unit and a vector function unit as the operation cores of all boards; the operation cores are used to execute the operations of the GPT model; among them, the matrix processing unit is used for matrix operations of the matrix multiplication layer, and the vector function unit is used for vector operations of the LayerNorm layer and activation functions; the operation types of the matrix processing unit include vector-matrix multiplication operations and matrix-matrix multiplication operations;

[0085] In the embodiments of the present invention, the operation cores can use different sharding strategies according to the different operation characteristics of matrix-matrix multiplication and vector-matrix multiplication. Since the main computational load of the GPT model is matrix operations, the matrix processing unit of the present invention adopts the design concept of a cyclic architecture. By multiplexing the matrix processing unit, matrix-matrix multiplication operations and vector-matrix multiplication operations can be completed, and the inference task of the entire model can be completed in a low-latency form when the input data volume is small. As for the vector function unit, it is responsible for vector addition, accumulation, division, exponentiation and other vector-related operations. In addition, the activation function can be GELU, SoftMax, etc., and the embodiments of the present invention are not limited.

[0086] 102. During the operation of the GPT model, for each target layer, when the operation core corresponding to the target layer is a matrix processing unit, shard the input parameters and weight parameters based on the operation type corresponding to the matrix processing unit of the target layer, and execute the operation of the target layer based on the sharded input parameters and sharded weight parameters; when the operation core corresponding to the target layer is a vector function unit, shard the input parameters and execute the operation of the target layer based on the sharded input parameters;

[0087] In the embodiments of the present invention, it can be understood that the sharding strategy can improve the data reuse rate of the weight parameters and at the same time improve the computing intensity of the matrix processing unit;

[0088] 104. For each target layer, obtain the target parameters through data synchronization operations between all boards to complete the operation of the target layer.

[0089] In the embodiments of the present invention, it can be understood that although the deployment method of multi-card partitioning can effectively reduce the theoretical communication volume required during cluster synchronization, it will cause the following problems: After completing the operations of self-attention and fully connected layers, each board in the cluster will only have partial output parameters. In the case of asynchronous communication, based on these incomplete output parameters of their respective boards, each board in the cluster cannot independently complete the subsequent operations corresponding to LayerNorma, QKV generation, and fully connected layers. Therefore, it is necessary to obtain target parameters through data synchronization operations between all boards to complete the operations of this target level.

[0090] It can be seen that the present invention can partition the weight parameters of the target level of the GPT model onto each board before the operation of the GPT model; construct a matrix processing unit and a vector function unit as the operation cores of all boards; during the operation of the GPT model, perform parameter sharding according to the operation cores of the target level and execute the operations of the target level; obtain target parameters through data synchronization between boards to complete the operations of the target level. The present invention can partition the weight parameters of the target level of the GPT model onto each board, construct operation cores of different levels, perform operations according to the sharded parameters during the operation, and obtain target parameters through data synchronization to complete the operations, which is beneficial to making full use of the computing resources of the reconfigurable cluster and improving the operation efficiency of the GPT model.

[0091] In an alternative embodiment, partitioning the weight parameters of the target level of the GPT model onto each board in the board cluster may include:

[0092] For the QKV generation layer, partition the weight parameters of the QKV generation layer onto each board in the board cluster according to head averaging;

[0093] For the fully connected layer and the LayerNorm layer, partition the weight parameters of the fully connected layer and the LayerNorm layer onto each board in the board cluster according to column averaging.

[0094] In this alternative embodiment, the head can be understood as the head in multi-head attention calculation.

[0095] It can be seen that this alternative embodiment can partition the weight parameters of the QKV generation layer, fully connected layer, and LayerNorm of the GPT model onto each board in the board cluster according to the corresponding partitioning methods, which is beneficial to improving the accuracy of board partitioning of the GPT model, and further beneficial to making full use of the computing resources of the reconfigurable cluster and improving the operation efficiency of the GPT model.

[0096] In another alternative embodiment, before sharding the input parameters and weight parameters according to the operation type corresponding to the matrix processing unit of this target level, the method may further include:

[0097] Pack several rows of input parameters into a group;

[0098] For the input parameters of different rows in the same group, input them into the target level in parallel;

[0099] For the input parameters of different groups, input them into the target level using a pipelining algorithm.

[0100] In this alternative embodiment, it can be understood that when each board processes the input parameters (input matrix), first, d rows of input parameters need to be packed into a group, and the value of d is not limited. The input parameters of different rows in the same group are processed in parallel, and a pipelining algorithm is used to accelerate between different groups. Considering that the bit width of the stream port in the communication IP core is usually much larger than the data bit width of the GPT model, packing d rows of input parameters into a group for simultaneous operation and transmission can also effectively reduce the bandwidth waste of the stream communication port. More importantly, in the case of limited resources, the packing and intra-group parallel algorithms can better ensure that the operation delay of each board can cover more of the communication delay without causing additional delay increase.

[0101] In this alternative embodiment, it can be understood that assuming that the bit width of the stream port in the communication IP core is d times the data bit width of the GPT model, and at the same time, the number of rows of input parameters in the same group is also d; N represents the maximum number of input parameters supported by the summator (using a binary tree for summation); the size of the input parameter (matrix) is seqlen×dmodel (dmodel refers to the word vector dimension of the GPT model); it takes K cycles to wait for the stream port to return the partial sum of other boards, and seqlen refers to the length of the entire GPT model input token. Taking the average value operation process of two boards as an example:

[0102] If each board does not group and process the input parameters, the total delay formula for the average value operation of each board is as follows:

[0103]

[0104] The total delay of the average value operation for packing d rows of input parameters in the same group for parallel processing is as follows:

[0105]

[0106] During the calculation process of the GPT model, it is necessary to mask the communication latency as much as possible with the operation latency. By comparing the maximum value operations in the above two formulas, it can be found that using grouping makes it easier to achieve this goal. In the worst case, compared with the existing solutions, the synchronization strategy of the present invention does not introduce additional communication or operation latency.

[0107] It can be seen that this optional embodiment can pack the input parameters into a group and input the input parameters to the target level in a parallel or pipelined manner, which is beneficial to improving the data transmission efficiency, increasing the possibility of masking the communication latency with the operation latency, and thus is beneficial to making full use of the computing resources of the reconfigurable cluster and improving the operation efficiency of the GPT model.

[0108] In another optional embodiment, the matrix processing unit is constructed by arranging a number of sub-processing units; each sub-processing unit includes a multiplier, an adder, and an accumulation register;

[0109] In the embodiments of the present invention, it can be understood that the multiplier is a general module unit for implementing multiplication calculation functions, not a specific module unit limited to use in the matrix processing unit. The same applies to the adder and the accumulation register, and the same applies to the exponentiation operator, the square operation operator, the adder, and the accumulation register in the following vector function unit.

[0110] And, fragmenting the input parameters based on the operation type corresponding to the matrix processing unit of the target level, and fragmenting the weight parameters based on the weight fragmenting strategy, may include:

[0111] When the operation type of the matrix processing unit is matrix multiplication, fragment the matrix-form input parameters into a number of sub-input matrices, and fragment the matrix-form weight parameters into a number of sub-weight matrices; wherein, the dimensions of the sub-input matrices and the sub-weight matrices are the same;

[0112] When the operation type of the matrix processing unit is vector multiplication by matrix, fragment the vector-form input parameters into a number of sub-input vectors, and fragment the matrix-form weight parameters into a number of sub-weight matrices; wherein, the length of the sub-input vectors is the same as the dimension of the sub-weight matrices.

[0113] In this optional embodiment, the matrix processing unit is as Figure 3 shown in the MPU unit of Figure 4 For matrix multiplication, the shown fragmenting strategy is adopted to improve the data reuse rate of the weight parameters. After fragmenting the large input matrix into a number of small matrices with dimensions of d×d, taking a single matrix as the basic unit, traverse the input and weight parameters in the order of column-first then row-first and row-first then column-first respectively. Then, use the broadcast algorithm to further accelerate the processing of the fragmented small matrices. Specifically, according to Figure 3Arrange d×d processing elements (PEs) in the manner shown by the MPU unit to complete the construction of the matrix processing unit. Each processing element includes a multiplier, an adder, and an accumulation register (reg). When multiplying two d×d matrices, the input parameter (matrix) of the i-th row needs to be broadcast to the processing elements of the i-th row, and the weight parameter (matrix) of the j-th column needs to be broadcast to the processing elements of the j-th column. Since all processing elements run in parallel and the internal processing elements are accelerated by the pipelining algorithm, theoretically, it only takes d clock cycles to complete the d×d dimensional matrix multiplication.

[0114] In this alternative embodiment, further, for the vector-matrix multiplication operation, the Figure 5 sharding strategy in is adopted to accelerate the vector-matrix multiplication operation, and a high-bandwidth memory (HBM) is used to meet the high-bandwidth requirements of the weight parameters. Complete the multiplication between a d-dimensional vector and L different d×d matrices within one processing cycle, and this process can be completed by reusing the matrix processing unit. Specifically, first broadcast the d-dimensional vector to all processing elements, and sequentially send these L different d×d matrices to the processing elements of L rows respectively. For each row of processing elements, send the j-th column data of the corresponding d×d matrix in this row to the j-th processing element. After d multiply-accumulate operations, all vector-matrix multiplication operations within one processing cycle can be completed.

[0115] It can be seen that this alternative embodiment can shard the input parameters and weight parameters according to the operation type of the matrix processing unit, which is beneficial to improving the data reuse rate of the weight parameters through data sharding, thereby improving the calculation efficiency of the matrix processing unit and further improving the operation efficiency of the GPT model.

[0116] In yet another alternative embodiment, the vector function unit is constructed by a summation module and a scaling module; each summation module includes several accumulators and adders arranged in a binary tree form, and each accumulator includes an exponential operation unit, a square operation unit, an adder, and an accumulation register; the scaling module is used to perform multiplication, division, addition, and subtraction calculations between vectors;

[0117] And, when the operation core corresponding to the target level is the vector function unit, sharding the input parameters and performing the operation of the target level based on the sharded input parameters may include:

[0118] For each target level, when the operation core corresponding to the target level is the vector function unit, split the matrix-form input parameters of the target level into several target vectors row by row, and perform the operation of the target level based on all the target vectors.

[0119] In this alternative embodiment, the vector function unit is as Figure 3As shown in the VFU unit, this unit is mainly responsible for completing vector operations such as the LayerNorm layer, GLUE, and SoftMax. If the input parameter is a matrix, it is first split into several vectors by rows and then calculated separately. The most important module of this unit is the summation module. First, let several accumulators perform accumulation operations on specific parts of the input parameter (vector) in parallel. Then, a binary tree is used to sum the results of all accumulators. The input data of the scaling module is usually independent of each other, so the processing can also be further accelerated by replicating this module. To cooperate with the synchronization strategy and further improve the operation parallelism, the vector function unit can be replicated several times to complete the parallel processing of the input data rows in the same group.

[0120] It can be seen that in this alternative embodiment, when the operation core is a vector function unit, the matrix-form input parameter of the target layer can be split into several target vectors by rows to execute the operation of the target layer, which is beneficial to improving the calculation efficiency of the vector function unit.

[0121] In another alternative embodiment, for each target layer, obtaining the target parameter through data synchronization operations between all boards to complete the operation of the target layer may include:

[0122] For the LayerNorm layer in the vector function layer on each board, calculate the sum of each row of the input parameter according to the input parameter of each board to obtain the sum result of each row of the input parameter on each board; and determine the target parameter based on all the sum results, and synchronize the target parameter to all other boards through broadcast communication; when each board receives the target parameters on all other boards, based on all the sum results on this board and the received target parameters, determine the output parameter of the LayerNorm layer on this board;

[0123] For the matrix multiplication layer on each board, determine the input parameter and target parameter of each matrix multiplication layer based on the operation result before each matrix multiplication layer on this board, and synchronize the target parameter to all other boards through broadcast communication; when each board receives the target parameters on all other boards, based on the input parameter of each matrix multiplication layer on this board and the received target parameters, determine the output parameter of the matrix multiplication layer on this board.

[0124] In this alternative embodiment, it can be understood that when the operation proceeds to before the LayerNorm, QKV generation, and fully connected layers, all boards must start the synchronization process to obtain global data, as Figure 2 shown. Only start the synchronization process before the layer operations that require global data, which will not increase the theoretical operation load of other layers.

[0125] In this alternative embodiment, the specific synchronization steps need to focus on whether they can minimize the system blockage caused by synchronous communication. To achieve this goal, specific operations and synchronous communication steps can be formulated for each level in the GPT model that requires synchronous communication, so as to maximize the masking of the operation latency of both. Generally speaking, the operation order of specific levels can be adjusted, and as soon as the target parameters to be synchronized are generated, they should be transmitted to the other boards immediately. As described in the above steps for the target levels, the levels that must be synchronized in the GPT model can be divided into the vector function layer and the matrix multiplication layer. The former includes the LayerNorm layer, and the latter includes the QKV generation and the fully connected layer.

[0126] In this alternative embodiment, further, for the vector function layer, as Figure 6 shown, when calculating the mean and variance of the input parameters of each row in the LayerNorm layer, each board first calculates the partial sum based on its local input parameters (first calculate the sum of the partial input parameters and then calculate the mean and variance after synchronization). Then, the boards immediately synchronize these partial sums through broadcast communication. As for the remaining operation, each board can complete it with its local input parameters. This synchronization strategy compresses the data volume required for synchronizing the LayerNorm layer from the original entire input matrix into several partial sums.

[0127] In this alternative embodiment, further, for the matrix multiplication layer, Figure 7 shows the operation and synchronization steps of each board for the fully connected layer. First, all matrix multiplication operations between the local input parameters and the weight parameters of the board are completed, and the intermediate results are stored in the output buffer. Each board completes the remaining operations after obtaining the target parameters of other single boards.

[0128] It can be seen that this alternative embodiment can determine the corresponding synchronization method according to whether the target level is the vector function layer or the matrix multiplication layer to complete the synchronization of the target parameters, which is beneficial to improving the synchronization accuracy of the target parameters. At the same time, it can also reduce unnecessary synchronization operations and improve the operation efficiency of the GPT model.

[0129] In another alternative embodiment, the method may further include:

[0130] Suspending the process for broadcast communication according to the completion time when the target parameters are synchronized to all other boards.

[0131] In this alternative embodiment, it can be understood that each board needs to suspend as early as possible the processes responsible for sending its local output parameters (target parameters) to the other boards and receiving the input parameters (target parameters) of the other boards, so as to utilize a longer operation latency to overlap the synchronization latency.

[0132] Further, for the first fully connected layer of each decoding layer of the GPT model, as Figure 7 shown, before this fully connected layer, each board needs to multiply nhead pairs of input parameters (matrices) of size seqlen×seqlen and seqlen×dhead in sequence (nhead refers to the number of heads in the self-attention operation, seqlen refers to the length of the input tokens of the entire GPT model, and dhead refers to the row dimension of the QKV matrix of a single head in the self-attention operation). This step can be called SmulV. After calculating the result corresponding to one head in SmulV, each board can sequentially complete marking, packing, and broadcasting this result to other boards. After the results (target parameters) corresponding to all heads are synchronized, each board also needs to suspend the process responsible for receiving data from the other boards.

[0133] In this optional embodiment, further, for the remaining two fully connected layers of each decoding layer of the GPT model, as Figure 7 shown, the operations before these two layers are scaling in the LayerNorm layer and the GELU layer. The input parameters in these two scaling processes are calculated independently, and their operation steps are similar. Each board can suspend the processes responsible for broadcasting and receiving data from the other boards according to the completion time of synchronizing the calculated target parameters to all other boards. The synchronization steps of the QKV generation layer are almost the same as those of the fully connected layer and will not be elaborated here.

[0134] It can be seen that this optional embodiment can suspend the process for broadcast communication according to the completion time of synchronizing the target parameters to all other boards, which is beneficial to reducing resource occupancy during data transmission, and thus beneficial to making full use of the computing resources of the reconfigurable cluster and improving the operation efficiency of the GPT model.

[0135] In yet another optional embodiment, the method may further include:

[0136] Monitoring the computing progress of the computing core through the data sending module;

[0137] When the computing progress of the computing core reaches the target node, packing, marking the target parameters, and putting them into the write interface of the stream port;

[0138] Monitoring the read interface of the stream port through the data receiving module. When the target parameters are obtained, unpacking and analyzing the target parameters and putting them into the corresponding on-chip buffer; the on-chip buffer is used to prevent the loss of target parameters and send and receive target parameters according to the first-in, first-out rule;

[0139] Monitor the progress of the computing unit through the computing blocking module. If the data receiving module fails to obtain the target parameter, perform a blocking operation on the computing operation corresponding to the target parameter.

[0140] It can be seen that this optional embodiment can monitor whether the target parameter is successfully synchronized by the data sending module, the data receiving module, and the computing blocking module, and pause the computing operation in a timely manner when the synchronization is not completed, which is beneficial to improving the computing accuracy of the GPT model and reducing resource consumption.

[0141] Embodiment 2

[0142] Please refer to Figure 8 , Figure 8 which is a schematic structural diagram of an operation acceleration device for a GPT model based on a reconfigurable cluster disclosed in an embodiment of the present invention. Among them, Figure 8 the method shown can be applied to the operation scenario of a GPT model based on a reconfigurable cluster. The number of boards in the reconfigurable cluster can be two or more, which is not limited in the embodiments of the present invention. As Figure 8 shown, the operation acceleration device for a GPT model based on a reconfigurable cluster may include:

[0143] A partitioning module 201, configured to partition the weight parameters of the target layer of the GPT model onto each board in the board cluster before the GPT model performs an operation; at least two boards are included in the board cluster; the target layer includes a vector function layer and a matrix multiplication layer; the vector function layer includes a LayerNorm layer, and the matrix multiplication layer includes a QKV generation layer and all fully connected layers;

[0144] A construction module 202, configured to construct a matrix processing unit and a vector function unit as the operation cores of all boards; the operation cores are used to perform the operations of the GPT model; among them, the matrix processing unit is used for matrix operations of the matrix multiplication layer, and the vector function unit is used for vector operations of the LayerNorm layer and the activation function; the operation types of the matrix processing unit include vector-matrix multiplication operations and matrix-matrix multiplication operations;

[0145] A sharding module 203, configured to, during the operation of the GPT model, for each target layer, when the operation core corresponding to the target layer is a matrix processing unit, shard the input parameters and weight parameters based on the operation type corresponding to the matrix processing unit of the target layer, and perform the operation of the target layer based on the sharded input parameters and the sharded weight parameters; when the operation core corresponding to the target layer is a vector function unit, shard the input parameters and perform the operation of the target layer based on the sharded input parameters;

[0146] The synchronization module 204 is configured to obtain target parameters through data synchronization operations among all boards for each target level to complete the operations of that target level.

[0147] It can be seen that the device described in the embodiments of the present invention can, before the operation of the GPT model, divide the weight parameters of the target level of the GPT model onto each board; construct a matrix processing unit and a vector function unit as the operation cores of all boards; during the operation of the GPT model, perform parameter sharding according to the operation cores of the target level and execute the operations of the target level; obtain target parameters through data synchronization among the boards to complete the operations of the target level. It can be seen that the present invention can divide the weight parameters of the target level of the GPT model onto each board, construct operation cores of different levels, perform operations according to the sharded parameters during the operation, and obtain target parameters through data synchronization to complete the operations, which is beneficial to making full use of the computing resources of the reconfigurable cluster and improving the operation efficiency of the GPT model.

[0148] In an optional embodiment, the partitioning module 201 divides the weight parameters of the target level of the GPT model onto each board in the board cluster, and the specific method includes:

[0149] For the QKV generation layer, the weight parameters of the QKV generation layer are divided onto each board in the board cluster according to head averaging;

[0150] For the fully connected layer and the LayerNorm layer, the weight parameters of the fully connected layer and the LayerNorm layer are divided onto each board in the board cluster according to column averaging.

[0151] It can be seen that the device described in the embodiments of the present invention can divide the weight parameters of the QKV generation layer, the fully connected layer, and the LayerNorm of the GPT model onto each board in the board cluster according to the corresponding partitioning methods, which is beneficial to improving the accuracy of board partitioning of the GPT model, and further beneficial to making full use of the computing resources of the reconfigurable cluster and improving the operation efficiency of the GPT model.

[0152] In another optional embodiment, as Figure 9 shown, the device may further include:

[0153] The packing module 205 is configured to pack several rows of input parameters into a group before the sharding module 203 shards the input parameters and weight parameters based on the operation type corresponding to the matrix processing unit of the target level;

[0154] The input module 206 is configured to input the input parameters of different rows in the same group into the target level in a parallel manner; for the input parameters of different groups, input them into the target level using a pipeline algorithm.

[0155] It can be seen that implementing the device described in the embodiments of the present invention can pack input parameters into a group and input the input parameters into the target level in a parallel or pipelined manner, which is beneficial to improving the efficiency of data transmission and the possibility of covering the operation delay with the communication delay. Furthermore, it is beneficial to make full use of the computing resources of the reconfigurable cluster and improve the operation efficiency of the GPT model.

[0156] In another optional embodiment, the matrix processing unit is constructed by arranging a number of sub-processing units; each sub-processing unit includes a multiplier, an adder, and an accumulation register;

[0157] Moreover, the sharding module 203 shards the input parameters based on the operation type corresponding to the matrix processing unit at the target level and shards the weight parameters based on the weight sharding strategy. The specific methods include:

[0158] When the operation type of the matrix processing unit is matrix multiplication operation, the matrix-form input parameters are sharded into a number of sub-input matrices, and the matrix-form weight parameters are sharded into a number of sub-weight matrices; wherein, the dimensions of the sub-input matrices and the sub-weight matrices are the same;

[0159] When the operation type of the matrix processing unit is vector multiplication operation, the vector-form input parameters are sharded into a number of sub-input vectors, and the matrix-form weight parameters are sharded into a number of sub-weight matrices; wherein, the length of the sub-input vectors is the same as the dimension of the sub-weight matrices.

[0160] It can be seen that implementing the device described in the embodiments of the present invention can shard the input parameters and weight parameters according to the operation type of the matrix processing unit, which is beneficial to improving the data reuse rate of the weight parameters through data sharding, and further improving the computing efficiency of the matrix processing unit, thereby improving the operation efficiency of the GPT model.

[0161] In another optional embodiment, the vector function unit is constructed by a summation module and a scaling module; each summation module includes a number of accumulators and adders arranged in a binary tree form, and each accumulator includes an exponential operation unit, a square operation unit, an adder, and an accumulation register; the scaling module is used to perform multiplication, division, addition, and subtraction calculations between vectors;

[0162] Moreover, when the operation core corresponding to the target level in the sharding module 203 is the vector function unit, the input parameters are sharded and the operation of the target level is performed based on the sharded input parameters. The specific methods include:

[0163] For each target level, when the operation core corresponding to the target level is the vector function unit, the matrix-form input parameters at the target level are split row by row into a number of target vectors, and the operation of the target level is performed based on all the target vectors.

[0164] It can be seen that when the operation core is a vector function unit, the device described in the embodiments of the present invention can split the input parameters in the form of a matrix at the target level into several target vectors row by row to perform the operations at the target level, which is beneficial to improving the calculation efficiency of the vector function unit.

[0165] In another optional embodiment, the synchronization module 204 may include:

[0166] The first determination sub-module 2041 is configured to, for each LayerNorm layer in the vector function layer on each board, calculate the sum of each row of the input parameters according to the input parameters of each board to obtain the sum result of each row of the input parameters on each board; and determine the target parameter based on all the sum results, and synchronize the target parameter to all other boards through broadcast communication; when each board receives the target parameters on all other boards, based on all the sum results on the board and the received target parameters, determine the output parameters of the LayerNorm layer on the board;

[0167] The second determination sub-module 2042 is configured to, for each matrix multiplication layer on each board, determine the input parameters and target parameters of each matrix multiplication layer based on the operation results before each matrix multiplication layer on the board, and synchronize the target parameters to all other boards through broadcast communication; when each board receives the target parameters on all other boards, based on the input parameters of each matrix multiplication layer on the board and the received target parameters, determine the output parameters of the matrix multiplication layer on the board.

[0168] It can be seen that the device described in the embodiments of the present invention can determine the corresponding synchronization method according to whether the target level is a vector function layer or a matrix multiplication layer to complete the synchronization of the target parameter, which is beneficial to improving the synchronization accuracy of the target parameter, and at the same time can reduce unnecessary synchronization operations and improve the operation efficiency of the GPT model.

[0169] In another optional embodiment, the synchronization module 204 may further include:

[0170] The suspension sub-module 2043 is configured to suspend the process for broadcast communication according to the completion time when the target parameter is synchronized to all other boards.

[0171] It can be seen that the device described in the embodiments of the present invention can suspend the process for broadcast communication according to the completion time when the target parameter is synchronized to all other boards, which is beneficial to reducing the resource occupation during the data transmission process, and further beneficial to making full use of the computing resources of the reconfigurable cluster and improving the operation efficiency of the GPT model.

[0172] Embodiment 3

[0173] Please refer to Figure 10 , Figure 10 which is a schematic structural diagram of an accelerator for accelerating the operation of a GPT model based on a reconfigurable cluster disclosed in an embodiment of the present invention. Among them, Figure 10 the described accelerator can be applied in an application server. As Figure 10 shown, the accelerator may include:

[0174] A memory 301 storing executable program code;

[0175] A processor 302 coupled to the memory 301;

[0176] The processor 302 invokes the executable program code stored in the memory 301 and executes the steps in the operation acceleration method of the GPT model based on the reconfigurable cluster described in Embodiment 1 of the present invention.

[0177] Embodiment 4

[0178] An embodiment of the present invention discloses a computer-storable medium. The computer storage medium stores computer instructions, which are used to execute the steps in the operation acceleration method of the GPT model based on the reconfigurable cluster described in Embodiment 1 of the present invention when the computer instructions are invoked.

[0179] Embodiment 5

[0180] An embodiment of the present invention discloses a computer program product. The computer program product includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute the steps in the operation acceleration method of the GPT model based on the reconfigurable cluster described in Embodiment 1 of the present invention.

[0181] Experimental setup

[0182] A cluster prototype was built using an AUS ESC8000A-E11 rack-mounted server, an AMD EPYC 7543 CPU, and two Alveo U280 data center accelerator cards for evaluation. This cluster prototype was developed using the Vitis 2022.2 and Vivado 2022.2 suites, and used half-precision floating-point as the model data bit width, with a working frequency of 200 MHz.

[0183] Xilinx's Aurora 64B / 66B communication IP core was used to complete all communication transmission operations. For simplicity, it was configured to work in stream and full-duplex modes, as Figure 11As shown. Since the present invention can be applied to an end-to-end accelerator, after the initialization process, each board will continuously communicate through this IP core. When the s_axi_tx_tvalid signal is set high, the routing module will write the corresponding data to the AXI stream port. If there is no new available data, the Aurora 64B / 66B core will automatically send an idle command and maintain the connection between each board. Therefore, when calculating the link latency, the initialization process of all IP cores can be ignored.

[0184] The aforementioned proposed synchronization process will frequently initiate micro-data communication requirements. Therefore, it is necessary to verify whether the link latency of the Aurora 64 / 66B IP core is relatively low. Design a test bench as shown in Figure 11 according to the Xilinx official tutorial, and instantiate 2 Aurora 64B / 66B IP cores on the U280 board and let them communicate with each other. Here, the number of channels is configured to 1, and the link rate is configured to 12.5 Gbps. It can be found through Vivado simulation that the link latency of the Aurora 64B / 66B IP core is about 266 ns, which can be ignored compared with the operation latency.

[0185] According to the suggestion of Xilinx, if the link rate is reduced by about 50% from the maximum line rate supported by the IP / Transceiver, better timing performance can be obtained, and no other additional synthesis strategies are required. The theoretical maximum link rate of the U280 board is 28.01664 Gbs. Therefore, in the final deployment, the number of channels is configured to 4, and the link rate is configured to 12.8 Gbps. In this way, the bit width of the stream interface of each transmission IP core reaches 256 bits, which is 16 times the data bit width of the accelerator.

[0186] Configure the size of the processing unit array in the matrix processing unit to 32×32, and a total of 4096 DSPs are used. Then, set the parallelism within each vector function unit to 2, and copy each vector function unit 16 times. The following table summarizes the resource consumption corresponding to the low-latency GPT model deployed on each U280 board.

[0187]

[0188] The above prototype is used to evaluate the performance of the system. The test models include the open-source 345M GPT-2 model of NVIDIA Megatron-LM and the 774M GPT-2 model of OpenAI. DFX is used as the benchmark for domain-specific accelerators. DFX can be configured with four or more FPGA boards. Here, only the performance of its 1-board and 2-board versions is compared. The NVIDIA V100 GPU is used as the GPU benchmark, and the GPT-2 model is run using the NVIDIA GPU-optimized Megatron-LM source code and CUDA Toolkit 11.1.

[0189] Result Analysis

[0190] Figure 12 The latency of DFX and the prototype of the present invention when deploying the 345M GPT-2 model on a single U280 board is compared. In this case, the average acceleration ratio of the present invention relative to DFX is 2.7 times, and the maximum acceleration ratio is 3.9 times. In the preprocessing stage, the computing latency of the present invention is 4 times faster than that of DFX. In the decoding stage, the computing latency of the present invention is 2.7 times faster than that of DFX. The acceleration effect of the present invention in the preprocessing stage comes from two aspects. First, the sharding strategy adopted by the present invention can significantly improve the data reuse rate of weight parameters in matrix operations. Second, the broadcast array of the present invention has a higher DSP efficiency than the multiply-accumulate tree of DFX. The acceleration effect of the present invention in the decoding stage mainly comes from the high computing intensity of the broadcast array.

[0191] Figure 12 The latency of DFX and the prototype of the present invention when deploying the 774M GPT-2 model on two U280s is also compared. In this case, the average acceleration ratio of the present invention relative to DFX is 3.6 times, and the maximum acceleration ratio is 4.1 times. After evaluation, the synchronization algorithm of the present invention saves at least 10 ms of system synchronization latency in the preprocessing stage. In addition, if the decoding stage needs to iterate 1024 times, the synchronization method of the present invention can save an additional 80 ms of synchronization latency. Although the saved synchronization latency is relatively small here, the present invention has covered almost all synchronization delays with operations. When expanding the cluster in the future, the proportion of synchronization delay in the inference latency of the GPT model will further increase, and the optimization effect of the synchronization algorithm of the present invention will become more significant.

[0192] Figure 13The latency of the V100 GPU, DFX, and the prototype of the present invention was compared when deploying the 345M GPT-2 model in single-card deployment. The average acceleration ratio of the present invention relative to the V100 GPU is 9.1 times, and the maximum acceleration ratio can reach 13.4 times. It can be found that when the length of the output token is small, the performance of the V100 GPU is better than that of DFX and the present invention. As the length of the input token increases, the gap between the V100 GPU and the FPGA accelerator will become larger and larger. For the preprocessing stage of the GPT model, its main load is matrix multiplication. Therefore, a large number of parallel units inside the GPU can quickly process all input tokens. However, as the length of the input token increases, the computing intensity of the GPU will also increase continuously. In this case, the inference latency of the preprocessing stage of the FPGA accelerator will be much lower than that of the GPU.

[0193] Figure 14 The latency of the V100 GPU, DFX, and the present invention was compared when deploying the 774M GPT-2 model on dual cards. The average acceleration ratio of the present invention compared to the V100 GPU is 12.8 times, and the highest acceleration ratio can reach 18.3 times. It can be found that when the length of the input token is 32 and the length of the output token is 1, the inference latency of the present invention is slightly lower than that of the V100 GPU. This further illustrates the high computing intensity of the computing unit of the present invention. Compared with the V100 GPU, the latency improvement of the present invention on the 774M GPT-2 model exceeds that of the 345M GPT-2 model. The former needs to process more vector-matrix multiplications in the decoding stage. The GPU will face obvious performance degradation due to insufficient computing intensity when processing this type of input. Therefore, as the scale of the GPT model increases, the acceleration effect of the present invention has a tendency to be further improved.

[0194] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed to multiple network modules. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0195] Through the specific descriptions of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, and the storage medium includes read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc memories, magnetic disk memories, tape memories, or any other medium that can be used to carry or store data and is computer-readable.

[0196] Finally, it should be noted that the operation acceleration method and device of the GPT model based on the reconfigurable cluster disclosed in the embodiments of the present invention only disclose the preferred embodiments of the present invention, and are only used to illustrate the technical solutions of the present invention, rather than limiting them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An operation acceleration method for the GPT model based on a reconfigurable cluster, characterized in that The method includes: Before the GPT model operates, dividing the weight parameters of the target layer of the GPT model onto each board in the board cluster; at least two boards are included in the board cluster; the target layer includes a vector function layer and a matrix multiplication layer; the vector function layer includes a LayerNorm layer, and the matrix multiplication layer includes a QKV generation layer and all fully connected layers; Constructing a matrix processing unit and a vector function unit as the operation cores of all the boards; the operation cores are used to execute the operations of the GPT model; among them, the matrix processing unit is used for matrix operations of the matrix multiplication layer, and the vector function unit is used for vector operations of the LayerNorm layer and activation functions; the operation types of the matrix processing unit include vector-matrix multiplication operations and matrix-matrix multiplication operations; During the operation of the GPT model, for each target layer, when the operation core corresponding to the target layer is the matrix processing unit, sharding the input parameters and the weight parameters based on the operation type corresponding to the matrix processing unit of the target layer, and performing the operation of the target layer based on the sharded input parameters and the sharded weight parameters; when the operation core corresponding to the target layer is the vector function unit, sharding the input parameters and performing the operation of the target layer based on the sharded input parameters; For each target layer, obtaining target parameters through data synchronization operations between all the boards to complete the operation of the target layer.

2. The operation acceleration method of the GPT model based on the reconfigurable cluster according to claim 1, wherein The dividing the weight parameters of the target layer of the GPT model onto each board in the board cluster includes: For the QKV generation layer, dividing the weight parameters of the QKV generation layer onto each board in the board cluster according to head averaging; For the fully connected layer and the LayerNorm layer, dividing the weight parameters of the fully connected layer and the LayerNorm layer onto each board in the board cluster according to column averaging.

3. The operation acceleration method of the GPT model based on the reconfigurable cluster according to claim 1, characterized in that Before sharding the input parameters and the weight parameters based on the operation type corresponding to the matrix processing unit of the target layer, the method further includes: Packing several rows of the input parameters into a group; For the input parameters in different rows of the same group, inputting them into the target layer in a parallel manner; For the input parameters in different groups, inputting them into the target layer using a pipeline algorithm.

4. The operation acceleration method of the GPT model based on the reconfigurable cluster according to claim 1, wherein The matrix processing unit is constructed by arranging several sub-processing units; each sub-processing unit includes a multiplier, an adder, and an accumulation register; And, sharding the input parameters based on the operation type corresponding to the matrix processing unit of the target layer and sharding the weight parameters based on the weight sharding strategy includes: When the operation type of the matrix processing unit is the matrix-matrix multiplication operation, sharding the input parameters in matrix form into several sub-input matrices, and sharding the weight parameters in matrix form into several sub-weight matrices; among them, the dimensions of the sub-input matrices and the sub-weight matrices are the same; When the operation type of the matrix processing unit is vector-matrix multiplication, the vector-form input parameter is sliced into several sub-input vectors, and the matrix-form weight parameter is sliced into several sub-weight matrices; wherein, the length of the sub-input vector is the same as the dimension of the sub-weight matrix.

5. The operation acceleration method of the GPT model based on the reconfigurable cluster according to claim 1, wherein The vector function unit is constructed by a summation module and a scaling module; each summation module includes several accumulators and adders, and each accumulator includes an exponentiator, a squarer, an adder, and an accumulation register; the scaling module is used to perform multiplication, division, addition, and subtraction operations between vectors; And, when the operation core corresponding to the target layer is the vector function unit, slicing the input parameter and performing the operation of the target layer based on the sliced input parameter, including: For each target layer, when the operation core corresponding to the target layer is the vector function unit, the matrix-form input parameter of the target layer is split into several target vectors row by row, and the operation of the target layer is performed based on all the target vectors.

6. The operation acceleration method of the GPT model based on the reconfigurable cluster according to any one of claims 1-5, characterized in that For each target layer, obtaining the target parameter through data synchronization operations between all the boards to complete the operation of the target layer, including: For the LayerNorm layer in the vector function layer on each board, calculate the sum of each row of the input parameter according to the input parameter of each board to obtain the sum result of each row of the input parameter on each board; and determine the target parameter based on all the sum results, and synchronize the target parameter to all the other boards through broadcast communication; when each board receives the target parameters on all the other boards, based on all the sum results on this board and all the received target parameters, determine the output parameter of the LayerNorm layer on this board; For the matrix multiplication layer on each board, determine the input parameter and the target parameter of each matrix multiplication layer based on the operation results before each matrix multiplication layer on this board, and synchronize the target parameter to all the other boards through broadcast communication; when each board receives the target parameters on all the other boards, based on the input parameter of each matrix multiplication layer on this board and the received target parameters, determine the output parameter of the matrix multiplication layer on this board.

7. The operation acceleration method of the GPT model based on the reconfigurable cluster according to claim 6, wherein, The method further includes: Suspending the process for broadcast communication according to the completion time when the target parameter is synchronized to all the other boards.

8. An operation acceleration device for a GPT model based on a reconfigurable cluster, characterized in that, The device includes: A partitioning module, configured to partition the weight parameters of the target layer of the GPT model to each board in the board cluster before the GPT model operates; at least two boards are included in the board cluster; the target layer includes a vector function layer and a matrix multiplication layer; the vector function layer includes a LayerNorm layer, and the matrix multiplication layer includes a QKV generation layer and all fully connected layers; A building block for building a matrix processing unit and a vector function unit as the operation cores of all the boards; the operation cores are used to execute the operations of the GPT model; wherein, the matrix processing unit is used for matrix operations in the matrix multiplication layer, and the vector function unit is used for vector operations in the LayerNorm layer and the activation function; the operation types of the matrix processing unit include vector-matrix multiplication operations and matrix-matrix multiplication operations; A sharding module, during the operation of the GPT model, for each target level, when the operation core corresponding to the target level is the matrix processing unit, shard the input parameters and the weight parameters based on the operation type corresponding to the matrix processing unit of the target level, and execute the operation of the target level based on the sharded input parameters and the sharded weight parameters; when the operation core corresponding to the target level is the vector function unit, shard the input parameters and execute the operation of the target level based on the sharded input parameters; A synchronization module, for each target level, to obtain target parameters through data synchronization operations among all the boards to complete the operation of the target level.

9. An accelerator for accelerating the operation of a GPT model based on a reconfigurable cluster, characterized in that, The accelerator includes: A memory storing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory and executes the operation acceleration method of the GPT model based on the reconfigurable cluster according to any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which are used to execute the operation acceleration method of the GPT model based on the reconfigurable cluster according to any one of claims 1-7 when the computer instructions are called.

Images