Transform high throughput reasoning pipeline scheduling method

By adopting the X-Y stacking pipeline parallel strategy and dynamic programming partitioning and mapping algorithm in the Transformer model, the memory access bottleneck problem of the Transformer model in high-throughput reasoning scenarios is solved, and efficient high-throughput reasoning execution is achieved.

CN119356739APending Publication Date: 2025-01-24ARTIFICIAL INTELLIGENCE RES INST OF HEFEI COMPREHENSIVE NAT SCI CENT (ANHUI ARTIFICIAL INTELLIGENCE LAB)
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411521418.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-29
Publication Date
2025-01-24

AI Technical Summary

Technical Problem

The Transformer model has a storage access bottleneck problem in high-throughput inference scenarios, and the existing acceleration methods are difficult to effectively solve the mapping problem of high-throughput inference.

Method used

A pipeline scheduling method for Transformer high-throughput inference is proposed. It adopts the X-Y stacking pipeline parallel strategy, divides the Transformer model into segments by blocks, and uses a dynamic programming-based partition mapping algorithm to achieve high-throughput pipeline parallel execution.

Benefits of technology

Through this method, the efficient execution of the Transformer model in high throughput reasoning scenarios has been implemented, which significantly improves throughput and solves the problem of visiting the bottleneck.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119356739A_ABST
    Figure CN119356739A_ABST
Patent Text Reader

Abstract

The invention discloses an assembly line scheduling method for Transform high throughput reasoning, which relates to the technical field of artificial intelligence, and comprises the following steps: step 1, an X-Y stacking assembly line parallel strategy: cutting a whole network model into segments according to Transform blocks, sequentially mapping each segment to a chip node array according to a set rule, each segment is executed on the chip node array in a pipeline parallel mode to form a pipeline stage, and the interior of the pipeline stage is executed in a tensor parallel mode; step 2, constructing a division mapping algorithm based on dynamic programming by using an X-Y stacking assembly line parallel strategy, and obtaining a highest throughput assembly line parallel division and mapping scheme under a given network and a chip node array based on the division mapping algorithm; according to the pipeline scheduling method, high throughput reasoning of the Transform model is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a pipeline scheduling method for Transformer high-throughput reasoning. Background Art

[0002] The self-attention mechanism is a powerful tool that can be used to model long-term dependencies in sequential data (such as language). Deep neural networks based on this mechanism (such as Vision Transformer) are also widely used in the field of computer science (CV), including tasks such as image classification, object detection, and semantic segmentation, and have achieved remarkable results. However, operations such as self-attention in the Transformer model involve complex data movement, high memory requirements, low data reuse, and slow operation on CPU and GPU platforms. The multi-core / storage-computing integrated architecture has scalable memory bandwidth and low memory access overhead, which helps to effectively alleviate the memory access bottleneck problem of the Transformer model, and thus becomes a potential solution to accelerate the Transformer.

[0003] There are currently many Transformer model acceleration methods based on the many-core / storage-computing integrated architecture, such as head-level parallelism, patch-level parallelism, and head-patch-level parallelism. These methods are mainly aimed at low-latency scenario requirements (such as autonomous driving), and these methods cannot solve the mapping problem of high-throughput reasoning well. Summary of the invention

[0004] Based on the technical problems existing in the background technology, the present invention proposes a pipeline scheduling method for high-throughput reasoning of Transformer, and realizes high-throughput reasoning of the Transformer model.

[0005] The pipeline scheduling method for Transformer high-throughput reasoning proposed in the present invention includes the following steps:

[0006] Step 1: XY stacking pipeline parallel strategy: The entire network model is divided into segments according to the Transformer blocks. Each segment is mapped to the chip node array in sequence according to the set rules. Each segment is executed in a pipeline parallel manner on the chip node array to form a pipeline stage. The pipeline stage is executed in a tensor parallel manner.

[0007] Step 2: Use the XY stacking pipeline parallel strategy to build a partitioning and mapping algorithm based on dynamic programming, and obtain the highest throughput pipeline parallel partitioning and mapping scheme under a given network and chip node array based on the partitioning and mapping algorithm.

[0008] Furthermore, in step 1, the entire network model is divided into segments according to the Transformer blocks, and each segment is sequentially mapped to the chip node array according to the set rules. The set rules are as follows:

[0009] The sub-array of each pipeline stage is rectangular, and the sub-array is the area occupied by the pipeline stage in the chip node array;

[0010] For any pipeline stage, the previous pipeline stage and the subarray of this pipeline stage form a rectangle.

[0011] Furthermore, in step 2, the input of the dynamic programming partition mapping algorithm is the number of rows X and columns Y of the chip node array, and the output is the Transformer block partition result with the highest throughput of the complete Transformer model on the entire PNM system and the sub-array coordinates CG[L][X][Y] allocated to all segments under LG[L][X][Y], where the partition results are: fragment partition LG[L][X][Y] under the minimization of the maximum delay of all segments, and the maximum delay TM[L][X][Y] in the left and right segments under LG[L][X][Y], where L is the total number of Transformer blocks.

[0012] Furthermore, the partition-map algorithm sets four nested loops, where the nesting from the inside to the outside is the boundary s∈[0,l-1], the number of columns of the subarray y∈[1,Y], the number of rows of the subarray x∈[1,X], and the number of Transformer blocks l∈[1,L];

[0013] The partition mapping algorithm uses the total number of Transformer blocks L as the total number of iterations. In the lth iteration, the dynamic programming partition mapping algorithm is as follows:

[0014] Set the division boundary s∈[0,l-1] to divide segment 0 to segment l into segment A, (0,s) and segment B, (s+1,l);

[0015] Split the subarray x*y into two parts and assign them to segment A and segment B respectively;

[0016] Set the direction value direction to indicate the direction of the split. A direction value of 0 means splitting the subarray x*y with a horizontal line, and a direction value of 1 means splitting the subarray (x*y) with a vertical line, corresponding to the Y stacking and X stacking of segments A and B respectively.

[0017] Setting the segmentation distance distance represents the distance from the lower boundary of the horizontal segmentation line ion array (x*y) or the right boundary of the vertical segmentation line ion array (x*y).

[0018] For each determined (s, l, x, y) array, the partitioning strategy is expressed as (direction, distance), and the segmented maximum delay of the partitioning strategy is calculated through the state transition equation;

[0019] Traverse all boundaries s and array partitions (direction, distance), calculate the maximum segment delay TM[l][x][y] under the current (l, x, y) condition, the fragment partition LG[l][x][y] and the subarray coordinates TCG[l][x][y] under the minimum maximum delay of all segments.

[0020] Furthermore, the state transfer equation is specifically:

[0021] When splitting the subarray x*y with direction value 0:

[0022] temp latency =max(Lac tab [s],[l],[x],y-distance),TM[s],[x],[distance])

[0023] When splitting the subarray x*y with direction value 1:

[0024] temp latency =max(Lac tab [s],[l],[x-distance,[y]),TM[s],[distance],[y])

[0025] Among them, temp latency is the maximum delay of the current segmentation strategy, (Lac tab [s],[l],[x],y-distance) and max(Lac tab [s],[l],[x-distance,[y]) are the delay cost evaluation tables corresponding to horizontal and vertical partitioning respectively. TM[s],[x],[distance] and TM[s],[distance],[y] are the maximum delays of the optimal partitioning strategies corresponding to horizontal and vertical partitioning within segment A.

[0026] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned pipeline scheduling method when executing the computer program.

[0027] A computer-readable storage medium stores a plurality of classification programs, wherein the plurality of classification programs are used to be called by a processor and execute the pipeline scheduling method as described above.

[0028] The advantages of the Transformer high-throughput reasoning pipeline scheduling method provided by the present invention are: the Transformer high-throughput reasoning pipeline scheduling method provided in the structure of the present invention is adapted to different intra-block tensor parallel methods, reduces the huge solution space of the mapping problem, and avoids the intricate data flow between pipeline levels; the proposed dynamic programming-based partitioning and mapping mathematical model can solve the partitioning and mapping problem of the network model at one time, and realize the high-throughput reasoning of the Transformer model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the process of the present invention;

[0030] Figure 2 It is a schematic diagram of the segmentation of the XY stacking pipeline parallel strategy;

[0031] Figure 3 Schematic diagram of mapping the segments of the XY stacking pipeline parallel strategy on the chip node array, where the arrows are the data flow directions;

[0032] Figure 4 Schematic diagram of the partition mapping algorithm based on dynamic programming. DETAILED DESCRIPTION

[0033] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0034] This embodiment is aimed at a type of storage-computing integrated architecture with multiple computing cores, specifically an expandable 2D computing node array connected by a grid-like on-chip network (NoC). Each computing node integrates computing and storage units, including a neural network computing engine (NN engine), an on-chip network router, a memory submodule, a memory submodule controller, and an auxiliary computing unit (ACU). The neural network computing engine is mainly composed of a buffer SRAM and a processing element (PE) array for multiplication and accumulation (MAC) operations in DNN. The PE array can take different forms such as a systolic array, an NVDLA-type array, or a coarse-grained reconfigurable array (CGRA).

[0035] The Transformer model is composed of stacked Transformer blocks. In each block, the feature map is normalized (layer normal), multi-head self-attention (MSA), residual connection (resudial), normalized, fully connected, and residual connected. In order to extract local features of the feature map, the Transformer model often introduces a local attention mechanism, divides the feature map into several fixed-size regions, and performs independent and parallel Transformer block calculations in each region, as follows.

[0036] like Figures 1 to 4 As shown, the pipeline scheduling method for Transformer high-throughput reasoning proposed in the present invention includes the following steps:

[0037] Step 1: XY stacking pipeline parallel strategy: The entire network model is divided into segments according to the Transformer blocks. Each segment is mapped to the chip node array in sequence according to the set rules. Each segment is executed in a pipeline parallel manner on the chip node array to form a pipeline stage. The pipeline stage is executed in a tensor parallel manner.

[0038] Transformer is composed of several stacked blocks. In order to achieve high-throughput reasoning under certain delay constraints, the entire network is divided into segments by blocks. The segments are executed in a pipeline parallel manner on each chip sub-array, forming a pipeline stage, which is executed in a tensor parallel manner.

[0039] XY stacking pipeline parallel strategy Figure 2 and 3 As shown. Starting from the lower left corner of the chip node array, each pipeline stage extends to the right (X) or upward (Y) in order. At the same time, the extension must follow the following rules:

[0040] 1) The subarray of each pipeline stage must be rectangular, and the subarray is the area occupied by the pipeline stage in the chip node array.

[0041] Each pipeline stage contains several Transformer blocks, which are executed in a tensor-parallel manner on the chip. Existing Transformer block tensor parallel methods are generally head-level parallelism, patch-level parallelism, and head-patch-level parallelism. These parallel methods require the array shape to be rectangular due to their internal specific data flow methods. In order to adapt to different tensor parallel methods, this embodiment stipulates that the subarray of each pipeline stage must be rectangular.

[0042] 2) For any pipeline stage, the subarray of its previous pipeline stage and this pipeline stage must form a rectangle.

[0043] The mapping problem of multiple segments on the chip node array is similar to the two-dimensional bin packing problem, which is an NP-hard problem with a huge solution space. Stipulating that for any pipeline stage, its previous pipeline stage and the subarray of this pipeline stage must form a rectangle can significantly reduce the solution space. At the same time, this rule can also ensure that data communication between different layer groups will not cross other pipeline stages, and the data flow is more regular.

[0044] Step 2: Use the XY stacking pipeline parallel strategy to build a partitioning and mapping algorithm based on dynamic programming, and obtain the highest throughput pipeline parallel partitioning and mapping scheme under a given network and chip node array based on the partitioning and mapping algorithm.

[0045] According to the XY stacking pipeline parallel strategy given above, this embodiment establishes a mathematical model based on dynamic programming to solve the partitioning problem and the mapping problem in one step. Specifically, the partitioning and mapping algorithm based on dynamic programming is as follows: Figure 4 As shown. The input of the algorithm is the number of rows X and columns Y of the node array, and the output is three arrays TM, LG and CG, which are used to record information about segment division and node array segmentation, that is, the output is the Transformer block division result with the highest throughput of the complete Transformer model on the entire PNM system and the subarray coordinates CG[L][X][Y] assigned to all segments under LG[L][X][Y]. The division results are: fragment division LG[L][X][Y] under the minimization of the maximum delay of all segments, and the maximum delay TM[L][X][Y] in the left and right segments under LG[L][X][Y]. L is the total number of Transformer blocks.

[0046] The partition mapping algorithm sets up four layers of nested loops, with the nesting from the inside to the outside being the boundary s∈[0,l-1], the number of columns of the subarray y∈[1,Y], the number of rows of the subarray x∈[1,X], and the number of Transformer blocks l∈[1,L]. For the first l Transformer blocks and the subarray of shape x*y, LG[l][x][y] records the segment partitioning under the condition of minimizing the maximum delay of all segments, and TM[l][x][y] records the maximum delay in all segments under the corresponding LG[l][x][y]. CG[l][x][y] records the subarray shape / coordinates assigned to all segments under the corresponding condition.

[0047] After initializing the three arrays, the dynamic programming process will gradually iterate l, x, y until the total number of Transformer blocks of the entire Transformer model L and the chip node array width X and height Y of the entire PNM system are reached. In each iteration, the maximum delay of the first l Transformer blocks on the subarray of shape x*y and the corresponding segment division and node array division are searched.

[0048] The partition mapping algorithm uses the total number of Transformer blocks L as the total number of iterations. In the lth iteration, the dynamic programming partition mapping algorithm is as follows:

[0049] (a1) Set the partition boundary s∈[0,l-1] to divide segment 0 to segment l into segment A, (0,s) and segment B, (s+1,l);

[0050] (a2) After fixing the boundary s, the subarray x*y is split into two parts and assigned to segment A and segment B respectively;

[0051] (a3) Set the direction value to indicate the direction of the segmentation. A direction value of 0 means that the subarray x*y is segmented with a horizontal line (corresponding to Figure 4 5), direction value 1 means to split the subarray (x*y) with a vertical line (corresponding to Figure 4 12), corresponding to the Y stack and X stack of segments A and B respectively;

[0052] (a4) Set the segmentation distance distance to represent the lower boundary of the horizontal segmentation line ion array (x*y) (corresponding to Figure 4 4) or the vertical line dividing line ion array (x*y) right border distance (corresponding to Figure 4 line 11);

[0053] In the case of horizontal line segmentation, the distance value is [1,Y], and in the case of vertical line segmentation, the distance value is [1,X]. That is, the array obtained under each segmentation strategy (direction, distance) is the subarray (x*y).

[0054] (a5) For each set of determined (s, l, x, y) arrays, the partitioning strategy is expressed as (direction, distance), and the state transition equation (corresponding to Figure 4 Lines 6 and 13 of ) calculate the segment maximum delay of the partitioning strategy;

[0055] The state transfer equation uses the delay cost evaluation table Lac_Tab[s][l][x][y] obtained by the tensor parallel method and the maximum delay of the optimal partition within segment A previously obtained. The specific formula is as follows:

[0056] When splitting the subarray x*y with direction value 0:

[0057] temp latency =max(Lac tab [s],[l],[x],y-distance),TM[s],[x],[distance])

[0058] When splitting the subarray x*y with direction value 1:

[0059] temp latency =max(Lac tab [s],[l],[x-distance,[y]),TM[s],[distance],[y])

[0060] Among them, temp latency is the maximum delay of the current segmentation strategy, (Lac tab [s],[l],[x],y-distance) and max(Lac tab [s],[l],[x-distance,[y]) are the delay cost evaluation tables corresponding to horizontal and vertical partitioning respectively. TM[s],[x],[distance] and TM[s],[distance],[y] are the maximum delays of the optimal partitioning strategies corresponding to horizontal and vertical partitioning within segment A.

[0061] (a6) Traverse all boundaries s and array partitions (direction, distance), calculate the maximum segment delay TM[l][x][y] under the current (l, x, y) condition, the segment partition LG[l][x][y] under the minimum maximum delay of all segments, and the subarray coordinates TCG[l][x][y].

[0062] Through (a1) to (a6), in the four-layer nested loop of the partition mapping algorithm, in the lth iteration, set (l,x,y), dynamically transform the boundary s∈[0,l-1] at the innermost layer, and obtain TM[l][x][y], LG[l][x][y] and TCG[l][x][y] under the fixed condition (l,x,y). That is, first set l,x unchanged, dynamically adjust y∈[1,Y] according to the horizontal line segmentation, and each time y is adjusted, s∈[0,l-1] needs to be dynamically transformed, so as to obtain TM[l][x][y], LG[l][x][y] and TCG[l][x][y] corresponding to each adjustment of y, until y=Y. Then fix l, y, and dynamically adjust x∈[1,X] according to the vertical line segmentation. Each time x is adjusted, s∈[0,l-1] needs to be dynamically transformed to obtain TM[l][x][y], LG[l][x][y], and TCG[l][x][y] corresponding to each adjustment of x until x=X.

[0063] Finally, the partition mapping algorithm is iterated until l=L, and finally the segment partition LG[L][X][Y] with the maximum delay of all segments minimized, the maximum delay TM[L][X][Y] in the left and right segments under LG[L][X][Y], and the subarray coordinates CG[L][X][Y] allocated to all segments under LG[L][X][Y] are output. The throughput under this partition result is 1 / (TM[L][X][Y]).

[0064] According to steps 1 and 2, the current Transformer model acceleration method based on the on-chip distributed storage architecture is mainly aimed at low-latency scenario requirements, and no one has proposed a Transformer acceleration method for high-throughput scenarios. The method proposed in this embodiment achieves a multiple increase in throughput compared to the existing method, and realizes high-throughput reasoning of the Transformer model.

[0065] Specifically, the pipeline parallel strategy between XY stacked layer groups proposed in this embodiment has the following advantages: ① adapting to different intra-block tensor parallel methods, ② reducing the huge solution space of the mapping problem, and ③ avoiding the intricate data flow between pipeline stages. The proposed partition mapping mathematical model based on dynamic programming can solve the partition mapping problem of the network model at one time and realize high-throughput reasoning of the Transformer model.

[0066] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A pipeline scheduling method for high-throughput reasoning with Transformer, characterized in that: The steps include: Step 1: XY stacking pipeline parallel strategy: The entire network model is divided into segments according to the Transformer blocks. Each segment is mapped to the chip node array in sequence according to the set rules. Each segment is executed in a pipeline parallel manner on the chip node array to form a pipeline stage. The pipeline stage is executed in a tensor parallel manner. Step 2: Use the XY stacking pipeline parallel strategy to build a partitioning and mapping algorithm based on dynamic programming, and obtain the highest throughput pipeline parallel partitioning and mapping scheme under a given network and chip node array based on the partitioning and mapping algorithm.

2. The pipeline scheduling method for Transformer high-throughput reasoning according to claim 1, characterized in that: In step 1, the entire network model is divided into segments according to the Transformer blocks, and each segment is mapped to the chip node array in sequence according to the set rules. The set rules are as follows: The sub-array of each pipeline stage is rectangular, and the sub-array is the area occupied by the pipeline stage in the chip node array; For any pipeline stage, the previous pipeline stage and the subarray of this pipeline stage form a rectangle.

3. The pipeline scheduling method for Transformer high-throughput reasoning according to claim 1, characterized in that: In step 2, the input of the dynamic programming partition mapping algorithm is the number of rows X and columns Y of the chip node array, and the output is the Transformer block partition result with the highest throughput of the complete Transformer model on the entire PNM system and the sub-array coordinates CG[L][X][Y] assigned to all segments under LG[L][X][Y]. The partition results are: fragment partition LG[L][X][Y] under the minimization of the maximum delay of all segments, and the maximum delay TM[L][X][Y] in the left and right segments under LG[L][X][Y], where L is the total number of Transformer blocks.

4. The pipeline scheduling method for high-throughput reasoning of Transformer according to claim 3, characterized in that: The partition mapping algorithm sets up four layers of nested loops, with the nesting from the inside to the outside being the boundary s∈[0, l-1], the number of columns of the subarray y∈[1, Y], the number of rows of the subarray x∈[1, X], and the number of Transformer blocks l∈[1, L]; The partition mapping algorithm uses the total number of Transformer blocks L as the total number of iterations. In the lth iteration, the dynamic programming partition mapping algorithm is as follows: Set the partition boundary s∈[0, l-1] to divide segment 0 to segment l into segment A, (0, s) and segment B, (s+1, l); Split the subarray x*y into two parts and assign them to segment A and segment B respectively; Set the direction value direction to indicate the direction of the split. A direction value of 0 means splitting the subarray x*y with a horizontal line, and a direction value of 1 means splitting the subarray (x*y) with a vertical line, corresponding to the Y stacking and X stacking of segments A and B respectively. Setting the segmentation distance distance represents the distance from the lower boundary of the horizontal segmentation line ion array (x*y) or the right boundary of the vertical segmentation line ion array (x*y). For each set of determined (s, l, x, y) arrays, the partitioning strategy is expressed as (direction, distance), and the segmented maximum delay of the partitioning strategy is calculated through the state transition equation; Traverse all boundaries s and array partitions (direction, distance), calculate the maximum segment delay TM[l][x][y] under the current (l, x, y) condition, the fragment partition LG[l][x][y] and the subarray coordinates TCG[l][x][y] under the minimum maximum delay of all segments.

5. The pipeline scheduling method for high-throughput reasoning of Transformer according to claim 4, characterized in that: The state transfer equation is specifically: When splitting the subarray x*y with direction value 0: temp latency =max(Lac tab [s],[l],[x],y-distance),TM[s],[x],[distance]) When splitting the subarray x*y with direction value 1: temp latency =max(Lac tab [s],[l],[x-distance,[y]),TM[s],[distance],[y]) Among them, temp latency is the maximum delay of the current segmentation strategy, (Lac tab [s], [l], [x], y-distance) and max(Lac tab [s], [l], [x-distance, [y]) are the delay cost evaluation tables corresponding to horizontal and vertical partitioning respectively, TM[s], [x], [distance] and TM[s], [distance], [y] are the maximum delays of the optimal partitioning strategies corresponding to horizontal and vertical partitioning within segment A.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the pipeline scheduling method according to any one of claims 1 to 5 is implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of classification programs, and the plurality of classification programs are used to be called by a processor and execute the pipeline scheduling method according to any one of claims 1 to 5.