Transform low-delay high-throughput reasoning mapping method

Abstract

The invention discloses a Transform low-delay high-throughput reasoning mapping method, relates to the technical field of artificial intelligence, and deploys a network model on a chip of an on-chip distributed storage architecture so as to improve the reasoning speed of the network model on hardware and improve the reasoning speed of the network model on hardware. The method specifically comprises the following steps: step 1, an X-Y stacking assembly line parallel strategy: a whole network model is segmented into layer groups according to Transform blocks, each layer group is sequentially mapped to a chip node array according to a set rule, the layer groups are executed in different areas of a chip in an assembly line parallel mode, and the interior of each layer group is executed in a tensor parallel mode; step 2, constructing an objective function and constraint conditions of a mathematical model based on division and mapping of integer linear programming so as to realize network model high throughput reasoning under delay constraint; according to the mapping method for low-delay high-throughput reasoning, high-throughput reasoning under the delay constraint of the Transform model is realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a mapping method for low-latency and high-throughput reasoning of a Transformer. Background Art

[0002] The self-attention mechanism is a powerful tool for modeling long-term dependencies in sequential data, such as language. Deep neural networks based on the self-attention mechanism have been widely used in various computer science (CV) tasks, such as image classification, object detection, and semantic segmentation, achieving excellent results. Computing architectures based on on-chip distributed storage offer scalable memory bandwidth and low memory access overhead, effectively alleviating the memory access bottleneck of the Transformer model, making it a potential solution for accelerating the Transformer.

[0003] Currently, there are many Transformer model acceleration methods based on on-chip distributed storage architectures, such as head-level parallelism, patch-level parallelism, and head-patch-level parallelism. These methods are primarily targeted at low-latency scenarios (such as autonomous driving). However, in practice, some scenarios require less stringent latency requirements but also high throughput. These include multi-camera batch inference tasks in autonomous driving and batch inference tasks on edge cloud servers. Previous technical approaches focused solely on low latency and were unable to achieve the highest inference throughput under a certain latency requirement in these scenarios. Summary of the Invention

[0004] Based on the technical problems existing in the background technology, the present invention proposes a mapping method for low-latency and high-throughput reasoning of Transformer to achieve high-throughput reasoning under the delay constraints of the Transformer model.

[0005] The Transformer low-latency and high-throughput inference mapping method proposed in this invention deploys the network model on a chip with an on-chip distributed storage architecture to improve the inference speed of the network model on hardware. The method for deploying the network model on the on-chip distributed storage architecture includes the following steps: Step 1: XY stacking pipeline parallel strategy: The entire network model is divided into layer groups according to Transformer blocks. Each layer group is sequentially mapped to the chip node array according to the set rules. The layer groups are executed in pipeline parallelism in different areas of the chip, and in tensor parallelism within the layer groups. Step 2: Based on the partitioning and mapping of integer linear programming, the objective function and constraints of the mathematical model are constructed to achieve high-throughput reasoning of the network model under latency constraints. The objective function of the mathematical model is to minimize the maximum delay in all layer groups as the optimization goal, specifically: ; in, Represents the first Transformer block of the current layer group, represents the first Transformer block of the next layer group, Respectively represent the number of columns and rows of the chip node array occupied by the current layer group, represents the product, Represents the first Transformer block To the last Transformer block Is it mapped to a size of On the subarray, Represents the first Transformer block in the current layer group To the last Transformer block In size Execution time on an array of chip nodes.

[0006] Furthermore, in step 1, the entire network model is divided into layer groups by block, and each layer group is sequentially mapped to the chip node array according to the set rules. The set rules are as follows: The chip node array occupied by each layer group is rectangular; For any layer group, the previous layer group and the chip node array occupied by the layer group form a rectangle.

[0007] Furthermore, in step 2, the constraints of the mathematical model are: Shape constraint of the chip node array occupied by the layer group: For any layer group, the previous layer group and the chip node array occupied by the layer group form a rectangle; The Transformer blocks contained in adjacent layer groups are continuous; Each Transformer block is assigned to a layer group; All chip nodes are used; The inference delay of the entire network model does not exceed the maximum execution time .

[0008] Furthermore, the formula corresponding to the constraint conditions of the mathematical model is expressed as follows: ; ; ; ; ; ; ; ; ; in, Represents an integer constant, and Respectively represent the number of rows and columns of the entire chip node array, Respectively represent The number of columns and rows of the chip node array occupied by each layer group, Indicates the The stacking direction of the layer groups, Respectively represent Layer groups and The number of columns and rows of the chip node array occupied by the previous layer group, Indicates the The first Transformer block number of the layer group, The first Transformer block of the 0th layer group is numbered 0. Indicates the The first Transformer block number of the layer group, Indicates the Layer groups contain Transformer blocks , Indicates the Layer groups contain Transformer blocks , Indicates the Layer groups relative The stacking direction of the previous layer group and the size of the chip node array it occupies, D = 0 means stacking to the right, D = 1 means stacking upwards, Represents the first Transformer block in the current layer group To the last Transformer block In size The execution time on the chip node array of is given by the Transformer block tensor parallel method, which is a constant multidimensional array, Indicates the number of layer groups, is the maximum execution time, Indicates the number of Transformer blocks.

[0009] The advantages of the proposed Transformer mapping method for low-latency, high-throughput inference are as follows: the proposed inter-group pipeline parallelism strategy for XY stacking layers can 1) adapt to different intra-block tensor parallelism methods; 2) reduce the vast solution space of the mapping problem; and 3) avoid the complex data flow between pipeline stages. Furthermore, the proposed mathematical model for partitioning and mapping based on integer linear programming (ILP) can solve the partitioning and mapping problems of the network model in a single step. These methods enable high-throughput inference of Transformer models under latency constraints in batch inference scenarios such as autonomous driving and edge cloud servers. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 Schematic diagram of the layer group division strategy; Figure 3 for Figure 2 Schematic diagram of the mapping strategy for the divided layer groups. DETAILED DESCRIPTION

[0011] The technical solutions of the present invention are described in detail below through specific embodiments. Numerous specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0012] For scenarios such as multi-camera data processing in autonomous driving and multi-user data processing in edge cloud devices, which have less stringent latency requirements but still demanding throughput, this embodiment targets a general-purpose distributed computing architecture (such as a near-memory computing architecture based on DRAM or SRAM). It comprises a scalable 2D spatial array of computing nodes connected via a mesh-based network-on-chip (NoC). Each computing node integrates computing and storage units (8MB) and consists of a neural network computing engine (NN engine), an on-chip network router, a memory submodule, a memory submodule controller, and an auxiliary computing unit (ACU). The NN engine consists of an SRAM buffer for multiply-accumulate (MAC) operations in the DNN and a processing element (PE) array. The PE array can be a systolic array, an NVDLA-type array, or a coarse-grained reconfigurable array (CGRA).

[0013] The Transformer model is composed of stacked Transformer blocks, typically consisting of four stages, with each stage containing several blocks processed serially. Each block includes layer normalization, fully connected (FC), residual connections (resudial), and multi-head self-attention (MSA) mechanisms, all of which can be processed serially. Transformer models often incorporate a local attention mechanism, partitioning the feature map into several fixed-size regions. Within each region, independent, parallelized Transformer block computations are performed.

[0014] This example proposes a mapping method for low-latency, high-throughput Transformer inference. The Transformer model is executed on the chip in a pipeline-parallel manner, with each pipeline stage executing in tensor parallelism. First, an XY stacking pipeline parallelism strategy for multi-stage pipelines on a node array is introduced. Then, a partitioning and mapping algorithm based on integer linear programming (ILP) is presented, which is described in detail below.

[0015] like Figures 1 to 3 As shown, the Transformer low-latency and high-throughput reasoning mapping method proposed in the present invention deploys the network model on a chip with an on-chip distributed storage architecture to improve the reasoning speed of the network model on hardware. The method for deploying the network model on the on-chip distributed storage architecture includes the following steps: Step 1: XY stacking pipeline parallel strategy: The entire network model is divided into layer groups according to Transformer blocks. Each layer group is sequentially mapped to the chip node array according to the set rules. The layer groups are executed in pipeline parallelism in different areas of the chip, and in tensor parallelism within the layer groups. The Transformer model is composed of several stacked blocks. In order to achieve high-throughput reasoning under certain delay constraints, the entire network model is divided into layer groups by blocks. The layer groups are executed in a pipeline parallel manner in different areas of the chip, and are executed in a tensor parallel manner within the layer groups. Existing Transformer block tensor parallel methods generally include: head-level parallelism, patch-level parallelism, and head-patch-level parallelism. These methods require that the chip node array occupied by the block be rectangular in shape. The inter-layer-group pipeline parallelism method proposed in this embodiment can adapt to these different block tensor parallel methods, and each layer group is executed in a block serial manner on different rectangular node arrays.

[0016] The pipeline parallel strategy between XY stacking layers is as follows Figure 2 and 3 As shown, the entire network model is divided into several layer groups by block, and each layer group is mapped to the chip node array in sequence according to the following rules: 1) The chip node array occupied by each layer group is rectangular; Each layer group consists of several serial Transformer blocks. Existing Transformer block tensor parallelism methods generally require the chip node array occupied by the block to be rectangular or linear in shape. This embodiment specifies that the node array occupied by each layer group must be rectangular, which allows for adaptability to different intra-block tensor parallelism methods.

[0017] 2) For any layer group, the previous layer group and the chip node array occupied by the layer group form a rectangle; The mapping of multiple layer groups onto a chip node array is similar to the two-dimensional bin packing problem, an NP-hard problem with a vast solution space. To reduce this solution space, for each layer group, its predecessor and the chip node array occupied by the layer group must form a rectangle. This rule also ensures that data communication between different layer groups does not cross other layer groups, maintaining a more regular data flow.

[0018] Step 2: Based on the partitioning and mapping of integer linear programming, the objective function and constraints of the mathematical model are constructed to achieve high-throughput reasoning of the network model under latency constraints. Based on the partitioning and mapping strategy defined in step 1, this embodiment establishes the partitioning and mapping problem of the Transformer model as a mathematical model based on integer linear programming to achieve high-throughput reasoning of network models under latency constraints. First, the variables in the mathematical model are explained: binary variables Indicates the Layer groups, including Transformer blocks . Binary variable :Indicates the The stacking direction and array size of each layer group relative to the previous layer group. D = 0 means stacking to the right, and D = 1 means stacking upward.

[0019] Constraints: (1); (2); (3); (4); (5); (6); (7); (8); (9); in, Represents an integer constant, and Respectively represent the number of rows and columns of the entire chip node array, Respectively represent The number of columns and rows of the chip node array occupied by each layer group, Indicates the The stacking direction of the layer groups, Respectively represent Layer groups and The number of columns and rows of the chip node array occupied by the previous layer group, Indicates the The first Transformer block number of the layer group, The first Transformer block of the 0th layer group is numbered 0. Indicates the The first Transformer block number of the layer group, Indicates the Layer groups contain Transformer blocks , Indicates the Layer groups contain Transformer blocks , Indicates the Layer groups relative The stacking direction of the previous layer group and the size of the chip node array it occupies, D = 0 means stacking to the right, D = 1 means stacking upwards, Represents the first Transformer block in the current layer group To the last Transformer block In size The execution time on the chip node array of is given by the Transformer block tensor parallel method, which is a constant multidimensional array, Indicates the number of layer groups, Indicates the number of Transformer blocks.

[0020] Constraints (1) and (2) are shape constraints for the chip node array occupied by the layer group, ensuring that for any layer group, its previous layer group and the chip node array occupied by the layer group form a rectangle. , indicating the The number of rows in the array that each layer group occupies. , indicating the The number of array columns that each layer group occupies. , indicating the The number of rows in the array that the layer group and the layer groups before it occupy. , indicating the The number of rows in the array that the layer group and its predecessor occupy, is the indicator function.

[0021] Constraints (3) and (4) ensure that the Transformer blocks contained in adjacent layer groups are continuous. Indicates the The first Transformer block contained in the layer group. , Indicates the The last Transformer block of the layer group. Constraint (5) ensures that each Transformer block is assigned to a layer group.

[0022] Constraints (6) and (7) ensure that all chip node arrays are used.

[0023] Constraints (8) and (9) indicate that the inference delay of the entire network model does not exceed .

[0024] in, ; Represents a Transformer block In size The execution time on the chip node array of is given by the Transformer block tensor parallel method, which is a constant multidimensional array.

[0025] Objective function: ; in, Represents the first Transformer block of the current layer group, represents the first Transformer block of the next layer group, Respectively represent the number of columns and rows of the chip node array occupied by the current layer group, represents the product, Represents the first Transformer block To the last Transformer block Is it mapped to a size of On the subarray, Represents the first Transformer block in the current layer group To the last Transformer block In size Execution time on an array of chip nodes.

[0026] This embodiment is based on an integer linear programming (ILP) partitioning and mapping algorithm, and the optimization goal is to minimize the maximum delay in all layer groups.

[0027] Currently, the Transformer model acceleration method based on on-chip distributed storage architecture is mainly aimed at low-latency scenario requirements. The method proposed in this embodiment can achieve an exponential increase in throughput while sacrificing some latency, thereby realizing high-throughput reasoning under the latency constraints of the Transformer model.

[0028] Specifically, the inter-group pipeline parallelism strategy for XY stacking layers proposed in this embodiment can 1. adapt to different intra-block tensor parallelism methods; 2. reduce the vast solution space of the mapping problem; and 3. avoid the complex data flow between pipeline stages. The proposed mathematical model for partitioning and mapping based on integer linear programming (ILP) can solve the partitioning and mapping problems of the network model in one go, enabling high-throughput inference within the latency constraints of the Transformer model.

[0029] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. Transformer low-latency high-throughput reasoning mapping method, characterized by: Deploy the network model on a chip with an on-chip distributed storage architecture to improve the inference speed of the network model on hardware; The method for deploying the network model on the on-chip distributed storage architecture comprises the following steps: Step 1: XY stacking pipeline parallel strategy: The entire network model is divided into layer groups according to the Transformer blocks. Each layer group is mapped to the chip node array in sequence according to the set rules. The layer groups are executed in pipeline parallel in different areas of the chip, and in tensor parallel within the layer groups. Step 2: Construct the objective function and constraint conditions of the mathematical model based on the partitioning and mapping of integer linear programming to achieve high-throughput reasoning of the network model under delay constraints; The objective function of the mathematical model is to minimize the maximum delay in all layer groups as the optimization goal, specifically: in, Represents the first Transformer block of the current layer group, represents the first Transformer block of the next layer group, Respectively represent the number of columns and rows of the chip node array occupied by the current layer group, represents the product, Represents the first Transformer block To the last Transformer block Is it mapped to a size of On the subarray of Represents the first Transformer block in the current layer group To the last Transformer block In size The execution time on an array of chip nodes.

2. The mapping method for low-latency and high-throughput reasoning of Transformer according to claim 1, characterized in that: In step 1, the entire network model is divided into layer groups by block, and each layer group is sequentially mapped to the chip node array according to the set rules. The set rules are as follows: The chip node array occupied by each layer group is rectangular; For any layer group, the previous layer group and the chip node array occupied by the layer group form a rectangle.

3. The mapping method for low-latency and high-throughput reasoning of Transformer according to claim 1, characterized in that: In step 2, the constraints of the mathematical model are: Shape constraint of chip node array occupied by layer group: for any layer group, the chip node array occupied by the previous layer group and the layer group forms a rectangle; The Transformer blocks contained in adjacent layer groups are continuous; Each Transformer block is assigned to a layer group; All chip nodes are used; The inference delay of the entire network model does not exceed the maximum execution time .

4. The mapping method for low-latency and high-throughput reasoning of Transformer according to claim 1, characterized in that: The formula corresponding to the constraints of the mathematical model is expressed as follows: in, Represents an integer constant, and Respectively represent the number of rows and columns of the entire chip node array, Respectively represent The number of columns and rows of the chip node array occupied by each layer group, Indicates The stacking direction of the layer groups, Respectively represent Layer groups and The number of columns and rows of the chip node array occupied by the previous layer group, Indicates The first Transformer block number of the layer group, The first Transformer block of the 0th layer group is numbered 0. Indicates The first Transformer block number of the layer group, Indicates Layer groups contain Transformer blocks , Indicates Layer groups contain Transformer blocks , Indicates Layer groups relative The stacking direction of the previous layer group and the size of the chip node array it occupies. D = 0 means stacking to the right, and D = 1 means stacking upward. Represents the first Transformer block in the current layer group To the last Transformer In size The execution time on the chip node array of is given by the Transformer block tensor parallel method, which is a constant multidimensional array, Indicates the number of layer groups, is the maximum execution time, Represents the number of Transformer blocks.

Images