Transformer mapping method for low latency and high throughput reasoning

Abstract

This invention discloses a mapping method for low-latency, high-throughput inference of Transformer, relating to the field of artificial intelligence technology. It deploys the network model on a chip with an on-chip distributed storage architecture to improve the inference speed of the network model on hardware. Specifically, it includes the following steps: Step 1: X-Y stacked pipeline parallel strategy: The entire network model is divided into layers according to Transformer blocks. Each layer is mapped sequentially to the chip node array according to a set rule. Layers are executed in pipeline parallelism in different regions of the chip, and within each layer, they are executed in tensor parallelism. Step 2: Constructing the objective function and constraints of the mathematical model based on integer linear programming partitioning and mapping to achieve high-throughput inference of the network model under latency constraints. This low-latency, high-throughput inference mapping method realizes high-throughput inference of the Transformer model under latency constraints.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a mapping method for low-latency and high-throughput reasoning of a Transformer. Background Art

[0002] The self-attention mechanism is a powerful tool for modeling long-term dependencies in sequential data (such as language). Deep neural networks based on the self-attention mechanism are widely used in various computer (CV) tasks, such as image classification, object detection, and semantic segmentation, and have achieved good results. The computing architecture based on on-chip distributed storage has scalable memory bandwidth and low memory access overhead, which can effectively alleviate the memory access bottleneck problem of the Transformer model, making it a possible solution to accelerate the Transformer.

[0003] There are many Transformer model acceleration methods based on on-chip distributed storage architecture, such as head-level parallelism, patch-level parallelism, and head-patch-level parallelism, which are mainly aimed at low-latency scenario requirements (such as autonomous driving). In actual applications, there are some scenarios that do not have such high requirements for latency but also have requirements for throughput. Such as multi-camera batch reasoning tasks in autonomous driving and batch reasoning tasks of edge cloud servers. Previous technical methods only used low latency as an optimization goal and could not achieve the highest throughput reasoning under a certain latency in these scenarios. Summary of the invention

[0004] Based on the technical problems existing in the background technology, the present invention proposes a mapping method for low-latency and high-throughput reasoning of Transformer to achieve high-throughput reasoning under the delay constraints of the Transformer model.

[0005] The mapping method for low-latency and high-throughput reasoning of Transformer proposed in the present invention deploys the network model on a chip with an on-chip distributed storage architecture to improve the reasoning speed of the network model on hardware; the method for deploying the network model on the on-chip distributed storage architecture includes the following steps:

[0006] Step 1: XY stacking pipeline parallel strategy: The entire network model is divided into layer groups according to the Transformer blocks. Each layer group is mapped to the chip node array in sequence according to the set rules. The layer groups are executed in pipeline parallel in different areas of the chip, and in tensor parallel within the layer groups.

[0007] Step 2: Construct the objective function and constraint conditions of the mathematical model based on the partitioning and mapping of integer linear programming to achieve high-throughput reasoning of the network model under delay constraints;

[0008] The objective function of the mathematical model is to minimize the maximum delay in all layer groups as the optimization goal, specifically:

[0009] ;

[0010] in, Represents the first Transformer block of the current layer group, represents the first Transformer block of the next layer group, Respectively represent the number of columns and rows of the chip node array occupied by the current layer group, represents the product, Represents the first Transformer block To the last Transformer block Is it mapped to a size of On the subarray of Represents the first Transformer block in the current layer group To the last Transformer block In size The execution time on an array of chip nodes.

[0011] Furthermore, in step 1, the entire network model is divided into layer groups by block, and each layer group is sequentially mapped to the chip node array according to the set rules. The set rules are as follows:

[0012] The chip node array occupied by each layer group is rectangular;

[0013] For any layer group, the previous layer group and the chip node array occupied by the layer group form a rectangle.

[0014] Furthermore, in step 2, the constraints of the mathematical model are:

[0015] Shape constraint of chip node array occupied by layer group: for any layer group, the chip node array occupied by the previous layer group and the layer group forms a rectangle;

[0016] The Transformer blocks contained in adjacent layer groups are continuous;

[0017] Each Transformer block is assigned to a layer group;

[0018] All chip nodes are used;

[0019] The inference delay of the entire network model does not exceed the maximum execution time .

[0020] Furthermore, the formula corresponding to the constraint conditions of the mathematical model is expressed as follows:

[0021] ;

[0022] ;

[0023] ;

[0024] ;

[0025] ;

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] in, Represents an integer constant, and Respectively represent the number of rows and columns of the entire chip node array, Respectively represent The number of columns and rows of the chip node array occupied by each layer group, Indicates The stacking direction of the layer groups, Respectively represent Layer groups and The number of columns and rows of the chip node array occupied by the previous layer group, Indicates The first Transformer block number of the layer group, The first Transformer block of the 0th layer group is numbered 0. Indicates The first Transformer block number of the layer group, Indicates Layer groups contain Transformer blocks , Indicates Layer groups contain Transformer blocks , Indicates Layer groups relative The stacking direction of the previous layer group and the size of the chip node array it occupies. D = 0 means stacking to the right, and D = 1 means stacking upward. Represents the first Transformer block in the current layer group To the last Transformer block In size The execution time on the chip node array of is given by the Transformer block tensor parallel method, which is a constant multidimensional array, Indicates the number of layer groups, is the maximum execution time, Represents the number of Transformer blocks.

[0031] The advantages of the Transformer low-latency high-throughput reasoning mapping method provided by the present invention are: the proposed XY stacking layer group pipeline parallel strategy can ① adapt to different intra-block tensor parallel methods; ② reduce the huge solution space of the mapping problem; ③ avoid the intricate data flow between pipeline levels; the proposed partitioning and mapping mathematical model based on integer linear programming (ILP) can solve the partitioning and mapping problems of the network model at one time. Through these methods, high-throughput reasoning of the Transformer model under delay constraints can be realized in batch reasoning scenarios such as autonomous driving and edge cloud servers. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the process of the present invention;

[0033] Figure 2 Schematic diagram of the layer group division strategy;

[0034] Figure 3 for Figure 2 Schematic diagram of the mapping strategy for the layer groups after division. DETAILED DESCRIPTION

[0035] Below, the technical solution of the present invention is described in detail through specific embodiments. Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific implementation disclosed below.

[0036] For the scenarios of multi-camera data processing in autonomous driving and multi-user data processing of edge cloud devices, which do not have such high requirements for latency but also have requirements for throughput, this embodiment is aimed at a general distributed computing architecture (such as a near-memory computing architecture based on DRAM or SRAM). It is an expandable 2D spatial computing node array connected by a mesh-based network on chip (NoC). Each computing node integrates computing and storage units (8MB), and consists of a neural network computing engine (NN engine) for computing, a network on chip router, a memory submodule, a memory submodule controller, and an auxiliary computing unit (ACU). Among them, the neural network computing engine consists of a buffer SRAM and a processing element (PE) array for multiplication and accumulation (MAC) operations in DNN. The PE array can be a systolic array, an NVDLA-type array, or a coarse-grained reconfigurable array (CGRA), etc.

[0037] The Transformer model is composed of stacked Transformer blocks, which generally have 4 stages, with several serially processed blocks in each stage. Each block contains the normalization (layer normal), full connection (FC), residual connection (resudial), and multi-head self-attention (MSA) mechanisms that can be processed in series. The Transformer model often introduces a local attention mechanism, which divides the feature map into several fixed-size regions and performs independent and parallel-processable Transformer block calculations in each region.

[0038] This embodiment proposes a mapping method for low-latency and high-throughput reasoning of Transformer. The Transformer model is executed on the chip in a pipeline parallel manner, and each level of the pipeline is executed in a tensor parallel manner. First, an XY stacking pipeline parallel strategy for multi-level pipelines on a node array is introduced. Then, a partitioning and mapping algorithm based on integer linear programming (ILP) is given, which is described in detail below.

[0039] like Figures 1 to 3 As shown, the mapping method for low-latency and high-throughput reasoning of Transformer proposed in the present invention deploys the network model on a chip with an on-chip distributed storage architecture to improve the reasoning speed of the network model on hardware. The method for deploying the network model on the on-chip distributed storage architecture includes the following steps:

[0040] Step 1: XY stacking pipeline parallel strategy: The entire network model is divided into layer groups according to the Transformer blocks. Each layer group is mapped to the chip node array in sequence according to the set rules. The layer groups are executed in pipeline parallel in different areas of the chip, and in tensor parallel within the layer groups.

[0041] The Transformer model is composed of several stacked blocks. In order to achieve high-throughput reasoning under certain delay constraints, the entire network model is divided into layer groups by blocks. The layer groups are executed in a pipeline parallel manner in different areas of the chip, and the layer groups are executed in a tensor parallel manner. The existing Transformer block tensor parallel methods generally include: head-level parallelism, patch-level parallelism, and head-patch-level parallelism. These methods require that the chip node array occupied by the block is rectangular in shape. The inter-layer group pipeline parallelism method proposed in this embodiment can adapt to these different block tensor parallel methods, and each layer group is executed in a block serial manner on different rectangular node arrays.

[0042] The XY stacking layer group pipeline parallel strategy is as follows Figure 2 and 3 As shown, the entire network model is divided into several layer groups by block, and each layer group is mapped to the chip node array in sequence according to the following rules:

[0043] 1) The chip node array occupied by each layer group is rectangular;

[0044] Each layer group is composed of several serial Transformer blocks. The existing Transformer block tensor parallel method generally requires that the chip node array occupied by the block is rectangular or linear. This embodiment stipulates that the node array occupied by each layer group must be rectangular, so that different intra-block tensor parallel methods can be adapted.

[0045] 2) For any layer group, the previous layer group and the chip node array occupied by the layer group form a rectangle;

[0046] The mapping problem of multiple layer groups on the chip node array is similar to the two-dimensional bin packing problem, which is an NP-hard problem with a huge solution space. In order to reduce the solution space, it is stipulated that for any layer group, its previous layer group and the chip node array occupied by this layer group must form a rectangle. At the same time, this rule can also ensure that data communication between different layer groups will not cross other layer groups, and the data flow is relatively regular.

[0047] Step 2: Construct the objective function and constraint conditions of the mathematical model based on the partitioning and mapping of integer linear programming to achieve high-throughput reasoning of the network model under delay constraints;

[0048] According to the partitioning and mapping strategy defined in step 1, this embodiment establishes the partitioning and mapping problem of the Transformer model as a mathematical model based on integer linear programming to achieve high-throughput reasoning of the network model under delay constraints. First, the variables in the mathematical model are explained:

[0049] Binary variables Indicates Layer groups, including Transformer blocks Binary variables :Indicates the The stacking direction and array size of each layer group relative to the previous layer group. D = 0 means stacking to the right, and D = 1 means stacking upward.

[0050] Constraints:

[0051] (1);

[0052] (2);

[0053] (3);

[0054] (4);

[0055] (5);

[0056] (6);

[0057] (7);

[0058] (8);

[0059] (9);

[0060] in, Represents an integer constant, and Respectively represent the number of rows and columns of the entire chip node array, Respectively represent The number of columns and rows of the chip node array occupied by each layer group, Indicates The stacking direction of the layer groups, Respectively represent Layer groups and The number of columns and rows of the chip node array occupied by the previous layer group, Indicates The first Transformer block number of the layer group, The first Transformer block of the 0th layer group is numbered 0. Indicates The first Transformer block number of the layer group, Indicates Layer groups contain Transformer blocks , Indicates Layer groups contain Transformer blocks , Indicates Layer groups relative The stacking direction of the previous layer group and the size of the chip node array it occupies. D = 0 means stacking to the right, and D = 1 means stacking upward. Represents the first Transformer block in the current layer group To the last Transformer block In size The execution time on the chip node array of is given by the Transformer block tensor parallel method, which is a constant multidimensional array, Indicates the number of layer groups, Represents the number of Transformer blocks.

[0061] Constraints (1) and (2) are shape constraints for the chip node array occupied by the layer group, ensuring that for any layer group, its previous layer group and the chip node array occupied by the layer group form a rectangle. , indicating the The number of rows in the array that each layer group occupies. , indicating the The number of columns in the array that each layer group occupies. , indicating the The number of rows in the array that the layer group and its predecessor occupy. , indicating the The number of columns in the array that the layer group and its previous layer groups occupy, is the indicator function.

[0062] Constraints (3) and (4) ensure that the Transformer blocks contained in adjacent layer groups are continuous. Indicates The first Transformer block contained in the layer group. , Indicates The last Transformer block of a layer group.

[0063] Constraint (5) ensures that each Transformer block is assigned to a layer group.

[0064] Constraints (6) and (7) ensure that all chip node arrays are used.

[0065] Constraints (8) and (9) mean that the inference delay of the entire network model does not exceed .

[0066] in, ; Represents a Transformer block In size The execution time on the chip node array of is given by the Transformer block tensor parallel method, which is a constant multidimensional array.

[0067] Objective function:

[0068] ;

[0069] in, Represents the first Transformer block of the current layer group, represents the first Transformer block of the next layer group, Respectively represent the number of columns and rows of the chip node array occupied by the current layer group, represents the product, Represents the first Transformer block To the last Transformer block Is it mapped to a size of On the subarray of Represents the first Transformer block in the current layer group To the last Transformer block In size The execution time on an array of chip nodes.

[0070] This embodiment is based on the partition mapping algorithm of integer linear programming (ILP), and the optimization goal is to minimize the maximum delay in all layer groups.

[0071] Currently, the Transformer model acceleration method based on the on-chip distributed storage architecture is mainly aimed at low-latency scenario requirements. The method proposed in this embodiment can achieve an exponential increase in throughput on the basis of sacrificing some latency, thereby realizing high-throughput reasoning under the latency constraints of the Transformer model.

[0072] Specifically, the pipeline parallel strategy between XY stacked layer groups proposed in this embodiment can ① adapt to different intra-block tensor parallel methods; ② reduce the huge solution space of the mapping problem; ③ avoid the intricate data flow between pipeline stages. The proposed mathematical model of partitioning and mapping based on integer linear programming (ILP) can solve the partitioning and mapping problems of the network model at one time, and realize high-throughput reasoning under the delay constraint of the Transformer model.

[0073] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. Transformer low-latency high-throughput reasoning mapping method, characterized by: Deploy the network model on a chip with an on-chip distributed storage architecture to improve the inference speed of the network model on hardware; The method for deploying the network model on the on-chip distributed storage architecture comprises the following steps: Step 1: XY stacking pipeline parallel strategy: The entire network model is divided into layer groups according to the Transformer blocks. Each layer group is mapped to the chip node array in sequence according to the set rules. The layer groups are executed in pipeline parallel in different areas of the chip, and in tensor parallel within the layer groups. Step 2: Construct the objective function and constraint conditions of the mathematical model based on the partitioning and mapping of integer linear programming to achieve high-throughput reasoning of the network model under delay constraints; The objective function of the mathematical model is to minimize the maximum delay in all layer groups as the optimization goal, specifically: in, Represents the first Transformer block of the current layer group, represents the first Transformer block of the next layer group, Respectively represent the number of columns and rows of the chip node array occupied by the current layer group, represents the product, Represents the first Transformer block To the last Transformer block Is it mapped to a size of On the subarray of Represents the first Transformer block in the current layer group To the last Transformer block In size Execution time on an array of chip nodes; The formula corresponding to the constraints of the mathematical model is expressed as follows: in, Represents an integer constant, and Respectively represent the number of rows and columns of the entire chip node array, Respectively represent The number of columns and rows of the chip node array occupied by each layer group, Indicates The stacking direction of the layer groups, Respectively represent Layer groups and The number of columns and rows of the chip node array occupied by the previous layer group, Indicates The first Transformer block number of the layer group, The first Transformer block of the 0th layer group is numbered 0. Indicates The first Transformer block number of the layer group, Indicates Layer groups contain Transformer blocks , Indicates Layer groups contain Transformer blocks , Indicates Layer groups relative The stacking direction of the previous layer group and the size of the chip node array it occupies. D = 0 means stacking to the right, and D = 1 means stacking upward. Represents the first Transformer block in the current layer group To the last Transformer In size The execution time on the chip node array of is given by the Transformer block tensor parallel method, which is a constant multidimensional array, Indicates the number of layer groups, is the maximum execution time, Represents the number of Transformer blocks.

2. The mapping method for low-latency and high-throughput reasoning of Transformer according to claim 1, characterized in that: In step 1, the entire network model is divided into layer groups by block, and each layer group is sequentially mapped to the chip node array according to the set rules. The set rules are as follows: The chip node array occupied by each layer group is rectangular; For any layer group, the previous layer group and the chip node array occupied by the layer group form a rectangle.

3. The mapping method for low-latency and high-throughput reasoning of Transformer according to claim 1, characterized in that: In step 2, the constraints of the mathematical model are: Shape constraint of chip node array occupied by layer group: for any layer group, the chip node array occupied by the previous layer group and the layer group forms a rectangle; The Transformer blocks contained in adjacent layer groups are continuous; Each Transformer block is assigned to a layer group; All chip nodes are used; The inference delay of the entire network model does not exceed the maximum execution time .

Images