A multi-head attention parallel granularity adaptive method for a transformer model operation

Abstract

The application belongs to the technical field of machine learning model acceleration based on Transformer, and particularly relates to a multi-head attention parallel granularity adaptive method for Transformer model operation, which comprises the following steps: quantitatively analyzing the calculation time Tc and the memory access time Tm of each layer of a Transformer model by respectively equating the parallel degree and the memory efficiency of each attention layer in a model inference process to the calculation time Tc and the memory access time Tm of each layer of the Transformer model; based on the quantitative analysis, an efficient interval jump search algorithm is proposed, the search space is divided into two intervals, and the slope of the secant line is calculated to find the interval where the optimal granularity is located. In the inference process of the Transformer model, the algorithm can quickly estimate and lock the optimal parallel granularity (i.e. the number of attention heads processed at the same time) under the current sequence length by using the execution time (calculation time and memory access time) information of the previous several layers during the forward inference of the model, so as to realize the collaborative optimization of the parallel degree and the memory bandwidth utilization at the root.

Description

Technical Field

[0001] This invention belongs to the technical field of Transformer-based machine learning model acceleration, and more specifically, relates to a multi-head attention parallel granularity adaptive method for Transformer model operation. Background Technology

[0002] Since its introduction in 2017, the Transformer model has demonstrated powerful capabilities in artificial intelligence tasks such as natural language processing, image processing, and speech recognition. Its core self-attention mechanism effectively captures long-distance dependencies between elements in an input sequence, providing robust foundational model support for various complex tasks. However, due to the complex memory access patterns and frequent transfers of large amounts of intermediate data involved in self-attention computation, the Transformer model exhibits relatively low energy efficiency on general-purpose processors (such as CPUs and GPUs), typically achieving only 5%-15% of its theoretical peak performance.

[0003] To address this issue, academia and industry have proposed various dedicated Transformer hardware accelerator designs in recent years. These accelerators significantly improve the execution efficiency of Transformer models through customized dataflow architectures, dedicated computing units, and optimized memory hierarchies. For example, existing technologies such as FACT and FTRANS have achieved up to 10 times the energy efficiency of GPUs in specific scenarios. These accelerators generally employ head-level parallelism strategies, such as... Figure 1 As shown, this involves processing the computations of multiple attention heads simultaneously in parallel to fully utilize hardware computing resources. Figure 1 (a) in the diagram represents the model structure of the Transformer, which mainly consists of several attention layers, each of which contains multiple attention heads that can be computed in parallel. Figure 1 In (b), it represents the parallel computation method of multi-head attention in the Transformer model, which divides the Q and K matrices of different heads and performs matrix multiplication in parallel.

[0004] However, existing dedicated accelerators face a key challenge in practical applications: they cannot efficiently adapt to diverse input sequence lengths. In practical application scenarios, the sequence length requirements vary greatly for different tasks. For example, document summarization applications typically process long texts containing thousands of tokens, while chat-based applications usually only need to process short sequences of dozens to hundreds of tokens. Since the Transformer model needs to exhaustively analyze the correlation between each pair of tokens, the amount of intermediate results is quadratically related to the sequence length. When the sequence length increases from 512 to 4096, the amount of intermediate results for a single layer can surge from dozens of MB to several GB.

[0005] Existing accelerators (such as FACT, FTRANS) generally adopt a fixed parallelization strategy, that is, they always maximize the computing parallelism and process all attention heads in parallel simultaneously. This design philosophy can fully utilize the computing resource utilization rate in short sequence scenarios (such as when the sequence length is 512), because all intermediate results can be cached in the on-chip memory. However, when the sequence length increases, these intermediate results quickly exceed the on-chip cache capacity, resulting in frequent off-chip memory accesses. As shown in Figure 2 (a) shows the proportion of the sources of memory access latency at different sequence lengths. As the sequence length increases, the off-chip memory access latency increases significantly; Figure 2 (b) shows the impact of the parallel granularity of attention heads on the model inference performance at different sequence lengths. As the sequence length increases, the optimal parallel granularity of attention heads changes dynamically. The experimental data in the figure shows that when the sequence length increases from 512 to 4096, the memory access latency of accelerators using a fixed maximum parallel granularity increases by 5.47 times, and the overall performance may even be 4.17 times lower than the sequential processing method, resulting in a negative acceleration effect.

[0006] More specifically, existing accelerators are generally designed based on an assumption: the on-chip cache is large enough to hold all intermediate results (i.e., , , , and , where 1 < i < H). Under this assumption, the computing efficiency is proportional to the computing parallelism, so existing works adopt a fixed maximum granularity parallel strategy. However, when running large models such as Llama3 - 8B, the amount of intermediate results for a single layer surges from 32 MB when the sequence length is 512 to 2 GB when the sequence length is 4096. In contrast, the on-chip cache capacity of typical Transformer accelerators is usually only in the MB level and cannot hold the intermediate results under long sequences. This huge gap leads to a sharp decline in the performance of existing accelerators when the sequence is long. That is, the existing technology cannot continuously maintain the efficient coordination of computing parallelism and memory efficiency under changing sequence lengths. Summary of the Invention

[0007] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a Transformer accelerator design method with adaptive parallel granularity. Its purpose is to propose a Transformer adaptive acceleration mode that can dynamically adjust the parallel granularity and take into account both hardware resource utilization and memory efficiency.

[0008] To achieve the above objectives, according to one aspect of the present invention, a multi-head attention parallel granularity adaptive method for Transformer model operation is provided, comprising: The parallelism and memory efficiency of each attention layer during the Transformer model inference process are respectively equated to the computation time of that layer. and memory access time Quantization is performed to determine the computation time corresponding to the current layer. Search space and memory access time The search space; the computation time and memory access time The current search space is divided into two intervals, and the computation time is... The two corresponding intervals are denoted as ( , )and( , ), memory access time The two corresponding intervals are denoted as ( , )and( , ), execute the interval jump search algorithm to calculate the secant slope. and The optimal parallel granularity is determined by how close the slopes of the secant lines in the two intervals are to -1. That is, if the inequality... If this holds true, it indicates that the optimal granularity is in the previous granularity interval, i.e. ( , )and( , The parallel granularity interval corresponding to ) is, otherwise it is in the next granularity interval; Select a parallel granularity for the current attention layer from the granularity range containing the optimal granularity, and determine the computation time of the current attention layer under that parallel granularity. and memory access time This reduces the computation time corresponding to the granularity range where the optimal granularity is located. and memory access time The space is divided into two intervals, and the interval jump search algorithm is re-executed until the preset stopping condition is met, and the desired solution is found. Minimize the optimal parallel granularity .

[0009] Furthermore, when the amount of data to be processed can be fully accommodated in the on-chip cache, the memory access time of each layer will be reduced. ;in, and The sum of these values ​​represents the amount of data to be processed; when the amount of data to be processed cannot be fully contained in the on-chip cache, the memory access time... ,in, Indicates the on-chip cache capacity. and These represent the bandwidth of the on-chip and off-chip memory, respectively. The ratio of the number of computations to the accelerator's computing power is used as the computation time for each layer. , is represented as:

[0010] In the formula, This indicates the accelerator's computing power; To vary with parallel granularity A scaling factor that increases logarithmically; D is the dimension of the hidden layer. Dimensions for each attention head; The sequence length is the input to the Transformer model; This indicates the number of operations performed.

[0011] According to another aspect of the present invention, a method for accelerating the operation of a Transformer model with multi-head attention parallel granularity adaptive is provided, comprising: S1. Run the first three layers of the Transformer model with granularities H, H / 2, and 1 respectively, where H represents the total number of attention heads in the model's attention layer; after each layer is executed, record the computation time of each layer. and memory access time ; S2. Granularity based on the nearest 3 layers and corresponding computation time and memory access time Using the multi-head attention parallel granularity adaptive method described above, a single interval jump search algorithm is executed to determine the granularity interval where the optimal granularity is located. From this interval, a parallel granularity is selected for the next layer to perform the computation of the next layer, and the computation time of the next layer is recorded. and memory access time ; S3. Repeat S2, each time reducing the search space of the parallel granularity by half until the termination condition of the interval jump search algorithm is reached. Then, complete the operation of the remaining attention layer with the optimal granularity finally determined by the search algorithm.

[0012] According to another aspect of the present invention, a multi-head attention parallel granularity adaptive Transformer model accelerator is provided, comprising: a tensor prefetch controller, a multi-granularity scheduler, a processing unit array, and a matrix connection unit. The tensor prefetch controller is located at the front end of the accelerator's data path and is responsible for managing data transfer from off-chip memory to on-chip cache. The multi-granularity scheduler contains a time logger, an index table, a comparator, a scheduling module, and a memory controller. The time logger records the computation time of each layer at different parallel granularities. and memory access time The results are then stored in an index table; the index table is a two-dimensional lookup table where rows represent different parallel granularities and columns represent different layers of the Transformer model. The table stores the corresponding granularity-layer pairs. and The values ​​are selected to provide historical data support for subsequent granularity selection; the comparator is used to execute the interval jump search algorithm in the multi-head attention parallel granularity adaptive method described above, and to retrieve and analyze different granularities using the index table. and The data determines the optimal granularity range and selects the granularity from it; the scheduling module is used to allocate the computation tasks of each head to the processing unit array according to the optimal granularity of each layer; the memory controller is used to decode a specific address according to the optimal granularity of each layer and obtain the computation tensor of the current layer through the tensor prefetch controller. The processing unit array is used to perform calculations based on the computation tensors required by the current layer to obtain output blocks; the matrix connection unit is used to concatenate the output blocks of multiple attention heads to form a complete attention output.

[0013] Furthermore, the index table is implemented using an SRAM array, which supports the simultaneous reading of historical data at multiple granularities.

[0014] Furthermore, the processing unit array includes multiple matrix multiplication units (MMUs) and multiple partial sum accumulators; each MMU is used to calculate the partial sum in matrix multiplication; the partial sum accumulator is used to accumulate the partial sums generated by multiple MMUs to obtain the complete matrix multiplication result; wherein, the accumulator adopts a pipelined design and is executed in parallel with the calculation of the MMU.

[0015] Furthermore, the scheduling module is also used to: when there is a batch with a small matrix multiplication size in the current layer at the determined optimal granularity of the current layer, that is, the batch cannot fully utilize the processing unit array, divide the processing unit array into two parts. One part is used to perform the Q, K, V matrix block generation calculation for the batch. Once the Q, K, V matrix block calculation for a certain token is completed, the obtained Q, K, V matrix block is transferred to the other part of the processing unit array to start calculation. .

[0016] Furthermore, the scheduling module of the multi-granularity scheduler adopts an input-oriented matrix multiplication scheduling strategy: the attention head calculation of each batch is divided into matrix multiplication calculations of several matrix blocks, wherein the calculation tasks of different input matrix blocks are assigned to different processing units in the processing unit array, thereby improving the parallelism of matrix multiplication calculation.

[0017] According to another aspect of the present invention, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method described above.

[0018] According to another aspect of the invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is run by a processor, it controls the device where the storage medium is located to perform the steps of the method described above.

[0019] In summary, compared with the prior art, the technical solutions conceived by this invention have the following main advantages: 1. This invention proposes an adaptive multi-head attention parallel granularity method for Transformer model operation. It quantifies the parallelism and memory efficiency of each attention layer during model inference by equating them to the computation time Tc and memory access time Tm of each Transformer model layer. Based on this quantification, an efficient interval jump search algorithm is proposed, which divides the search space into two intervals and calculates their secant slopes. and The optimal parallel granularity is determined by how close the slope of the secant line between two intervals is to -1. That is, if the inequality... If the condition is met, it indicates that the optimal granularity is in the previous granularity interval; otherwise, it is in the next granularity interval. During the Transformer model inference process, this algorithm utilizes the execution time (computation time and memory access time) information of the previous several layers during model forward inference to quickly predict and lock the optimal parallel granularity (i.e., the number of attention heads processed simultaneously) for the current sequence length, thereby achieving synergistic optimization of parallelism and memory bandwidth utilization at the root. Therefore, this invention can dynamically select the optimal parallel granularity based on the real-time input sequence length, overcoming the rigidity of existing acceleration strategies. It can maintain near-optimal performance under various sequence lengths and has strong adaptability. In addition, by synergistically optimizing parallelism and memory efficiency, compared with high-end GPUs (such as RTX4090) and existing advanced accelerators (FTRANS, FACT), this invention can achieve a performance acceleration of up to 2.11 times and an average of more than 1.74 times in actual tests. At the same time, due to the significant reduction of unnecessary off-chip memory access, the energy efficiency ratio of this invention is more than 20 times higher than that of GPUs and also has significant advantages over other dedicated acceleration strategies.

[0020] 2. This invention further proposes to improve hardware utilization by using staged fusion multiplication and input-oriented scheduling strategies. This effectively solves the resource utilization problem when the optimal granularity is small or the optimal granularity is not a divisor of the number of heads, ensuring that the hardware computing unit can maintain high utilization in various scenarios. Attached Figure Description

[0021] Figure 1 The diagram below illustrates the Transformer model structure and head-level parallelization in the prior art provided by this invention; wherein, (a) is a diagram of the Transformer model structure; and (b) is a diagram illustrating the parallel computation method of multi-head attention in the Transformer model. Figure 2 The present invention provides a memory access latency decomposition diagram and performance comparison diagram under different models and sequence lengths; wherein, (a) is a trend diagram of the proportion of sources of memory access latency under different sequence lengths; and (b) is a diagram of the impact of the parallel granularity of attention heads on model inference performance under different sequence lengths. Figure 3 This is an overall architecture diagram of the MGTrans accelerator provided in an embodiment of the present invention; Figure 4 A detailed flowchart of the multi-granularity scheduler provided in this embodiment of the invention; Figure 5The diagram illustrates a comparison between the traditional output-oriented scheduling and the input-oriented scheduling strategy adopted in this invention, as provided in the embodiments of the present invention. In this diagram, (a) represents the block-based form of matrix multiplication, where, in actual execution, the matrix multiplication calculation of the attention head is divided into several matrix blocks that conform to the input size of the MMU for computation; (b) represents the output-oriented scheduling method, where multiple input matrix blocks belonging to the same output matrix block are assigned to the same MMU for computation; and (c) represents the input-oriented scheduling method, where different input matrix blocks are assigned to different MMUs for computation. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0023] To facilitate understanding, the technical terms used in this invention will be explained first.

[0024] Parallel granularity refers to the number of attention heads processed simultaneously in multi-head attention computation. Parallel granularity determines the balance between computational parallelism and memory efficiency, and is a core optimization parameter of this invention.

[0025] Multi-head attention: The core computational unit in the Transformer model, it divides the input data into multiple independent "heads", each head calculates attention weights, and finally merges the results to capture the feature representations of different subspaces in the input sequence.

[0026] Attention Head: The basic computational unit in a multi-head attention mechanism, where each head independently processes a portion of the input features and calculates attention weights.

[0027] Head-level parallelism refers to a computational strategy that processes multiple attention heads simultaneously. Existing accelerators typically employ head-level parallelism with a fixed maximum granularity, while this invention dynamically adjusts the parallel granularity based on the sequence length.

[0028] Batch: refers to dividing the total number of heads in the attention layer into several batches according to the number of heads determined by the parallel granularity and executing them serially. The processing unit array can only process the calculation of one batch of attention heads at a time.

[0029] Stage-fused multiplication: A matrix multiplication optimization strategy that integrates computationally dependent stages (such as Q, K, and V generation stages) in multi-head attention with... The computation stage is streamlined to improve hardware resource utilization, and is especially suitable for scenarios where the optimal granularity is not an approximation of the total number of heads or the granularity is small.

[0030] On-chip buffer / cach: A high-speed storage unit located inside the accelerator chip, used to temporarily store intermediate computation results and reduce latency when accessing off-chip memory. Its capacity is limited, typically in the MB range, and is a key factor limiting the granularity of parallelism.

[0031] Off-chip memory refers to storage devices (such as DRAM) located outside the accelerator chip. It has a large capacity but high access latency. When intermediate results exceed the on-chip cache capacity, frequent access to off-chip memory is required, significantly reducing system performance.

[0032] Intermediate Results: Temporary data generated during multi-head attention computation, including matrices such as Q, K, V, S(P), and O. The amount of intermediate results data is proportional to the square of the sequence length and is a key factor determining memory access efficiency.

[0033] MGTrans: The name of the parallel granularity adaptive Transformer model accelerator proposed in this invention, representing "Multi-Granularity Transformer Accelerator".

[0034] Tensor Fetch Controller: A module in the accelerator responsible for managing data transfer from off-chip memory to on-chip cache. It prefetches data according to computational needs, reducing the waiting time of computational units.

[0035] Processing Element Array (PE Tile): A computational array consisting of multiple processing element tiles. Each processing element tile contains multiple matrix multiplication units and partial sum accumulators for performing matrix operations.

[0036] Index Table: A key data structure in the multi-granularity scheduler, used to store the computation time and memory access time of each layer under different parallel granularities, providing a basis for the selection of granularity of subsequent layers.

[0037] Example 1 A multi-head attention parallel granularity adaptive method for running Transformer models includes: The parallelism and memory efficiency of each attention layer during the Transformer model inference process are respectively equated to the computation time of that layer. and memory access time Quantization is performed to determine the computation time corresponding to the current layer. Search space and memory access time The search space; the computation time and memory access time The current search space is divided into two intervals, and the computation time is... The two corresponding intervals are denoted as ( , )and( , ), memory access time The two corresponding intervals are denoted as ( , )and( , ), execute the interval jump search algorithm to calculate the secant slope. and The optimal parallel granularity is determined by how close the slopes of the secant lines in the two intervals are to -1. That is, if the inequality... If this holds true, it indicates that the optimal granularity is in the previous granularity interval, i.e. ( , )and( , The parallel granularity interval corresponding to ) is, otherwise it is in the next granularity interval; Select a parallel granularity for the current attention layer from the granularity range containing the optimal granularity, and determine the computation time of the current attention layer under that parallel granularity. and memory access time This reduces the computation time corresponding to the granularity range where the optimal granularity is located. and memory access time The space is divided into two intervals, and the interval jump search algorithm is re-executed until the preset stopping condition is met, and the desired solution is found. Minimize the optimal parallel granularity .

[0038] In this embodiment, the parallelism and memory efficiency of each attention layer during the model inference process are equivalent to the computation time Tc and memory access time Tm of each layer of the Transformer model, and then quantitatively analyzed.

[0039] Specifically, this embodiment takes into account the amount of intermediate result data in multi-head attention calculation. Perform accurate modeling:

[0040] Where, in the formula, ; ; This indicates the parallel granularity, i.e., the number of attention heads processed simultaneously; The sequence length is the input to the Transformer model; Dimensions for each attention head; This represents the number of bytes occupied by each value in each intermediate result data of the model.

[0041] Based on this data volume model, when the amount of data to be processed can be fully contained in the on-chip cache, the memory access time of each layer is... ;in, and The sum of these values ​​represents the amount of data to be processed, which is determined based on the sequence length of the input Transformer model and the parallel granularity. When the amount of data to be processed cannot be fully contained in the on-chip cache, memory access time... ,in, Indicates the on-chip cache capacity. and These represent the bandwidth of on-chip and off-chip memory, respectively. Meanwhile, the ratio of the number of operations (also determined based on the sequence length of the input Transformer model and the parallel granularity) to the accelerator's computing power is used as the computation time. It can be represented as:

[0042] Where S represents the computing power of the accelerator, and D is the dimension of the hidden layer. For follow A scaling factor that increases logarithmically.

[0043] Based on the modeling of computation time Tc and access latency Tm, the relationship between the total processing time T of each layer of the Transformer model and these two factors can be obtained. The total processing time is limited by the one that takes longer, i.e.:

[0044] Therefore, the optimization objective is transformed into finding... Minimize the optimal parallel granularity Based on the above quantitative analysis, it can be seen that Tc varies with parallel granularity. Tm decreases as the parallel granularity increases. As the number of cells increases, experimental data and theoretical analysis show that the two approximately satisfy an inverse relationship: Tc × Tm ≈ C, that is... , where C is a constant. Therefore, we have At the equilibrium point where T reaches its minimum value, Tc and Tm should be equal, i.e. Further obtained .

[0045] Based on this quantitative analysis, this embodiment proposes an efficient interval jump search algorithm, which divides the search space into two intervals and calculates their secant slopes. and The optimal parallel granularity is determined by how close the slope of the secant line between two intervals is to -1. That is, if the inequality... If the condition is met, it indicates that the optimal granularity is in the previous granularity interval; otherwise, it is in the next granularity interval. During the Transformer model inference process, this algorithm utilizes the execution time (computation time and memory access time) information of the previous several layers during model forward inference to quickly predict and lock the optimal parallel granularity (i.e., the number of attention heads processed simultaneously) for the current sequence length, thereby achieving synergistic optimization of parallelism and memory bandwidth utilization at the root.

[0046] In this embodiment, the computation time for each layer is determined. and memory access time In this case, it can be determined either through actual records or through calculation.

[0047] Example 2 A method for accelerating the execution of Transformer models with multi-head attention parallel granularity adaptation includes: S1. Run the first three layers of the Transformer model with granularities H, H / 2, and 1 respectively, where H represents the total number of attention heads in the model's attention layer; after each layer is executed, record the computation time of each layer. and memory access time ; S2. Granularity based on the nearest 3 layers and corresponding computation time and memory access time The multi-head attention parallel granularity adaptive method as described in claim 1 is employed. A single interval jump search algorithm is executed to determine the granularity interval containing the optimal granularity. From this interval, a parallel granularity is selected for the next layer to execute the next layer's computation, and the computation time of the next layer is recorded. and memory access time ; S3. Repeat S2, each time reducing the search space of the parallel granularity by half until the termination condition of the interval jump search algorithm is reached (set the maximum number of iterations to ensure that the algorithm converges within a finite number of steps), and then complete the operation of the remaining attention layer with the optimal granularity finally determined by the search algorithm.

[0048] The core of this embodiment lies in intelligent decision-making at the software level. The relevant technical solutions are the same as above and will not be repeated here.

[0049] Example 3 A multi-head attention parallel granularity adaptive Transformer model accelerator includes: a tensor prefetch controller, a multi-granularity scheduler, a processing unit array, and a matrix connection unit. The tensor prefetch controller is located at the front end of the accelerator's data path and is responsible for managing data transfer from off-chip memory to on-chip cache. The multi-granularity scheduler contains a time logger, an index table, a comparator, a scheduling module, and a memory controller. The time logger records the computation time of each layer at different parallel granularities. and memory access time The results are then stored in an index table; the index table is a two-dimensional lookup table where rows represent different parallel granularities and columns represent different layers of the Transformer model. The table stores the corresponding granularity-layer pairs. and The values ​​are selected to provide historical data support for subsequent granularity selection; the comparator is used to execute the interval jump search algorithm in the multi-head attention parallel granularity adaptive method as described in claim 1, and to retrieve and analyze different granularities by combining the index table. and The data determines the optimal granularity range and selects the granularity from it; the scheduling module is used to allocate the computation tasks of each head to the processing unit array according to the optimal granularity of each layer; the memory controller is used to decode a specific address according to the optimal granularity of each layer and obtain the computation tensor of the current layer through the tensor prefetch controller. The processing unit array is used to perform calculations based on the computation tensors required by the current layer to obtain output blocks; the matrix connection unit is used to concatenate the output blocks of multiple attention heads to form a complete attention output.

[0050] As a preferred embodiment, the processing unit array includes multiple matrix multiplication units (MMUs) and multiple partial sum accumulators; each MMU is used to calculate the partial sum in matrix multiplication; the partial sum accumulator is used to accumulate the partial sums generated by multiple MMUs to obtain the complete matrix multiplication result; wherein, the accumulator adopts a pipelined design and is executed in parallel with the calculation of the MMUs.

[0051] As a preferred implementation, the scheduling module is further configured to: when there is a batch with a small matrix multiplication size in the current layer at the determined optimal granularity of the current layer, i.e., the batch cannot fully utilize the processing unit array, divide the processing unit array into two parts. One part is used to perform the Q, K, V matrix block generation calculation for the batch. Once the Q, K, V matrix block calculation for a certain token is completed, the obtained Q, K, V matrix block is transferred to the other part of the processing unit array to start calculation. .

[0052] As a preferred implementation, the scheduling module of the multi-granularity scheduler adopts an input-oriented matrix multiplication scheduling strategy: the attention head calculation of each batch is divided into matrix multiplication calculations of several matrix blocks, wherein the calculation tasks of different input matrix blocks are assigned to different processing units in the processing unit array, thereby improving the parallelism of matrix multiplication calculation.

[0053] This embodiment mainly describes a dedicated hardware accelerator for Transformer models based on a multi-head attention parallel granularity selection strategy that balances parallelism and memory efficiency. A specific implementation example is described in detail below: MGTrans hardware architecture implementation: such as Figure 3 As shown, the accelerator mainly includes a tensor prefetch controller, a multi-granularity scheduler, a processing unit array, and a matrix connection unit. Its specific functions are as follows: The tensor prefetch controller, located at the front end of the accelerator's data path, manages data transfer from off-chip memory to on-chip cache. This controller comprises two main modules: an address generator and a request arbitrator. The address generator pre-calculates the storage addresses of the required tensors based on the computational needs of the current layer; the request arbitrator coordinates multiple data requests, ensuring that high-priority tasks are executed first. In its implementation, the tensor prefetch controller employs a double-buffering mechanism: while one buffer is providing data to the computation unit, another buffer pre-loads the next batch of data from off-chip memory, achieving overlap between data transfer and computation and maximizing bandwidth utilization. The tensor prefetch controller works closely with the multi-granularity scheduler, dynamically adjusting the data prefetching strategy based on the determined optimal parallel granularity to ensure that only the data required for the current batch is prefetched, avoiding unnecessary data transfer.

[0054] The multi-granularity scheduler is a core component, such as Figure 4As shown, the system includes a time recorder, an index table, a comparator, a scheduling module, and a memory controller. The time recorder accurately records the computation time (Tc) and memory access time (Tm) for each layer at different parallel granularities. This module uses a high-precision counter to record execution time in clock cycles and stores the results in the index table. The index table is a two-dimensional lookup table where rows represent different parallel granularities (1, 2, 4, ..., H) and columns represent different layers of the Transformer. The table stores the Tc and Tm values ​​corresponding to each granularity-layer pair, providing historical data support for subsequent granularity selection. In the hardware implementation, the index table is implemented using an SRAM array, supporting simultaneous reading of historical data from multiple granularities. The comparator module analyzes the Tc and Tm data at different granularities to determine the interval where the equilibrium point lies. This module implements the core logic of the interval jump search algorithm, quickly locating the search range of the optimal parallel granularity by comparing the differences in the rates of change of Tc and Tm between adjacent granularities. The comparator module contains a dedicated computation unit for calculation... Key performance indicators (KPIs) are as follows. The scheduling module is responsible for allocating the computational tasks of each head to the processing unit array according to the determined optimal granularity, and dynamically adjusting resource allocation during the computation process to ensure maximum computational efficiency. The memory controller is responsible for decoding specific addresses according to the determined granularity to obtain the computational tensors of the current layer. This module optimizes the data access mode, adjusts the memory access granularity according to the parallel granularity, and reduces the number of memory accesses. The memory controller is also responsible for managing the storage location of intermediate results, prioritizing the retention of frequently accessed data in the on-chip cache.

[0055] The processing unit array consists of 16 processing unit blocks, each containing 16 matrix multiplication units (MMUs) and 16 partial sum accumulators. Each MMU comprises a 4×4 processing unit (PE) responsible for calculating the partial sums in matrix multiplication. The partial sum accumulators are responsible for accumulating the partial sums generated by multiple MMUs to obtain the complete matrix multiplication result. The accumulators employ a pipelined design, allowing them to execute in parallel with the MMU computations, thus improving overall throughput. Under an input-oriented scheduling strategy, the accumulators can promptly merge partial sums from different MMUs without waiting for the generation of a complete output block.

[0056] The auxiliary computation unit includes a Softmax unit, a layer normalization unit, and a Dropout unit, which are used to perform non-matrix multiplication operations in the Transformer model.

[0057] The matrix connection network is responsible for concatenating the computation results of multiple attention heads to form a complete attention output. This network employs a tree-like topology, progressively merging the outputs of 16 processing unit blocks to ultimately generate a complete output tensor. The connection network supports dynamic configuration, adjusting the connection method according to different numbers of heads and dimensions.

[0058] Matrix multiplication distribution strategy implementation: When the optimal granularity is small, or the optimal granularity is not a divisor of the total number of heads, the corresponding matrix multiplication scale of each batch or the last batch is small, which may not fully utilize the computing units. Specifically, for example, when the optimal granularity is 10 and the total number of heads H=32, the last batch contains only 2 heads. In this case, the processing unit array is divided into two parts, one part performs the generation stage calculation of the last batch (…). Once the Q, K, V matrix block for a given token has been calculated and obtained, it is immediately transmitted via the on-chip network to another part of the processing unit array to begin calculation. In terms of specific hardware implementation, such as Figure 5 As shown, Figure 5 In the example, (a) represents the block form of matrix multiplication. In actual execution, the matrix multiplication calculation of the attention head will be divided into several matrix blocks that conform to the input size of the MMU for operation. Figure 5 (b) in the diagram represents an output-oriented scheduling method, in which multiple input matrix blocks belonging to the same output matrix block will be assigned to the same MMU for computation; Figure 5 In this example, (c) represents an input-oriented scheduling method, where different input matrix blocks are assigned to different MMUs for computation. In this embodiment, the matrix multiplication computation of the attention head is divided into several matrix blocks that conform to the MMU input size for operation (e.g., ...). Figure 5 As shown in (a)), during the computation process, the matrix multiplication task scheduling adopts a scheduling method oriented towards the reusability of the input matrix. Figure 5 As shown in (c), the strategy of allocating computational tasks of different input matrix blocks to different processing units improves computational parallelism by distributing a portion of the computational task of an output matrix block to multiple processing units and ensures that data blocks can be generated and transmitted as early as possible. Compared with the strategy of allocating computational tasks of the same output matrix block to the same processing unit, this avoids output-oriented scheduling. Figure 5 As shown in (b) of the diagram, there is an inefficiency problem where the entire output block must be waited for before it can be passed.

[0059] The relevant technical solutions are the same as above, and will not be repeated here.

[0060] Example 4 This application also relates to an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.

[0061] The electronic device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The memory can be used to store computer programs and / or modules. The processor performs various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory.

[0062] The relevant technical solutions are the same as above, and will not be repeated here.

[0063] Example 5 This application also relates to a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described above.

[0064] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0065] The relevant technical solutions are the same as above, and will not be repeated here.

[0066] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-head attention parallel granularity adaptive method for Transformer model operation, characterized in that, include: The parallelism and memory efficiency of each attention layer during the Transformer model inference process are respectively equated to the computation time of that layer. and memory access time Quantization is performed to determine the computation time corresponding to the current layer. Search space and memory access time The search space; the computation time and memory access time The current search space is divided into two intervals, and the computation time is... The two corresponding intervals are denoted as ( , )and( , ), memory access time The two corresponding intervals are denoted as ( , )and( , ), execute the interval jump search algorithm to calculate the secant slope. and The optimal parallel granularity is determined by how close the slopes of the secant lines in the two intervals are to -1. That is, if the inequality... If this holds true, it indicates that the optimal granularity is in the previous granularity interval, i.e. ( , )and( , The parallel granularity interval corresponding to ) is, otherwise it is in the next granularity interval; Select a parallel granularity for the current attention layer from the granularity range containing the optimal granularity, and determine the computation time of the current attention layer under that parallel granularity. and memory access time This reduces the computation time corresponding to the granularity range where the optimal granularity is located. and memory access time The space is divided into two intervals, and the interval jump search algorithm is re-executed until the preset stopping condition is met, and the desired solution is found. Minimize the optimal parallel granularity .

2. The multi-head attention parallel granularity adaptive method as described in claim 1, characterized in that, When the amount of data to be processed can be fully contained in the on-chip cache, the memory access time of each layer is reduced. ;in, and The sum of these values ​​represents the amount of data to be processed; when the amount of data to be processed cannot be fully contained in the on-chip cache, the memory access time... ,in, Indicates the on-chip cache capacity. and These represent the bandwidth of the on-chip and off-chip memory, respectively. The ratio of the number of computations to the accelerator's computing power is used as the computation time for each layer. , is represented as: In the formula, This indicates the accelerator's computing power; To vary with parallel granularity A scaling factor that increases logarithmically; D is the dimension of the hidden layer. Dimensions for each attention head; The sequence length is the input to the Transformer model; This indicates the number of operations performed.

3. A method for accelerating the execution of a Transformer model with multi-head attention and adaptive parallel granularity, characterized in that, include: S1. Run the first three layers of the Transformer model with granularities H, H / 2, and 1 respectively, where H represents the total number of attention heads in the model's attention layer; after each layer is executed, record the computation time of each layer. and memory access time ; S2. Granularity based on the nearest 3 layers and corresponding computation time and memory access time The multi-head attention parallel granularity adaptive method as described in claim 1 is employed. A single interval jump search algorithm is executed to determine the granularity interval containing the optimal granularity. From this interval, a parallel granularity is selected for the next layer to execute the next layer's computation, and the computation time of the next layer is recorded. and memory access time ; S3. Repeat S2, each time reducing the search space of the parallel granularity by half until the termination condition of the interval jump search algorithm is reached. Then, complete the operation of the remaining attention layer with the optimal granularity finally determined by the search algorithm.

4. A Transformer model accelerator with multi-head attention parallel granularity adaptive, characterized in that, include: Tensor prefetch controller, multi-granularity scheduler, processing unit array, and matrix connection unit; The tensor prefetch controller is located at the front end of the accelerator's data path and is responsible for managing data transfer from off-chip memory to on-chip cache. The multi-granularity scheduler contains a time logger, an index table, a comparator, a scheduling module, and a memory controller. The time logger records the computation time of each layer at different parallel granularities. and memory access time The results are then stored in an index table; the index table is a two-dimensional lookup table where rows represent different parallel granularities and columns represent different layers of the Transformer model. The table stores the corresponding granularity-layer pairs. and The values ​​are selected to provide historical data support for subsequent granularity selection; the comparator is used to execute the interval jump search algorithm in the multi-head attention parallel granularity adaptive method as described in claim 1, and to retrieve and analyze different granularities by combining the index table. and The data determines the optimal granularity range and selects the granularity from it; the scheduling module is used to allocate the computation tasks of each head to the processing unit array according to the optimal granularity of each layer; the memory controller is used to decode a specific address according to the optimal granularity of each layer and obtain the computation tensor of the current layer through the tensor prefetch controller. The processing unit array is used to perform calculations based on the computation tensors required by the current layer to obtain output blocks; the matrix connection unit is used to concatenate the output blocks of multiple attention heads to form a complete attention output.

5. The Transformer model accelerator as described in claim 4, characterized in that, The index table is implemented using an SRAM array, which supports the simultaneous reading of historical data at multiple granularities.

6. The Transformer model accelerator as described in claim 4, characterized in that, The processing unit array contains multiple matrix multiplication units (MMUs) and multiple partial sum accumulators; each MMU is used to calculate the partial sum in matrix multiplication; the partial sum accumulator is used to accumulate the partial sums generated by multiple MMUs to obtain the complete matrix multiplication result; the accumulator adopts a pipelined design and is executed in parallel with the calculation of the MMUs.

7. The Transformer model accelerator as described in claim 4, characterized in that, The scheduling module is further configured to: when there are batches with small matrix multiplication sizes in the current layer at the determined optimal granularity of the current layer, i.e., the batch cannot fully utilize the processing unit array, divide the processing unit array into two parts. One part is used to perform the Q, K, V matrix block generation calculation for the batch. Once the Q, K, V matrix block calculation for a certain token is completed, the obtained Q, K, V matrix block is transferred to the other part of the processing unit array to start calculation. .

8. The Transformer model accelerator as described in claim 4, characterized in that, The scheduling module of the multi-granularity scheduler adopts an input-oriented matrix multiplication scheduling strategy: the attention head calculation of each batch is divided into matrix multiplication calculations of several matrix blocks, and the calculation tasks of different input matrix blocks are assigned to different processing units in the processing unit array to improve the parallelism of matrix multiplication calculation.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 3.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein the computer program, when executed by a processor, controls the device on which the storage medium resides to perform the steps of the method as described in any one of claims 1 to 3.

Images