Compilation optimization method and device for accelerating Attention calculation

Abstract

The invention discloses a compilation optimization method and device for accelerating Attention calculation, and belongs to the field of compilation optimizing.The method comprises the following steps that multiple scheduling strategies are generated according to the number, shape and Attention type of input matrixes and a target hardware platform; defining a cost function, evaluating time and space overhead of each scheduling strategy, and selecting an optimal strategy based on hardware platform characteristics and user requirements; the method comprises the following steps of: performing multi-core parallel division on input data in sequence length and Head dimension, and reducing delay through asynchronous communication overlapping calculation and communication; in a training stage, a random floating point matrix generated by hardware and an attention weight are used for carrying out Hadamard product operation, and a Dropout function is realized. According to the method, the characteristics of different hardware platforms can be flexibly adapted through the generated Attention operator template, excellent performance can be achieved under various shapes, and efficient data division is achieved by optimizing balance between calculation and communication.

Description

Technical Field

[0001] The present invention relates to the field of compilation optimization methods, and in particular to a compilation optimization method and device for accelerating Attention calculations. Background Art

[0002] Attention is a widely used operator in deep learning, demonstrating particularly strong performance in fields such as natural language processing and computer vision. It dynamically assigns attention weights, enabling the model to focus on the most important parts of the input. However, as model size grows, the computational and storage costs of the Attention operator increase significantly, particularly when processing long sequences. This overhead becomes a major performance bottleneck. This high complexity directly leads to excessive memory consumption and inefficiencies on resource-constrained hardware. Furthermore, the extensive communication involved in distributed training further exacerbates communication latency and bandwidth limitations, hindering model scalability.

[0003] To address these issues, researchers have proposed various optimization methods, including sparse attention, low-rank approximation, and flash attention. These methods reduce complexity by eliminating unnecessary computation or leveraging local information, but they also carry the potential risk of reduced model accuracy. Furthermore, attention optimization must also take into account hardware characteristics, such as the parallel computing capabilities of GPUs / TPUs and network topology, to strike a balance between performance and resource utilization.

[0004] Therefore, efficiently optimizing the Attention operator while ensuring model performance has become one of the core challenges in the current deep learning field. Solving this problem will not only improve model efficiency but also promote the practical application of large-scale models.

[0005] Therefore, those skilled in the art provide a compilation optimization method and device for accelerating Attention calculation to solve the problems raised in the above background technology. Summary of the Invention

[0006] To address the deficiencies in the prior art, the present invention provides a compilation optimization method for accelerating Attention calculation, comprising the following steps:

[0007] Generate multiple scheduling strategies based on the number and shape of input matrices, attention type, and target hardware platform;

[0008] Define a cost function to evaluate the time and space overhead of each scheduling strategy, and select the optimal strategy based on hardware platform characteristics and user needs;

[0009] The input data is divided into multiple cores in parallel based on sequence length and head dimension, and computation and communication are overlapped through asynchronous communication to reduce latency.

[0010] During the training phase, the random floating-point matrix generated by hardware is used to perform Hadamard product operations with the attention weights to implement the Dropout function.

[0011] As a further solution of the present invention: generating a scheduling strategy includes:

[0012] Determine the block size and loop traversal order based on the Attention operator type, the sequence length of the input tensor, and the number of heads;

[0013] Combined with the target platform's L2 cache capacity and intermediate variable lifecycle, a scheduling template containing block calculation kernels and control flows is generated.

[0014] As a further solution of the present invention: the evaluation index of the cost function includes:

[0015] Run time, storage usage, power consumption, and hardware parallelism, and perform multi-objective optimization with user-defined weights.

[0016] As a further solution of the present invention, the multi-core parallel partitioning includes one of the following strategies:

[0017] Distribute sequence lengths to different cores;

[0018] Assign Head to different cores;

[0019] The sequence length and Head are combined and divided into different cores.

[0020] As a further solution of the present invention: the multi-core parallel partitioning further includes:

[0021] Adjust the parallel dimension based on the number of cores of the target device, and pad the divided dimension to an integer multiple of the number of cores.

[0022] Select the collective communication method based on the hardware topology and embed the communication function into the scheduling template.

[0023] As a further solution of the present invention: the generation of the random floating-point matrix includes:

[0024] Generates a half-precision matrix whose element values ​​are 0 or 1 / (1-p) with probability p, and directly performs mixed-precision Hadamard product operations through vector components.

[0025] As a further solution of the present invention: it also includes:

[0026] During the compilation phase, users are supported to manually replace the automatically generated scheduling templates to achieve performance tuning on specific hardware platforms.

[0027] As a further solution of the present invention: the generation of the scheduling strategy further includes:

[0028] Dynamically adjust the block size and inter-core data allocation strategy based on the dynamic length of the input sequence and the number of heads.

[0029] As a further solution of the present invention: the asynchronous communication includes:

[0030] When transferring data between cores, high-precision data is converted to low-precision format to reduce communication bandwidth usage.

[0031] This application also discloses a compilation optimization device for accelerating Attention calculation, and a compilation optimization method for accelerating Attention calculation, including:

[0032] Multi-core AI accelerator, each core contains a tensor component, a vector component, an IMM component, and a DMA component;

[0033] The tensor component is used to perform matrix multiplication operations;

[0034] The vector component is used to perform element-by-element operations and Softmax calculations;

[0035] The IMM component is used to generate a random floating-point matrix to support the Dropout operation;

[0036] The DMA component is used to realize data movement between memory and cache and inter-core communication;

[0037] The multi-core AI accelerator supports data distribution across sequence length and Head dimension, and reduces communication overhead through low-precision conversion.

[0038] The beneficial effects of the present invention are embodied in:

[0039] The Attention operator template generated by this invention can flexibly adapt to the characteristics of different hardware platforms and achieve excellent performance under various shapes. Secondly, it fully considers the parallel computing capabilities and network topology characteristics of the hardware platform, achieving efficient data partitioning by optimizing the balance between computation and communication. Furthermore, by leveraging the hardware's ability to generate random number matrices, it accelerates the implementation of Dropout on the target platform, significantly improving the training speed of the Attention operator. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly describes the drawings required for the specific embodiments or the description of the prior art. Similar elements or parts are generally identified by similar reference numerals throughout the drawings. Elements or parts in the drawings are not necessarily drawn to scale.

[0041] Figure 1 This is a flowchart of a compilation optimization method to accelerate Attention calculation;

[0042] Figure 2 A schematic diagram of scheduling input in a compilation optimization method to accelerate Attention calculation;

[0043] Figure 3 A schematic diagram of scheduling template generation in a compilation optimization method to accelerate Attention computation;

[0044] Figure 4 A schematic diagram of optimal template selection in a compilation optimization method for accelerating Attention calculations;

[0045] Figure 5 Generate a schematic diagram for the parallel strategy in a compilation optimization method to accelerate Attention computation;

[0046] Figure 6 This is a hardware structure diagram of a compilation optimization device that accelerates Attention calculations. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0048] As mentioned in the background technology of this application, research has found that existing methods such as sparse attention, low-rank approximation and flash attention reduce complexity by reducing unnecessary calculations or utilizing local information, but at the same time also bring the potential risk of reduced model accuracy and have certain defects.

[0049] To address the above-mentioned shortcomings, this application discloses a compilation optimization method and device for accelerating Attention calculations. The generated Attention operator template can flexibly adapt to the characteristics of different hardware platforms and achieve excellent performance under various shapes. Secondly, the parallel computing capabilities and network topology characteristics of the hardware platform are fully considered. By optimizing the balance between computing and communication, efficient data partitioning is achieved. In addition, by leveraging the hardware's ability to generate random number matrices, the implementation of Dropout on the target platform is accelerated, thereby significantly improving the training speed of the Attention operator.

[0050] The following will describe in detail how the solution of this application solves the above technical problems with reference to the accompanying drawings.

[0051] See also Figure 1 In an embodiment of the present invention, a compilation optimization method for accelerating Attention calculation includes the following steps:

[0052] Generate a variety of scheduling strategies based on the number and shape of input matrices, the attention type, and the target hardware platform. More specifically, the present invention automatically generates multiple scheduling templates based on constraints such as the attention operator input type, number of inputs, sequence length, and number of heads. Each template includes submodules such as the size of each dimension's block, parallelization strategy, kernel execution order, and control flow generation. To maximize performance, users can also manually replace scheduling templates.

[0053] Define a cost function, evaluate the time and space overhead of various scheduling strategies, and select the optimal strategy based on the hardware platform characteristics and user requirements. More specifically, compile solution optimization involves automatically generating templates that are fed into the cost function for a specific hardware backend and generate corresponding performance analysis results. This section selects the scheduling template that best meets user requirements.

[0054] The input data is divided into multi-core parallel divisions based on the sequence length and head dimension, and asynchronous communication is used to overlap calculation and communication to reduce latency. To be more specific, the multi-core parallel strategy of the present invention determines the data division and communication methods on different cards based on the number of cores of the target device, the input sequence length, the number of heads and other information, and uses overlapping calculation and communication to hide communication delays.

[0055] During the training phase, a hardware-generated random floating-point matrix is ​​used to perform a Hadamard product with the attention weights to implement dropout. Specifically, the target device's IMM component supports generating a random floating-point matrix X with a certain probability p directly in the L2 cache. Elements of the X matrix can only be 0 or 1 / (1-p) and are stored in half-precision. Dropout is achieved by performing a Hadamard product with the X matrix. The vector component supports half-precision and mixed-precision calculations, effectively reducing memory usage and improving computation speed.

[0056] This application adapts to different hardware platforms and input shapes through compile-time scheduling template generation and cost model evaluation to avoid performance fluctuations. It also combines a block strategy based on sequence length and head dimension to reduce global communication overhead and improve scalability in distributed scenarios.

[0057] In this embodiment, generating a scheduling strategy includes: determining the block size and loop traversal order based on the Attention operator type, the sequence length of the input tensor, and the number of heads; and generating a scheduling template including the block calculation kernel and control flow in combination with the L2 cache capacity and intermediate variable life cycle of the target platform.

[0058] In this embodiment, the evaluation indicators of the cost function include: running time, storage occupancy, power consumption and hardware parallelism, and multi-objective optimization is performed through user-defined weights.

[0059] In this embodiment, multi-core parallel partitioning includes one of the following strategies: allocating sequence lengths to different cores; allocating heads to different cores; merging sequence lengths and heads and then partitioning them to different cores.

[0060] In this embodiment, multi-core parallel partitioning also includes: adjusting the parallel dimension according to the number of cores of the target device, and making the dimension after partitioning an integer multiple of the number of cores by padding; selecting a collective communication method in combination with the hardware topology structure, and embedding the communication function into the scheduling template.

[0061] In this embodiment, the generation of the random floating-point matrix includes: generating a half-precision matrix with element values ​​of 0 or 1 / (1-p) with probability p, and directly performing a mixed-precision Hadamard product operation through a vector component.

[0062] In this embodiment, it also includes: supporting users to manually replace automatically generated scheduling templates during the compilation phase to achieve performance tuning under specific hardware platforms.

[0063] In this embodiment, the generation of the scheduling strategy further includes: dynamically adjusting the block size and the inter-core data allocation strategy according to the dynamic length of the input sequence and the number of heads.

[0064] In this embodiment, asynchronous communication includes: when transmitting data between cores, converting high-precision data into a low-precision format to reduce communication bandwidth usage.

[0065] In this embodiment, the specific process of optimizing the compilation scheme is as follows:

[0066] (1) Figure 2 As shown, the input received by this solution can be divided into three aspects. The first is the type of Attention operator. Currently, the main types of Attention include Group Query Attention (GQA), Multi Query Attention (MQA), Multi Head Attention (MHA), and Cross Attention. Only by clarifying the type of Attention can the basic logic of the calculation be determined. The second is the Attention input tensor. The sequence length, head, and number of input tensors not only affect the data movement capacity of the DMA component but also affect subsequent scheduling optimization, thereby affecting the final scheduling template. The last input is the operator attributes. Different attributes will affect the kernel in the scheduling template. For example, most Attention calculations usually involve scaling, requiring the vector component to be multiplied by a coefficient during calculation.

[0067] (2) After receiving the above input, if Figure 3 As shown in the figure, the Scheduler will generate multiple scheduling templates for the Attention operator. One scheduling template corresponds to one scheduling strategy. During the generation process, the loop order of the Attention operator will be considered first. By explicitly specifying the traversal order of the three matrices, Query, Key, and Value, a series of DMA Kernels will be generated. Then, the Scheduler module will further calculate the block size ( Figure 3 Template 1's q_unit and kv_unit. The scheduler then defines the specific loops that need to be expanded in the L1 cache and the size of the PE array, generating a series of computation kernels. Given the presence of certain special shapes, the resulting templates will include control flow to execute different branches, achieving optimal performance in different situations. Currently, there is still much room for exploration in Attention optimization, so users are allowed to handwrite templates instead of automatically generated ones.

[0068] (3) Figure 4As shown in the figure, the various attention templates generated by the scheduler are input into the cost model for evaluation. Based on the target platform's hardware information, such as processor architecture, memory hierarchy, and power constraints, the performance of each template is deeply analyzed. Specifically, the cost model measures key metrics such as runtime, memory usage, and power consumption for each template, providing baseline data for subsequent evaluation. After obtaining these metrics, the evaluation function comprehensively evaluates the template based on user requirements or goals (such as minimizing runtime, conserving memory resources, or reducing power consumption). The evaluation function not only considers a single dimension but also integrates multiple factors (such as the trade-off between runtime and power consumption) to ultimately select the template that best meets user requirements and platform characteristics. Finally, the optimal template from the evaluation results is selected and returned, providing a basis for subsequent task execution. This process completes a complete closed loop from template generation to performance evaluation and matching with user requirements, providing strong support for the efficient execution of complex computing tasks.

[0069] In this embodiment, the multi-core parallel strategy is specifically as follows:

[0070] (1) In different large models, the number of heads for query, key, and value varies greatly. At the same time, the sequence length of the attention operator is also a dynamic value, which makes it complicated to design a parallel strategy on the target platform. To solve this problem, Figure 5 As shown, the present invention first extracts relevant parameters from the input of the Attention operator, including the number of heads, sequence length and operator type.

[0071] (2) In the first step, these parameters are passed to the Parallelism Method module. This module provides three parallel strategies: the first is to distribute the sequence length to different computing cores; the second is to distribute the query, key, and value heads to different cores; and the third is to combine the sequence length with the head and then divide them into different cores.

[0072] (3) In the second step, based on different parallel strategies, the number of heads and sequence lengths of the query, key, and value matrices on each core can be determined, and this information can be passed to the runner module. The runner module needs to consider the computational complexity and communication time required by different parallel strategies. First, the three parallel strategies will cause the parallel dimension to be not divisible by the number of cores of the target device. This technical solution will comprehensively evaluate the impact of filling the head or sequence length to an integer multiple of the number of cores of the target device on the overall computational complexity and select the solution with the least impact. Then the runner module will consider various collective communication methods, and combine the topology of the target platform (ring structure or tree structure), communication bandwidth and energy consumption to finally determine the optimal parallel strategy and corresponding collective communication method. The corresponding communication function will also be embedded in the attention operator template generated in 1. Finally, in order to reduce the amount of communication data, the DMA component of the target device supports converting high-precision data into low-precision data during inter-core communication.

[0073] Through this systematic processing method, the present invention can effectively optimize the parallel computing in Attention, thereby improving the overall performance.

[0074] In this embodiment, Dropout optimization is specifically as follows: During the training process, Dropout can be directly applied to the attention weights. The IMM component of the target platform supports generating a random half-precision floating-point matrix X directly on the L2 cache with a certain probability p. The elements of the X matrix can only be 0 or 1 / (1-p). The Dropout function can be implemented on the target platform by performing a Hadamard product operation on the attention weights and the X matrix. Considering that the attention weight matrix is ​​stored in single-precision form during training, the vector component supports half-precision or mixed-precision calculations, thereby effectively reducing memory usage and improving calculation speed.

[0075] like Figure 6As shown, this application also discloses a compilation optimization device for accelerating Attention computations, which is applied to a compilation optimization method for accelerating Attention computations. The device comprises a multi-core AI accelerator, each core of which includes a tensor component, a vector component, an IMM component, and a DMA component. The tensor component is used to perform matrix multiplication operations; the vector component is used to perform element-by-element operations and softmax calculations; the IMM component is used to generate random floating-point matrices to support dropout operations; and the DMA component is used to implement data movement between memory and cache and inter-core communication. The multi-core AI accelerator supports data allocation based on sequence length and head dimension, and reduces communication overhead through low-precision conversion. More specifically, the tensor component implements tensor computations for operators such as MatMul, the vector component is used for vector computations for operators such as Mul, Add, and Softmax, the DMA component implements data movement from DDR to on-chip memory, and the IMM component is used to generate random number matrices. During computation, input data is first moved from DDR to the L2 cache. The tensor component and vector component load the required local data blocks from the L2 cache and store them in the L1 cache to accelerate computation. The L1 cache provides operands to PE, supporting high-speed computing.

[0076] To further illustrate the present invention, a compilation optimization method and device for accelerating Attention calculation provided by the present invention are described in detail below in conjunction with embodiments.

[0077] Example 1: Dynamic Schedule Generation and Evaluation

[0078] 1.1 Input parameter configuration:

[0079] Attention type: Multi-Head Attention (MHA);

[0080] Input shape: sequence length = 1024, number of heads = 16, head_dim = 128;

[0081] Target hardware: 8-core AI accelerator, L2 cache capacity = 16MB.

[0082] 1.2 Scheduling template generation:

[0083] Generate three blocking strategies (q_unit=256,kv_unit=512; q_unit=512,kv_unit=256; dynamic blocking).

[0084] Combined with the L2 cache capacity, the life cycle of intermediate variables is determined and control flow branches are generated.

[0085] 1.3 Cost model evaluation:

[0086] The running time of each template was evaluated (Strategy 1: 12ms; Strategy 2: 14ms; Strategy 3: 10ms), and Strategy 3 was selected.

[0087] Example 2: Multi-core parallel training optimization

[0088] Parallel strategy: Divide the sequence length (1024) into 8 blocks (128 per core), and allocate the number of heads (16) to 2 cores.

[0089] Communication optimization: A tree topology is used for AllReduce communication, and data precision is converted from FP32 to FP16, reducing bandwidth usage by 50%.

[0090] Example 3: Dropout Hardware Acceleration

[0091] The IMM component generates a half-precision matrix with probability p=0.1, and the element value is 0 or 1.11 (1 / (1-0.1)).

[0092] The attention weight (FP32) and the random matrix (FP16) perform mixed precision Hadamard product operations, reducing the computation time by 40%.

[0093] It should be noted that the following explanations are made for the terms in the above content:

[0094] MatMul: Matrix Multiplication;

[0095] Input sequence: an input matrix representing the word embedding or feature vector of the sequence, which is the basis for self-attention calculation on the input;

[0096] Query Matrix: Query matrix, obtained by linear transformation of the input sequence, used to calculate the correlation between the current element and other elements;

[0097] Key Matrix: Key matrix, obtained by linear transformation of the input sequence, used to determine the characteristics and positions of other elements;

[0098] Value Matrix: The value matrix is ​​obtained by linear transformation of the input sequence, which represents the specific content or information of each element;

[0099] Softmax: A commonly used activation function, mainly used in the output layer of multi-class classification problems. It converts a set of arbitrary real numbers into numerical values ​​representing a probability distribution, so that each number is between 0 and 1 and the sum of all numbers is 1;

[0100] GPU: Graphics Processing Unit, a processor dedicated to processing graphics and image calculations;

[0101] ASIC: Application-Specific Integrated Circuit, is an integrated circuit customized for a specific application or task;

[0102] FP8: 8-bit floating point number, a floating point format optimized for deep learning and high-performance computing;

[0103] Attention: A key technique used in deep learning models, especially when processing sequential data. It allows the model to focus on information at different locations when processing the input sequence, helping the model better capture important correlations and patterns in the sequence;

[0104] Head: In the multi-head attention mechanism, the model learns multiple sets of attention weights in parallel, each of which is called a "head." Each head has its own linear transformation of the query, key, and value, and learns different representations through these linear transformations.

[0105] head_num: refers to the number of heads in the multi-head attention mechanism;

[0106] head_dim: refers to the dimension of the feature vector processed by each attention head;

[0107] DDR: is a type of computer memory, Double Data Rate, used for storing and retrieving data;

[0108] DMA: Direct Memory Access is a computer data transfer method that allows peripheral devices to directly access system memory without CPU intervention.

[0109] L1 cache: This is the fastest storage layer in a computer system and is closest to the processor core. Its main function is to provide the processor with frequently used data and instructions, reducing the need to access main memory and improving overall system performance.

[0110] L2 cache: A cache layer used for data storage and transmission, commonly used in modern computing architectures (including CPUs, GPUs, and ASICs) to provide efficient data access and storage mechanisms. It usually appears between the L1 cache and main memory (or other external memory devices);

[0111] Hadamard product: It is an element-by-element product that acts on two matrices of the same dimension and produces a new matrix of the same dimension. Each element of the new matrix is ​​the product of the elements of the corresponding position in the original two matrices;

[0112] Dropout: It is a commonly used regularization technique that randomly discards the output of some neurons to prevent the model from over-relying on certain neurons, thereby reducing the risk of overfitting.

[0113] Kernel: A function that implements some core computing logic and can be executed on various accelerators.

[0114] PE: Processing Element, the basic unit of the computing array, used to perform accelerated computing.

[0115] The Attention operator template generated by this invention can flexibly adapt to the characteristics of different hardware platforms and achieve excellent performance under various shapes. Secondly, it fully considers the parallel computing capabilities and network topology characteristics of the hardware platform, achieving efficient data partitioning by optimizing the balance between computation and communication. Furthermore, by leveraging the hardware's ability to generate random number matrices, it accelerates the implementation of Dropout on the target platform, significantly improving the training speed of the Attention operator.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and description of the present invention.

Claims

1. A compilation optimization method for accelerating Attention calculation, characterized in that: The following steps are involved: Generate multiple scheduling strategies based on the number and shape of input matrices, attention type, and target hardware platform; Define a cost function to evaluate the time and space overhead of each scheduling strategy, and select the optimal strategy based on hardware platform characteristics and user needs; The input data is divided into multiple cores in parallel based on sequence length and head dimension, and computation and communication are overlapped through asynchronous communication to reduce latency. During the training phase, the random floating-point matrix generated by hardware is used to perform Hadamard product operations with the attention weights to implement the Dropout function.

2. A compilation optimization method for accelerating Attention calculation according to claim 1, characterized in that: The generation scheduling strategy includes: Determine the block size and loop traversal order based on the Attention operator type, the sequence length of the input tensor, and the number of heads; Combined with the target platform's L2 cache capacity and intermediate variable lifecycle, a scheduling template containing block calculation kernels and control flows is generated.

3. A compilation optimization method for accelerating Attention calculation according to claim 2, characterized in that: The evaluation indicators of the cost function include: Run time, storage usage, power consumption, and hardware parallelism, and perform multi-objective optimization with user-defined weights.

4. A compilation optimization method for accelerating Attention calculation according to claim 3, characterized in that: The multi-core parallel partitioning includes one of the following strategies: Distribute sequence lengths to different cores; Assign Head to different cores; The sequence length and Head are combined and divided into different cores.

5. A compilation optimization method for accelerating Attention calculation according to claim 4, characterized in that: The multi-core parallel partitioning further includes: Adjust the parallel dimension based on the number of cores of the target device, and pad the divided dimension to an integer multiple of the number of cores. Select the collective communication method based on the hardware topology and embed the communication function into the scheduling template.

6. A compilation optimization method for accelerating Attention calculation according to claim 5, characterized in that: The generation of the random floating point matrix includes: Generates a half-precision matrix whose element values ​​are 0 or 1 / (1-p) with probability p, and directly performs mixed-precision Hadamard product operations through vector components.

7. A compilation optimization method for accelerating Attention calculation according to claim 6, characterized in that: Also includes: During the compilation phase, users are supported to manually replace the automatically generated scheduling templates to achieve performance tuning on specific hardware platforms.

8. The compilation optimization method for accelerating Attention calculation according to claim 7, characterized in that: The generation of the scheduling strategy further includes: Dynamically adjust the block size and inter-core data allocation strategy based on the dynamic length of the input sequence and the number of heads.

9. A compilation optimization method for accelerating Attention calculation according to claim 8, characterized in that: The asynchronous communication includes: When transferring data between cores, high-precision data is converted to low-precision format to reduce communication bandwidth usage.

10. A compilation optimization device for accelerating Attention calculation, characterized in that: The compilation optimization method for accelerating Attention calculation according to any one of claims 1 to 9 comprises: Multi-core AI accelerator, each core contains a tensor component, a vector component, an IMM component, and a DMA component; The tensor component is used to perform matrix multiplication operations; The vector component is used to perform element-by-element operations and Softmax calculations; The IMM component is used to generate a random floating-point matrix to support the Dropout operation; The DMA component is used to realize data movement between memory and cache and inter-core communication; The multi-core AI accelerator supports data distribution across sequence length and Head dimension, and reduces communication overhead through low-precision conversion.

Images