Merculation NPU operator automatic tuning method and system based on fusion operator performance modeling

Abstract

The invention relates to a mercuric chloride NPU operator automatic tuning method and system based on fusion operator performance modeling, and the method comprises the steps: firstly constructing an end-to-end performance analysis model of a fusion operator for collaborative execution of a Cube core and a Vector core, and employing two modes to accurately predict execution delays under different parameter configurations through dynamic discrimination of a calculation speed relation between double cores; on the basis, a heuristic design space exploration algorithm with minimization of data movement amount as guidance is provided, and the algorithm is combined with the multilevel storage characteristic of the mercuric chloride NPU, adopts a two-stage strategy of block size optimization and buffer area allocation collaboration, and automatically searches for optimal parameter configuration meeting on-chip storage capacity constraints. According to the method, automatic performance adjustment and optimization of the fusion operator are realized, compared with traditional manual adjustment and optimization or compiler default configuration, near-optimal configuration can be found within a few minutes, the average performance prediction error is lower than 5%, the maximum performance speed-up ratio of part of operators can reach 1.46 times, and the efficiency of operator development and deployment on the mercuric chloride NPU is remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of AI operator optimization and hardware performance modeling, specifically involving an automatic operator tuning method and system for Ascend NPU based on the performance modeling of the Cube-Vector Fusion Operator for the collaborative execution of Cube cores and Vector cores in Huawei Ascend NPU. Background Technology

[0002] With the rapid development of artificial intelligence and deep learning, the Neural Processing Unit (NPU) has demonstrated significant advantages in energy efficiency and computational latency due to its highly customized design for deep learning computing. Huawei's Ascend NPU adopts the Da Vinci architecture, such as... Figure 2 As shown, it integrates a Cube core dedicated to matrix computation and a Vector core for vector computation. These two cores work closely together to efficiently execute complex fusion operators. This domain-specific hardware design achieves higher computational density and energy efficiency for specific AI workloads compared to general-purpose GPUs.

[0003] However, the advanced architecture of the Ascend NPU also brings unprecedented challenges in performance modeling and optimization. Its core complexity manifests in two aspects: First, it employs a complex multi-level storage hierarchy, requiring data to flow between off-chip memory (HBM), L2 cache, L1 cache, dedicated L0 cache (L0A / B / C), and unified buffer (UB), managed by multiple memory transfer engines (such as MTE2 and MTE1). Furthermore, mandatory format conversions are required when data moves between levels (e.g., ...). , These transformations introduce significant latency overhead. Secondly, the heterogeneous collaboration mechanism between the Cube core and the Vector core makes the execution data flow of the fusion operator more complex, involving data synchronization between cores and pipeline overlap.

[0004] In particular, performance modeling for Cube-Vector (CV) fusion operators (such as GeGLU and FFN in Transformers) presents unique and severe challenges. The typical pattern for these operators is that the Cube core first performs matrix operations (such as GEMM), and the intermediate results generated must immediately be handed over to the Vector core for a series of vectorization operations (such as activation function GeLU and element-wise operations). This collaborative process introduces pipeline dependencies and data synchronization overhead not present in traditional single-core operators. Its performance bottleneck no longer depends solely on the computational power of a single core, but is jointly determined by the relative processing speeds of the two cores, the efficiency of exchanging intermediate results at the storage level (usually through L2 cache as a shared workspace), and the degree of overlap between computation and data transfer under a double-buffering mechanism. As shown in the figure, the end-to-end latency of CV fusion operators exhibits two distinct patterns, corresponding to situations where the Cube core's computational efficiency is slower / faster than the Vector core's computational efficiency, respectively. Existing models for single-core or regular data flows cannot characterize this dynamic, data-dependent dual-core collaborative behavior.

[0005] Furthermore, the differences in architecture compared to other AI accelerators mean that performance models and optimization methods designed for traditional rule-based architectures (such as the systolic array used in Google TPU) cannot be directly applied to Ascend NPU. Systolic arrays rely on a regular array of two-dimensional processing units (PEs) and a relatively simple data flow; their performance can be approximated as a function of PE size, bandwidth, and data volume. In contrast, the execution mode of Ascend NPU involves pipelined operations of different computing cores, multi-stage data transmission, and format conversion.

[0006] Therefore, developers face a vast and multi-dimensional design space when manually optimizing operators for the Ascend NPU. For example... Figure 3 As shown, the matrix multiplication operator alone requires adjustment of the block size ( This involves multiple parameters, such as double buffer size and iteration order. Exhaustive searching for all possible combinations of these parameters incurs a staggering overhead: for example, the design space for a simple GEMM operator can reach [amount missing]. On a massive scale, exhaustive search through time-consuming on-board performance profiling takes over 2 hours. For more complex operators or diverse input shapes, this trial-and-error-based manual tuning method incurs enormous time and manpower costs, severely hindering operator development efficiency and deployment. Some research has attempted to address these issues, but all have limitations. On the one hand, performance modeling efforts such as Verrocchio, which built a unified model for the da Vinci core, simplify computation and memory access to a ratio of data volume to bandwidth, ignoring the fundamental differences in their behavior, leading to significant prediction errors for diverse operators. On the other hand, while learning-based black-box models (such as WMDRS) can make predictions using runtime metrics, they rely on feature collection and model training, introducing additional overhead. Furthermore, their interpretability is poor, and their generalization ability is significantly affected by feature selection, making them unreliable for various novel operators. In terms of automatic optimization, existing design space exploration (DSE) frameworks (such as Timeloop and GAMMA) mainly focus on mapping optimization of spatial architecture, failing to fully consider the heterogeneous core collaboration, format conversion overhead, and complex interactions between multi-level caches in Ascend NPU. Therefore, they cannot achieve efficient and refined collaborative optimization on such complex architectures.

[0007] In summary, the Ascend NPU's architectural advantages coexist with the deep optimization challenges it presents. Currently, there is a lack of a white-box performance analysis model that can accurately characterize its heterogeneous computing and complex data flow, as well as an automated method that can efficiently search for optimal configurations within a vast design space based on this model. This deficiency prevents the full utilization of hardware performance and restricts developer productivity. Therefore, an innovative solution integrating accurate performance modeling and efficient automated exploration is urgently needed to achieve rapid and optimal deployment of Ascend NPU fusion operators, which is the core motivation of this research. Summary of the Invention

[0008] This invention aims to address the problems in existing technologies, such as inaccurate performance modeling of Ascend NPU fusion operators, low efficiency of manual tuning, and poor performance of default compiler configurations. Specifically, it needs to solve the following problems: 1. How to accurately model the end-to-end performance of the Cube core and Vector core when they collaborate to execute fusion operators, especially the impact of data synchronization and pipeline overlap between cores.

[0009] 2. How to efficiently and automatically search for the optimal parameter configuration that meets hardware resource constraints and achieves the best performance within a large design space (involving multi-dimensional parameters such as partitioning, buffering, and iteration order), avoiding time-consuming exhaustive search or trial and error based on experience.

[0010] To solve the above-mentioned technical problems, the present invention provides the following technical solution: An automatic operator tuning method for Ascend NPU based on fusion operator performance modeling is characterized by the following steps: S1. Parse the input Ascend C source code and extract the operation sequence of the operators defined in the source code and the set of adjustable parameters including block size, buffer allocation strategy and iteration order; S2. Based on the operation sequence extracted in step (1), construct an end-to-end performance analysis model for the operator. The performance analysis model identifies whether the operator is a fusion operator executed collaboratively by the Cube core and the Vector core according to the operation sequence of the operator. S3. If the operator is identified as a fusion operator, then proceed to step (4); otherwise, adopt the performance model and optimization strategy for single-core operators. S4. Based on the design space defined by the set of adjustable parameters, and in combination with the hardware capacity constraints of the Ascend NPU on-chip memory, initialize an optimization problem aimed at minimizing end-to-end execution latency; S5. The optimization problem is solved using a data movement-oriented heuristic design space exploration algorithm, wherein each iteration of the algorithm includes: S5.1 Based on the current candidate parameter configuration, use the performance analysis model constructed in step S2 to predict its end-to-end execution latency; S5.2 Calculate the total amount of data moved from off-chip storage to on-chip storage corresponding to the current parameter configuration; S5.3 Generate the next set of candidate parameter configurations based on the predicted delay and total data movement, and following the heuristic rules; S5.4 Repeat step S5 until the preset search termination condition is met, and output the parameter configuration with the smallest prediction delay among all candidate configurations that meet the hardware constraints as the final tuning result.

[0011] Furthermore, in step S2, constructing the performance analysis model for the Cube-Vector fusion operator specifically includes: S2.1 The time for independently modeling the Cube core and Vector core to process a single computational block is denoted as S2.1. and The total time for each of the computational blocks to be processed independently is denoted as . , ; S2.2 According to and The comparison results model the end-to-end delay into two modes: -when When using the first mode, the overall delay is... -when When the second mode is used, the overall delay is reduced. ; S2.3 further supports fine-grained overlap of the computation pipeline between cores by dividing the cores into sub-blocks, and calculates the optimized latency accordingly. Finally, with min( , This is used as the final performance prediction value for this configuration.

[0012] Furthermore, the heuristic design space exploration algorithm guided by data movement volume described in step S5 aims to minimize the total data movement volume. As a key guiding principle in the search process, , and They refer to from HBM and L2 Cache The amount of data transferred to on-chip storage.

[0013] Furthermore, the heuristic design space exploration algorithm employs a phased collaborative optimization strategy: S4.1 First stage: Joint optimization of block size: Cube Starting with the minimum block size supported by the core, the block size parameters are adjusted sequentially according to preset heuristic rules aimed at improving on-chip cache utilization. baseM , baseN, baseK And after each adjustment, evaluate the performance through step (5.1), record the configuration with better performance, until the hardware capacity constraint is reached; S4.2 The second stage involves co-optimizing buffer allocation and iteration order: Based on the candidate block size obtained in the first stage, a greedy strategy is adopted to allocate double buffers for the input matrix participating in the calculation under the L1 buffer capacity constraint, for different calculation iteration orders of row priority and column priority respectively. The allocation objective is to minimize the total data movement while satisfying the capacity requirement. M data .

[0014] Furthermore, the heuristic rule in the first stage is as follows: prioritize increasing the size of baseK to improve the utilization of L0A / B cache, and then coordinately increase the sizes of baseM and baseN to optimize the utilization of LOC cache.

[0015] Furthermore, in the second stage, when the row-first iteration order is adopted, the execution logic of the greedy strategy is as follows: first, allocate sufficient buffer space for the left input matrix to accommodate all the block data involved in one iteration row; then allocate the remaining space of the L1 cache to the right input matrix to cache as much of its block data as possible.

[0016] Furthermore, when constructing the performance analysis model in step S2, the hardware-specific parameters on which it depends are obtained through micro-benchmark testing or by analyzing the actual hardware. These parameters include the block processing latency of the Cube core, the transmission bandwidth between different levels of storage, the data format conversion overhead coefficient, and the pipeline interval between operations within the Vector core.

[0017] This invention also provides an automatic operator tuning system for Ascend NPU, characterized in that the system includes: The code parsing and modeling module is used to parse the operator source code and build the corresponding performance analysis model. The automatic tuning engine module is used to automatically search for and output the optimal parameter configuration; The hardware constraint management module stores the on-chip memory capacity and architecture parameters of the Ascend NPU, and provides hardware constraint information for the code parsing and modeling module and the automatic tuning engine module.

[0018] Compared with existing technologies, it has the following beneficial effects: 1. High Modeling Accuracy: By separating and refining the modeling of computation, memory access, and inter-core collaboration, the constructed end-to-end performance analysis model can accurately predict the execution latency of fusion operators. Experiments show that for various representative fusion operators such as GEMM, GeGLU, FFN, and Attention, the average prediction error is less than 5% under diverse input shapes and parameter configurations.

[0019] 2. Automated and efficient tuning: The proposed DSE algorithm eliminates the need for repeated, time-consuming onboard testing. Relying solely on rapid analysis model evaluation, it can optimize within minutes within a vast design space (e.g., The search is completed within a few orders of magnitude. Compared to a brute-force search that would take hours, this represents a two-order-of-magnitude improvement in efficiency.

[0020] 3. Significant Performance Improvement: The automatically searched parameter configuration delivers a significant performance boost compared to the default configuration generated by the Ascend C compiler. Under various irregular input shapes, the performance speedup can reach up to 1.46 times. Even for regular shapes (dimensions powers of 2) that have already been well optimized by the compiler, the method of this invention still achieves comparable or better performance.

[0021] 4. Method versatility: The performance modeling method described delves into the details of hardware architecture and is not only applicable to Ascend NPU, but its analysis ideas based on pipelines and data movement can also provide a reference for the performance modeling and optimization of other AI accelerators with complex memory hierarchies and heterogeneous computing units. Attached Figure Description

[0022] Figure 1This is a flowchart illustrating the overall process of the automatic optimization method of the present invention. Figure 2 This is a schematic diagram of the Ascend NPU Da Vinci architecture. Figure 3 : This is the computational flow of the Cube-Vector fusion operator (taking GeGLU as an example). Figure 4 These are two computation modes of the Cube-Vector fusion operator. Figure 5 This diagram illustrates the default execution pipeline of the Cube-Vector fusion operator and a diagram showing how fine-grained sub-tile optimization of pipeline overlap between cores is used. Detailed Implementation

[0023] To make the technical problems to be solved, the technical solutions, and the beneficial effects of this method clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0024] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific examples disclosed below.

[0025] This invention provides an automatic tuning method for Ascend NPU operators based on fusion operator performance modeling. This method is executed by a software system (which can be called the NPUMeter automatic tuning framework) that implements the corresponding functions. The overall flowchart of this system is as follows: Figure 1 As shown, it includes two parts: performance analysis model construction and design space exploration algorithm.

[0026] 1. End-to-end performance analysis model of fusion operator First, the Ascend C code of the input operator is parsed to extract the operation sequence and adjustable parameters. Based on this, a refined performance model for the Cube-Vector (CV) fusion operator is constructed, identifying and modeling two typical modes of dual-core collaboration, and comparing them... Cube Core processing time for a single block and Vector Core processing time for a single block ,like Figure 4 As shown, the end-to-end modeling formula is divided into two modes: Mode 1 ( Cube Core computing efficiency is slower than or equal to Vector Core computing efficiency): when The overall end-to-end delay calculation formula is as follows:

[0027] in, for Vector The total time for the core to independently process all blocks.

[0028] Mode 2 ( Cube Core computing efficiency is faster than Vector Core computing efficiency): when The overall end-to-end delay calculation formula is as follows:

[0029] in, for Cube The total time for the core to independently process all blocks.

[0030] Further optimization of pipeline overlap between cores; the model supports fine-grained sub-tile strategies, allowing... Vector The core is Cube The e-core begins processing immediately after generating a sub-tile, thereby reducing overall latency. This model achieves high-precision prediction of end-to-end latency by separating modeling computation, storage access, and format conversion stages, and by incorporating measured hardware parameters (such as core block processing latency, bandwidth at each level, and format conversion coefficients).

[0031] 2. Design Space Exploration (DSE) Algorithm Based on Data Movement Based on the aforementioned performance model that can be quickly evaluated, this invention proposes a heuristic DSE algorithm for automatically searching for optimal parameter configurations. Specifically, the tuning problem is formalized as: minimizing the performance model prediction latency under on-chip memory (L1 Cache, L0A / B / C Cache) capacity constraints. The algorithm uses the total amount of data moved. This guides the search direction and is defined as the total amount of data transferred from off-chip storage (HBM) and L2 cache to on-chip storage, as shown in the following formula:

[0032] in, and These refer to the amount of data moved from HBM and L2 Cache, respectively. This metric effectively reflects the performance bottleneck in the fusion operator that is constrained by memory bandwidth.

[0033] The algorithm employs a two-stage search strategy: Phase 1: Joint Optimization of Block Size fromCube The core supports the smallest granularity of block partitioning (such as...) It starts by gradually increasing the block size. It uses heuristic rules to gradually increase the block size, for example, first increasing... To improve L0A / B cache utilization, a rebalanced increase and To improve LOC cache utilization, the configuration with the lowest latency is recorded. After each adjustment, the performance model is called to evaluate latency. Until the hardware capacity limit is reached or performance no longer improves.

[0034] Phase Two: Co-optimization of Buffer Allocation and Iteration Order Under a defined L1 cache capacity constraint, a greedy strategy is employed to allocate double buffer sizes for candidate iteration orders such as row-first or column-first. Taking row-first order as an example, the algorithm first allocates a buffer for the left input matrix to accommodate a complete row of blocks (achieving high reuse), then allocates a buffer for the right input matrix to accommodate as many of its blocks as possible. If space remains, it is then allocated to the left matrix. This process aims to minimize... This will reduce delays caused by data movement.

[0035] 3. Integration of automatic tuning processes The aforementioned models and algorithms are integrated into an automated tuning framework: after receiving the operator source code, the system automatically parses the parameters, instantiates the performance model, and runs the DSE algorithm to search for the optimal configuration, ultimately outputting the parameter combination that satisfies hardware constraints and has the lowest prediction latency. The entire process eliminates the need for repeated on-board testing, significantly reducing tuning time and costs.

[0036] The following example illustrates the implementation of this invention using automatic tuning of the GeGLU (Gated Linear Unit with GeLU activation) fusion operator in a Transformer model on an Ascend 910B NPU. On the Ascend NPU, the GeGLU operator is typically performed by matrix multiplication by the Cube core, and the result is then passed to the Vector core for sequential GeLU activation and element-wise multiplication operations, making it a typical Cube-Vector fusion operator.

[0037] Example: Automatic tuning of GeGLU operators The goal of this embodiment is to automatically search for a set of optimal compilation configuration parameters, given the Ascend C language implementation source code of the GeGLU operator, under the premise of satisfying the hardware resource constraints of the Ascend 910B NPU, so as to minimize the actual execution time of the operator.

[0038] Step S1: Source Code Analysis and Information Extraction The system receives the Ascend C source code file of the GeGLU operator input by the user. This source code defines the computation logic of the operator, which usually includes the matrix multiplication (GEMM) part executed by the Cube core and the activation function and element-wise operation part executed by the Vector core.

[0039] The parser performs lexical and syntactic analysis on the source code and extracts the following key information: Parameter configuration: Identifies adjustable parameters, including the block size of the Cube core ( The size of the double buffer allocated in the L1 cache for the two input matrices. Vector core processing block size And the order of iteration (e.g., row priority).

[0040] Model parameterization: Based on the hardware specifications of the Ascend 910B NPU, specific parameters required for the model are obtained through micro-benchmarking or profiling, such as the Cube core block processing latency. Assembly line start interval Bandwidth at each storage tier ( ), format conversion factor Vector core sub-operation switching delay wait.

[0041] Step S2: Building the GeGLU End-to-End Performance Analysis Model 1. Cube Part Model: Based on the following formula, using the input matrix shape (M, N, K) and parameters ( ) and the execution time of the instructions collected by the upper board and command transmission interval The time spent on production line downtime Calculate the total computation time for the Cube core to process all blocks. and single block time Meanwhile, data transfer times for stages such as MTE2, MTE1, and Fixpipe are calculated using memory access modeling formulas.

[0042]

[0043] 2. Vector Partial Model: Based on the following formula, combining the Vector sub-operation sequence (GeLU decomposes it into multiple sub-operations) and the values ​​of each sub-operation... , and the execution time of the instructions collected by the upper board. Calculate the total time for the Vector core to process all blocks. and single block time .

[0044]

[0045] 3. Fusion Modeling: Comparison With Assuming that, according to calculations, According to Mode 1 in the inventive solution, the initial end-to-end delay prediction value of GeGLU is:

[0046] 4. Fine-grained optimization evaluation: Furthermore, the model can evaluate even finer-grained pipeline overlap schemes (such as further dividing the blocks generated by the Cube into Sub-Tile). If this strategy is adopted, the delayed update is:

[0047] in This refers to the time it takes for the Cube core to calculate one sub-tile. The model will select... and The smaller value in the range is used as the final prediction delay under this configuration.

[0048] Step S3: Design Space Exploration and Optimal Configuration Search Given an input shape, such as (M=512, N=1792, K=1024), and L1 and L0 cache capacity constraints.

[0049] 1. Initialization and constraint settings: Set the initial block size to (16,16,16), double buffering to minimum allocation, and iteration order to row first.

[0050] 2. Two-stage heuristic search: Phase 1: Joint Optimization of Block Size Call the performance model to evaluate the latency of the current configuration. and Subsequently, adjustments were made according to the aforementioned heuristic rules ( For example, first... Observe the changes in lag by increasing the value from 16 to 32 and then to 64; then appropriately decrease the value. At the same time increase and Each adjustment is followed by a reassessment, and the configuration with lower latency is recorded. This process continues until the total block size approaches the L0 cache capacity limit.

[0051] Phase Two: Buffer and Iteration Order Optimization Several optimal block size combinations selected in the first stage are fixed. For each block combination, a greedy buffer allocation algorithm is executed for both "row-first" and "column-first" iteration orders.

[0052] For row-major order, the left matrix takes precedence (corresponding to...). (Divide) allocate buffers until they can hold a complete line ( The data is divided into blocks (number of blocks). Then, the right matrix (corresponding to...) is... The buffer is allocated in blocks, increasing the size of each block by half a matrix column until the L1 cache is full or... No further reduction. Calculate the latency for each buffer allocation scheme.

[0053] Iterate through all evaluated parameter combinations (block size, iteration order, buffer allocation) and select the configuration that satisfies all hardware constraints and minimizes the performance model's predicted latency as the output. In this example, assume the final output configuration is: , , The process is row-first iteration, and the L1 double buffer is allocated as (128KB, 256KB).

[0054] Step S4: Verification and Results The configurations automatically generated by NPUMeter and the default configuration of the Ascend C compiler were compiled and run separately on the board. The test results show that, with an input shape of (512, 1792, 1024), the execution time of the automatically tuned configuration was 435.26 μs, while the execution time of the default compiler configuration was 503.25 μs, representing a performance improvement of 1.16 times (speedup). Meanwhile, the entire search process only took approximately 15.3 minutes. The predicted latency of the performance model and the experimental error were within 5%, verifying the accuracy and efficiency of the method.

[0055] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An automatic operator tuning method for Ascend NPU based on fusion operator performance modeling, characterized in that, Includes the following steps: S1. Parse the input Ascend C source code and extract the operation sequence of the operators defined in the source code and the set of adjustable parameters including block size, buffer allocation strategy and iteration order; S2. Based on the operation sequence extracted in step (1), construct an end-to-end performance analysis model for the operator. The performance analysis model identifies whether the operator is a fusion operator executed collaboratively by the Cube core and the Vector core according to the operation sequence of the operator. S3. If the operator is identified as a fusion operator, then proceed to step (4); otherwise, adopt the performance model and optimization strategy for single-core operators. S4. Based on the design space defined by the set of adjustable parameters, and in combination with the hardware capacity constraints of the Ascend NPU on-chip memory, initialize an optimization problem aimed at minimizing end-to-end execution latency; S5. The optimization problem is solved using a data movement-oriented heuristic design space exploration algorithm, wherein each iteration of the algorithm includes: S5.1 Based on the current candidate parameter configuration, use the performance analysis model constructed in step S2 to predict its end-to-end execution latency; S5.2 Calculate the total amount of data moved from off-chip storage to on-chip storage corresponding to the current parameter configuration; S5.3 Generate the next set of candidate parameter configurations based on the predicted delay and total data movement, and following the heuristic rules; S5.4 Repeat step S5 until the preset search termination condition is met, and output the parameter configuration with the smallest prediction delay among all candidate configurations that meet the hardware constraints as the final tuning result.

2. The automatic tuning method for Ascend NPU operators based on fusion operator performance modeling according to claim 1, characterized in that, In step S2, the construction of a performance analysis model for the Cube-Vector fusion operator specifically includes: S2.1 The time for independently modeling the Cube core and Vector core to process a single computational block is denoted as S2.

1. and The total time for each of the computational blocks to be processed independently is denoted as . , ; S2.2 According to and The comparison results model the end-to-end delay into two modes: -when When using the first mode, the overall delay is... -when When the second mode is used, the overall delay is reduced. ; S2.3 further supports fine-grained overlap of the computation pipeline between cores by dividing the cores into sub-blocks, and calculates the optimized latency accordingly. Finally, with min( , This is used as the final performance prediction value for this configuration.

3. The automatic tuning method for Ascend NPU operators based on fusion operator performance modeling according to claim 1, characterized in that, Step S5 describes a heuristic design space exploration algorithm guided by data movement volume, aiming to minimize the total data movement volume. As a key guiding principle in the search process, , and They refer to from HBM and L2 Cache The amount of data transferred to on-chip storage.

4. The automatic tuning method for Ascend NPU operators based on fusion operator performance modeling according to claim 3, characterized in that, The heuristic design space exploration algorithm employs a phased collaborative optimization strategy: S4.1 First stage: Joint optimization of block size: Cube Starting with the minimum block size supported by the core, the block size parameters are adjusted sequentially according to preset heuristic rules aimed at improving on-chip cache utilization. baseM , baseN, baseK And after each adjustment, evaluate the performance through step (5.1), record the configuration with better performance, until the hardware capacity constraint is reached; S4.2 The second stage involves co-optimizing buffer allocation and iteration order: Based on the candidate block size obtained in the first stage, a greedy strategy is adopted to allocate double buffers for the input matrix participating in the calculation under the L1 buffer capacity constraint, for different calculation iteration orders of row priority and column priority respectively. The allocation objective is to minimize the total data movement while satisfying the capacity requirement. M data .

5. The automatic tuning method for Ascend NPU operators based on fusion operator performance modeling according to claim 4, characterized in that, The specific heuristic rule in the first stage is as follows: prioritize increasing the size of baseK to improve the utilization of L0A / B cache, and then coordinately increase the sizes of baseM and baseN to optimize the utilization of LOC cache.

6. The automatic tuning method for Ascend NPU operators based on fusion operator performance modeling according to claim 4, characterized in that, In the second stage, when the row-first iteration order is adopted, the execution logic of the greedy strategy is as follows: first, allocate enough buffer space to the left input matrix to accommodate all the block data involved in one iteration row; then allocate the remaining space of the L1 cache to the right input matrix to cache as much of its block data as possible.

7. The automatic tuning method for Ascend NPU operators based on fusion operator performance modeling according to claim 1, characterized in that, When constructing the performance analysis model in step S2, the hardware-specific parameters on which it depends are obtained through micro-benchmark testing or by analyzing the actual hardware. These parameters include the block processing latency of the Cube core, the transmission bandwidth between different levels of storage, the data format conversion overhead coefficient, and the pipeline interval between operations within the Vector core.

8. An automatic operator tuning system for Ascend NPU, characterized in that, The system includes: The code parsing and modeling module is used to parse the operator source code and build the corresponding performance analysis model. The automatic tuning engine module is used to automatically search for and output the optimal parameter configuration; The hardware constraint management module stores the on-chip memory capacity and architecture parameters of the Ascend NPU, and provides hardware constraint information for the code parsing and modeling module and the automatic tuning engine module.

Images