Artificial intelligence operator simulation method, simulator and computer equipment

Abstract

The invention provides an artificial intelligence operator simulation method, a simulator and computer equipment. The method comprises the steps that an operator description interface of a simulator is utilized, and a configuration file needed when a target operator is executed is set; the configuration file is used for describing various parameters involved when the target operator is executed; the target operator comprises a plurality of different types of artificial intelligence operators; according to the configuration file, extracting an instruction behavior of the target operator on the graphics processor; and mapping the instruction behavior of the target operator into a target transaction of a system-level design language of the simulator, generating a kernel instance of the target transaction, triggering the target transaction based on the kernel instance of the target transaction, and simulating operation of the target operator and a plurality of hardware modules of the simulator.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an artificial intelligence operator simulation method, simulator, and computer equipment. Background Technology

[0002] With the continuous development of artificial intelligence technology, deep learning models have been widely used in various applications. To improve the efficiency of model training and inference, it is necessary to evaluate the performance of the artificial intelligence operators involved in deep learning models. In the existing technology, the performance analysis of artificial intelligence operators mainly relies on simulators based on the NVIDIA CUDA architecture (such as GPGPU-Sim). These simulators can only be used with NVIDIA's graphics processing units (GPUs), and each generation of architecture requires writing corresponding operators separately, resulting in weak compatibility and scalability. Summary of the Invention

[0003] This application provides an artificial intelligence operator simulation method, simulator, and computer device, which can simulate different types of artificial intelligence operators and has strong compatibility and scalability.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides an artificial intelligence operator simulation method, including: using the simulator's operator description interface to set a configuration file required for the execution of a target operator; the configuration file is used to describe the various parameters involved in the execution of the target operator; the target operator includes multiple different types of artificial intelligence operators; and based on the configuration file, extracting the instruction behavior of the target operator on the graphics processor. The instruction behavior of the target operator is mapped to the target transaction of the simulator's system-level design language, and a kernel instance of the target transaction is generated. The target transaction is triggered based on the kernel instance of the target transaction to simulate the operation of the target operator and multiple hardware modules of the simulator.

[0005] This application provides a simulator including: an operator description interface and an execution module; the operator description interface is used to set the configuration file required when the target operator is executed; the configuration file is used to describe the various parameters involved in the execution of the target operator; the target operator includes multiple different types of artificial intelligence operators; the execution module is used to extract the instruction behavior of the target operator on the graphics processor based on the configuration file; map the instruction behavior of the target operator to the target transaction of the simulator's system-level design language, and generate a kernel instance of the target transaction; trigger the target transaction based on the kernel instance of the target transaction to simulate the operation of the target operator and multiple hardware modules of the simulator.

[0006] This application provides a computer device including an emulator, which is used to execute the steps in the above-described method.

[0007] In this embodiment, the simulator's operator description interface is used to set the configuration file required for the execution of the target operator. This unified operator description interface allows for the standardized configuration of different types of AI operators, including general-purpose and novel operators, enabling rapid integration of novel operators into existing frameworks and improving configuration flexibility and scalability. The instruction behavior of the target operator on the graphics processor is extracted from the configuration file and mapped to target transactions in the simulator's system-level design language. A kernel instance of the target transaction is generated, and the target transaction is triggered based on this kernel instance, simulating the operation of the target operator with multiple hardware modules of the simulator. This converts the target operator's instruction behavior into target transactions within the simulator. Through the scheduling and execution of these target transactions, the simulator can accurately simulate the performance of the target operator under different graphics processor architectures. Therefore, efficient performance modeling and analysis of multiple different types of AI operators under different graphics processor architectures is achieved, improving the flexibility and scalability of operator-level simulation and providing effective theoretical support for future research and optimization of graphics processor architectures. Attached Figure Description

[0008] Figure 1 This is a first flowchart illustrating the artificial intelligence operator simulation method provided in this application embodiment; Figure 2 This is a schematic diagram of the tensor memory engine in the simulator provided in the embodiments of this application; Figure 3 This is a schematic diagram of the second process of the artificial intelligence operator simulation method provided in the embodiments of this application; Figure 4 This is a flowchart illustrating the Flash Attention operator in the artificial intelligence operator simulation method provided in this application embodiment.

[0009] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0012] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0013] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0014] This application provides an artificial intelligence operator simulation method applied to a simulator, which can be implemented through the following steps 101 to 104: Step 101: Using the simulator's operator description interface, set the configuration file required for the execution of the target operator; the configuration file is used to describe the various parameters involved in the execution of the target operator; the target operator includes multiple different types of artificial intelligence operators.

[0015] Simulators are used to simulate the behavior of different types of artificial intelligence operators on graphics processing units (GPUs) to evaluate the performance of these different types of AI operators. In some implementations, the simulator can be an AI operator performance analysis tool based on a system design-level language (SystemC).

[0016] The simulator uses a unified operator description interface. This interface is used to configure the configuration files required for operator execution. In some implementations, the operator description interface can be a JSON interface or a YAML interface. If the operator description interface is a JSON interface, the configuration file can be in JSON format; if the operator description interface is a YAML interface, the configuration file can be in YAML format.

[0017] A target operator refers to one or more artificial intelligence operators that need to be simulated. These operators can be of the same or different types. An artificial intelligence operator (AI operator) is a basic unit in a deep learning model that performs a specific computational task. For example, AI operators can include, but are not limited to, operations such as matrix multiplication, attention mechanisms, and convolution. Each operator typically contains a set of input data, computational logic, and output results, and is a fundamental building block in the neural network inference and training process.

[0018] The configuration file describes various parameters involved in operator execution, reflecting the operator's execution logic and hardware resource allocation strategy. These parameters may include, but are not limited to, key information such as the operator type (e.g., GEMM, Flash Attention), input dimension, output dimension, thread block partitioning method, and data precision. For example, if the target operator is a conventional operator, its configuration file may include defining the dimensions of matrix A and matrix B, the data type of computation (FP32 or FP16), and whether to enable Tensor Core acceleration. If the target operator is a newer Transformer-optimized operator like Flash Attention, its configuration file may also include defining specific parameters such as the number of attention heads, query-key-value dimension, and input sequence length.

[0019] It should be noted that by setting a unified operator description interface, different types of artificial intelligence operators, such as general operators and novel operators, can be configured in a unified manner, enabling novel operators to be quickly integrated into the existing framework, thereby improving the flexibility and scalability of configuration.

[0020] In some implementations, in Flash Attention scenarios, the configuration file may specify the size of each thread block, preloading strategy, and memory access mode. These parameters directly affect the behavior of the Tensor Memory Engine and the Tensor Core Engine. Furthermore, the configuration file supports flexible adjustments to the hardware architecture, such as modifying L1 / L2 cache capacity and NoC bandwidth limits, thereby meeting the modeling needs of different GPU architectures. This design not only improves modeling efficiency but also provides theoretical support for future hardware optimization.

[0021] In some implementations, the simulator's operator description interface can be used to write the configuration file required by the target operator during execution, in JSON or YAML format.

[0022] Step 102: Based on the configuration file, extract the instruction behavior of the target operator on the graphics processor.

[0023] Instruction behavior refers to a series of operations triggered by the target operator throughout the execution process, including data loading, computation, storage, synchronization, and other behaviors. These instruction behaviors can be represented in the form of abstract instructions in the simulation environment, such as instruction set architecture (ISA) or transactions in system-level languages ​​(SystemC).

[0024] In some implementations, the configuration file can be parsed to identify the various execution stages of the target operator and extract the instruction behaviors of each execution stage. These execution stages include, but are not limited to, the prologue, the main loop, and the epilogue. For example, the instruction behaviors in the main loop stage of Flash Attention include: extracting Ki+1 and Vi from the Tensor Memory Engine (TME), triggering the calculation of QKi+1, and continuing to process operations such as PVi and Row Sum after QKi+1 is calculated.

[0025] Step 103: Map the instruction behavior of the target operator to the target transaction of the simulator's system-level design language, and generate a kernel instance of the target transaction.

[0026] Mapping refers to translating the instruction behavior of the target operator into a transactional form usable within the simulator, enabling the simulator to understand and execute these operations. A transaction is a type of event-driven operation defined within the simulator, used to trigger the behavior of hardware modules within the SystemC environment. These transactions can include attributes such as event name, operation type, required resources, and timestamps, simulating the specific details of hardware execution. A target transaction refers to the transaction used to simulate the execution process of the target operator.

[0027] System-level design language (SystemC) is a high-level abstract language used to describe the behavior of hardware systems, and is used in modeling environments such as SystemC. System-level design language can accurately describe the behavior, data flow, and control flow of hardware modules, facilitating subsequent simulation and verification.

[0028] In SystemC, a transaction refers to an event-driven unit of operation used to transfer control and data flow between hardware modules. By mapping operator instruction behavior to transactions, the simulator can accurately simulate the interaction behavior between hardware modules without relying on the underlying clock cycle details.

[0029] A kernel instance is a concrete instantiated object of the target transaction running within the simulator, containing the data, parameters, and state information required by the target transaction. After the kernel instance is generated, the simulator executes the transaction logic based on the kernel instance.

[0030] In some implementations, the kernel instance of the target transaction encapsulates the specific operations performed by the target operator, such as data reading, matrix multiplication, and normalization calculation, and carries relevant parameters (such as address, length, and data format). The kernel instance of the target transaction serves as a communication interface between the operator and the hardware module, enabling the simulator to understand and respond to the operator's behavior.

[0031] In some implementations, the interaction interface between the operator and the hardware module can be used to map the instruction behavior of the target operator to the target transaction of the simulator's system-level design language. The interaction interface is used to establish the connection between the operator and the hardware module. For example, a TME Load instruction can be mapped to a transaction named tme_load. The tme_load transaction carries information such as the data source address, target cache address, and data size, and triggers the corresponding data loading behavior in the simulator.

[0032] In some implementations, the instruction behavior of the target operator can be mapped to the target transaction of the system-level design language in the simulator based on the mapping rules between instruction behavior and transactions in the simulator's system-level design language. The mapping rules characterize the conversion method between instruction behavior and transactions in the system-level design language. For example, the mapping rules include mapping instruction behavior such as instruction encoding, operands, memory access behavior, and resource consumption to transaction attributes, and mapping execution timing to transaction methods.

[0033] In some implementations, the instruction behavior of the target operator can be parsed to obtain the instruction characteristics of the target operator (such as operands, addressing modes, etc.); based on the instruction characteristics of the target operator, a transaction class is defined and instantiated, and the instantiated transaction is the target transaction.

[0034] In some implementations, simulator instructions for the target operator can be constructed based on the instruction behavior of the target operator. The simulator instructions can then be used to map the instruction behavior of the target operator to the target transaction of the simulator's system-level design language.

[0035] In some implementations, after obtaining the target transaction, the simulator can automatically generate a kernel instance of the target transaction. For example, after obtaining the target transaction, a kernel template matching the operator type of the target operator can be determined from multiple kernel templates based on the operator type of the target operator; and a kernel instance of the target transaction can be generated based on the matching kernel template.

[0036] It should be noted that the simulator in this application is built based on a system design-level language. Therefore, during operator mapping, the instruction behavior of the operator needs to be mapped to transactions in the system design language. However, if the simulator is built based on a language other than a system design-level language, then during operator mapping, the instruction behavior of the operator needs to be mapped to transactions recognizable by that other language. By converting the instruction behavior of the target operator into target transactions in the simulator's system-level design language, transaction-level modeling of the target operator's execution process is achieved. Through transaction scheduling and execution, the simulator can accurately simulate the performance of the target operator under different graphics processor architectures, including key indicators such as computational latency, memory bandwidth utilization, and bottleneck locations.

[0037] Step 104: Trigger the target transaction based on the kernel instance of the target transaction to simulate the operation of the target operator and multiple hardware modules of the simulator.

[0038] In some implementations, the processing logic of the target transaction can be executed based on the kernel instance of the target transaction, thereby simulating the operation of the target operator and multiple hardware modules of the simulator.

[0039] In some implementations, after the kernel instance of the target transaction triggers the target transaction, it sends control signals to multiple hardware modules of the simulator and receives the return results from multiple hardware modules.

[0040] In this embodiment, the simulator's operator description interface is used to set the configuration file required for the execution of the target operator. This unified operator description interface allows for the standardized configuration of different types of AI operators, including general-purpose and novel operators, enabling rapid integration of novel operators into existing frameworks and improving configuration flexibility and scalability. The instruction behavior of the target operator on the graphics processor is extracted from the configuration file and mapped to target transactions in the simulator's system-level design language. A kernel instance of the target transaction is generated, and the target transaction is triggered based on this kernel instance, simulating the operation of the target operator with multiple hardware modules of the simulator. This converts the target operator's instruction behavior into target transactions within the simulator. Through the scheduling and execution of these target transactions, the simulator can accurately simulate the performance of the target operator under different graphics processor architectures. Therefore, efficient performance modeling and analysis of multiple different types of AI operators under different graphics processor architectures is achieved, improving the flexibility and scalability of operator-level simulation and providing effective theoretical support for future research and optimization of graphics processor architectures.

[0041] In some embodiments, step 101 above can be implemented by the following steps 1011 to 1012: Step 1011: Use the first operator description interface to set the configuration parameters of the target operator; the configuration parameters of the target operator are used to define the execution logic of the target operator.

[0042] The First Operator Description Interface (FIDMI) is a configuration interface for operator settings, used to define the behavior patterns and execution flow of the target operator. The FIDMI supports configuration information in multiple formats, such as JSON or YAML, facilitating the rapid construction and switching of different operator models. For example, in the Flash Attention scenario, users can specify key parameters such as matrix size, thread block partitioning method, and data precision through the FIDMI to precisely control the execution behavior of the target operator on the graphics processor.

[0043] In some implementations, when a user wants to perform performance analysis on a GEMM operator, a preset configuration file can be loaded through the first operator description interface. The simulator automatically generates the corresponding execution flow based on the configuration parameters of the target operator and starts the simulation.

[0044] Step 1012: Use the second operator description interface to set the configuration parameters of the hardware module; the configuration parameters of the hardware module are used to define the resource allocation logic of the hardware module; the operator description interface includes the first operator description interface and the second operator description interface, and the configuration file includes the configuration parameters of the target operator and the configuration parameters of the hardware module.

[0045] The second operator description interface is a configuration interface for hardware modules, allowing users to set operating parameters for different hardware modules (such as L1 / L2 cache, DDR memory, and NoC interconnect). Operating parameters may include, but are not limited to, bandwidth limits, latency, and capacity. These parameters control the behavior of the hardware modules during simulation. For example, users can use the second operator description interface to set the read latency of the TME module to 5 cycles, or to set the maximum L2 cache capacity to 2MB, to more closely resemble a real hardware environment.

[0046] The second operator description interface works in tandem with the first operator description interface. The first operator description interface defines the execution logic of the target operator, while the second operator description interface is responsible for adapting the execution logic of the target operator to the specific hardware environment. Together, the first and second operator description interfaces constitute a complete modeling framework that ensures the accuracy and consistency of the execution logic of the target operator under different hardware configurations.

[0047] In practical applications, when users want to evaluate the impact of a new GPU architecture on the Flash Attention operator, they can adjust the configuration parameters of the hardware module through the second operator description interface, such as increasing L2 cache bandwidth or reducing TME latency, and then rerun the simulator to observe performance changes. Parametric modeling significantly enhances the ability to explore hardware architecture optimization.

[0048] In this embodiment, the computational logic of the target operator can be flexibly adjusted through the first operator description interface, and the simulated hardware environment can be flexibly adjusted through the second operator description interface. This enables independent configuration of the target operator and the hardware modules on which the target operator depends, and can flexibly adapt to different types of artificial intelligence operators and diverse graphics processor architectures without modifying the underlying hardware modeling code, thereby improving modeling efficiency and maintainability.

[0049] In some embodiments, step 102 above can be implemented by the following steps 1021 to 1022: Step 1021: Determine the operator execution code pre-stored in the simulator.

[0050] Operator execution code refers to the specific logic of operator execution. In some implementations, operator execution code describes information such as the operator execution pipeline, the execution path in the graphics processor architecture, scheduling logic, and hardware resource usage.

[0051] In some implementations, operator execution code can be predefined and stored in a simulator (such as the SystemC modeling environment).

[0052] Step 1022: Based on the operator execution pipeline defined in the operator execution code, identify the configuration file to extract the instruction behavior corresponding to the target operator.

[0053] The instruction behavior corresponding to the target operator is used to reflect the execution characteristics of the target operator on the graphics processor architecture.

[0054] In some implementations, the configuration file can be parsed using the pipeline of operator execution defined in the operator execution code to identify each execution stage of the target operator and extract the instruction behavior of each execution stage.

[0055] In this embodiment, by pre-storing operator execution code in the simulator and dynamically identifying the configuration file of the target operator based on the pipeline of operator execution defined in the operator execution code, the instruction behavior of the target operator can be identified, thereby quickly building a performance analysis model for artificial intelligence operators.

[0056] In some embodiments, the "mapping the instruction behavior of the target operator to the target transaction of the system-level design language in the simulator" in step 103 above can be achieved through the following steps 1031 to 1032: Step 1031: Based on the instruction behavior of the target operator, construct the simulator instructions for the target operator.

[0057] Simulator instructions are used to interact with hardware modules in the SystemC modeling environment. Unlike machine code in traditional instruction set architectures (ISAs), these instructions are implemented within the SystemC modeling framework through transactions, facilitating rapid modeling and verification of operator behavior under different hardware configurations. For example, in the Flash Attention operator, simulator instructions may include TME Load (data read), TCE MMA (matrix multiplication acceleration), and FOP ADD / MUL (addition, subtraction, multiplication, and division operations). Simulator instructions do not involve low-level encoding or decoding logic; instead, they directly encapsulate operation types and parameter information for rapid parsing and execution.

[0058] The process of constructing simulator instructions is based on abstracting and extracting the behavioral characteristics of the target operator. For example, for a GEMM operator, the main behaviors of the GEMM operator include loading the weight matrix, performing matrix multiplication, and writing back the results. Each stage of these behaviors generates corresponding simulator instructions. The process of constructing simulator instructions is usually completed automatically within the simulator framework. Developers only need to provide operator configuration files, such as parameter definitions in YAML format.

[0059] Step 1032: Using the interactive interface, send simulator instructions to the simulator's hardware circuitry to map the instruction behavior of the target operator to the target transaction of the simulator's system-level design language.

[0060] The interaction interface acts as a communication bridge between the simulator software layer and the hardware modeling module. It receives instructions from the upper-level simulator and transforms these instructions into events suitable for hardware circuit processing. In the SystemC environment, the functionality implemented by the interaction interface typically manifests as a transaction-level modeling mechanism, specifically by driving the execution flow of the hardware module through event triggering.

[0061] In some implementations, after receiving simulator instructions, the hardware circuitry performs corresponding operations based on the instruction content, such as loading data into a cache, starting a computing unit, or synchronizing threads. Ultimately, the operations of loading data into a cache, starting a computing unit, and synchronizing threads are recorded as a transaction sequence in the simulator's system-level design language (such as SystemCTLM transactions), forming a complete execution trajectory. For example, a TME Load instruction is sent to the memory engine module via the interactive interface, triggering a data loading operation. The mechanism by which simulator instructions are passed to the hardware circuitry through the interactive interface ensures the continuity and consistency of the instruction flow, contributing to the accurate simulation of hardware behavior.

[0062] The advantage of using an interactive interface for instruction transmission is that the interactive interface can shield the specific implementation details of the hardware, allowing the same set of simulator instructions to run on different hardware models, thereby improving the portability and scalability of the system.

[0063] In some implementations, simulator instructions enter the hardware module's transaction queue through the input port of the interactive interface, and are then executed sequentially according to priority or time order. Each simulator instruction corresponds to one transaction, and after the transaction is completed, status information can be fed back to the simulator, thus forming a closed-loop control to ensure the controllability and observability of the entire simulation process.

[0064] Different simulator instructions correspond to different hardware function modules. For example, the TME Load instruction is used to read data from external storage (such as HBM / DDR) into the on-chip cache, and is suitable for the initialization phase or cyclic loading of data. The TCE MMA instruction is used to perform large-scale matrix multiplication operations, and is suitable for the core calculation part of GEMM-type operators.

[0065] In this embodiment, by constructing simulator instructions and sending the instructions to the hardware circuit using an interactive interface, it is possible to map the instruction behavior of the target operator to the target transaction of the simulator's system-level design language, thereby improving the flexibility and accuracy of modeling and meeting the simulation requirements of new graphics processor architectures.

[0066] In some embodiments, step 104 above can be implemented by the following step 1041: Step 1041: Send the kernel instance of the target transaction to the simulator's compute distribution manager, and then transmit it to the simulator's memory partitioning module via the compute distribution manager. This causes the memory partitioning module to trigger the target transaction based on the kernel instance of the target transaction, simulating the operation of the target operator and multiple hardware modules of the simulator.

[0067] In some implementations, the kernel instance of the target transaction is sent to the simulator's computation distribution manager; the computation distribution manager stores the contents of the kernel instance of the target transaction and sends the kernel instance of the target transaction to the memory partitioning module according to the specific contents of the kernel instance of the target transaction; the memory partitioning module records the data required for the execution of the target transaction and sends the loaded data to the simulator's computation unit (such as the tensor computation engine) for computation.

[0068] In some implementations, the compute distribution manager can distribute kernel instances of a target transaction according to a preset distribution method. Specifically, the kernel instances of a target transaction can be distributed based on the type of the target transaction. Alternatively, the kernel instances of a target transaction can be distributed based on the current load and resource availability.

[0069] In this embodiment, by introducing a computation distribution manager and a memory partitioning module, efficient interaction between operators and hardware modules is achieved, improving the accuracy of transaction scheduling, optimizing data access paths and resource utilization, and enhancing modeling accuracy and simulation efficiency.

[0070] In some embodiments, the artificial intelligence operator simulation method provided in this application further includes the following steps 104 to 105: Step 104: During the simulation, record the clock cycles, delay, and bandwidth utilization of the target operator from the start to the end of execution; the recorded clock cycles are not used for the clock cycles of the hardware modules of the simulator to simulate the target precision.

[0071] During the execution of the target operator in the simulator, multiple performance metrics are continuously collected and recorded, including but not limited to: clock cycle, latency, and bandwidth utilization. This data is used to evaluate the resource consumption of the target operator across the entire GPU architecture. The clock cycle refers to the total time required for the target operator to complete from startup, and can be determined by the number of operations within the target operator, hardware execution efficiency, and other factors.

[0072] Simulator hardware modules (such as Tensor Core, Memory Engine, etc.) usually have a fixed clock frequency. Therefore, the recorded clock cycle may differ from the clock cycle of the hardware module itself. This difference reflects the inconsistency between the execution of the target operator and the actual operation of the hardware.

[0073] Latency refers to the time required for a specific operation or instruction to be completed from the time it is issued. For example, the latency of a memory read operation is the time interval between issuing a read request and receiving the data. Latency statistics help identify performance degradation issues caused by waiting for certain resources.

[0074] Bandwidth utilization indicates whether the data access capabilities of memory or other storage devices are fully utilized within a certain period of time. High bandwidth utilization means frequent and efficient data transmission, while low bandwidth utilization may indicate data access bottlenecks or redundant operations.

[0075] The recorded clock cycles are not used to simulate the clock cycles of the hardware modules of the simulator with target precision. This means that this application does not aim for a completely accurate hardware module clock cycle, but rather for an estimated total number of cycles, latency, and bandwidth utilization. In this way, by comparing the estimated performance with the actual performance of the hardware modules, the bottleneck location of each hardware module can be effectively inferred.

[0076] Step 105: Based on the clock cycle, delay, and bandwidth utilization of the target operator and the scheduling timing of the target operator for the hardware module, determine the bottleneck location of the hardware module.

[0077] The scheduling order of a target operator on hardware modules refers to the order in which the target operator calls and the dependencies between various hardware modules (such as TME, Tensor Core, L2 Cache, etc.) during execution. The scheduling sequence includes the order of calls and dependencies.

[0078] In some implementations, the call order and dependencies of the target operator on each hardware module can be comprehensively analyzed based on the recorded clock cycle, recorded latency, and recorded bandwidth utilization to pinpoint the key modules that cause performance degradation.

[0079] For example, if a target operator frequently calls a certain cache module during execution, but the bandwidth utilization of this frequently called cache module remains low for an extended period, it indicates that this frequently called cache module may be a bottleneck for the target operator. Similarly, if an operation with significantly higher-than-average latency occurs, it may indicate that the relevant hardware module is experiencing resource contention or waiting issues when processing that operation.

[0080] In some implementations, hardware modules that frequently exhibit inefficient behavior during execution can be identified by cross-referencing the target operator's execution logs with the hardware module's performance logs. For example, in the Flash Attention operator, if the QK operation has high latency and is accompanied by low Tensor Core utilization, it indicates that the QK operation may be limited by data loading speed rather than computational power.

[0081] In this embodiment, by recording clock cycles, latency, bandwidth utilization, and scheduling timing information, the bottleneck location of the hardware module can be accurately identified. This allows for targeted adjustments to the hardware configuration or optimization of the target operator execution strategy, effectively alleviating performance bottlenecks and improving overall system operating efficiency and resource utilization.

[0082] In some embodiments, the artificial intelligence operator simulation method provided in this application further includes the following steps 106 to 110: Step 106: Obtain the log files generated by the multiple hardware modules of the simulator when the target operator is executed.

[0083] During the operation of the artificial intelligence operator simulation method, the target operator interacts with multiple hardware modules, generating a large amount of runtime performance data. This runtime performance data can be recorded in the form of log files. Log files may include, but are not limited to, information such as timestamps, event types (e.g., Load, Store, Compute), resource usage (e.g., register usage, cache hit rate), and execution latency.

[0084] By collecting log files from multiple hardware modules, a comprehensive understanding of the target operator's behavior on different hardware units can be achieved, providing a detailed data foundation for subsequent analysis. For example, during the execution of the GEMM or Flash Attention operator, the TME module is responsible for data loading, the Tensor Compute Engine (TCE) is responsible for matrix multiplication, and the Async module is used for synchronization operations. The logs of each module can reflect the load and bottlenecks of each module under specific operators.

[0085] Step 107: Use the roofline model to process the total computation and total access volume in the log file to obtain the computation intensity.

[0086] The roofline model is a theoretical model used to evaluate the upper limit of the performance of a computing system. It divides system performance into two dimensions: arithmetic strength and memory bandwidth. The core idea of ​​the roofline model is that when the arithmetic strength is above a certain threshold (i.e., ridge point computational strength), the system is computationally constrained; conversely, when the arithmetic strength is below a certain threshold, the system is memory constrained.

[0087] Arithmetic intensity is an important indicator for measuring the computational intensity of an operator. Arithmetic intensity can be defined as the number of floating-point operations performed per unit of memory access.

[0088] In some implementations, the number of computation instructions and memory accesses recorded in the log file can be statistically analyzed to obtain the total computational load and total memory access count. A division operation is then performed on the total computational load and total memory access count to obtain the arithmetic strength of the target operator. For example, if an operator executes 10... 9 FLOPs (floating-point operands that access 0 simultaneously) 8 If the total number of bytes read from and written to memory is Bytes, then the arithmetic strength of this operator is 10 FLOPs / Byte.

[0089] Step 108: Perform calculations on the peak computing power and memory bandwidth in the log file to obtain the ridge point computing intensity.

[0090] Peak hardware computing power refers to the maximum theoretical computing power that the hardware in the simulator can provide, and it can be measured in TFLOPS (trillion floating-point operations per second). Memory bandwidth refers to the amount of data that a storage unit (such as DDR) can transfer per unit of time, usually measured in GB / s.

[0091] Ridge computational intensity represents the balance point between computational power and memory bandwidth in a system. When the arithmetic intensity of the target operator is greater than the ridge computational intensity, performance is limited by computational resources; when the arithmetic intensity of the target operator is less than or equal to the ridge computational intensity, performance is limited by memory bandwidth. For example, if the ridge computational intensity is 20 FLOPs / Byte, then the arithmetic intensity of the target operator needs to exceed the ridge computational intensity by 20 FLOPs / Byte to fully utilize computational resources.

[0092] In some implementations, configuration parameters for the hardware modules can be extracted from the log file; the peak computing power and memory bandwidth of the hardware can be determined based on the configuration parameters; and the peak computing power and memory bandwidth can be divided to obtain the ridge point computing intensity. In this case, the ridge point computing intensity is the ratio of the peak computing power to the memory bandwidth.

[0093] For example, if a GPU has a peak hardware computing power of 10 TFLOPS and a memory bandwidth of 500 GB / s, then the ridge point computation intensity is (10 × 10^12) / (500 × 10^9) = 20 FLOPs / Byte. This means that when the arithmetic intensity of the target operator exceeds 20 FLOPs / Byte, the system executing the target operator will be primarily limited by computing resources; otherwise, it will be primarily limited by memory bandwidth.

[0094] Step 109: When the computational intensity is greater than that of the ridge point, it is determined that there is a computational bottleneck in the simulator.

[0095] If the computational intensity is greater than the ridge point computational intensity, it means that the arithmetic intensity of the target operator exceeds the ridge point computational intensity, indicating a computational bottleneck.

[0096] In some implementations, if a certain type of operator frequently triggers computational bottlenecks, the number of computing cores can be increased or the parallelism can be improved to enhance overall computing power utilization. Simultaneously, by optimizing operator scheduling strategies, idle computation periods can be reduced, thereby further unlocking the hardware's potential.

[0097] Step 110: If the computational intensity is less than or equal to the ridge computational intensity, determine that the simulator has a memory bottleneck.

[0098] If the computational intensity is less than or equal to the ridge computational intensity, it means that the computational intensity of the target operator does not exceed the ridge computational intensity, and a memory bottleneck exists at this time.

[0099] In some implementations, if a certain type of operator frequently triggers a memory bottleneck, the bottleneck can be mitigated by adjusting the data loading strategy, increasing cache capacity, or optimizing memory access patterns. Furthermore, hardware configuration information, such as L2 / L3 cache bandwidth and DDR memory frequency, can be used to further analyze the specific source of the memory bottleneck.

[0100] In some implementations, the computational intensity of each hardware module can be calculated. If there is a target hardware module with a computational intensity greater than the ridge point computational intensity, it is determined that the target hardware module has a computational bottleneck. If there is a target hardware module with a computational intensity less than or equal to the ridge point computational intensity, it is determined that the target hardware module has a memory bottleneck.

[0101] In this embodiment of the application, by obtaining the log files generated by multiple hardware modules in the artificial intelligence operator simulation method when the target operator is executed, and by using the roofline model to analyze the computational intensity and ridge computational intensity, it is possible to accurately determine whether the simulator has a computational bottleneck or memory bottleneck. In this way, performance problems can be effectively located, and targeted optimization measures can be taken accordingly to further improve the execution efficiency of artificial intelligence operators on GPUs.

[0102] In some embodiments, the target operator includes a Flash Attention operator. The Flash Attention operator is used to optimize the attention mechanism of the Transformer model. By reducing unnecessary memory accesses and utilizing computational reuse features, it significantly reduces computational latency and memory consumption, making it particularly suitable for large-scale natural language processing tasks. The following describes the artificial intelligence operator simulation method provided in this application embodiment using the Flash Attention operator. The simulation method of the Flash Attention operator includes the following steps A1 to A2: Step A1: Identify the configuration file required for the execution of the Flash Attention operator, map the instruction behavior of the Flash Attention operator to the operator transaction of the simulator's system-level design language, and generate a kernel instance of the operator transaction.

[0103] Operator transactions refer to transactions used to simulate the execution process of the Flash Attention operator.

[0104] In some implementations, the instruction behavior of the Flash Attention operator can be mapped to operator transactions in the simulator's system-level design language by utilizing the interaction interface between the operator and the hardware module.

[0105] In some implementations, the instruction behavior of the Flash Attention operator can be mapped to the operator transaction of the simulator's system-level design language based on the mapping rules between instruction behavior and transactions in the simulator's system-level design language.

[0106] In some implementations, simulator instructions for the Flash Attention operator can be constructed based on the instruction behavior of the Flash Attention operator. These simulator instructions can then be used to map the instruction behavior of the Flash Attention operator to operator transactions in the simulator's system-level design language.

[0107] In some implementations, simulator instructions for the FlashAttention operator can be constructed based on the instruction behavior of the FlashAttention operator. The simulator instructions of the FlashAttention operator can then be used to map the instruction behavior of the FlashAttention operator to operator transactions in the system-level design language of the simulator.

[0108] In some implementations, after obtaining the operator transaction, the simulator can automatically generate a kernel instance of the operator transaction. For example, after obtaining the operator transaction, a kernel template matching the operator type of the Flash Attention operator can be determined from multiple kernel templates based on the operator type of the Flash Attention operator; and a kernel instance of the target operator transaction can be generated based on the matching kernel template.

[0109] Step A2: Trigger operator transactions based on kernel instances of operator transactions to simulate the operation of the Flash Attention operator and multiple hardware modules of the simulator.

[0110] In some implementations, the processing logic of operator transactions can be executed based on the kernel instance of the operator transaction, thereby simulating the operation of the Flash Attention operator and multiple hardware modules of the simulator.

[0111] In some implementations, after a kernel instance based on an operator transaction triggers an operator transaction, it sends control signals to multiple hardware modules of the simulator and receives return results from multiple hardware modules.

[0112] For example, when a TME Load transaction is triggered, the simulator activates the TME module and simulates the process of the TME module reading data from off-chip memory to on-chip cache according to the transaction parameters; while when a TCE Matmul transaction is triggered, the simulator activates the computation engine and simulates the execution flow of matrix multiplication. The above transaction-driven simulation method enables the simulator to accurately reproduce the operation of operators under a specific GPU architecture without the need for real hardware support.

[0113] In this embodiment, by introducing the Flash Attention operator and its transactional modeling method, the simulator's support for emerging Artificial Intelligence (AI) operators can be effectively improved. This allows for more accurate simulation of the execution behavior of novel operators in different hardware environments, enabling rapid identification of performance bottlenecks and guiding architectural improvements, thereby driving iterative optimization of AI chip design.

[0114] In some embodiments, the operation of the multiple hardware modules of the above-described simulation Flash Attention operator and simulator can be implemented through the following steps B1 to B3: Step B1: Utilize multiple hardware modules of the simulator to execute multiple thread blocks of the Flash Attention operator.

[0115] A thread block is the basic unit of parallel computing in a GPU. Each thread block consists of multiple threads that can share resources (such as shared memory) and operate synchronously. In this embodiment, the FlashAttention operator can be divided into multiple thread blocks, each independently mapped to a different hardware module for execution, thereby achieving highly parallel performance modeling. By decomposing the FlashAttention operator into multiple thread blocks, parallel execution improves overall simulation efficiency and more closely resembles the actual behavior of a real GPU.

[0116] Step B2: After multiple thread blocks have completed execution, terminate the simulation operation of the Flash Attention operator and calculate the clock cycles, latency, and bandwidth utilization of the Flash Attention operator from the start to the end of execution.

[0117] In some implementations, once all thread blocks have finished executing, the simulator can automatically terminate the entire FlashAttention operator simulation process and summarize key performance indicators. Key performance indicators include: clock cycles, latency, and bandwidth utilization.

[0118] Step B3: Based on the clock cycle, latency, and bandwidth utilization of the Flash Attention operator and the scheduling timing of the Flash Attention operator for the hardware module, determine the bottleneck location of the hardware module.

[0119] The hardware modules include: a tensor storage engine, a tensor computation engine, and global memory. The execution operations of each thread block include: In the processing phase, the kernel of the Flash Attention operator sends a data loading instruction for the i-th loop to the tensor storage engine; the tensor storage engine loads the data required for the i-th loop; the kernel of the Flash Attention operator sends a computation instruction for the i-th loop to the tensor computation engine; the tensor computation engine responds to the computation instruction for the i-th loop and performs a first computation operation based on the data required for the i-th loop; i is a positive integer. In the final phase, after the asynchronous barrier of the normalization operation in the i-th loop is reached, the tensor computation engine performs a second computation operation and stores the final computation result in global memory.

[0120] The tensor storage engine handles data read and write operations, supporting loading data from global memory into on-chip cache and writing the results back to global memory after computation. The tensor computation engine primarily performs computationally intensive operations such as matrix multiplication and addition. Global memory is the storage unit for storing large amounts of data.

[0121] In some implementations, if the latency of a certain stage is significantly higher than that of other stages, or if the bandwidth utilization of a certain hardware module is saturated for a long period of time, then the hardware module can be identified as the bottleneck of the current system.

[0122] In some implementations, during each loop of the processing phase, the kernel of the Flash Attention operator first sends a data loading instruction to the tensor storage engine, which is responsible for loading the input data required for this loop. Subsequently, the kernel of the Flash Attention operator sends a computation instruction to the tensor computation engine, which performs the first round of computation operations based on the loaded data.

[0123] For example, the processing phase may include a preparation phase and a main loop phase. In the preparation phase, the kernel issues a TME data loading instruction, the TME loads Q0, K0, V0, and K1, and then the kernel issues a TCE instruction. After Q0 and K0 arrive, Q*K is calculated. In the main loop phase, the kernel issues a TME data loading instruction, the TME loads Ki+1 and Vi, and simultaneously executes Q*Ki+1 and a normalization operation (Softmax). After QK is calculated, P*Vi is calculated, along with Row Sum, Row Max, and other operations. The main loop phase repeats several times, depending on the configuration file.

[0124] For example, in the epilogue phase, after waiting for the asynchronous barrier of the normalization operation to arrive, the final P*V is executed, and the calculation result is stored in global memory. After storage, the next thread block is executed, and so on, until all thread blocks are executed. Then the simulation is terminated, and the number of running cycles is counted, and data such as computing power utilization and memory bandwidth utilization are calculated.

[0125] In this embodiment, the simulation process of the Flash Attention operator is described in detail, which can more comprehensively reflect its behavioral characteristics on the hardware module. This demonstrates that the artificial intelligence operator simulation method provided in this embodiment is applicable to novel operators such as Flash Attention, overcoming the problem of insufficient support for novel operators in traditional simulators.

[0126] It should be noted that the artificial intelligence operator simulation method provided in this application aims to solve the problem that existing GPU performance modeling tools have poor adaptability and difficulty in supporting emerging operators. This application introduces a unified operator description interface to set the configuration file required when the operator is executed. Based on the configuration file, the instruction behavior of the operator is mapped to the target transaction that the simulator can recognize, thereby realizing efficient performance modeling and analysis of various types of artificial intelligence operators under different graphics processor architectures.

[0127] This application provides a simulator, which includes: an operator description interface and an execution module; the operator description interface is used to set the configuration file required when the target operator is executed; the configuration file is used to describe the various parameters involved in the execution of the target operator; the target operator includes multiple different types of artificial intelligence operators; the execution module is used to extract the instruction behavior of the target operator based on the configuration file; map the instruction behavior of the target operator to the target transaction of the simulator's system-level design language, and generate a kernel instance of the target transaction; trigger the target transaction based on the kernel instance of the target transaction, and simulate the operation of the target operator and multiple hardware modules of the simulator.

[0128] In some implementations, the simulator is built on the system-level design language SystemC; the formula operations in the target operator are implemented using Python or Excel.

[0129] The execution module converts the parameter information in the configuration file into a specific sequence of hardware operation instructions (a sequence composed of multiple simulator instructions), such as loading data, performing calculations, and storing results. The execution module supports modeling multiple different types of artificial intelligence operators (such as GEMM and Flash Attention), and can dynamically adjust the execution flow of the target operator according to the configuration file.

[0130] The introduction of the execution module enhances the simulator's scalability and flexibility, enabling it to adapt to the evolving needs of new artificial intelligence operators. By separating the target operator description from the execution logic, new target operator types can be supported without modifying the underlying hardware model.

[0131] In some embodiments, the simulator further includes a computation distribution manager and a memory partitioning module; the computation distribution manager is used to transfer the kernel instance of the target transaction to the memory partitioning module according to a preset distribution strategy; the memory partitioning module is used to trigger the target transaction based on the kernel instance of the target transaction, simulating the operation of the target operator and multiple hardware modules of the simulator.

[0132] The Compute Distribution Manager (CDM) is a crucial component in the simulator for scheduling and distributing transactions. It receives transaction requests from operators or other modules and allocates them to the corresponding hardware modules according to a specific scheduling strategy. Based on the hardware resource information and distribution strategy in the configuration file, the CDM rationally distributes transaction instances across different hardware modules, ensuring maximum resource utilization during operator execution. For example, the CDM might send transactions that load data to the memory partitioning module and computation tasks to the Tensor Computation Engine (TCE).

[0133] The function of the memory partitioning module is to receive kernel instances from the compute distribution manager and trigger corresponding hardware operations based on these instances, thereby simulating the behavior of the target operator on a real GPU. For example, the memory partitioning module loads data from global memory into the on-chip cache, or writes computation results back to global memory.

[0134] The memory partitioning module can contain multiple physical or logical partitions, each capable of independently executing specific transaction operations. This partitioning design helps isolate the impact of different transactions, improving overall stability and scalability. For example, in the Flash Attention operator, the memory partitioning module can handle the data loading and computation tasks of the Q, K, and V matrices separately.

[0135] The memory partitioning module is also responsible for coordinating the interaction between multiple hardware modules. For example, in the TME (Tensor Memory Engine) module, the memory partitioning module triggers data loading instructions; in the TCE (Tensor Compute Engine) module, the memory partitioning module triggers matrix multiplication operation instructions. By coordinating the interaction between multiple hardware modules through the memory partitioning module and triggering relevant instructions in the TME and TCE modules respectively, the simulator can accurately reproduce the operation flow between operators and hardware.

[0136] Distribution strategies can be defined based on hardware architecture characteristics, load balancing requirements, or performance optimization goals, including but not limited to: Round Robin, priority sorting, and dynamic load awareness. By flexibly configuring distribution strategies, optimal task partitioning and execution efficiency can be achieved under different hardware topologies.

[0137] In some embodiments, the simulator further includes a single instruction multiple data execution unit (SMI), a single instruction multiple thread scheduler, a tensor memory engine, a multi-level cache, and an on-chip network. The SMI is used to simulate a target operator controlling multiple data units to perform parallel operations simultaneously through a single instruction. The single instruction multiple thread scheduler is used to simulate a target operator performing operations on multiple sets of data simultaneously through a single instruction. The tensor memory engine is used to execute tensor operations of the target operator. The multi-level cache is used to cache data of the target operator during execution. The on-chip network is used to transmit data of the target operator during execution in the form of data packets between multiple hardware modules of the simulator.

[0138] Single Instruction Multiple Data (SIMD) allows a graphics processing unit (GPU) to operate on multiple data elements simultaneously using a single instruction. SIMD execution units are suitable for vectorized computation tasks, such as matrix multiplication or pixel operations in image processing. In this application, the SIMD execution unit is designed to efficiently simulate the parallel computation behavior of artificial intelligence operators, particularly excelling in high-performance computing scenarios such as GEMM and Flash Attention.

[0139] Single Instruction Multiple Data (SIMD) is used to enable the same instruction to be executed concurrently by multiple threads, thereby achieving highly parallel computing capabilities. In this application, the role of the SIMD scheduler is to decompose the operation of the operator into multiple threads and allocate these threads to different hardware execution units to maximize parallelism and throughput.

[0140] The Tensor Memory Engine (TME) is a high-speed cache and data loading / storage unit specifically designed for tensor data structures. The TME can efficiently read and store tensor data and supports complex tensor transformation operations, such as matrix transpose, slicing, and concatenation. In this application, the TME is integrated into the simulator to support tensor operations in AI models, including weight updates and activation function calculations in convolutional neural networks.

[0141] A multi-level caching system may include L1, L2, and L3 caches, which reside at different levels of the graphics processor to accelerate data access. In this application, the multi-level cache is used to temporarily store frequently accessed data during operator execution, thereby reducing reliance on external storage devices and lowering latency.

[0142] Network-on-a-chip (NoC) is a communication architecture used to connect multiple computing and storage units within a single chip. NoC uses routing algorithms to send data packets from source nodes to destination nodes and supports various topologies such as mesh, tree, and ring. In this application, NoC is used to achieve efficient data exchange between different hardware modules, ensuring that each module can work collaboratively and maintain good data flow consistency. By introducing NoC, the communication efficiency between hardware modules can be significantly improved, data transmission bottlenecks can be reduced, thereby enhancing the parallel processing capability of the entire system.

[0143] The following describes the application of the artificial intelligence operator simulation method provided in the embodiments of this application in real-world scenarios.

[0144] This application aims to propose a flexible and scalable AI operator performance modeling method for analyzing the behavior of operators such as GEMM and FlashAttention on novel graphics processor architectures, in order to predict performance bottlenecks and guide future architecture optimization. The specific technical solution is as follows: 1. Construction of GPU abstract architecture model; In SystemC, a modular GPU system model is built that includes a single instruction multiple data execution unit, a single instruction multiple thread scheduler, a tensor memory engine, L1 / L2 / L3 caches, double data rate (DDR), and network-on-chip (NoC) interconnects.

[0145] The model is built in layers according to functional modules, supporting flexible replacement and parameterized configuration, making it easy to adapt to the evolution of different GPU architectures.

[0146] like Figure 2 As shown, the tensor memory engine is used to perform at least the following steps: 1. Calculate the address based on configuration information such as tensor descriptors, data location, and block dimensions; 2. Send a request to the memory subsystem; 3. Wait for the memory subsystem's response and return.

[0147] 2. AI operator behavior modeling (data flow + control flow); For operators such as GEMM and Flash Attention, their data loading, computation, and memory access instructions are extracted and mapped to transactions in SystemC. Extraction method: Operator execution code is set up in the simulator. This code defines the operator execution pipeline. During simulator runtime, the pipeline translates the operator execution into corresponding hardware instructions and passes them to the hardware unit. Mapping method: An interface is defined between operators and hardware units, similar to the ISA instruction set. It contains information similar to the ISA instruction set, but unlike ISA, it does not require encoding / decoding and can be used within the simulator framework to pass information from operators to hardware modules. The operator constructs a simulator instruction, sends it to the hardware unit, and the hardware unit executes the corresponding instruction.

[0148] The modeling focuses on the operator's scheduling logic, instruction triggering mechanism, data path length, and memory access reuse strategy, simulating the operator's interaction with the hardware module throughout the execution process. The simulated interaction methods are: the operator sends a data read instruction, corresponding to the TME performing a loading operation; the operator sends a matrix multiplication instruction, corresponding to the TCE performing matrix multiplication acceleration operations.

[0149] 3. Design of theoretical performance index evaluation methods; Instead of focusing on cycle accuracy, the core metrics are estimated total cycles, latency, and bandwidth utilization, combined with scheduling timing to infer the bottleneck locations of each module. Estimation method: During operator execution, data reading takes clock cycles, as do computation and data writing. The current cycle after operator execution is the total cycle count. Not focusing on cycle accuracy means this result is not cycle-accurate; it's used for comparison with the actual hardware cycle count. Latency and bandwidth utilization are statistically analyzed during simulator operation. For example, the current cycle is recorded before data reading and after reading, and subtracted from the initial cycle to obtain a latency sample. Statistical analysis is then performed based on all sample points. Scheduling timing refers to the order in which multiple modules are scheduled during operator execution. Bottleneck example: If data reading hasn't finished before matrix multiplication, preventing immediate multiplication, this indicates a memory bottleneck.

[0150] By analyzing the model's runtime logs, theoretical performance data such as operator memory access pressure, computation latency, and scheduling congestion are obtained. Each simulator hardware unit has many performance metrics, which are saved in log files after the simulation ends. Developers then analyze these log files. The analysis is based on the roofline model. After the arithmetic intensity reaches a threshold, the GPU's computing power utilization (TFLOPS) will reach a peak (e.g., 90%-99%). If it reaches such a range, it indicates a computational bottleneck; conversely, it indicates a memory bottleneck. Memory bottlenecks require examining the performance metrics of each level of memory unit. For example, exceptionally high L2 bandwidth likely indicates an L2 bottleneck, although specific examples require specific analysis.

[0151] 4. Parametric modeling and configuration mechanism; Provides a unified JSON / YAML interface to describe operator parameters (such as matrix size, tile size, and thread block).

[0152] Block partitioning and memory access strategies are automatically generated by the simulation environment based on the configuration, and the operator execution flow is automatically generated.

[0153] The model's configuration file can be modified to support quick switching between different operator scenarios for performance comparison and sensitivity analysis. Experimental data can be obtained from each performance test, allowing for analysis of the performance and sensitivity to different parameters by comparing the data.

[0154] The simulator's flowchart is as follows: like Figure 3As shown, taking Flash Attention as an example, the application of each module in the operator simulation process is illustrated: After obtaining the A computation operator, a configuration file is set; a kernel is generated according to the configuration file and sent to the computation distribution manager; the computation distribution manager sends the kernel to each module in the memory partitioning module. The memory partitioning module includes TME, Async, TCE, FOP, and SFU; among them, TCE is used for large-scale matrix multiplication; FOP is used for small-scale addition and multiplication; SFU is used for special operations, such as exponentiation; TME is used for data reading; Async is used for synchronous operations; for example, Flash Attention uses TCE for Q*K, FOP for accumulation, and SFU for Softmax. The simulator specifically executes the following steps: 1. The emulator generates a kernel instance by recognizing the configuration file and sends it to the Compute Distribution Manager (CDM). The CDM sends thread blocks to the Memory Partition Module (MP) based on different distribution methods, and the MP executes predefined kernel instructions. The YAML interface takes a configuration file as input, containing two YAML files: one for operators, defining types, parameters, configuration information to support flexible operation of various operators; and another for hardware, containing information such as the type, computing power, bandwidth, and capacity of each unit to support different hardware architectures.

[0155] 2. For example Figure 4 As shown, in the preparation phase (Prologue), the kernel issues a TME data loading instruction, TME loads Q0, K0, V0, K1, and then the kernel issues a TCE instruction. After Q0 and K0 arrive, Q*K is calculated. Here, row_max i-1 means: calculate the maximum row value of the data in the i-1th cycle; TCE[0]QK_0 1024 cycle means: tensor calculation unit 0 performs the Q*K matrix multiplication in the 0th cycle, which takes 1024 clock cycles.

[0156] During the preparation phase: The kernel has not yet entered the main computation stage. Its main task is to prepare the resources and data required for computation, including: preloading data from off-chip memory (HBM / DDR) to on-chip cache (SRAM / Shared Memory); initializing registers and warp states; and configuring hardware units (such as Tensor Cores and TMEs). If the data is prepared quickly, the computing units can immediately enter a full-load state, avoiding "idling".

[0157] 3. Main loop stage: The kernel issues a TME data loading instruction, TME loads Ki+1 and Vi, and executes Q*Ki+1 and normalization operation (Softmax). After QK calculation is completed, P*Vi is calculated, and RowSum, Row Max and other operations are performed. The Main loop stage will be repeated several times, depending on the configuration file. Among them, TCE256x128x128[0] 1024 Cycle Q K_i means: Tensor calculation engine 0 performs a multiplication operation on the 256×128 Q matrix and the 128×128 K matrix of the i-th cycle, which takes 1024 clock cycles. TCE 256x128x128 1024 Cycle[0]PV_i-1 means: Tensor calculation engine 0 performs a multiplication operation on the attention weight matrix P and the matrix of the (i-1)-th cycle with a dimension of 256×128×128, which takes 1024 clock cycles. The meaning of `EXP2 C_Softmax i-1 1024 cycle` is: the fused Softmax operation in the (i-1)th cycle, completed by the EXP2 exponent unit, taking 1024 clock cycles. The meaning of `row_Sum 256Cycle` is: a lightweight aggregation operation that sums the matrix row by row, taking 256 clock cycles. The meaning of `softmax init i256 Cycle` is: the lightweight initialization operation of Softmax in the i-th cycle, taking 256 clock cycles.

[0158] The main loop phase proceeds as follows: The TME data loading instruction loads the Key and Value values ​​needed for the next loop from the on-chip memory (SRAM). Simultaneously, the TCE instruction executes the Q*K, K*V, and Softmax operations for the current loop. After the calculations are complete, the Key and Value values ​​needed for the next loop are also read, allowing the next loop to begin directly. This method avoids TCE idle cycles.

[0159] 4. Epilogue: After waiting for the Softmax Async Barrier to arrive, the final P*V is executed, and the calculation result O is stored in global memory. After storage, the next thread block is executed until all thread blocks are executed, the simulation is terminated, the number of cycles is counted, and the computing power utilization, memory bandwidth utilization, and other data are calculated. Among them, TCE 256x128x128 [0]PV means: the tensor calculation engine 0 performs block matrix multiplication on the 256×128 attention weight matrix P and the 128×128 matrix V. EXP2 C_Softmax is a fusion acceleration operator designed for Softmax. row_Sum is the row dimension aggregation operation in the AI ​​operator. row_max i 512 Cycle means: the aggregation operation of extracting the maximum value row by row of the matrix in the i-th cycle, which takes 512 clock cycles. The TME Store O 2048 Cycle means that the memory access operation of the Tensor Store Engine to write the final output tensor of the operator into memory takes 2048 clock cycles.

[0160] It should be noted that, in the embodiments of this application, if the above methods are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, or the parts that contribute to related technologies, can be embodied in the form of software products. These software products are stored in a storage medium and include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0161] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0162] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0163] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0164] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0165] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0166] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0167] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0168] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0169] The above are merely embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for simulating artificial intelligence operators, characterized in that, The artificial intelligence operator simulation method, applied to a simulator, includes: Using the simulator's operator description interface, a configuration file is set up for the target operator to execute; the configuration file describes the various parameters involved in the execution of the target operator; the target operator includes multiple different types of artificial intelligence operators. Based on the configuration file, extract the instruction behavior of the target operator on the graphics processor; The instruction behavior of the target operator is mapped to the target transaction of the simulator's system-level design language, and a kernel instance of the target transaction is generated; The target transaction is triggered based on the kernel instance of the target transaction, simulating the operation of the target operator and multiple hardware modules of the simulator.

2. The artificial intelligence operator simulation method according to claim 1, characterized in that, The step of using the simulator's operator description interface to set the configuration file required for the execution of the target operator includes: Using the first operator description interface, the configuration parameters of the target operator are set; the configuration parameters of the target operator are used to define the execution logic of the target operator. The configuration parameters of the hardware module are set using the second operator description interface; the configuration parameters of the hardware module are used to define the resource allocation logic of the hardware module; the operator description interface includes the first operator description interface and the second operator description interface, and the configuration file includes the configuration parameters of the target operator and the configuration parameters of the hardware module.

3. The artificial intelligence operator simulation method according to claim 1, characterized in that, The step of extracting the instruction behavior of the target operator on the graphics processor based on the configuration file includes: Determine the operator execution code pre-stored in the simulator; Based on the operator execution pipeline defined in the operator execution code, the configuration file is identified to extract the instruction behavior corresponding to the target operator.

4. The artificial intelligence operator simulation method according to claim 1, characterized in that, The process of mapping the instruction behavior of the target operator to the target transaction of the simulator's system-level design language includes: Based on the instruction behavior of the target operator, construct the simulator instructions for the target operator; Using an interactive interface, the simulator instructions are sent to the simulator's hardware circuitry to map the instruction behavior of the target operator to the target transaction of the simulator's system-level design language.

5. The artificial intelligence operator simulation method according to claim 1, characterized in that, The kernel instance based on the target transaction triggers the target transaction, simulating the operation of the target operator and multiple hardware modules of the simulator, including: The kernel instance of the target transaction is sent to the compute distribution manager of the simulator, and then transmitted to the memory partitioning module of the simulator via the compute distribution manager. The memory partitioning module triggers the target transaction based on the kernel instance of the target transaction, simulating the operation of the target operator and multiple hardware modules of the simulator.

6. The artificial intelligence operator simulation method according to any one of claims 1 to 5, characterized in that, The artificial intelligence operator simulation method also includes: During the simulation, the clock cycles, latency, and bandwidth utilization of the target operator from the start to the end of execution are recorded; the recorded clock cycles are not used to simulate the clock cycles of the simulator's hardware modules with the target precision. Based on the clock cycle, latency, and bandwidth utilization corresponding to the target operator and the scheduling timing of the target operator on the hardware module, the bottleneck location of the hardware module is determined.

7. The artificial intelligence operator simulation method according to any one of claims 1 to 5, characterized in that, The target operator includes the Flash Attention operator; The artificial intelligence operator simulation method also includes: Identify the configuration file required when the Flash Attention operator is executed, map the instruction behavior of the Flash Attention operator to the operator transaction of the system-level design language of the simulator, and generate a kernel instance of the operator transaction; The operator transaction is triggered based on the kernel instance of the operator transaction to simulate the operation of the Flash Attention operator and multiple hardware modules of the simulator.

8. The artificial intelligence operator simulation method according to claim 7, characterized in that, The simulation of the FlashAttention operator and the operation of multiple hardware modules of the simulator includes: The multiple hardware modules of the simulator are used to execute multiple thread blocks of the Flash Attention operator respectively; When all the thread blocks have been executed, the simulation operation of the Flash Attention operator is terminated, and the clock cycles, latency, and bandwidth utilization of the Flash Attention operator from the start to the end of execution are calculated. Based on the clock cycle, latency, and bandwidth utilization corresponding to the Flash Attention operator and the scheduling timing of the Flash Attention operator for the hardware module, the bottleneck location of the hardware module is determined. The hardware module includes: a tensor storage engine, a tensor computation engine, and global memory; the execution operations of each thread block include: During the processing phase, the kernel of the Flash Attention operator sends a data loading instruction for the i-th loop to the tensor storage engine; the tensor storage engine loads the data required for the i-th loop; the kernel of the Flash Attention operator sends a calculation instruction for the i-th loop to the tensor computation engine; the tensor computation engine responds to the calculation instruction for the i-th loop and performs a first calculation operation based on the data required for the i-th loop; i is a positive integer; In the final stage, after the asynchronous barrier of the normalization operation in the i-th loop is reached, the tensor computation engine executes the second computation operation and stores the final computation result in global memory.

9. The artificial intelligence operator simulation method according to any one of claims 1 to 5, characterized in that, The artificial intelligence operator simulation method also includes: Obtain the log files generated by multiple hardware modules of the simulator when the target operator is executed; The computational intensity is obtained by calculating the total computational load and total access volume in the log files using the roofline model. The peak computing power and memory bandwidth of the hardware in the log file are processed to obtain the ridge computing intensity; If the computational intensity is greater than the ridge computational intensity, it is determined that the simulator has a computational bottleneck. If the computational intensity is less than or equal to the ridge computational intensity, it is determined that the simulator has a memory bottleneck.

10. A simulator, characterized in that, The simulator includes: an operator description interface and an execution module; The operator description interface is used to set the configuration file required when the target operator is executed; the configuration file is used to describe the various parameters involved when the target operator is executed; the target operator includes multiple different types of artificial intelligence operators; The execution module is configured to extract the instruction behavior of the target operator on the graphics processor based on the configuration file; map the instruction behavior of the target operator to the target transaction of the simulator's system-level design language, and generate a kernel instance of the target transaction; trigger the target transaction based on the kernel instance of the target transaction to simulate the operation of the target operator and multiple hardware modules of the simulator.

11. The simulator according to claim 10, characterized in that, The simulator also includes a computing distribution manager and a memory partitioning module; The computation distribution manager is used to transmit the kernel instance of the target transaction to the memory partition module according to a preset distribution strategy. The memory partitioning module is used to trigger the target transaction based on the kernel instance of the target transaction, simulating the operation of the target operator and multiple hardware modules of the simulator.

12. The simulator according to claim 10, characterized in that, The simulator also includes a single instruction multiple data execution unit, a single instruction multiple thread scheduler, a tensor memory engine, a multi-level cache, and an on-chip network; The single instruction multiple data execution unit is used to simulate the target operator controlling multiple data units to perform parallel operations simultaneously through one instruction. The single-instruction multi-threaded scheduler is used to simulate the target operator performing operations on multiple sets of data simultaneously through a single instruction. The tensor memory engine is used to perform tensor operations on the target operator; The multi-level cache is used to cache the data of the target operator during execution; The on-chip network is used to transmit data during the execution of the target operator in the form of data packets between multiple hardware modules of the simulator.

13. The simulator according to any one of claims 10 to 12, wherein the simulator is built based on the system-level design language SystemC; and the formula operations in the target operator are implemented using Python or Excel.

14. A computer device, characterized in that, The computer device includes a simulator for performing the steps of the method according to any one of claims 1 to 9.

Images