Method and apparatus for obtaining model acceleration performance, device and storage medium
By identifying and skipping irrelevant acceleration instructions in the functional simulator, synchronizing results, and inserting architectural checkpoints, the problems of slow simulation speed and low flexibility in the acceleration of large language models are solved, achieving efficient and accurate performance evaluation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- BEIJING INSTITUTE OF OPEN SOURCE CHIP
- Filing Date
- 2026-01-13
- Publication Date
- 2026-07-24
AI Technical Summary
Existing technologies, when evaluating acceleration of large language models, suffer from slow simulation speeds and architectural checkpoints that cannot adapt to hardware design and model updates, resulting in low flexibility and efficiency.
By identifying irrelevant acceleration instructions in the functional simulator, skipping their execution, synchronizing the results on the host, inserting architectural checkpoints, and dynamically controlling the simulation process, the cost of manual adjustments can be reduced.
It improves the simulation efficiency and evaluation accuracy of large language model reasoning tasks, adapts to rapid changes in hardware and models, and reduces evaluation costs.
Smart Images
Figure CN121503706B_ABST
Abstract
Description
Technical Field
[0001] This application relates to, but is not limited to, the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for acquiring model acceleration performance. Background Technology
[0002] With the rapid development of Large Language Model (LLM) technology, its application in natural language processing tasks (such as text generation, translation, and question answering) is becoming increasingly widespread, covering multiple fields such as cloud services and personal assistants. These application scenarios have placed diverse demands on the performance, latency, and energy efficiency of hardware accelerators, driving the rise of heterogeneous accelerator architectures.
[0003] Currently, the industry primarily employs sampling methods to evaluate the acceleration process of large language models. This involves simulating representative regions of the workload rather than the entire workload to reduce simulation time and improve evaluation efficiency. This method typically relies on a functional simulator to run the full workload, identify representative regions, and generate architectural checkpoints. These checkpoints are then reconstructed in a high-fidelity simulator for performance analysis.
[0004] However, this approach results in slow simulation because large language model inference tasks involve a large number of accelerator instructions, which are typically executed through interpretation rather than optimization. Furthermore, with changes in hardware design, model updates, or inference frameworks, previously generated architecture checkpoints often become unusable, requiring manual reset after each update. This significantly increases manpower and time costs, reducing the flexibility and efficiency of the large language model acceleration process. Summary of the Invention
[0005] In view of this, embodiments of this application provide at least one method, apparatus, device, and storage medium for obtaining model acceleration performance.
[0006] The technical solution of this application embodiment is implemented as follows:
[0007] On one hand, embodiments of this application provide a method for obtaining model acceleration performance, the method comprising:
[0008] Identify irrelevant acceleration instructions for the large language model inference task running in the recognition function simulator, wherein the execution result of the irrelevant acceleration instructions does not affect the control flow of other parts of the large model inference task running in the function simulator;
[0009] When the execution control flow of the functional simulator reaches the irrelevant acceleration instruction, the functional simulator is controlled to skip the execution phase of the irrelevant acceleration instruction, and the instruction execution result of the irrelevant acceleration instruction from the host independent of the functional simulator is synchronized to the functional simulator.
[0010] Based on the positions of skip operations and synchronization operations in the operation control flow, architectural checkpoints are inserted in the operation control flow;
[0011] The acceleration performance of the large language model inference task is evaluated based on the simulator state collected from the architecture checkpoints.
[0012] On the other hand, embodiments of this application provide a device for obtaining model acceleration performance, the device comprising:
[0013] The identification module is used to identify irrelevant acceleration instructions for the large language model inference task running in the functional simulator. The execution result of the irrelevant acceleration instructions does not affect the other parts of the operation control flow of the functional simulator to continue running the large model inference task.
[0014] The control module is configured to, when the operation control flow of the function simulator reaches the irrelevant acceleration instruction, control the function simulator to skip the execution phase of the irrelevant acceleration instruction, and synchronize the instruction execution result of the irrelevant acceleration instruction from the host independent of the function simulator to the function simulator.
[0015] Based on the positions of skip operations and synchronization operations in the operation control flow, architectural checkpoints are inserted in the operation control flow;
[0016] An evaluation module is used to evaluate the acceleration performance of the large language model inference task based on the simulator state collected from the architecture checkpoints.
[0017] In another aspect, embodiments of this application provide a computer device, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method for obtaining model acceleration performance.
[0018] In another aspect, embodiments of this application provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method for obtaining model acceleration performance.
[0019] In another aspect, embodiments of this application provide a computer program including computer-readable code. When the computer-readable code is run in a computer device, the processor in the computer device executes some or all of the steps in the method for obtaining the above-described model acceleration performance.
[0020] This application embodiment identifies and skips the execution phase of irrelevant acceleration instructions during the large language simulation and deduction task running in the function simulator. By combining the host with the function simulator to synchronize the execution results of the skipped instructions, the stability of the large language simulation and deduction task is ensured while reducing the runtime of the function simulator. This improves the performance evaluation efficiency of the large language simulation and deduction task in the function simulator. Furthermore, based on the location of the skipped and synchronized operations, architecture checkpoints are automatically added to the runtime control flow for sampling to evaluate the acceleration performance of the large language simulation and deduction task. This allows the architecture checkpoints to be dynamically controlled to adapt to the situation of the large language simulation and deduction task, reducing the manual and time costs of adjusting the architecture checkpoints, thereby further improving the performance evaluation efficiency of the large language simulation and deduction task.
[0021] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this application. Attached Figure Description
[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.
[0023] Figure 1 A schematic diagram illustrating the implementation process of a method for obtaining model acceleration performance provided in an embodiment of this application;
[0024] Figure 2 A schematic diagram illustrating the principle of a method for obtaining model acceleration performance provided in an embodiment of this application;
[0025] Figure 3 A system flowchart illustrating a method for obtaining model acceleration performance as provided in an embodiment of this application;
[0026] Figure 4 A schematic diagram of the composition structure of a model acceleration performance acquisition device provided in an embodiment of this application;
[0027] Figure 5 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application. Detailed Implementation
[0028] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0029] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.
[0030] The terms “first / second / third” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first / second / third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.
[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.
[0032] This application provides a method for obtaining model acceleration performance, which can be executed by a server-side processor. The server-side can refer to computer devices such as servers, laptops, tablets, desktop computers, smart TVs, set-top boxes, and mobile devices (e.g., mobile phones, portable video players, personal digital assistants, dedicated messaging devices, portable gaming devices). Figure 1 This is a schematic diagram illustrating the implementation process of a method for obtaining model acceleration performance provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes:
[0033] Step 101: Identify irrelevant acceleration instructions for the large language model inference task running in the functional simulator. The execution result of the irrelevant acceleration instructions does not affect the operation control flow of other parts of the large model inference task running in the functional simulator.
[0034] In this application embodiment, the Instruction Set Simulator (ISS) is a software tool used to simulate the target hardware instruction set architecture to execute and test programs or workloads. Large language model inference tasks refer to natural language processing tasks executed through large-scale language models (LLMs), such as text generation or question answering. Irrelevant acceleration instructions are instructions in the hardware acceleration unit whose execution results are unaffected by the computational results of the ISS, and are specifically designed to accelerate specific computational tasks.
[0035] The system first analyzes the large language model inference task running in the functional simulator to identify irrelevant acceleration instructions. By parsing the instruction set architecture and program code, the system determines which accelerator instructions' execution results are independent of the simulator's computational results. The system uses static properties, such as tensor dimensions and model hyperparameters, to filter these instructions, ensuring they do not affect control flow. The system records the location and type of these irrelevant acceleration instructions, providing a basis for subsequent skip operations.
[0036] Step 102: When the operation control flow of the function simulator reaches the irrelevant acceleration instruction, control the function simulator to skip the execution phase of the irrelevant acceleration instruction, and synchronize the instruction execution result of the irrelevant acceleration instruction from the host independent of the function simulator to the function simulator.
[0037] In this embodiment, the execution control flow refers to the execution order of code and its branching logic during program execution. Irrelevant acceleration instructions are accelerator instructions to be processed in the functional simulator. The host is the computing device running the simulator, responsible for executing the actual computational tasks. The instruction execution result is the data or state generated after the accelerator instructions are executed.
[0038] The system monitors the execution control flow of the function simulator. When the control flow reaches an irrelevant acceleration instruction, the system controls the function simulator to skip the execution phase of that instruction. The system only retains the instruction acquisition and decoding phases to avoid computationally intensive operations. Simultaneously, the system obtains the instruction execution result from the host and synchronizes the result to the function simulator. The system uses a lightweight synchronization mechanism to ensure that the simulator state is consistent with the host's calculation results, preventing control flow deviation.
[0039] Step 103: Based on the positions of skip operations and synchronization operations in the operation control flow, insert architecture checkpoints in the operation control flow.
[0040] In this embodiment, the skip operation is the behavior of the system control function simulator skipping the execution phase of irrelevant acceleration instructions. The synchronization operation is the behavior of the system synchronizing the host computation results to the function simulator. The execution control flow is the execution order of code and its branching logic during program execution. The architecture checkpoint is the system state saved during the simulation, used for restoring the simulation and performance evaluation.
[0041] The system determines when to insert architectural checkpoints based on the positions of skip and synchronization operations within the runtime control flow. Architectural checkpoints are inserted at critical points in the control flow, such as after a skip operation or upon completion of a synchronization operation. The system saves the current simulator's architectural state, including register values and memory contents. These architectural checkpoints are used for subsequent high-fidelity simulations to ensure the accuracy and recoverability of the sampling process.
[0042] Step 104: Based on the simulator status collected by the architecture checkpoint, evaluate the acceleration performance of the large language model inference task.
[0043] In this embodiment, the architecture checkpoint is the system state saved during the simulation. The simulator state is the internal data of the functional simulator at a specific moment, such as registers, memory, and program counter. Large language model inference tasks are natural language processing tasks performed using large-scale language models. Acceleration performance typically refers to latency, throughput, and speedup ratio. Latency and throughput are metrics for evaluating the efficiency and effectiveness of hardware accelerators in processing tasks, while speedup ratio is a metric for evaluating functional simulator acceleration methods.
[0044] The system evaluates the acceleration performance of large language model inference tasks based on simulator states collected from architecture checkpoints. The system loads the architecture checkpoints into a high-fidelity simulator for subsequent simulation and analysis. The system measures metrics such as task execution time, resource utilization, and instruction throughput, generating a performance report. By comparing data before and after skipping operations, the system verifies the acceleration effect, ensuring the accuracy and reliability of the evaluation results.
[0045] This application embodiment identifies and skips the execution phase of irrelevant acceleration instructions during the large language simulation and deduction task running in the function simulator. By combining the host with the function simulator to synchronize the execution results of the skipped instructions, the stability of the large language simulation and deduction task is ensured while reducing the runtime of the function simulator. This improves the performance evaluation efficiency of the large language simulation and deduction task in the function simulator. Furthermore, based on the location of the skipped and synchronized operations, architecture checkpoints are automatically added to the runtime control flow for sampling to evaluate the acceleration performance of the large language simulation and deduction task. This allows the architecture checkpoints to be dynamically controlled to adapt to the situation of the large language simulation and deduction task, reducing the manual and time costs of adjusting the architecture checkpoints, thereby further improving the performance evaluation efficiency of the large language simulation and deduction task.
[0046] In some embodiments, step 101 includes: analyzing the control flow dependencies of each acceleration instruction in the large language model inference task, and filtering out accelerator instructions whose control flow dependencies depend only on static attributes as irrelevant acceleration instructions.
[0047] In this application embodiment, a large language model inference task refers to a computational process that uses a large-scale language model to perform natural language processing operations, such as text generation or translation. Acceleration instructions are commands specifically designed in hardware acceleration units to efficiently execute specific computational tasks. Control flow dependencies represent the degree to which the program execution order and branch logic depend on the instruction execution results. Static properties are parameters that remain unchanged in a program or model, such as tensor dimensions or model hyperparameters. Irrelevant acceleration instructions refer to accelerator instructions whose execution results do not affect the program control flow.
[0048] The system performs large language model inference tasks and analyzes the control flow dependencies of each accelerated instruction. The system checks whether the execution results of these instructions depend only on static properties, such as tensor dimensions or model hyperparameters, and are unaffected by runtime computation values. Through this analysis, the system identifies accelerator instructions whose control flow dependencies depend only on static properties and classifies them as irrelevant acceleration instructions. This process involves parsing the instruction set architecture, evaluating the impact of instruction execution on program branches and loops, and filtering based on predefined rules to ensure that only instructions whose execution can be skipped without causing control flow deviation are selected.
[0049] It's worth noting that the determination process for this irrelevant acceleration instruction can be pre-executed, thereby accelerating the running efficiency of the functional simulator. Alternatively, it can be executed during the execution of the large language model inference task, depending on actual needs. Furthermore, the method for determining the irrelevant acceleration instruction can be pre-set, and users can configure it according to their actual needs; no limitations are imposed here.
[0050] This application embodiment, by systematically analyzing control flow dependencies and filtering irrelevant acceleration instructions, can accurately identify skippable operations, thereby achieving efficient skipping execution in subsequent steps, reducing simulation time, and maintaining accuracy through a synchronization mechanism. Ultimately, it improves the functional simulation efficiency of large language model inference tasks, supports rapid hardware evaluation, and adapts to dynamically changing workloads.
[0051] In some embodiments, step 101 includes: adjusting and testing the dynamic attributes in the large language model inference task, and taking acceleration instructions with the same test results under different dynamic attributes as irrelevant acceleration instructions.
[0052] In this application embodiment, a large language model inference task refers to a computational process that utilizes a large-scale language model to perform natural language processing operations, such as text generation or question answering. Dynamic attributes refer to parameters or states that may change during program execution, such as input data or runtime environment conditions. Test results refer to performance data or status information output by the system after simulating or executing a task. Acceleration instructions refer to dedicated commands in the hardware acceleration unit used to improve the efficiency of specific computational tasks. Irrelevant acceleration instructions refer to those acceleration instructions that have no substantial impact on task results or control flow; skipping them will not affect accuracy.
[0053] The system first analyzes the large language model inference task, identifying dynamic attributes such as input sequence length or model parameters. The system creates multiple test scenarios by modifying the values of these dynamic attributes, such as adjusting input data or changing execution conditions. The system runs the task in each scenario and records the corresponding test results, such as execution time or output accuracy. The system compares the test results under different dynamic attributes; if certain acceleration instructions produce the same output or performance metrics in all tests, the system marks these instructions as irrelevant acceleration instructions.
[0054] It's worth noting that the determination process for this irrelevant acceleration instruction can be pre-executed, thereby accelerating the running efficiency of the functional simulator. Alternatively, it can be executed during the execution of the large language model inference task, depending on actual needs. Furthermore, the method for determining the irrelevant acceleration instruction can be pre-set, and users can configure it according to their actual needs; no limitations are imposed here.
[0055] This application's embodiments efficiently identify irrelevant acceleration instructions that can be skipped in large language model inference tasks, thereby reducing computational overhead during simulation, improving the efficiency of functional simulation, and maintaining the accuracy of evaluation. It is suitable for rapidly changing hardware and model environments.
[0056] In some embodiments, step 102 includes:
[0057] Sub-step 1021: When the execution control flow of the function simulator reaches the irrelevant acceleration instruction, the instruction fetch stage and the decoding stage of the irrelevant acceleration instruction are executed sequentially.
[0058] In this embodiment, a functional simulator is a software tool used to simulate a target hardware instruction set architecture, execute and test programs or workloads. The execution control flow refers to the execution order of program code and its branching logic. Irrelevant acceleration instructions are hardware acceleration unit instructions specifically designed to accelerate specific computing tasks. Irrelevant acceleration instructions refer to accelerator instructions that do not affect the control flow during simulation. The instruction fetch phase is the process of retrieving instructions from memory. The decoding phase is the process of parsing the fetched instructions into executable operations.
[0059] When the system runs an LLM inference task in the functional simulator, it monitors whether the runtime control flow reaches an irrelevant acceleration instruction. When the runtime control flow reaches an irrelevant acceleration instruction, the system identifies it. The system first executes the instruction fetching phase for the irrelevant acceleration instruction, retrieving instruction data from simulated memory. Next, the system executes the decoding phase, parsing the instruction into specific operations. This process ensures that instruction fetching and parsing are completed completely, providing accurate instruction information for subsequent steps while maintaining the continuity of program control flow.
[0060] Step 1022: After the decoding stage, skip the execution stage of the irrelevant acceleration instructions and synchronize the instruction execution results of the irrelevant acceleration instructions from the host independent of the function simulator to the function simulator.
[0061] In this embodiment, the execution phase is the process of actually executing instructions in the simulator. The host is a physical or virtual computing device running the simulator. The instruction execution result is the instruction operation output data calculated by the host. Synchronization is the process of transferring data from the host to the simulator to maintain a consistent state.
[0062] After decoding irrelevant acceleration instructions, the system skips their execution phase to avoid computationally intensive operations. The system then synchronizes the host computer's execution results for those instructions to the functional simulator. This synchronization is achieved by inserting synchronization instructions, ensuring that the simulator state matches the host computer's computation results. This mechanism prevents control flow deviations caused by skipping the execution phase, maintains simulation accuracy, and significantly reduces simulation time.
[0063] This application embodiment significantly improves simulation speed by efficiently handling the workload of large language model inference tasks in a functional simulator, skipping the execution phase of irrelevant acceleration instructions and synchronizing host results, while maintaining sampling accuracy, adapting to rapidly changing hardware and instruction sets, reducing evaluation costs, and supporting agile development.
[0064] In some embodiments, sub-step 1022 includes:
[0065] Sub-step 10221: Insert a synchronization instruction into the runtime control flow after skipping the execution phase of the irrelevant acceleration instruction, and send the input data of the irrelevant acceleration instruction to the host through the synchronization instruction.
[0066] In this embodiment, the synchronization instruction is a software-level primitive used to establish communication between the simulator and the host computer, ensuring data consistency. The host computer refers to the physical or virtual computing device running the simulator, capable of executing computational tasks and returning results. Input data consists of initial parameters or operands required for the execution of irrelevant acceleration instructions, such as tensor dimensions or vector values. Control flow refers to the sequence of instructions and branching logic during program execution, determining the execution path of the code.
[0067] After skipping the execution phase of irrelevant acceleration instructions, the system analyzes the current control flow to determine where synchronization instructions need to be inserted. The system generates synchronization instructions and embeds them into specific points in the control flow, such as the next instruction sequence after the skipped operation. The system packages the input data for the irrelevant acceleration instructions and sends it to the host through the interface between the simulator and the host. The system ensures that the data transmission process is synchronized with the simulator's execution state to avoid interrupting the continuity of the control flow.
[0068] Step 10222: Receive the instruction execution result sent by the host in response to the synchronization instruction, and write the instruction execution result into the register or memory of the functional simulator.
[0069] In this embodiment, the instruction execution result is the output value calculated by the host based on the input data, such as the product or sum of vector operations. The functional simulator is a software tool used to simulate the execution process of the target hardware instruction set architecture. Registers are high-speed storage units in the simulator used to temporarily store data or instructions. Memory is the main storage area in the simulator used to store program code and runtime data.
[0070] The system receives the execution results of instructions sent by the host through the communication channel between the simulator and the host. The system parses the received data packets and extracts the calculation results. Based on the simulator's architecture, the system writes the instruction execution results to a specified register or memory address. The system ensures that the write operation matches the simulator's current execution context, such as updating vector register values or modifying tensor data in memory. The system verifies the consistency of the state after the write operation to prevent data conflicts or control flow deviations.
[0071] The embodiments of this application ensure that the simulator state is consistent with the host calculation results through a synchronization mechanism, which reduces the functional simulation time, improves the sampling efficiency of the LLM workload, and avoids control flow errors caused by skip instructions in traditional methods.
[0072] In some embodiments, step 103 includes inserting an architecture checkpoint in the runtime control flow at a position following the skip operation and the synchronization operation.
[0073] In this embodiment, the system automatically inserts architectural checkpoints at specific locations in the control flow after skip and synchronization operations are completed. The system generates a complete system state record by saving the current simulator's architectural state, including register values, memory contents, and key data such as the program counter. These architectural checkpoints are then stored in a designated storage medium for loading and use by the high-fidelity simulator to support subsequent performance evaluation and analysis.
[0074] By sequentially executing skip operations, synchronization operations, and architecture checkpoint insertion, the system can efficiently complete the sampling and evaluation process of LLM workloads, significantly reducing simulation time while maintaining evaluation accuracy. It also supports rapid adaptation to hardware and instruction set changes, improving the scalability and efficiency of the overall evaluation process.
[0075] In some embodiments, step 104 includes:
[0076] Step 1041: Collect the simulator state of the functional simulator at the sampling point and record the position of the sampling point. The simulator state includes at least: register value, memory content and program counter.
[0077] In this embodiment, a sampling point refers to a specific point in time selected during the simulation process to capture a snapshot of the system state. A functional simulator is a software tool used to simulate the execution of a target hardware instruction set architecture. The simulator state includes the system's runtime data at a specific moment, such as register values, memory contents, and a program counter. Register values are temporary data stored in processor registers. Memory contents are the programs and data stored in system memory. The program counter is a register that indicates the address of the next instruction to be executed.
[0078] When the system performs a large language model inference task in the functional simulator, it collects the simulator state at predefined sampling points. The system records the current values of registers, memory, and the program counter as the simulator state. Simultaneously, the system records the location information of the sampling points, such as instruction addresses or timestamps, to ensure that the state data corresponds to specific execution points. The data collection process is automated to ensure data integrity and accuracy, providing reliable input for subsequent steps.
[0079] Step 1042: Input the location of the sampling point into the high-fidelity simulator and restore the simulator state.
[0080] In this embodiment, the high-fidelity simulator is a high-precision simulation tool capable of simulating hardware behavior in detail for performance analysis. The sampling point location refers to the specific address or identifier recorded when sampling occurs. Simulator state restoration refers to loading the collected state data into the simulator, restoring it to the conditions of a specific execution point.
[0081] The system uses the sampling point locations recorded in step 1041 as input and transmits them to the high-fidelity simulator. The high-fidelity simulator parses the location information and loads the corresponding simulator state data, including register values, memory contents, and the program counter. The system restores the state through a data interface or file transfer mechanism to ensure that the simulation environment is consistent with the original functional simulator. The restoration process includes initializing the simulator memory and registers, and setting the program counter to point to the correct instruction, laying the foundation for subsequent task execution.
[0082] Step 1043: Execute the large language model inference task in the high-fidelity simulator and obtain index parameters reflecting the acceleration performance of the large language model inference task.
[0083] In this application embodiment, the large language model inference task refers to a computational job based on a large-scale language model, such as text generation or translation. Acceleration performance metrics are measures that quantify system efficiency, such as execution time, throughput, or resource utilization.
[0084] The system initiates the execution of a large language model inference task in a high-fidelity simulator, resuming operation from the sampling point based on the recovered simulator state. The simulator simulates hardware behavior, tracks the task execution process, and collects performance data. The system calculates key performance indicators, such as instruction cycle count, cache hit rate, or power consumption, and outputs the results through built-in analysis tools. The execution process ensures environmental consistency and avoids external interference, thereby providing accurate performance evaluation reports.
[0085] This application embodiment reduces the time overhead of the entire simulation by collecting and restoring the state of sampling points, while using a high-fidelity simulator to ensure evaluation accuracy, improve the iteration efficiency of accelerator design, support rapidly changing hardware and load requirements, and enhance scalability and practicality.
[0086] In some embodiments, refer to Figure 2 ,in:
[0087] add: The addition instruction adds two operands and stores the result;
[0088] vmul: Vector multiplication command, performs multiplication operations on vector data;
[0089] `vred`: Vector reduction command, reduces vector data to scalar results;
[0090] ld: Load instruction, reads data from memory into a register;
[0091] vld: Vector load instruction, loads vector data from memory;
[0092] `st`: Store instruction, writes register data to memory;
[0093] `vst`: Vector storage instruction, which stores vector data into memory;
[0094] mv: The move command, which copies data from one location to another;
[0095] `beq`: A conditional branch instruction that jumps to the target address when two operands are equal.
[0096] Before skipping irrelevant acceleration instructions for a large language model inference task using the embodiments of this application, the irrelevant acceleration instructions in the function simulator's execution control flow (Instruction sequence in a basic block) consist of four parts: vld, vmul, vrd, and vst. Only the fetch and decode stages can be retained, and the execution stage can be skipped. This yields the execution control flow after skipping the execution of irrelevant acceleration instructions using the embodiments of this application (After Skip).
[0097] In some embodiments, according to Figure 3 In the original Large Language Model Inference Task (Origin Simulation) without applying the embodiments of this application, the Large Language Model Inference Task (LLMWorkload) will be directly input into the Functional Simulator (ISS) for execution.
[0098] In the SkipSync Simulation of the large language model inference task using the embodiments of this application, the large language model inference task is run simultaneously in the host and the functional simulator (ISS). The host and the functional simulator first execute the execution phase of other instructions and other operations. When the control flow reaches the execution phase of the irrelevant acceleration instruction (Target Operator), the functional simulator skips the execution phase of the irrelevant acceleration instruction (Skip On~Skip Off), while the host executes the irrelevant acceleration instruction completely. Then, the execution results of the instruction execution phase skipped by the functional simulator (Target Operator with Skip) are synchronized to the functional simulator.
[0099] Based on the foregoing embodiments, this application provides a device for acquiring model acceleration performance. The device includes various units and modules included in each unit, which can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.
[0100] Some embodiments of this application can achieve at least the following beneficial effects:
[0101] Improve simulation efficiency: SkipSync can significantly reduce the simulation time of LLM inference tasks while ensuring simulation accuracy. It is especially suitable for large-scale models and multi-core scenarios, and can quickly complete the sampling process and generate accurate checkpoints.
[0102] Reduced costs associated with hardware updates: SkipSync’s scalability allows users to adapt to new accelerator instruction sets and hardware architecture updates without frequently updating the simulation framework, reducing development costs and debugging time caused by hardware updates.
[0103] Highly efficient support for agile development: SkipSync enables efficient evaluation under constantly changing hardware and workload conditions, providing strong support for agile development of the accelerator and improving the iteration speed in the design and verification process.
[0104] Figure 4This is a schematic diagram of the composition structure of a model acceleration performance acquisition device provided in an embodiment of this application, as shown below. Figure 4 As shown, the model acceleration performance acquisition device 20 includes:
[0105] The identification module 201 is used to identify irrelevant acceleration instructions for the large language model inference task running in the functional simulator. The execution result of the irrelevant acceleration instructions does not affect the other parts of the operation control flow of the functional simulator to continue running the large model inference task.
[0106] The control module 202 is used to control the functional simulator to skip the execution phase of the irrelevant acceleration instruction when the operation control flow of the functional simulator reaches the irrelevant acceleration instruction, and to synchronize the instruction execution result of the irrelevant acceleration instruction from the host independent of the functional simulator to the functional simulator.
[0107] Based on the positions of skip operations and synchronization operations in the operation control flow, architectural checkpoints are inserted in the operation control flow;
[0108] Evaluation module 203 is used to evaluate the acceleration performance of the large language model inference task based on the simulator state collected by the architecture checkpoint.
[0109] In some embodiments, the identification module 201 is further configured to:
[0110] Analyze the control flow dependencies of each acceleration instruction in the large language model inference task, and select accelerator instructions whose control flow dependencies depend only on static attributes as irrelevant acceleration instructions.
[0111] In some embodiments, the identification module 201 is further configured to:
[0112] The dynamic attributes in the large language model inference task are adjusted and tested, and acceleration instructions with the same test results under different dynamic attributes are regarded as irrelevant acceleration instructions.
[0113] In some embodiments, the control module 202 is further configured to:
[0114] When the execution control flow of the functional simulator reaches the irrelevant acceleration instruction, the instruction fetch stage and the decoding stage of the irrelevant acceleration instruction are executed sequentially.
[0115] After the decoding stage, the execution stage of the irrelevant acceleration instructions is skipped, and the instruction execution results of the irrelevant acceleration instructions from the host, which is independent of the functional simulator, are synchronized to the functional simulator.
[0116] In some embodiments, the control module 202 is further configured to:
[0117] A synchronization instruction is inserted into the runtime control flow after skipping the execution phase of the irrelevant acceleration instruction, and the input data of the irrelevant acceleration instruction is sent to the host through the synchronization instruction.
[0118] The system receives the instruction execution result from the host in response to the synchronization instruction and writes the instruction execution result into the register or memory of the functional simulator.
[0119] In some embodiments, the control module 202 is further configured to:
[0120] In the operation control flow, an architecture checkpoint is inserted at a position after the skip operation and the synchronization operation.
[0121] In some embodiments, the evaluation module 203 is further configured to:
[0122] The simulator state of the functional simulator is collected at the sampling point, and the position of the sampling point is recorded. The simulator state includes at least: register value, memory content, and program counter.
[0123] Input the location of the sampling point into the high-fidelity simulator and restore the simulator state;
[0124] The large language model inference task is executed in the high-fidelity simulator to obtain index parameters reflecting the acceleration performance of the large language model inference task.
[0125] This application embodiment identifies and skips the execution phase of irrelevant acceleration instructions during the large language simulation and deduction task running in the function simulator. By combining the host with the function simulator to synchronize the execution results of the skipped instructions, the stability of the large language simulation and deduction task is ensured while reducing the runtime of the function simulator. This improves the performance evaluation efficiency of the large language simulation and deduction task in the function simulator. Furthermore, based on the location of the skipped and synchronized operations, architecture checkpoints are automatically added to the runtime control flow for sampling to evaluate the acceleration performance of the large language simulation and deduction task. This allows the architecture checkpoints to be dynamically controlled to adapt to the situation of the large language simulation and deduction task, reducing the manual and time costs of adjusting the architecture checkpoints, thereby further improving the performance evaluation efficiency of the large language simulation and deduction task.
[0126] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this application can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.
[0127] It should be noted that, in the embodiments of this application, if the above-described method for obtaining model acceleration performance is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.
[0128] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.
[0129] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.
[0130] This application provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.
[0131] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.
[0132] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.
[0133] It should be noted that, Figure 5 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this application, such as... Figure 5 As shown, the hardware entity of the computer device 700 includes: one or more processors 701, a communication interface 702, and a memory 703, wherein:
[0134] Processor 701 typically controls the overall operation of computer device 700.
[0135] Communication interface 702 enables computer devices to communicate with other terminals or servers over a network.
[0136] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 701 and various modules in the computer device 700. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 701, the communication interface 702, and the memory 703 can be performed via bus 704. Only one processor is shown in the figure; each processor 700 includes one or more cores.
[0137] It should be noted that the computer device may include multiple processors 701, and each processor 701 can interact with each other through aggregated communication methods such as all-to-all, all-gather, or all-reduce. The processors 701 may be central processing units (CPUs), graphics processing units (GPUs), embedded neural network processing units (NPUs), tensor processing units (TPUs), data processing units (DPUs), accelerated processing units (APUs), floating-point processing units (FPUs), or application-specific integrated circuits (ASICs). The processors may also be single-core or multi-core processors. The processor may consist of a CPU and hardware chips. The hardware chips may be ASICs, PLDs, or combinations thereof. The PLDs may be complex programmable logic devices (CPLDs), FPGAs, generic array logic (GALs), or any combination thereof. The processor can also be implemented using logic devices with built-in processing logic, such as FPGAs or digital signal processors (DSPs).
[0138] The communication interface 702 can be a wired interface or a wireless interface, used to communicate with other modules or devices. The wired interface can be an Ethernet interface, a local interconnect network (LIN), etc., and the wireless interface can be a cellular network interface or a wireless LAN interface, etc.
[0139] Memory 703 can be non-volatile memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Memory 703 can also be volatile memory, which can be random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synclink dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM), direct rambus DRAM (DRDRAM), and rambus DRAM.
[0140] The 704 bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc.
[0141] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.
[0142] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0143] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.
[0144] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.
[0145] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.
[0146] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.
[0147] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.
[0148] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.
Claims
1. A method for obtaining model acceleration performance, characterized in that, The method includes: Identify irrelevant acceleration instructions for the large language model inference task running in the recognition function simulator, wherein the execution result of the irrelevant acceleration instructions does not affect the other parts of the operation control flow of the function simulator to continue running the large model inference task; The irrelevant acceleration command is identified in at least one of the following ways: Perform a large language model inference task and analyze the control flow dependency of each acceleration instruction in the large language model inference task; check whether the execution result of each acceleration instruction depends only on static attributes, identify acceleration instructions whose control flow dependency depends only on static attributes, and treat them as irrelevant acceleration instructions; Analyze the large language model inference task and identify its dynamic attributes; create multiple test scenarios by modifying the values of the dynamic attributes; run the large language model inference task in each test scenario and record the corresponding test results; compare the test results under different dynamic attributes, and use acceleration instructions that produce the same output or performance index in all test scenarios as irrelevant acceleration instructions. When the execution control flow of the functional simulator reaches the irrelevant acceleration instruction, the instruction fetch stage and the decoding stage of the irrelevant acceleration instruction are executed sequentially. After the decoding stage, the execution stage of the irrelevant acceleration instructions is skipped, and the instruction execution results of the irrelevant acceleration instructions from the host, which is independent of the functional simulator, are synchronized to the functional simulator. Based on the positions of skip operations and synchronization operations in the operation control flow, architectural checkpoints are inserted in the operation control flow; The acceleration performance of the large language model inference task is evaluated based on the simulator state collected from the architecture checkpoints.
2. The method according to claim 1, characterized in that, The step of synchronizing the instruction execution results of the unrelated acceleration instructions from a host independent of the functional simulator to the functional simulator includes: A synchronization instruction is inserted into the runtime control flow after skipping the execution phase of the irrelevant acceleration instruction, and the input data of the irrelevant acceleration instruction is sent to the host through the synchronization instruction. The system receives the instruction execution result from the host in response to the synchronization instruction and writes the instruction execution result into the register or memory of the functional simulator.
3. The method according to any one of claims 1 or 2, characterized in that, The insertion of architectural checkpoints into the runtime control flow based on the positions of skip and synchronization operations within the runtime control flow includes: In the operation control flow, architecture checkpoints are inserted at positions following skip operations and synchronization operations, respectively.
4. The method according to any one of claims 1 or 2, characterized in that, The evaluation of the acceleration performance of the large language model inference task based on the simulator state collected from the architecture checkpoints includes: The simulator state of the functional simulator is collected at sampling points, and the location of the sampling points is recorded. The simulator state includes at least: register values, memory contents, and program counter. Input the location of the sampling point into the high-fidelity simulator and restore the simulator state; The large language model inference task is executed in the high-fidelity simulator to obtain index parameters reflecting the acceleration performance of the large language model inference task.
5. A device for acquiring model acceleration performance, characterized in that, The device includes: The identification module is used to identify irrelevant acceleration instructions for the large language model inference task running in the functional simulator. The execution result of the irrelevant acceleration instructions does not affect the other parts of the operation control flow of the functional simulator to continue running the large model inference task. The irrelevant acceleration command is identified in at least one of the following ways: Perform a large language model inference task and analyze the control flow dependency of each acceleration instruction in the large language model inference task; check whether the execution result of each acceleration instruction depends only on static attributes, identify acceleration instructions whose control flow dependency depends only on static attributes, and treat them as irrelevant acceleration instructions; Analyze the large language model inference task and identify its dynamic attributes; create multiple test scenarios by modifying the values of the dynamic attributes; run the large language model inference task in each test scenario and record the corresponding test results; compare the test results under different dynamic attributes, and use acceleration instructions that produce the same output or performance index in all test scenarios as irrelevant acceleration instructions. The control module is configured to, when the operation control flow of the function simulator reaches the irrelevant acceleration instruction, sequentially execute the instruction fetch stage and the decoding stage of the irrelevant acceleration instruction; after the decoding stage, skip the execution stage of the irrelevant acceleration instruction and synchronize the instruction execution result of the irrelevant acceleration instruction from the host independent of the function simulator to the function simulator. Based on the positions of skip operations and synchronization operations in the operation control flow, architectural checkpoints are inserted in the operation control flow; An evaluation module is used to evaluate the acceleration performance of the large language model inference task based on the simulator state collected from the architecture checkpoints.
6. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the method for obtaining model acceleration performance according to any one of claims 1 to 4.
7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps in the method for obtaining the model acceleration performance as described in any one of claims 1 to 4.