Performance evaluation method and device and equipment cluster

Abstract

The embodiment of the invention discloses a performance evaluation method and device and an equipment cluster, relates to the technical field of computers, and aims at solving the problem that the performance of an AI chip cannot be accurately evaluated so as to effectively support optimization of the AI chip. The performance evaluation method comprises the steps of obtaining performance information of a first AI chip; the performance information of the first AI chip comprises a first sub-time period and a second sub-time period, the first sub-time period is a time period when an operation operator corresponding to an operation layer of the model in the first AI chip is scheduled to execute an operation process, and the second sub-time period is a time period when the first AI chip schedules the operation operator and forwards data required by the operation operator when the model runs; determining an evaluation result of the first AI chip according to the performance information of the first AI chip; the evaluation result of the first AI chip comprises a first index for indicating the performance of the first AI chip in the processing operation process and a second index for indicating the performance of the first AI chip in the processing scheduling and forwarding process; and displaying the evaluation result of the first AI chip.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a performance evaluation method, apparatus, and device cluster. Background Art

[0002] With the rapid development of artificial intelligence (AI) applications such as neural network models, artificial intelligence (AI) chips have become widely used. AI chips, also known as AI accelerators or computing cards, generally refer to hardware modules specifically designed to handle computing tasks in AI applications. The performance of AI chips affects the cost and efficiency of running neural network models, making accurate evaluation and optimization of AI chip performance crucial.

[0003] Currently, the performance differences between different chips can be evaluated by comparing the time it takes them to execute a model end-to-end. However, this approach cannot accurately assess the performance of AI chips during processing, making it difficult to effectively optimize them. Summary of the Invention

[0004] This application provides a performance evaluation method, apparatus, and device cluster for improving the problem of being unable to accurately evaluate the performance of AI chips, so as to effectively support the optimization of AI chips.

[0005] In a first aspect, the present application provides a performance evaluation method, comprising: obtaining performance information of a first AI chip that provides computing power for model execution, determining an evaluation result of the first AI chip based on the performance information of the first AI chip, and further displaying the evaluation result of the first AI chip. The first AI chip includes an operator corresponding to a computing layer of the model, the operator being used to implement the computing process of the corresponding computing layer; the performance information of the first AI chip includes a first sub-period and a second sub-period of the operator corresponding to the computing layer of the model on the first AI chip. The first sub-period is the period during which the operator in the first AI chip executes the computing process after being scheduled. The second sub-period is the period during which the first AI chip schedules the operator and forwards data required by the operator. The first evaluation result includes a first indicator and a second indicator. The first indicator is used to indicate the performance of the first AI chip in processing the computing process. The second indicator is used to indicate the performance of the first AI chip in processing the scheduling and forwarding processes.

[0006] In this technical solution, the operation time of the operator included in the AI ​​chip can be used as the granularity to obtain the operation time of the operator, the time required for scheduling the operator and forwarding the data of the operator, and the AI ​​chip index evaluation and display according to the operation time of the operator, the time required for scheduling the operator and forwarding the data of the operator. Compared with evaluating the AI ​​chip according to the time of the end-to-end processing process based on the model, the technical solution provided by this application evaluates the AI ​​chip according to the finer-grained time, achieves a more refined evaluation, and improves the accuracy of the AI ​​chip evaluation. At the same time, the AI ​​chip is evaluated from multiple indicators, and the performance evaluation results in different dimensions are realized. The performance of the first AI chip is intuitively displayed, which facilitates the analysis of the performance of the first AI chip in different processing logics, so as to effectively support the optimization of the AI ​​chip.

[0007] In a possible implementation, the first indicator is the sum of the first sub-periods; and the second indicator is the sum of the second sub-periods.

[0008] In one possible implementation, the performance information of the first AI chip also includes a third sub-period, which is the period during which a communication operator in the first AI chip is scheduled to execute a communication process. The communication operator is an operator corresponding to the operation layer of the model, which is used to be scheduled by the first AI chip to implement a communication process with the outside world. The first evaluation result also includes a third indicator, which is the difference between the first duration and the second duration. The first duration is the sum of each third sub-period, and the second duration is the sum of the overlapping periods between the first sub-period and the third sub-period.

[0009] In one possible implementation, the performance information of the first AI chip also includes a first total period, which is the period used by the first AI chip to complete the model operation process; the second sub-period is obtained by removing other sub-periods included in the performance information of the first AI chip from the first total period.

[0010] In one possible implementation, the evaluation results of the first AI chip also include indicators corresponding to multiple types of operators; the indicators corresponding to each type of operator are used to characterize the processing performance of each type of operator.

[0011] In one possible implementation, the indicators corresponding to each of the multiple types of operators include at least one of the following: a fourth indicator, a fifth indicator, and a sixth indicator; the fourth indicator is used to indicate the performance of the first AI chip in processing matrix operations, the fifth indicator is used to indicate the performance of the first AI chip in processing vector operations, and the sixth indicator is used to indicate the performance of the first AI chip in processing fusion operations; the fourth indicator includes the sum of the first sub-time periods corresponding to matrix operation operators whose operation type is matrix operation; the fifth indicator includes the sum of the first sub-time periods corresponding to vector operation operators whose operation type is vector operation; and the sixth indicator includes the sum of the first sub-time periods corresponding to fusion operation operators whose operation type is fusion operation.

[0012] In one possible implementation, the performance information of the first AI chip also includes the amount of first storage resources used by the first AI chip during model execution; the evaluation result of the first AI chip also includes a seventh indicator corresponding to the first storage resource amount.

[0013] In one possible implementation, the performance evaluation method further includes: obtaining performance information of a second AI chip; the performance information of the second AI chip includes at least one of the following: a second total time period, a fourth sub-time period, a fifth sub-time period, and an amount of second storage resources used by the second AI chip during model operation; the second total time period is the time period used by the second AI chip to complete the model operation process; the fourth sub-time period is the time period during which an operator corresponding to the operation layer of the model in the second AI chip is scheduled to perform the operation process; the fifth sub-time period is the time period during which a communication operator in the second AI chip is scheduled to perform the communication process, where the communication operator is an operator corresponding to the operation layer of the model that is scheduled by the second AI chip to implement a communication process with the outside world; determining an evaluation result of the second AI chip based on the performance information of the second AI chip; and displaying the evaluation result of the second AI chip.

[0014] In one possible implementation, the evaluation result of the second AI chip includes an eighth indicator; the eighth indicator corresponds to the sixth indicator, and the eighth indicator is the sum of the fourth sub-time period corresponding to the fusion operation operator in the second AI chip; the operation process implemented by the fusion operation operator in the second AI chip is the same as that implemented by the fusion operation operator in the first AI chip.

[0015] In a second aspect, a performance evaluation device is provided, comprising: an acquisition module, a processing module, and a display module;

[0016] an acquisition module, configured to acquire performance information of a first AI chip; the first AI chip is configured to provide computing power for model execution, the performance information of the first AI chip including a first sub-period and a second sub-period, the first sub-period being a period during which an operator corresponding to a computing layer of the model in the first AI chip is scheduled to execute a computation, and the second sub-period being a period during which the first AI chip schedules the operator and forwards data required by the operator during model execution;

[0017] a processing module, configured to determine an evaluation result of the first AI chip based on the performance information of the first AI chip acquired by the acquisition module; the evaluation result of the first AI chip including a first indicator and a second indicator; the first indicator indicating the performance of the first AI chip in processing an operation; and the second indicator indicating the performance of the first AI chip in processing a scheduling and forwarding process.

[0018] The display module is configured to display the evaluation result of the first AI chip determined by the processing module.

[0019] In a possible implementation, the first indicator is the sum of the first sub-periods; and the second indicator is the sum of the second sub-periods.

[0020] In one possible implementation, the performance information of the first AI chip also includes a third sub-period, which is the period during which a communication operator in the first AI chip is scheduled to execute a communication process. The communication operator is an operator corresponding to the operation layer of the model, which is used to be scheduled by the first AI chip to implement a communication process with the outside world. The first evaluation result also includes a third indicator, which is the difference between the first duration and the second duration. The first duration is the sum of each third sub-period, and the second duration is the sum of the overlapping periods between the first sub-period and the third sub-period.

[0021] In one possible implementation, the performance information of the first AI chip also includes a first total period, which is the period used by the first AI chip to complete the model operation process; the second sub-period is obtained by removing other sub-periods included in the performance information of the first AI chip from the first total period.

[0022] In one possible implementation, the evaluation results of the first AI chip also include indicators corresponding to multiple types of operators; the indicators corresponding to each type of operator are used to characterize the processing performance of each type of operator.

[0023] In one possible implementation, the indicators corresponding to each of the multiple types of operators include at least one of the following: a fourth indicator, a fifth indicator, and a sixth indicator; the fourth indicator is used to indicate the performance of the first AI chip in processing matrix operations, the fifth indicator is used to indicate the performance of the first AI chip in processing vector operations, and the sixth indicator is used to indicate the performance of the first AI chip in processing fusion operations; the fourth indicator includes the sum of the first sub-time periods corresponding to matrix operation operators whose operation type is matrix operation; the fifth indicator includes the sum of the first sub-time periods corresponding to vector operation operators whose operation type is vector operation; and the sixth indicator includes the sum of the first sub-time periods corresponding to fusion operation operators whose operation type is fusion operation.

[0024] In one possible implementation, the performance information of the first AI chip also includes the amount of first storage resources used by the first AI chip during model execution; the evaluation result of the first AI chip also includes a seventh indicator corresponding to the first storage resource amount.

[0025] In one possible implementation, the acquisition module is further configured to acquire performance information of the second AI chip; the performance information of the second AI chip includes at least one of the following: a second total period, a fourth sub-period, a fifth sub-period, and an amount of second storage resources used by the second AI chip during model execution; the second total period is the period used by the second AI chip to complete the model execution; the fourth sub-period is the period during which an operator corresponding to the computation layer of the model in the second AI chip is scheduled to execute the computation; and the fifth sub-period is the period during which a communication operator in the second AI chip is scheduled to execute the communication. The communication operator is an operator corresponding to the computation layer of the model that is scheduled by the second AI chip to implement communication with the outside world.

[0026] The processing module is further configured to determine an evaluation result of the second AI chip based on the performance information of the second AI chip acquired by the acquisition module;

[0027] The display module is further used to display the evaluation result of the second AI chip determined by the processing module.

[0028] In one possible implementation, the evaluation result of the second AI chip includes an eighth indicator; the eighth indicator corresponds to the sixth indicator, and the eighth indicator is the sum of the fourth sub-time period corresponding to the fusion operation operator in the second AI chip; the operation process implemented by the fusion operation operator in the second AI chip is the same as that implemented by the fusion operation operator in the first AI chip.

[0029] In a third aspect, the present application provides a computing device cluster comprising at least one computing device, each computing device comprising a processor and a memory; the processor of at least one computing device is used to execute instructions stored in the memory of at least one computing device, so that the operating steps of the method in the first aspect or any possible implementation of the first aspect are executed.

[0030] It should be noted that the memory and processor described in this application can be integrated on a single chip or can be separately arranged on different chips. This application does not limit the type of memory and the arrangement of the memory and processor.

[0031] In a fourth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing device cluster, enables the operation steps of the method in the first aspect or any possible implementation of the first aspect to be executed.

[0032] In a fifth aspect, the present application provides a computer-readable storage medium comprising computer program instructions, which, when executed by a computing device cluster, enables the operation steps of the method in the first aspect or any possible implementation of the first aspect to be executed.

[0033] It can be understood that any of the performance evaluation devices or computing device clusters or computer-readable storage media or computer program products provided above are used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A schematic diagram of the structure of a performance evaluation system provided in an embodiment of the present application;

[0035] Figure 2 A schematic diagram of the structure of a front-end device provided in an embodiment of the present application;

[0036] Figure 3 A flow chart of a performance evaluation method provided in an embodiment of the present application;

[0037] Figure 4 A schematic diagram of time period information provided in an embodiment of the present application;

[0038] Figure 5 A schematic diagram of indicator information provided in an embodiment of the present application;

[0039] Figure 6 A schematic diagram of an indicator display interface provided in an embodiment of the present application;

[0040] Figure 7A schematic diagram of another indicator display interface provided in an embodiment of the present application;

[0041] Figure 8 A flow chart of another performance evaluation method provided in an embodiment of the present application;

[0042] Figure 9 A schematic diagram of the structure of a performance evaluation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The terms "first", "second", "third" and "fourth" in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects rather than to limit a specific order.

[0044] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0045] The following is a brief introduction to the terms and technologies involved in the embodiments of the present application to facilitate readers' understanding.

[0046] With the popularization of computer technology, artificial intelligence applications such as neural network models have developed rapidly, especially large language models (LLMs). LLMs generally refer to deep learning models with billions or more parameters. As a general model, LLMs perform well in a wide range of tasks. LLMs can be trained using self-supervised learning or semi-supervised learning with a large amount of unlabeled text as training data, and are not trained for specific tasks such as sentiment analysis, named entity recognition, or mathematical reasoning. The training process of LLMs can include tasks such as predicting the next word in a sentence. With sufficient training and parameter count, LLMs can capture most of the syntax and semantics of human language. After training, LLMs can "remember" a large number of facts and can efficiently and accurately display a considerable amount of common sense.

[0047] For example, a widely used LLM is the chat generative pre-trained transformer (ChatGPT). ChatGPT is an artificial intelligence-driven natural language processing tool that uses a transformer neural network architecture. It can process sequence data and has language understanding and text generation capabilities. When training ChatGPT, it is connected to a large corpus containing real-world conversations, which enables ChatGPT to understand everything from astronomy to geography, and to interact based on the context of the chat content, enabling chat conversations that are almost indistinguishable from real humans. In addition to chat conversations, ChatGPT can also complete writing tasks such as emails, video scripts, copywriting, translation, and code.

[0048] The rapid development of neural network models has led to a rapid increase in demand for computing power, and AI chips such as neural processing units (NPUs) and graphics processing units (GPUs) have been widely used. For example, the initial training of ChatGPT requires the computing power of 10,000 GPUs, and the computing power consumption for subsequent daily tuning training is approximately 3640 petabytes (P). The number of floating point operations per second (FLOPS) performed requires the support of 7 to 8 data centers with a computing power of 500PFLOPS, which is very expensive. For example, the training process of the large language model meta AI (LLaMA) requires 1 million GPU hours. The training process of the pathway language model (PaLM) requires a computing power of 2.56e24FLOPS.

[0049] The computing performance of AI chips is crucial for neural network models, directly impacting their training cost and efficiency. Optimizing the computing performance of a specific AI chip requires understanding its strengths and weaknesses, thereby enhancing its strengths and addressing its weaknesses, ultimately improving the overall computing performance of the AI ​​chip.

[0050] However, the pain point in AI chip performance analysis is that it is often only possible to compare the duration of the end-to-end processing of the model executed by different AI chips. The end-to-end processing process of the model can be a model training process or a model inference process. A model training process is a complete training cycle including a forward training step, a reverse training step, and a model operation layer parameter update step. A model inference process is a forward propagation process in which the model independently processes the original input data and outputs the final result. For example, based on an analyzer (Profiler) for analyzing model performance, the execution duration of the model end-to-end processing process can be collected. The relevant analyzer can collect a model inference process in the following way.

[0051]

[0052] As shown in Table 1, the analyzer implemented using this method can collect information such as the execution time and duration percentage of each operation process in the central processing unit (CPU) and compute unified device architecture (CUDA) during the end-to-end processing of the ResNet18 model. The operations involved in the end-to-end processing of the ResNet18 model include two-dimensional convolution operations, convolution operations, synchronization operations, continuous storage operations, and CUDA deep neural network library (cuDNN) convolution operations. CUDA is the parallel computing engine on the GPU, which provides the GPU with powerful parallel computing capabilities, making the GPU widely used in various high-performance computing scenarios. The units of time in Table 1 are microseconds (μs).

[0053] The "CPU Time Percentage" indicates the proportion of the CPU execution time of a computation process to the total execution time of that computation process. The "CPU Total Time Percentage" indicates the time a computation process spends on the CPU. The "CPU Time Percentage" indicates the proportion of the CPU time occupied by a computation process during the end-to-end processing of the ResNet18 model. The "CPU Total Time Percentage" indicates the total CPU execution time of a computation process during the end-to-end processing of the ResNet18 model. The "CPU Average Time Percentage" indicates the average CPU execution time of each computation process. The "CUDA Time Percentage" indicates the proportion of the GPU execution time of a computation process during the end-to-end processing of the ResNet18 model. The "CUDA Total Time Percentage" indicates the total GPU execution time of a computation process. The "CUDA Average Time Percentage" indicates the average GPU execution time of each computation process. The "Number of Schedules" indicates the number of times a computation process is scheduled. Due to differences in input data, the duration parameters corresponding to the same computation process in Table 1 may vary.

[0054] Table 1

[0055]

[0056] However, this approach simply collects the time consumption information of each operation process according to the end-to-end processing of the model, and cannot convert the collected information into specific performance data. It is difficult to support the division of the time it takes for AI chips to execute the end-to-end processing of the model into performance indicators of multiple dimensions. This makes it difficult to conduct fine-grained comparative evaluations between AI chips, understand the strengths and weaknesses of AI chips, and intuitively and effectively support the optimization of AI chips. For example, as shown in Table 1, this approach only lists the time consumption information of each operation process during the end-to-end processing of the ResNet18 model, and does not consider classifying these operations by operation type, etc. This makes it difficult to intuitively support the overall analysis of identical or similar operations and cannot support fine-grained evaluation of AI chips.

[0057] To this end, embodiments of the present application provide a performance evaluation method, apparatus, and device cluster. The method provided herein may include: obtaining performance information of a first AI chip that provides computing power for model execution, determining an evaluation result of the first AI chip based on the performance information of the first AI chip, and further displaying the evaluation result of the first AI chip. The first AI chip includes an operator corresponding to the model's computing layer, and the operator is used to implement the computing process of the corresponding computing layer; the performance information of the first AI chip includes a first sub-period and a second sub-period of the operator corresponding to the computing layer of the model on the first AI chip. The first sub-period is the period during which the operator in the first AI chip executes the computing process after being scheduled. The second sub-period is the period during which the first AI chip schedules the operator and forwards data required by the operator. The first evaluation result includes a first indicator and a second indicator. The first indicator is used to indicate the performance of the first AI chip in processing the computing process. The second indicator is used to indicate the performance of the first AI chip in processing the scheduling and forwarding processes.

[0058] Based on this, the present application can use the operators included in the AI ​​chip as the granularity to obtain the running time of the operators, the time required for scheduling the operators and forwarding the data required for the operators, and evaluate the indicators of the AI ​​chip and display them according to the running time of the operators and the time required for scheduling the operators and forwarding the data required for the operators. Compared with evaluating the AI ​​chip according to the time of the end-to-end processing process based on the model, the technical solution provided by the present application evaluates the AI ​​chip according to a finer-grained time, achieves a more refined evaluation, and improves the accuracy of the AI ​​chip evaluation. At the same time, the AI ​​chip is evaluated from multiple indicators to achieve performance evaluation results in different dimensions, and intuitively display the performance of the first AI chip, which facilitates the analysis of the performance of the first AI chip in different processing logics, so as to effectively support the optimization of the AI ​​chip.

[0059] The performance evaluation method can be applied to a front-end device in a performance evaluation system. The performance evaluation system provided in the embodiment of the present application is described below.

[0060] Figure 1 Schematic diagram of a performance evaluation system 100 provided in an embodiment of the present application. The performance evaluation system 100 may include a front-end device 10 and a computing device 20. The front-end device 10 and the computing device 20 may be connected via a wired network or a wireless network.

[0061] Optional, Figure 1 The front-end device 10 and the computing device 20 may be functional modules integrated into the same device, or may be two independently configured devices, which is not limited in this application.

[0062] It is easy to understand that when the front-end device 10 and the computing device 20 are functional modules integrated into the same device, the communication between the front-end device 10 and the computing device 20 is based on communication between internal modules of the devices. In this case, the communication process between the two is the same as the communication process between the front-end device 10 and the computing device 20 when they are independently installed.

[0063] For ease of understanding, this application is mainly explained by taking the example of the front-end device 10 and the computing device 20 being independently configured.

[0064] Figure 1 The front-end device 10 can be used to obtain information about the first AI chip and / or the second AI chip during model operation from the computing device 20, analyze and process the information about the first and second AI chips during model operation to obtain fine-grained indicator data, and display the obtained indicator data, thereby supporting intuitive and effective optimization of the AI ​​chips. The model operation process is the end-to-end processing process of the model, which can be a model training process or a model inference process.

[0065] Figure 1 The computing device 20 can be configured with one or more AI chips, such as a first AI chip and a second AI chip, etc. The computing device 20 can collect information about the first AI chip and the second AI chip during the model operation process and send the collected information to the front-end device 10.

[0066] For example, Figure 1 The front-end device 10 in the example can be configured with an input module, a transceiver module, a processing module, a display module, and the like. The input module can be a peripheral device such as a keyboard or a mouse. The transceiver module can be a communication interface, and the like. The display module can be an LCD screen, and the like. The staff can perform an evaluation start operation directed to the first AI chip (for example, clicking a start button) through the input module configured on the front-end device 10. In response to the evaluation start operation, the front-end device 10 can send an information acquisition request to the computing device 20 through the transceiver module to obtain information about the first AI chip during the model operation process. In response to the information acquisition request, the computing device 20 can execute the model operation process on the first AI chip, collect information about the first AI chip during the model operation process, and send the collected information to the front-end device 10. Furthermore, the front-end device 10 can obtain the information about the first AI chip during the model operation process through the transceiver module, and analyze and process the information about the first AI chip during the model operation process into various fine-grained indicator data through the processing module, so as to further display the disassembled indicator data through the display module.

[0067] Optionally, the front-end device 10 and the computing device 20 may be terminals. The terminal may be a personal computer such as a desktop, tablet computer, or laptop computer, or a remote terminal, a user terminal (TE), or a mobile device. This application does not limit the form of the terminal. The device for implementing the function of the terminal may be a terminal, or a device that can support the terminal to implement the function, such as a chip system. The device may be installed in the terminal or used in conjunction with the terminal. In the embodiment of the present application, the chip system may be composed of a chip, or may include a chip and other discrete devices.

[0068] Optionally, the computing device 20 may also be a server. The server may be a single server, or a server cluster consisting of multiple servers. Optionally, a server cluster may also be referred to as a computing device cluster. In some embodiments, the server cluster may also be a distributed cluster. This application does not limit the form of the server.

[0069] It should be noted that Figure 1 What is shown is only an example of the device form of the front-end device 10 and the computing device 20. Figure 1 The performance evaluation system 100 shown is only an example and does not limit the system architecture to which the performance evaluation method provided in the embodiments of the present application is applicable. For example, the performance evaluation system may include multiple front-end devices 10. For another example, the performance evaluation system may include multiple computing devices 20.

[0070] Figure 2 Schematic diagram of the hardware structure of a front-end device 10 provided in an embodiment of the present application. The front-end device 10 may include a processor 1001, a memory 1002, and a transceiver 1003. The processor 1001, the memory 1002, and the transceiver 1003 may be interconnected via a bus 1004.

[0071] The processor 1001 may be one or more CPUs. In the case where the processor 1001 is a CPU, the CPU may be a single-core CPU or a multi-core CPU. The processor 1001 may be used to read information stored in the memory 1002, perform information disassembly operations, etc.

[0072] Memory 1002 may be random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM). Memory 1002 may be used to store information and related indicator data of the first and second AI chips during model execution.

[0073] The transceiver 1003 is configured to receive / send data in response to instructions from the processor 1001 and the memory 1002 .

[0074] The embodiment of the present application does not limit the number of these components. For example, the front-end device 10 may include one or more processors 1001. In another example, the front-end device 10 may include one or more memories 1002.

[0075] The following, combined Figure 1 or Figure 2 The front-end device shown illustrates the performance evaluation method provided in the embodiment of the present application.

[0076] Figure 3 A flow chart of a performance evaluation method provided in an embodiment of the present application. Figure 3 The method shown can be executed by a front-end device of a performance evaluation system. Figure 3 The front-end device in the method shown may specifically include: Figure 2 As shown in the parts. Figure 3 As shown, the performance evaluation method may include: S101-S103.

[0077] S101. A front-end device obtains performance information of a first AI chip.

[0078] The performance information of the first AI chip includes a first sub-period and a second sub-period.

[0079] The first AI chip can provide computing power for model operation. Model operation may include but is not limited to model training and model inference. Model training may include a complete training cycle of forward training, backward training of the initial model, and parameter updates of the model's computational layer. Model inference may refer to the process of using a trained model to input input parameters into the model and obtain output parameters (or output results) through calculations in the model's computational layer.

[0080] In the present application, the operation layer of the model includes one or more. For example, the neural network model may include an input layer, a convolutional layer, a synchronization layer, an output layer, etc. One operation layer of the model corresponds to an operation process. The operation processes corresponding to different operation layers may be the same or different, without limitation. There is a sequential execution logic between the operation layers of the model. A complete operation process of the model may include executing the logic in the order of the operation layers of the model, starting from the first operation layer of the model until the last operation layer of the model ends. The operation of each operation layer of the model can rely on the computing power provided by the AI ​​chip. The AI ​​chip may include an operation operator corresponding to the operation layer, which can implement the operation process of the operation layer. In other words, the model operation process involves a large number of numerical operation processes, and different operation processes require different operation operators to execute. These operation processes can be scheduled to be executed by each first operation operator on the first AI chip, thereby effectively improving the operation efficiency.

[0081] In this application, the model may be an AI model, such as ChatGPT or LLaMA, and the first AI chip may be an AI chip, such as an NPU or GPU. The performance information of the first AI chip may be used to characterize the performance of the first AI chip when providing computing power for the model operation.

[0082] Specifically, the model includes one or more operation layers, the first AI chip includes operation operators corresponding to the operation layers of the model, and the performance parameters of the first AI chip may include a first sub-period and a second sub-period corresponding to each operation operator.

[0083] A first sub-period may include the period during which an operator (or first operator) in the first AI chip is scheduled to execute a computation. Multiple operators correspond to multiple computation layers of a model, and one operator can be used to implement the computation of a corresponding computation layer in the model. That is, the first sub-period is the period during which an operator corresponding to a computation layer in the model is scheduled to execute a computation in the first AI chip.

[0084] A second sub-period may include the period during which the first AI chip schedules an operator and forwards the data required for the operator's execution. In other words, the second sub-period includes the period during which the first AI chip schedules and forwards data during the operator's computation. Specifically, the second sub-period is the period during which the first AI chip schedules the operator and forwards the data required by the operator while the model is running.

[0085] It should be understood that the time period described in this application can be replaced by descriptions of duration, time period, time, duration corresponding to a time period, time length or other names without limitation.

[0086] Optionally, the front-end device obtaining the performance information of the first AI chip may include: the front-end device obtaining the performance information of the first AI chip from a computing device that includes the first AI chip. Alternatively, when the computing device is a functional module deployed on the front-end device, the front-end device may be configured with the first AI chip and an analyzer for analyzing model performance. The analyzer can be used to collect time information and memory consumption during the model execution process. The front-end device may execute the model execution process through the first AI chip, and at the same time, collect the start and end times of the model execution process, as well as the time period during which the calculation operators and communication operators in the first AI chip are scheduled for execution, through the analyzer. In addition, the front-end device may also collect the usage of the memory resources of the first AI chip during the model execution process through the analyzer.

[0087] In a possible embodiment, for a type of model (or called a common model) that requires lower computing power, a single AI chip (such as a first AI chip) can support the implementation of its operation process. That is, a single AI chip can complete the operation process of the common model by scheduling and forwarding between the configured operators. In this case, the duration of the end-to-end operation process of the common model executed by the single AI chip can be considered to be broken down into multiple first sub-periods and multiple second sub-periods. That is, the front-end device can respectively obtain the first sub-period in which each operator in the single AI chip is scheduled to execute the operation process, and the second sub-period in which the single AI chip schedules each operator and forwards the data required for the execution of each operator. Furthermore, the performance of the single AI chip can be analyzed based on the multiple first sub-periods and the multiple second sub-periods.

[0088] In one possible approach, the performance information of the first AI chip may further include a first total period. The first total period may be the period used by the first AI chip to complete the model execution process. In this case, the multiple second sub-periods may be obtained by removing the other sub-periods included in the performance information of the first AI chip from the first total period. For example, the front-end device may collect the start and end times of the model execution process through an analyzer to obtain the first total period. Furthermore, the front-end device may collect the start and end times of each operator in the first AI chip after being scheduled for execution through an analyzer to obtain multiple first sub-periods. Furthermore, the front-end device may remove the multiple first sub-periods from the first total period to obtain multiple second sub-periods.

[0089] In one possible embodiment, for a type of model (or large model) that requires higher computing power, the implementation of its operation process can be supported by multiple AI chips. In this case, the operation process of the large model includes the communication process between different AI chips. For example, the operation operator that generates the communication process in the AI ​​chip can be a broadcast operator, an all-reduce operator, a reduce operator, an all-gather operator, a gather operator, a scatter operator, a reduce-scatter operator, an all-to-all operator, a send operator, a receive operator, a non-blocking send operator, a non-blocking receive operator, a batch non-blocking send and receive operator (batch_isend_irecv) operator, etc., that is, these operators require a communication process during the execution of the operation.

[0090] To facilitate analysis of the performance of the first AI chip executing the communication process, the performance information of the first AI chip may further include multiple third sub-periods. The third sub-period may be the period during which the communication operator in the first AI chip is scheduled to execute the communication process. A communication operator is an operator corresponding to the computational layer of the model, and is used to be scheduled by the first AI chip to implement the communication process with the external world. For example, the front-end device may use an analyzer to collect the start and end times of each communication operator in the first AI chip after being scheduled to execute, thereby obtaining multiple third sub-periods.

[0091] In one possible embodiment, the performance information of the first AI chip may further include the memory resource usage of the first AI chip during model execution (or referred to as the first storage resource usage). The first storage resource usage may include the memory resource usage of the first AI chip at various moments during model execution. For example, the front-end device may use an analyzer to collect information such as the time and size of storage resource allocation in the first AI chip to obtain the memory resource usage of the first AI chip at various moments during model execution.

[0092] S102. The front-end device determines an evaluation result of the first AI chip based on the performance information of the first AI chip.

[0093] The evaluation result of the first AI chip (or referred to as the first evaluation result) may include a first indicator and a second indicator.

[0094] In this application, the first indicator is used to indicate the processing performance of the operation operator in the first AI chip, that is, the performance of the first AI chip in processing the operation process. The first indicator can be determined based on the first sub-period corresponding to each operation operator.

[0095] For example, the first indicator can be the sum of the first sub-periods corresponding to each operator. Another example is that the first indicator can be the average of the first sub-periods corresponding to each operator, i.e., the first indicator is the ratio of the sum of the first sub-periods corresponding to each operator to the number of first sub-periods. Another example is that the first indicator can be the proportion of the sum of the first sub-periods corresponding to each operator in the total running time of the model. The total running time of the model is the first total period.

[0096] In this application, the second indicator is used to indicate the performance of the first AI chip in processing the scheduling and forwarding process. The second indicator can be determined based on the second sub-period corresponding to each operator. For example, the second indicator can be the sum of the second sub-periods corresponding to each operator. For another example, the second indicator can be the average of the second sub-periods corresponding to each operator, that is, the ratio between the sum of the second sub-periods corresponding to each operator and the number of second sub-periods. For another example, the second indicator can also be the proportion of the sum of the second sub-periods corresponding to each operator in the total running time of the model.

[0097] In one possible embodiment, for large models requiring high computing power, the first evaluation result may further include a third indicator. The third indicator may be used to indicate the performance of the first AI chip in processing the communication process. Optionally, the third indicator may be the sum of the third sub-period corresponding to the operator that triggers the communication process, or the mean of the third sub-period corresponding to the operator that triggers the communication process.

[0098] It should be understood that the triggering communication process described in this application can replace the description of the generating communication process, without limitation. The operator that triggers the communication process can refer to an operator whose processing result needs to be sent to other AI chips for processing through the communication process.

[0099] One possible way is Figure 4As shown, the first AI chip may perform the communication process and the calculation process simultaneously, that is, there may be overlapping periods and non-overlapping periods between the third sub-period and the first sub-period. For example, assume that the first sub-period for operator A to perform the calculation process may be 10:00-10:05, and the third sub-period corresponding to the communication process triggered by operator A is 10:02-10:07, and the first sub-period for operator B to perform the calculation process is 10:06-10:07. The overlapping periods are 10:02-10:05 and 10:06-10:07, and the non-overlapping period is 10:05-10:06. During the overlapping periods, the first AI chip can still perform calculations, and the speed at which the first AI chip processes the calculation process is not affected. The non-overlapping period is the period during which the first AI chip waits for the data to be processed to complete the interaction, that is, the idle period during which the first AI chip is not performing calculations. Therefore, the length of the non-overlapping period may affect the speed at which the first AI chip processes the calculation process.

[0100] In this case, the third indicator can also be the sum of the non-overlapping time periods to indicate the performance of the first AI chip in processing the communication process. The front-end device can then determine the sum of the third sub-periods corresponding to the operators that trigger the communication process as the first duration, and the sum of the overlapping periods between the first sub-periods corresponding to each operator and the third sub-periods corresponding to each operator as the second duration, and then the difference between the first duration and the second duration can be determined as the third indicator. Alternatively, the third indicator can also be the proportion of the sum of the non-overlapping time periods in the total running time of the model.

[0101] Optionally, the operation processes / operation types / operation capabilities of different operators may be the same / similar, or different. In this application, operators may be classified based on their operation processes into multiple categories of operators. Operators of the same category have the same / similar computing capabilities. The first evaluation result also includes indicators corresponding to each of the multiple categories of operators. The indicators corresponding to each category of operators can be used to characterize the processing performance of each category of operators, that is, the performance of operators of the same category with the same / similar computing capabilities.

[0102] In one possible embodiment, multiple types of operators may include, but are not limited to, matrix operators, vector operators, and fusion operators. Matrix operators may be operators used to perform matrix operations, i.e., operators with a matrix operation type. Matrix operations include various operations such as addition, subtraction, multiplication, transposition, and multiplication performed on matrices. For example, matrix operators may include the matrix multiplication (Matmul) operator and the batch matrix multiplication (BatchMatmul) operator. Vector operators may be operators used to perform vector operations, i.e., operators with a vector operation type. Vector operations include various operations such as addition, subtraction, multiplication, scalar product, and cross product performed on vectors. Fusion operators may be operators used to perform fusion operations, i.e., operators with a fusion operation type. Fusion operations can fuse different matrix operators or different vector operators to reduce memory usage during the operation process and the execution time of the assigned operation tasks.

[0103] The indicators corresponding to each of the multiple types of operators include at least one of the following: a fourth indicator, a fifth indicator, and a sixth indicator. For example, the front-end device may divide the first sub-period corresponding to each operator into a first sub-period corresponding to each matrix operator, a first sub-period corresponding to each vector operator, and a first sub-period corresponding to each fusion operator, based on the operation type of the operator corresponding to each first sub-period. Furthermore, the front-end device may determine the fourth indicator, and / or the fifth indicator, and / or the sixth indicator based on the division results.

[0104] The fourth indicator can be used to indicate the processing performance of the matrix operation operator in the first AI chip, that is, the matrix computing power of the first AI chip. The fourth indicator can be determined based on the first sub-periods corresponding to each matrix operation operator. For example, the fourth indicator can be the sum of the first sub-periods corresponding to each matrix operation operator. For another example, the fourth indicator can be the average of the first sub-periods corresponding to each matrix operation operator. For another example, the fourth indicator can also be the proportion of the sum of the first sub-periods corresponding to each matrix operation operator in the total running time of the model. For another example, the fourth indicator can also be the number of first sub-periods corresponding to the matrix operation operator.

[0105] The fifth indicator is used to indicate the processing performance of the vector operation operator in the first AI chip, that is, the vector computing power of the first AI chip. The fifth indicator can be determined based on the first sub-periods corresponding to each vector operation operator. For example, the fifth indicator can be the sum of the first sub-periods corresponding to each vector operation operator. For another example, the fifth indicator can be the mean of the first sub-periods corresponding to each vector operation operator. For another example, the fifth indicator can also be the proportion of the sum of the first sub-periods corresponding to each vector operation operator in the total running time of the model. For another example, the fifth indicator can also be the number of first sub-periods corresponding to the vector operation operator, etc.

[0106] The sixth indicator is used to indicate the processing performance of the fusion operation operator in the first AI chip. The sixth indicator can be determined based on the first sub-periods corresponding to each fusion operation operator. For example, the sixth indicator is the sum of the first sub-periods corresponding to each fusion operation operator. For another example, the sixth indicator can be the average of the first sub-periods corresponding to each fusion operation operator. For another example, the sixth indicator can also be the proportion of the sum of the first sub-periods corresponding to each fusion operation operator in the total running time of the model. For another example, the sixth indicator can also be the number of first sub-periods corresponding to the fusion operation operator. Since the fusion operation operator is used to fuse different matrix operation operators or different vector operation operators to reduce the memory usage of the operation process and the execution time of the operation task, the sixth indicator can effectively reflect whether the design and integration of the first AI chip need further optimization, thereby helping to improve the operation performance and efficiency.

[0107] In one possible approach, when the model running process is a model training process, the sixth indicator can also be split into a first sub-indicator and a second sub-indicator.

[0108] The first sub-indicator is used to indicate the processing performance of the fusion operation operator in the first AI chip during the forward training process. The first sub-indicator can be determined based on the first sub-period corresponding to each fusion operation operator during the forward training process. For example, the front-end device can collect the start and end times of the forward training process during the model operation through an analyzer to obtain the period corresponding to the forward training process. Furthermore, the front-end device can determine the first sub-indicator based on the first sub-period corresponding to the fusion operation operator within the period corresponding to the forward training process. For example, the first sub-indicator is the sum of the first sub-periods corresponding to each fusion operation operator during the forward training process. For another example, the first sub-indicator can be the average of the first sub-periods corresponding to each fusion operation operator during the forward training process. For another example, the first sub-indicator can also be the proportion of the sum of the first sub-periods corresponding to each fusion operation operator during the forward training process in the total running time of the model. For another example, the first sub-indicator can also be the number of first sub-periods corresponding to the fusion operation operator during the forward training process, etc.

[0109] The second sub-indicator is used to indicate the processing performance of the fusion operator in the first AI chip during the reverse training process. The second sub-indicator can be determined based on the first sub-period corresponding to each fusion operator during the reverse training process. Specifically, the determination method of the first sub-indicator can be referred to and will not be repeated here.

[0110] In one possible embodiment, the first evaluation result may further include a seventh indicator. The seventh indicator is used to indicate the storage performance of the first AI chip. The seventh indicator may be determined based on the memory resource usage of the first AI chip during model execution. For example, the seventh indicator may be the maximum memory resource usage of the first AI chip at various times during model execution, or the average memory resource usage of the first AI chip at various times during model execution.

[0111] In one possible embodiment, in order to accurately demonstrate the performance of the first AI chip during model operation, Figure 5 As shown, the first evaluation result can simultaneously include the first total time period, and the seventh indicator corresponding to the usage of storage resources, as well as the first indicator corresponding to the operation process, the second indicator corresponding to the scheduling and forwarding process, and the third indicator corresponding to the communication process based on the first total time period, and the fourth indicator corresponding to the matrix operation, the fifth indicator corresponding to the vector operation, and the sixth indicator corresponding to the fusion operation further divided based on the first sub-time period corresponding to each operation operator.

[0112] S103. The front-end device displays the evaluation result of the first AI chip.

[0113] In one possible example, the front-end device may display the evaluation results of the first AI chip in the manner shown in Table 2 below, where the unit of duration is seconds (s) and the unit of storage resources is gigabytes (GB).

[0114] Table 2

[0115]

[0116] In one possible example, after the front-end device collects the usage of the first AI chip's memory resources (i.e., storage resources) during the model operation process through the analyzer, it can also display the collected indicator information in conjunction with the visualization tool TensorBoard. Figure 6 The diagram of the indicator display interface shown in Figure 1 shows the peak memory usage of the first AI chip during the model operation. The vertical axis is used to indicate the memory usage in megabytes (MB). The horizontal axis is used to indicate the time in milliseconds (MS). Figure 6 As shown in the figure, the peak memory usage is 63756.9MB. For another example, the front-end device can Figure 7 The diagram of the indicator display interface shown in Figure 1 shows the memory usage of the first AI chip during the model operation. The vertical axis is used to indicate the memory usage in MB. The horizontal axis is used to indicate the time. Figure 7As shown, the peak memory usage is 651.43MB.

[0117] Based on the description of S101-S103 above, it can be seen that the front-end device can obtain performance information of the first AI chip in multiple dimensions. For example, the running time of each operator in the first AI chip, the time it takes for the first AI chip to schedule operators and forward data required for operators, the time it takes for the first AI chip to execute communication processes triggered by operators, and the amount of memory resources used by the first AI chip during model execution. Furthermore, the front-end device can evaluate and display the first AI chip's performance information based on these multiple dimensions. Compared to evaluating AI chips based on the duration of the end-to-end processing of the model, the technical solution provided in this application evaluates AI chips based on more granular performance information, achieving a more refined evaluation and improving the accuracy of AI chip evaluation. Furthermore, by evaluating AI chips based on multiple indicators, performance evaluation results from different dimensions are achieved, and the performance of the first AI chip is intuitively displayed, facilitating analysis of the performance of the first AI chip in different processing logics, effectively supporting AI chip optimization.

[0118] In one embodiment, Figure 8 As shown, an embodiment of the present application provides a flow chart of another performance evaluation method. Figure 8 The method shown in can support fine-grained performance comparison between two different chips to accurately reflect the advantages and disadvantages of the chips. Figure 8 As shown, the performance evaluation method may include: S201-S203.

[0119] S201. The front-end device obtains performance information of the second AI chip.

[0120] The second AI chip can provide computing power for model execution. The performance information of the second AI chip can be used to characterize the performance of the second AI chip when providing computing power for model execution. Furthermore, to accurately compare the performance of the first and second AI chips, the model execution process executed by the second AI chip can be identical to the model execution process executed by the first AI chip. For example, the front-end device can run the same model on the first and second AI chips to process the same input, and collect the performance information of the first and second AI chips respectively through the analyzer.

[0121] Optionally, the second AI chip and the first AI chip can be two different AI chips. For example, the first AI chip can be a GPU, and the second AI chip can be an NPU. Alternatively, the second AI chip and the first AI chip can be two different versions of the same AI chip. For example, the first AI chip can be version 1.0 of the NPU, and the second AI chip can be version 2.0 of the NPU.

[0122] The performance information of the second AI chip may include at least one of the following: a second total period, a fourth sub-period, a fifth sub-period, and an amount of second storage resources used by the second AI chip during model execution.

[0123] The second total period may be the period used by the second AI chip to complete the model running process.

[0124] A fourth sub-period may include the period during which an operator (or second operator) in the second AI chip is scheduled to execute a computation. In other words, a fourth sub-period may be the period during which an operator corresponding to a computation layer of the model in the second AI chip is scheduled to execute a computation. Multiple second operators correspond to multiple computation layers of the model; a second operator can be used to implement the computation of a corresponding computation layer in the model and can be scheduled by the second AI chip to implement the computation in the model.

[0125] A fifth sub-period may be a period during which a communication operator in the second AI chip is scheduled to perform a communication process. The communication operator is an operator corresponding to the operation layer of the model, and is used to be scheduled by the second AI chip to implement a communication process with the outside world.

[0126] The second storage resource amount refers to the amount of memory resources used by the second AI chip during the model running process.

[0127] Optionally, the manner in which the front-end device obtains the performance information of the second AI chip can refer to the detailed description of the front-end device obtaining the performance information of the first AI chip in S101 above, and will not be repeated here.

[0128] In one possible approach, the performance information of the second AI chip may further include multiple sixth sub-periods. A sixth sub-period may include a period during which the second AI chip schedules an operator and forwards data required for the execution of the operator. In other words, the second sub-period includes a period during which the second AI chip schedules and forwards data during the operation of the operator. That is, the period during which the second AI chip schedules the second operator and forwards data that needs to be processed by the second operator. For example, the front-end device can collect the idle period of the second AI chip during the model operation process through an analyzer to obtain multiple sixth sub-periods. The idle period is the period during which neither the second operator nor the communication operator in the second AI chip is executed. Alternatively, the front-end device can also remove multiple fourth sub-periods and multiple fifth sub-periods from the second total period to obtain multiple sixth sub-periods.

[0129] S202. The front-end device determines an evaluation result of the second AI chip based on the performance information of the second AI chip.

[0130] The evaluation result of the second AI chip may include an eighth indicator corresponding to the sixth indicator. The eighth indicator may be used to indicate the processing performance of the fusion operation operator in the second AI chip.

[0131] In one possible way, the front-end device can determine the eighth indicator based on the fourth sub-period corresponding to each second operation operator whose operation type is fusion operation by the same calculation method as the sixth indicator. For example, the eighth indicator can be the sum of the fourth sub-periods corresponding to each fusion operation operator in the second AI chip. For another example, the eighth indicator can be the average of the fourth sub-periods corresponding to each fusion operation operator in the second AI chip. For another example, the eighth indicator can also be the proportion of the sum of the fourth sub-periods corresponding to each fusion operation operator in the second AI chip in the total running time of the model (i.e., the second total period). For another example, the eighth indicator can also be the number of fourth sub-periods corresponding to the fusion operation operator in the second AI chip, etc.

[0132] In one possible approach, to accurately compare the fusion operation performance of the first and second AI chips, the computational process implemented by the fusion operator corresponding to the fourth sub-period used to determine the eighth metric can be the same as the computational process implemented by the fusion operator corresponding to the first sub-period used to determine the sixth metric. That is, the computational process implemented by the fusion operator in the second AI chip is the same as that implemented by the fusion operator in the first AI chip. The front-end device can then determine, from all the operators included in the second AI chip, individual second operators that have the same computational process as the individual fusion operators in the first AI chip, and determine the eighth metric based on the fourth sub-periods corresponding to each of the identified second operators.

[0133] For example, considering that different AI chips may have different definitions of the operation process of the operator, a mapping relationship table can be pre-established in the front-end device. The mapping relationship table may include the correspondence between the first operator and the second operator that process the same operation process. For example, if the operation process implemented by a first operator is the same as the operation process implemented by two second operators, then there is a correspondence between the first operator and the two second operators. If the operation process implemented by three first operators is the same as the operation process implemented by one second operator, then there is a correspondence between the three first operators and the second operator.

[0134] For another example, a first AI chip may include a fusion operator A for performing a scaling operation, a fusion operator B for performing a masking operation, and a fusion operator C for performing a normalization operation. A second AI chip may include a fusion operator D for performing scaling, masking, and normalization operations. That is, the computational process performed by fusion operator D is identical to the computational process performed by fusion operators A, B, and C. Thus, there is a correspondence between fusion operators A, B, and C and fusion operator D. Furthermore, when the front-end device determines the sixth indicator of the first AI chip based on the first sub-periods corresponding to fusion operators A, B, and C, it can determine the eighth indicator of the second AI chip based on the fourth sub-period corresponding to fusion operator D. This allows for an accurate comparative evaluation of the processing performance of the fusion operators in the first and second AI chips based on the sixth and eighth indicators.

[0135] In one possible embodiment, the evaluation result of the second AI chip may also include at least one of the following: a ninth indicator corresponding to the first indicator, a tenth indicator corresponding to the second indicator, an eleventh indicator corresponding to the third indicator, a twelfth indicator corresponding to the fourth indicator, a thirteenth indicator corresponding to the fifth indicator, and a fourteenth indicator corresponding to the seventh indicator.

[0136] In one possible approach, to accurately compare the performance of the first and second AI chips in processing fusion operations, the front-end device may determine a ninth indicator based on multiple fourth sub-periods using the same calculation method as the first indicator. Furthermore, the front-end device may determine a tenth indicator based on multiple sixth sub-periods using the same calculation method as the second indicator. Furthermore, the front-end device may determine an eleventh indicator based on multiple fifth sub-periods using the same calculation method as the third indicator. Furthermore, the front-end device may determine a twelfth indicator based on the fourth sub-period corresponding to each second operator whose operation type is a matrix operation using the same calculation method as the fourth indicator. Furthermore, the front-end device may determine a thirteenth indicator based on the fourth sub-period corresponding to each second operator whose operation type is a vector operation using the same calculation method as the fifth indicator. Furthermore, the front-end device may determine a fourteenth indicator based on the second storage resource amount using the same calculation method as the seventh indicator. The calculation method for each indicator in the evaluation result of the second AI chip can be found in the description of S102 and the eighth indicator above and will not be repeated here.

[0137] S203. The front-end device displays the evaluation result of the second AI chip.

[0138] In one possible example, the front-end device may display the evaluation results of the second AI chip while displaying the evaluation results of the first AI chip in the manner shown in Table 3 below.

[0139] Table 3

[0140]

[0141] As shown in Table 3, the performance indicators of the first and second AI chips in multiple dimensions, such as processing operations, scheduling and forwarding, communication, matrix operations, vector operations, fusion operations, storage resource usage, and total operating time, can effectively support comparative evaluation. For example, the performance of the first AI chip in processing communication and fusion operations is significantly different from that of the second AI chip. Therefore, it is necessary to optimize the communication operator in the first AI chip and the second operation operator whose operation type is fusion operation to improve the performance of the first AI chip.

[0142] Based on this, for the same model running process, the front-end device can respectively obtain the performance information of the second AI chip and the first AI chip in the same multiple dimensions, and then determine the evaluation results of the second AI chip corresponding to the evaluation results of the first AI chip, so as to support accurate comparative analysis of the performance of the first AI chip and the second AI chip, realize fine-grained performance comparison between different AI chips, facilitate accurate analysis of the advantages and disadvantages of different AI chips, and thus effectively support the optimization of AI chip performance.

[0143] Furthermore, based on the above Figure 3 and Figure 8 The method process can be conveniently coded to realize process toolization, so as to support the efficient determination of the performance evaluation results of different AI chips after collecting information of different AI chips during the model operation process, realize accurate and fine-grained performance comparison, and reduce the difficulty of performance comparison analysis between different AI chips.

[0144] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of method. In order to realize the above functions, it includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily appreciate that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0145] The embodiment of the present application can divide the functional modules of the front-end device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. In actual implementation, there may be other division methods.

[0146] Combined with the above Figures 3 to 8 , describes in detail the performance evaluation method provided by the embodiment of the present application, and will be combined with Figure 9 , describes a performance evaluation device provided according to an embodiment of the present application.

[0147] Figure 9 The performance evaluation device 30 is a schematic diagram of a structure of a performance evaluation device 30 provided in an embodiment of the present application. The performance evaluation device 30 can be used to perform Figure 3 or Figure 8 The performance evaluation method shown. The performance evaluation device 30 can be applied to the front-end device in the performance evaluation system. The performance evaluation device 30 can include: an acquisition module 3001, a processing module 3002 and a display module 3003. For example, in combination with Figure 3 , the acquisition module 3001 can be used to execute S101, the processing module 3002 can be used to execute S102, and the display module 3003 can be used to execute S103. Figure 8 The acquisition module 3001 can be used to execute S201, the processing module 3002 can be used to execute S202, and the display module 3003 can be used to execute S203.

[0148] Acquisition module 3001 is configured to acquire performance information of a first AI chip; the first AI chip is configured to provide computing power for model execution, and the performance information of the first AI chip includes a first sub-period and a second sub-period. The first sub-period is the period during which operators corresponding to the computation layer of the model in the first AI chip are scheduled to execute computations, and the second sub-period is the period during which the first AI chip schedules operators and forwards data required by the operators during model execution.

[0149] Processing module 3002 is configured to determine an evaluation result of the first AI chip based on the performance information of the first AI chip acquired by acquisition module 3001. The evaluation result of the first AI chip includes a first indicator and a second indicator. The first indicator indicates the performance of the first AI chip in processing operations. The second indicator indicates the performance of the first AI chip in processing scheduling and forwarding.

[0150] The display module 3003 is configured to display the evaluation result of the first AI chip determined by the processing module 3002 .

[0151] In a possible implementation, the first indicator is the sum of the first sub-periods; and the second indicator is the sum of the second sub-periods.

[0152] In one possible implementation, the performance information of the first AI chip also includes a third sub-period, which is the period during which a communication operator in the first AI chip is scheduled to execute a communication process. The communication operator is an operator corresponding to the operation layer of the model, which is used to be scheduled by the first AI chip to implement a communication process with the outside world. The first evaluation result also includes a third indicator, which is the difference between the first duration and the second duration. The first duration is the sum of each third sub-period, and the second duration is the sum of the overlapping periods between the first sub-period and the third sub-period.

[0153] In one possible implementation, the performance information of the first AI chip also includes a first total period, which is the period used by the first AI chip to complete the model operation process; the second sub-period is obtained by removing other sub-periods included in the performance information of the first AI chip from the first total period.

[0154] In one possible implementation, the evaluation results of the first AI chip also include indicators corresponding to multiple types of operators; the indicators corresponding to each type of operator are used to characterize the processing performance of each type of operator.

[0155] In one possible implementation, the indicators corresponding to each of the multiple types of operators include at least one of the following: a fourth indicator, a fifth indicator, and a sixth indicator; the fourth indicator is used to indicate the performance of the first AI chip in processing matrix operations, the fifth indicator is used to indicate the performance of the first AI chip in processing vector operations, and the sixth indicator is used to indicate the performance of the first AI chip in processing fusion operations; the fourth indicator includes the sum of the first sub-time periods corresponding to matrix operation operators whose operation type is matrix operation; the fifth indicator includes the sum of the first sub-time periods corresponding to vector operation operators whose operation type is vector operation; and the sixth indicator includes the sum of the first sub-time periods corresponding to fusion operation operators whose operation type is fusion operation.

[0156] In one possible implementation, the performance information of the first AI chip also includes the amount of first storage resources used by the first AI chip during model execution; the evaluation result of the first AI chip also includes a seventh indicator corresponding to the first storage resource amount.

[0157] In one possible implementation, acquisition module 3001 is further configured to acquire performance information of a second AI chip; the performance information of the second AI chip includes at least one of the following: a second total period, a fourth sub-period, a fifth sub-period, and an amount of second storage resources used by the second AI chip during model execution; the second total period is the period used by the second AI chip to complete the model execution; the fourth sub-period is the period during which an operator corresponding to the computation layer of the model in the second AI chip is scheduled to execute the computation; and the fifth sub-period is the period during which a communication operator in the second AI chip is scheduled to execute the communication. A communication operator is an operator corresponding to the computation layer of the model that is scheduled by the second AI chip to implement communication with the outside world.

[0158] The processing module 3002 is further configured to determine an evaluation result of the second AI chip based on the performance information of the second AI chip acquired by the acquisition module 3001;

[0159] The display module 3003 is further configured to display the evaluation result of the second AI chip determined by the processing module 3002 .

[0160] In one possible implementation, the evaluation result of the second AI chip includes an eighth indicator; the eighth indicator corresponds to the sixth indicator, and the eighth indicator is the sum of the fourth sub-time period corresponding to the fusion operation operator in the second AI chip; the operation process implemented by the fusion operation operator in the second AI chip is the same as that implemented by the fusion operation operator in the first AI chip.

[0161] The explanation of the relevant contents and description of the beneficial effects in this embodiment can refer to the above method embodiment, which will not be repeated here. Figure 1 , the performance evaluation device 30 in this embodiment can be Figure 1 The front-end device 10 in the embodiment of the present invention can be used to implement the acquisition module 3001, the processing module 3002 and the display module 3003. Figure 1 The functions of the front-end device 10 shown in FIG.

[0162] As an example, combining Figure 2 , the performance evaluation device 30 in this embodiment can be Figure 2 In the front-end device 10, part or all of the above-mentioned acquisition module 3001, processing module 3002 and display module 3003 can also be implemented by the processor 2001.

[0163] It should be understood that the performance evaluation device 30 of the embodiment of the present application can be implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), and the PLD can be a complex programmable logical device (CPLD), a field programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof. Figure 3 or Figure 8 When the performance evaluation method is shown, the performance evaluation device 30 and its various modules may also be software modules.

[0164] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using a software program, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer execution instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more media that can be integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium (eg, a solid state disk (SSD)).

[0165] The above is only a specific embodiment of the present application. Those skilled in the art may conceive of changes or substitutions based on the specific embodiment provided in this application, and all such changes or substitutions shall fall within the scope of protection of this application.

Claims

1. A performance evaluation method, characterized in that: include: Obtaining performance information of a first AI chip; the first AI chip is used to provide computing power for model execution, the performance information of the first AI chip including a first sub-period and a second sub-period, the first sub-period being a period during which an operator corresponding to a computing layer of the model in the first AI chip is scheduled to perform a computing process, and the second sub-period being a period during which the first AI chip schedules the operator and forwards data required by the operator when the model is running; Determining an evaluation result of the first AI chip based on the performance information of the first AI chip; the evaluation result of the first AI chip includes a first indicator and a second indicator; the first indicator is used to indicate the performance of the first AI chip in processing a calculation process; the second indicator is used to indicate the performance of the first AI chip in processing a scheduling and forwarding process; Display the evaluation results of the first AI chip.

2. The performance evaluation method according to claim 1, wherein: The first indicator is the sum of the first sub-period; the second indicator is the sum of the second sub-period.

3. The performance evaluation method according to claim 1 or 2, characterized in that: The performance information of the first AI chip further includes a third sub-period, where the third sub-period is a period during which a communication operator in the first AI chip is scheduled to perform a communication process. The communication operator is an operator corresponding to the operation layer of the model and is scheduled by the first AI chip to implement a communication process with the outside world. The evaluation result of the first AI chip also includes a third indicator, which is the difference between the first duration and the second duration, the first duration is the sum of the third sub-periods, and the second duration is the sum of the overlapping periods between the first sub-period and the third sub-period.

4. The performance evaluation method according to any one of claims 1 to 3, characterized in that: The performance information of the first AI chip also includes a first total period, which is the period used by the first AI chip to complete the model operation process; the second sub-period is obtained by removing other sub-periods included in the performance information of the first AI chip from the first total period.

5. The performance evaluation method according to any one of claims 1 to 3, characterized in that: The evaluation results of the first AI chip also include indicators corresponding to multiple types of operators; the indicators corresponding to each type of operator are used to characterize the processing performance of each type of operator.

6. The performance evaluation method according to claim 5, characterized in that: The indicators corresponding to each of the multiple types of operation operators include at least one of the following: a fourth indicator, a fifth indicator, and a sixth indicator; The fourth indicator is used to indicate the performance of the first AI chip in processing matrix operations, the fifth indicator is used to indicate the performance of the first AI chip in processing vector operations, and the sixth indicator is used to indicate the performance of the first AI chip in processing fusion operations; The fourth indicator includes the sum of the first sub-periods corresponding to the matrix operation operator whose operation type is matrix operation; the fifth indicator includes the sum of the first sub-periods corresponding to the vector operation operator whose operation type is vector operation; the sixth indicator includes the sum of the first sub-periods corresponding to the fusion operation operator whose operation type is fusion operation.

7. The performance evaluation method according to any one of claims 1 to 6, characterized in that: The performance information of the first AI chip also includes the amount of first storage resources used by the first AI chip during the operation of the model; the evaluation result of the first AI chip also includes a seventh indicator corresponding to the first storage resource amount.

8. The performance evaluation method according to any one of claims 1 to 7, characterized in that: Also includes: Acquiring performance information of a second AI chip; the performance information of the second AI chip includes at least one of the following: a second total time period, a fourth sub-time period, a fifth sub-time period, and an amount of second storage resources used by the second AI chip during the model operation process; the second total time period is the time period used by the second AI chip to complete the model operation process; the fourth sub-time period is the time period during which an operator corresponding to the operation layer of the model in the second AI chip is scheduled to perform the operation process; and the fifth sub-time period is the time period during which a communication operator in the second AI chip is scheduled to perform the communication process. The communication operator is an operator corresponding to the operation layer of the model, which is used to be scheduled by the second AI chip to implement a communication process with the outside world; determining an evaluation result of the second AI chip based on the performance information of the second AI chip; Display the evaluation results of the second AI chip.

9. The performance evaluation method according to claim 8, characterized in that: The evaluation result of the second AI chip includes an eighth indicator; The eighth indicator corresponds to the sixth indicator, and the eighth indicator is the sum of the fourth sub-time period corresponding to the fusion operation operator in the second AI chip; the fusion operation operator in the second AI chip and the fusion operation operator in the first AI chip implement the same operation process.

10. A performance evaluation device, characterized in that: The device includes: an acquisition module, a processing module and a display module; The acquisition module is configured to acquire performance information of a first AI chip; the first AI chip is configured to provide computing power for model execution, the performance information of the first AI chip including a first sub-period and a second sub-period, the first sub-period being a period during which an operator corresponding to a computing layer of the model in the first AI chip is scheduled to execute a computing process, and the second sub-period being a period during which the first AI chip schedules the operator and forwards data required by the operator during model execution; the processing module being configured to determine an evaluation result of the first AI chip based on the performance information of the first AI chip acquired by the acquisition module; the evaluation result of the first AI chip including a first indicator and a second indicator; the first indicator being configured to indicate the performance of the first AI chip in processing a calculation process; and the second indicator being configured to indicate the performance of the first AI chip in processing a scheduling and forwarding process. The display module is configured to display the evaluation result of the first AI chip determined by the processing module.

11. A computing device cluster, characterized in that: comprising at least one computing device, each computing device including a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in a memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 9.

12. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device cluster, the computing device cluster is caused to perform the method according to any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that The method comprises computer program instructions, which, when executed by a computing device cluster, cause the computing device cluster to perform the method according to any one of claims 1 to 9.

Images