Modulation efficiency test method, system, device, equipment, medium and product

Abstract

The invention relates to the technical field of artificial intelligence, and particularly discloses a modular computation efficiency testing method, system, device and equipment, a medium and a product. Determining a model performance parameter, a model data throughput parameter and a calculation performance parameter according to the model calculation task, and determining the model calculation efficiency of the tested model calculation system according to the model performance parameter, the model data throughput parameter and the calculation performance parameter, so as to carry out quantitative evaluation on the model performance improvement effect of the calculation power improvement. Therefore, quantitative evaluation of the model calculation performance of the tested model calculation system comprising the tested model, the tested software framework and the tested equipment is realized. As an evaluation scheme suitable for various model calculation tasks, the comparability indexes of different models running on different devices are enhanced, and a reference basis is provided for improvement and perfection of the model and a model calculation system.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, system, device, equipment, medium and product for testing analog computing efficiency. Background Art

[0002] With the rapid development of artificial intelligence technology, artificial intelligence models are showing trends such as large data processing volume and high computational complexity. The computing power requirements of model computing tasks for hardware devices are rapidly increasing, and a large amount of resources are consumed. At present, the evaluation indicators for model computing performance are usually performance evaluation indicators for the model itself, such as model accuracy. The relevant technology lacks evaluation indicators for the model computing system composed of software and hardware used to perform model calculations, as well as performance testing methods for the model computing system.

[0003] How to test and quantitatively evaluate the model computing performance of a model computing system is a technical problem that needs to be solved by those skilled in the art. Summary of the invention

[0004] The purpose of the present invention is to provide a model calculation efficiency testing method, system, device, equipment, medium and product for testing and evaluating the model calculation performance of the model calculation system.

[0005] In order to solve the above technical problems, the present invention provides a modular efficiency testing method, comprising: Get the dataset; Controlling the tested device to run the tested software framework to call the tested model and the data set to perform model calculation tasks; Determining model performance parameters, model data throughput parameters and computing performance parameters according to the model computing task; Determine the model computing efficiency of the tested model computing system according to the model performance parameter, the model data throughput parameter and the computing performance parameter to obtain a test result of the tested model computing system; The model under test computing system includes the model under test, the software framework under test and the device under test.

[0006] On the one hand, determining the model computing efficiency of the tested model computing system according to the model performance parameter, the model data throughput parameter and the computing performance parameter includes: Determining an effective throughput evaluation parameter of the tested model computing system according to the model performance parameter and the model data throughput parameter; The modular calculation efficiency is determined according to the effective throughput evaluation parameter and the calculation performance parameter.

[0007] On the other hand, the larger the effective throughput corresponding to the effective throughput evaluation parameter is, the greater the modular efficiency is.

[0008] On the other hand, the model data throughput parameter includes a unit model data calculation amount; The unit model data calculation amount is the calculation amount required for the tested model to process unit input data; The better the model performance corresponding to the model performance parameter, the larger the effective throughput evaluation parameter; the effective throughput evaluation parameter is negatively correlated with the unit model data calculation amount.

[0009] On the other hand, the model under test is a language model, and the unit model data calculation amount is the calculation amount required for the model under test to process each input tag.

[0010] On the other hand, the model under test is an image processing model, and the unit model data calculation amount is the calculation amount required for the model under test to process each input image.

[0011] On the other hand, the model data throughput parameter includes a throughput rate; The throughput rate is the rate at which the tested model processes input data per unit time; The better the model performance corresponding to the model performance parameter is, the larger the effective throughput evaluation parameter is; and the effective throughput evaluation parameter is positively correlated with the throughput rate.

[0012] On the other hand, the model under test is a language model, and the throughput rate is the number of input tokens processed by the model under test per unit time.

[0013] On the other hand, the model under test is an image processing model, and the throughput rate is the number of input images processed by the model under test per unit time.

[0014] On the other hand, determining the model computing efficiency of the tested model computing system according to the model performance parameter, the model data throughput parameter and the computing performance parameter includes: If the model performance parameter is within the range of the benchmark model performance parameter, determining the model calculation efficiency according to the model performance parameter, the model data throughput parameter and the calculation performance parameter; If the model performance parameter is not within the range of the benchmark model performance parameter, the model performance parameter is set to 0, and the modular efficiency is determined based on the model performance parameter, the model data throughput parameter and the computing performance parameter.

[0015] On the other hand, the model performance parameter is the model accuracy of the tested model, and the model accuracy is determined by the following formula: ; The model measured accuracy is the measured accuracy of the tested model in the model calculation task, and the benchmark accuracy is the minimum allowable accuracy of the tested model.

[0016] In another aspect, the computing performance parameter includes computing efficiency; The computing efficiency is positively correlated with the measured computing performance parameters of the model computing system under test for executing the model computing task, and the computing efficiency is negatively correlated with the theoretical peak performance parameters of the computing system of the device under test; The computing performance corresponding to the computing performance parameter is positively correlated with the computing efficiency.

[0017] On the other hand, the step of determining the measured calculation performance parameter includes: Obtaining the total measured amount of calculations performed by the tested model computing system in executing the model computing task and the total measured time consumed by the tested model computing system in executing the model computing task; The measured calculation performance parameter is determined according to the measured total calculation amount and the measured total time consumption.

[0018] On the other hand, the measured computing performance parameter is the number of floating-point operations per second of the tested device executing the model computing task; The computing system theoretical peak performance parameter is the theoretical peak number of floating-point operations per second of the device under test.

[0019] On the other hand, the step of determining the measured computing performance parameter and the step of determining the theoretical peak performance parameter of the computing system include: Obtaining a theoretical peak performance parameter of the computing system corresponding to a first data type of the device under test; If the actual data types in the model calculation tasks are all the first data types, the calculation performance parameters corresponding to the actual data types in the model calculation tasks are used as the measured calculation performance parameters; If the model calculation task contains an actual data type that is not the first data type, the calculation performance parameters corresponding to the actual data type in the model calculation task are equated with the calculation performance parameters corresponding to the first data type to obtain the measured calculation performance parameters.

[0020] On the other hand, the step of determining the measured computing performance parameter and the step of determining the theoretical peak performance parameter of the computing system include: Obtaining a theoretical peak performance parameter of the computing system corresponding to a second data type of the device under test; If the actual data types in the model calculation task are all the second data type, the calculation performance parameters corresponding to the actual data types in the model calculation task are used as the measured calculation performance parameters; If the actual data type that is not the second data type exists in the model calculation task, the calculation performance parameters corresponding to the actual data type in the model calculation task are converted into the calculation performance parameters corresponding to the second data type to obtain the measured calculation performance parameters.

[0021] On the other hand, the model computing efficiency of the tested model computing system is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, and is calculated by the following formula: ; Among them, the model accuracy is the model performance parameter; the unit model data calculation amount is the calculation amount required for the tested model to process unit input data; and the calculation efficiency is the calculation performance parameter.

[0022] On the other hand, the model computing efficiency of the tested model computing system is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, and is calculated by the following formula: ; Among them, the model accuracy is the model performance parameter; the measured calculation amount per input tag is the model data throughput parameter, which represents the calculation amount required for the tested model to process each input tag; and the calculation efficiency is the calculation performance parameter.

[0023] On the other hand, the model computing efficiency of the tested model computing system is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, and is calculated by the following formula: ; Among them, the model accuracy is the model performance parameter; the unit model data calculation amount is the calculation amount required for the tested model to process unit input data; the ratio of the measured computing performance parameters of the tested model computing system to execute the model computing task to the theoretical peak performance parameters of the computing system of the tested device represents the computing efficiency, and the computing efficiency is the computing performance parameter.

[0024] On the other hand, the model computing efficiency of the tested model computing system is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, and is calculated by the following formula: ; Among them, the model accuracy is the model performance parameter; the unit model data calculation amount is the calculation amount required for the tested model to process unit input data; the ratio of the measured total calculation amount to the measured total time consumption of the model calculation task is the measured calculation performance parameter of the tested model calculation system to perform the model calculation task; the ratio of the measured calculation performance parameter to the theoretical peak performance parameter of the calculation system of the tested device represents the calculation efficiency, and the calculation efficiency is the calculation performance parameter.

[0025] On the other hand, the model computing efficiency of the tested model computing system is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, and is calculated by the following formula: ; Among them, the model accuracy is the model performance parameter; the unit model data calculation amount is the calculation amount required for the tested model to process unit input data; the ratio of the product of the unit model data calculation amount and the total number of input data to the measured total time consumed by the model calculation task is the measured calculation performance parameter of the tested model calculation system to perform the model calculation task; the ratio of the measured calculation performance parameter to the theoretical peak performance parameter of the calculation system of the tested device represents the calculation efficiency, and the calculation efficiency is the calculation performance parameter.

[0026] On the other hand, the model computing efficiency of the tested model computing system is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, and is calculated by the following formula: ; Among them, the model accuracy is the model performance parameter; the ratio of the total number of input data to the measured total time consumption is the model data throughput parameter; and the theoretical peak performance parameter of the computing system is the computing performance parameter.

[0027] On the other hand, the model computing efficiency of the tested model computing system is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, and is calculated by the following formula: ; Among them, the model accuracy is the model performance parameter; the throughput rate is the model data throughput parameter, which represents the rate at which the tested model processes input data per unit time; and the computing system theoretical peak performance parameter is the computing performance parameter.

[0028] On the other hand, determining the model computing efficiency of the tested model computing system according to the model performance parameter, the model data throughput parameter and the computing performance parameter to obtain a test result of the tested model computing system includes: Determine the comprehensive modular computing efficiency of the tested model computing system according to the modular computing efficiency of the tested model computing system in multiple test scenarios; Taking the comprehensive simulation efficiency as the test result; The test scenarios correspond to the data sets one by one.

[0029] On the other hand, determining the model computing efficiency of the tested model computing system according to the model performance parameter, the model data throughput parameter and the computing performance parameter to obtain a test result of the tested model computing system includes: Determine the comprehensive modular computing efficiency of the tested model computing system according to the modular computing efficiency of the tested model computing system in multiple test scenarios; Taking the comprehensive simulation efficiency as the test result; Among them, one of the test scenarios corresponds to one or more of the data sets.

[0030] On the other hand, the step of determining the modular efficiency corresponding to a single test scenario includes: If the test scenario corresponds to a data set, the analog calculation efficiency corresponding to the data set is used as the analog calculation efficiency corresponding to the test scenario; If the test scenario corresponds to multiple data sets, the analog computing efficiency corresponding to the test scenario is determined according to the analog computing efficiencies corresponding to each of the data sets corresponding to the test scenario.

[0031] On the other hand, determining the analog calculation efficiency corresponding to the test scenario according to the analog calculation efficiency corresponding to each of the data sets corresponding to the test scenario includes: The arithmetic mean of the analog calculation efficiencies corresponding to the data sets is taken as the analog calculation efficiency corresponding to the test scenario.

[0032] On the other hand, determining the analog calculation efficiency corresponding to the test scenario according to the analog calculation efficiency corresponding to each of the data sets corresponding to the test scenario includes: The geometric mean of the analog calculation efficiency corresponding to each of the data sets is used as the analog calculation efficiency corresponding to the test scenario.

[0033] On the other hand, the step of determining the modular efficiency corresponding to a single test scenario includes: If the test scenario corresponds to a data set, the analog calculation efficiency corresponding to the data set is used as the analog calculation efficiency corresponding to the test scenario; If the test scenario corresponds to multiple data sets, the single-scenario model performance parameters corresponding to the test scenario are determined according to the model performance parameters corresponding to each data set corresponding to the test scenario; the single-scenario model data throughput parameters corresponding to the test scenario are determined according to the model data throughput parameters corresponding to each data set corresponding to the test scenario; the single-scenario computing performance parameters corresponding to the test scenario are determined according to the computing performance parameters corresponding to each data set corresponding to the test scenario; The model calculation efficiency corresponding to the test scenario is determined according to the single-scenario model performance parameter, the single-scenario model data throughput parameter and the single-scenario calculation performance parameter.

[0034] On the other hand, determining the single-scenario model performance parameter corresponding to the test scenario according to the model performance parameters corresponding to each data set corresponding to the test scenario includes: taking the arithmetic mean of the model performance parameters corresponding to each data set as the single-scenario model performance parameter; Determining the single-scenario model data throughput parameter corresponding to the test scenario according to the model data throughput parameters corresponding to each data set corresponding to the test scenario, including: taking the arithmetic mean of the model data throughput parameters corresponding to each data set as the single-scenario model data throughput parameter; Determining the single-scenario computing performance parameter corresponding to the test scenario according to the computing performance parameters corresponding to each data set corresponding to the test scenario includes: taking the arithmetic mean of the computing performance parameters corresponding to each data set as the single-scenario computing performance parameter.

[0035] On the other hand, determining the single-scenario model performance parameter corresponding to the test scenario according to the model performance parameters corresponding to each data set corresponding to the test scenario includes: taking the geometric mean of the model performance parameters corresponding to each data set as the single-scenario model performance parameter; Determining the single-scenario model data throughput parameter corresponding to the test scenario according to the model data throughput parameters corresponding to each data set corresponding to the test scenario, including: taking the geometric mean of the model data throughput parameters corresponding to each data set as the single-scenario model data throughput parameter; Determining a single-scenario computing performance parameter corresponding to the test scenario according to the computing performance parameters corresponding to each data set corresponding to the test scenario includes: taking a geometric mean of the computing performance parameters corresponding to each data set as the single-scenario computing performance parameter.

[0036] On the other hand, according to the modular computing efficiency of the tested model computing system in multiple test scenarios, determining the comprehensive modular computing efficiency of the tested model computing system includes: The arithmetic mean of the analog calculation efficiencies corresponding to the multiple test scenarios is taken as the comprehensive analog calculation efficiency.

[0037] On the other hand, the arithmetic mean of the modular efficiency corresponding to the multiple test scenarios is used as the comprehensive modular efficiency, which is calculated by the following formula: ; in, represents the comprehensive modular efficiency, Indicates that the model computing system under test is in The modular computing efficiency under the test scenario is Indicates that the model computing system under test is in A first weight coefficient under the test scenario.

[0038] On the other hand, according to the modular computing efficiency of the tested model computing system in multiple test scenarios, determining the comprehensive modular computing efficiency of the tested model computing system includes: The geometric mean of the analog calculation efficiencies corresponding to the multiple test scenarios is used as the comprehensive analog calculation efficiency.

[0039] On the other hand, the geometric mean of the analog calculation efficiencies corresponding to the multiple test scenarios is used as the comprehensive analog calculation efficiency, which is calculated by the following formula: ; in, represents the comprehensive modular efficiency, Indicates that the model computing system under test is in The module calculation efficiency under the test scenario is represents cumulative multiplication calculation, Indicates that the model computing system under test is in A second weight coefficient under the test scenario.

[0040] On the other hand, controlling the tested device to run the tested software framework to call the tested model and the data set to perform the model calculation task includes: Loading the tested software framework on the tested device and initializing the tested software framework; The tested software framework is started to call the tested model and the data set to execute the model calculation task.

[0041] On the other hand, loading the tested software framework on the tested device and initializing the tested software framework includes: Loading the library file of the model computing system under test on the device under test; Acquire the type of the interface function of the tested model computing system through the calling interface of the library file; Initialize the interface function according to the type of the interface function.

[0042] On the other hand, the types of the interface functions include at least: initialization function, completion function, data loading function, data unloading function, model calculation control function, and test parameter return function; The initialization function is the interface function that performs initialization according to the test configuration information; the completion function is the interface function of the task performed by the tested model computing system after completing the model computing task; the data loading function is the interface function for loading model computing data; the data unloading function is the interface function for unloading the model computing data; the model computing control function is the interface function for executing the model computing task; the test parameter return function is the output for calculating the test parameters required for at least one of the model performance parameters, the model data throughput parameters and the computing performance parameters.

[0043] On the other hand, starting the tested software framework to call the tested model to execute the model calculation task includes: Starting the tested software framework to execute the model calculation task; Obtaining a first result returned by calling the test parameter return function after the tested software framework calls the data loading function to load the model calculation data; Obtaining a second result of calling the test parameter return function after the tested software framework calls the model calculation control function to execute the model calculation task; Waiting for the tested software framework to output a third result after completing the model calculation task; Obtaining a fourth result returned by calling the test parameter return function after the tested software framework calls the data unloading function to unload the model calculation data; The model accuracy of the tested model is calculated according to the execution result of the model calculation task, including: The model accuracy is calculated according to at least one of the first result, the second result, the third result, and the fourth result.

[0044] In order to solve the above technical problems, the present invention also provides a test system for modular efficiency, including a test device and a device to be tested; Wherein, the device under test is used to run the software framework under test to perform model calculation tasks according to the model under test and the data set; The test device is used to obtain the data set, control the tested device to run the tested software framework to call the tested model and the data set to perform the model calculation task; determine the model performance parameters, model data throughput parameters and calculation performance parameters according to the model calculation task; determine the model calculation efficiency of the tested model calculation system according to the model performance parameters, the model data throughput parameters and the calculation performance parameters to obtain the test results of the tested model calculation system; The model under test computing system includes the model under test, the software framework under test and the device under test.

[0045] In order to solve the above technical problems, the present invention also provides a test device for modular efficiency, comprising: An acquisition unit, used for acquiring a data set; A control unit, used to control the tested device to run the tested software framework to call the tested model and the data set to perform model calculation tasks; A determination unit, configured to determine a model performance parameter, a model data throughput parameter, and a computing performance parameter according to the model computing task; and determine a model computing efficiency of the tested model computing system according to the model performance parameter, the model data throughput parameter, and the computing performance parameter, so as to obtain a test result of the tested model computing system; The model under test computing system includes the model under test, the software framework under test and the device under test.

[0046] In order to solve the above technical problems, the present invention also provides a test device for modular efficiency, comprising: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the modular efficiency testing method as described in any one of the above items are implemented.

[0047] In order to solve the above technical problems, the present invention also provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the modular efficiency testing method as described in any one of the above items are implemented.

[0048] In order to solve the above technical problems, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the modular efficiency testing method as described in any one of the above items.

[0049] The modular calculation efficiency testing method provided by the present invention has the beneficial effect of determining the model performance parameters, model data throughput parameters and computing performance parameters according to the model calculation task after controlling the tested software framework to call the tested model to perform the model calculation task according to the data set on the tested device, and determining the modular calculation efficiency of the tested model calculation system according to the model performance parameters, the model data throughput parameters and the computing performance parameters, so as to quantitatively evaluate the effect of computing power improvement on model performance improvement, thereby realizing the quantitative evaluation of the model calculation performance of the tested model calculation system including the tested model, the tested software framework and the tested device. The modular calculation efficiency and the testing method provided by the present invention are used as an evaluation scheme suitable for various types of model calculation tasks, which enhances the comparability indicators of different models running on different devices, and provides a reference basis for the improvement and perfection of models and model calculation systems, and then the accuracy of different models and multivariate computing power performance can be quantitatively evaluated, thereby ensuring the effective traction of the improvement of model capabilities and the reduction of application costs.

[0050] The analog arithmetic efficiency testing system, analog arithmetic efficiency testing device, analog arithmetic efficiency testing equipment, non-volatile storage medium and computer program product provided by the present invention have the above-mentioned beneficial effects, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0052] Figure 1 A flowchart of a modular calculation efficiency testing method provided by an embodiment of the present invention; Figure 2 A timing diagram of a modular efficiency testing method provided by an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a modular efficiency testing system provided by an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a test device for modular efficiency provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The core of the present invention is to provide a model calculation efficiency testing method, system, device, equipment, medium and product for testing and evaluating the model calculation performance of the model calculation system.

[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] To facilitate understanding of the technical solution provided by the embodiment of the present invention, some key terms used in the embodiment of the present invention are first explained here.

[0056] Computing power: It is used to measure the ability of computing systems (such as computers, servers, data centers, etc.) to process information and perform calculations. The most commonly used unit is floating point operations per second (FLOPS).

[0057] Model Computational Efficiency: A solution for evaluating the efficiency of large models. It is used to reflect the comprehensive efficiency of the tested model under the hardware system and software framework. The evaluation objects are the model itself, the tested software framework, and the tested AI bare metal server (computing power base). It is a comprehensive evaluation of the above three types of performance factors.

[0058] Floating point operations per second (FLOPS) is an indicator to measure the processing power of a computer system or computing device. It refers to the number of floating point operations (addition, subtraction, multiplication, division, etc.) that a system or device can perform per unit time.

[0059] Measured computing performance (in TFLOPS): The actual amount of floating-point operations such as addition, subtraction, multiplication, and division performed by the statistical model per unit time during inference and training.

[0060] The measured total computing power (in TFLOPS) is the total amount of floating-point operations such as addition, subtraction, multiplication, and division that the model actually performs during training or inference.

[0061] Theoretical peak performance of the computing system (in TFLOPS): a theoretical value calculated based on the main frequency, number of cores and other characteristics of the chip hardware itself, provided by the corresponding chip manufacturer; the single-precision floating-point theoretical peak performance can be used as the basis.

[0062] Throughput: The number of samples processed per unit time. It has different meanings in different scenarios. In language-related large models, throughput is represented by tokens / s, and in image-related large models, throughput is represented by images / s. The meaning of throughput in other scenarios is not repeated here.

[0063] Figure 1 A flowchart of a modular efficiency testing method provided in an embodiment of the present invention.

[0064] like Figure 1 As shown, the modular efficiency testing method provided by the embodiment of the present invention includes: S101: Obtain a data set; S102: Control the tested device to run the tested software framework to call the tested model and data set to perform model calculation tasks; S103: Determine model performance parameters, model data throughput parameters, and computing performance parameters according to the model computing task; S104: determining the model computing efficiency of the tested model computing system according to the model performance parameter, the model data throughput parameter and the computing performance parameter to obtain a test result of the tested model computing system; The model computing system under test includes the model under test, the software framework under test and the device under test.

[0065] In view of the lack of evaluation schemes for the accuracy and efficiency of model computing power in related technologies, the embodiment of the present invention proposes an evaluation index for the model computing efficiency of the model computing system. The model computing efficiency provided by the embodiment of the present invention can be used as a unified measurement scale for model computing systems using different models, different software frameworks, and different devices, enhance the comparability of model computing performed by different devices, and provide directional guidance for the improvement of models, software frameworks, and hardware.

[0066] After determining the tested model, tested software framework and tested device included in the tested model computing system, the model computing efficiency testing method provided by the embodiment of the present invention can be executed.

[0067] The modular efficiency testing method provided in the embodiment of the present invention can be implemented in software by designing and deploying test scripts, and can be executed based on the tested device or another testing device in hardware.

[0068] When performing tests on the model computing system under test, the test environment requirements may include: The ambient temperature is (25±5)℃, the relative humidity is 45%~75%, and the atmospheric pressure is 86kPa~106kPa; For the equipment under test with a nominal power less than 1.5kW, the test power supply should be AC ​​(220±1%) V; for the equipment under test with a nominal power greater than 1.5kW, the test power supply should be AC ​​(220±4%) V; For the equipment under test with a nominal power less than 1.5kW, the total harmonic distortion of the test power supply shall not exceed 2%; for the equipment under test with a nominal power greater than 1.5kW, the total harmonic distortion of the test power supply shall not exceed 5%; The power input frequency of all tested equipment should be (50±1.0%) Hz; The test environment should avoid interference factors such as strong magnetic fields and strong vibrations.

[0069] Requirements for the device under test may include: The device under test must include the central processing unit (CPU), memory, hard disk, and operating system. The equipment under test should use a standard power supply. During the test, all power supplies should be connected to an AC power source. The tested device should set all software options to default state. Power management technology and / or power saving features can only be tested with the tested server enabled by default. The device under test should be tested using the operating system declared by the manufacturer.

[0070] It should be noted that the tested model in the embodiment of the present invention can be any type of model, which can be divided from the perspectives of function, structure and application field, including but not limited to supervised learning models (classification models, regression models, etc.), unsupervised learning models (clustering models, dimensionality reduction models, association rule learning models, etc.), reinforcement learning models (strategy learning models, value learning models, strategy optimization models, etc.), semi-supervised learning models (models trained with labeled and unlabeled data, such as self-training models, etc.), integrated learning models (combining multiple models to improve performance, such as Bagging, Boosting, Stacking, etc.), deep learning models (convolutional neural networks, recurrent neural networks, long short-term memory networks, gated recurrent units, Transformer models, etc.), generative models (generative adversarial networks, variational autoencoders, etc.), explanatory models (decision trees, rule engines, etc.), probability models (Bayesian networks, hidden Markov models, etc.), meta-learning models, etc.

[0071] The tested software framework in the embodiment of the present invention is software used to call the tested model to perform model calculation tasks according to a data set.

[0072] In the tested model computing system of the embodiment of the present invention, the number of tested devices can be one or more. When the number of tested devices is multiple, different tested devices can be devices of the same type or heterogeneous computing devices. In some optional implementations of the embodiment of the present invention, the tested device can be a server, which can be an artificial intelligence bare metal server (computing power base).

[0073] In the embodiment of the present invention, the model calculation task may be a model training task or a model reasoning task.

[0074] Among them, the model training task is to adjust the parameters of the model through the optimization algorithm so that it can accurately learn knowledge from the data set, so as to accurately process the unknown data. In an embodiment of the present invention, when the number of devices under test is multiple, the model training task can be a synchronous training task or an asynchronous training task. Among them, the synchronous training task is that all devices under test read the model parameters at the same time and use the same model parameters for calculation. After each iteration, all devices under test will update the model parameters synchronously. The asynchronous training task is that different devices under test read the model parameters and calculate independently, without waiting for other devices under test to complete, and the update of the model parameters will be asynchronously shared with other devices under test.

[0075] Among them, the model reasoning task is to use the tested model (a trained model) to process unknown data. In an embodiment of the present invention, when there are multiple devices under test, the model reasoning task can be a synchronous reasoning (SynchronousInference) task or an asynchronous reasoning (Asynchronous Inference) task. Among them, the synchronous reasoning task is that the tested device obtains the next copy of the data to be processed for corresponding processing after each model reasoning is completed. In synchronous reasoning, at any given time point, only one reasoning request is executed, and other reasoning requests must wait for the current reasoning request to be completed before they can start. Synchronous reasoning is usually deployed in a linear blocking manner. If there is a reasoning task being executed in the current process, other reasoning tasks need to wait for the reasoning task to be completed before they can be executed. Asynchronous reasoning tasks allow reasoning tasks and other thread tasks to be executed in parallel, thereby improving resource utilization. In asynchronous reasoning, a piece of data to be processed can be handed over to the model for reasoning, and the program immediately obtains the next piece of data to be processed for corresponding preprocessing, and then waits for the last reasoning calculation to be completed before handing the data to be processed to the model for reasoning.

[0076] In an embodiment of the present invention, the required data set is determined according to at least one of the tested model, the tested software framework, the tested device and the model calculation task. In practical applications, the same model may have the ability to perform multiple application tasks. For example, a language model can be used to perform problem-solving tasks, code generation tasks, language understanding tasks, etc., and different application tasks correspond to different data sets. Then in S101, the acquired data set may be a data set corresponding to the application task to which the tested model is applicable. In addition, according to the different data types used in the model calculation task and the different types of data to be processed, an application task may correspond to one or more data sets. For example, an image processing task may correspond to a cat image data set and a dog image data set. An application task may correspond to data sets of different data types, such as a data set of single-precision floating-point data and a data set of integer data.

[0077] In S102, the test script as described above is applied to control the device under test to run the software framework under test to call the model under test and the data set to perform the model calculation task.

[0078] After the model computing system under test completes the model computing task, the model computing efficiency of the model computing system under test is calculated.

[0079] The evaluation index of modular computing efficiency proposed in the embodiment of the present invention is intended to evaluate the performance of the model computing system from two perspectives: the model itself and the software and hardware framework. As a comparability index that can measure various combinations of model computing systems, it is evaluated from the perspective of the computing power accuracy efficiency of the model. The meaning of this index can be further expressed as the higher the model accuracy of the model computing task, the faster the execution rate, and the fewer resources occupied, the higher the modular computing efficiency. To this end, in an embodiment of the present invention, the modular computing efficiency of the model computing system under test is determined based on the model performance parameters of the model computing system under test, the model data throughput parameters of the model computing system under test, and the computing performance parameters of the model computing system under test.

[0080] In the embodiment of the present invention, the model performance parameter is used to quantitatively represent the performance of the tested model, which is an evaluation of the tested model itself. The performance of the tested model can be represented from the perspectives of the accuracy of the results, the reliability of the results, the stability of the model, etc. In practical applications, the type of model performance parameter is selected according to the requirements of the model calculation task and the characteristics of the data to be processed by the tested model.

[0081] In some optional implementations of the embodiments of the present invention, the model performance parameter may include at least one of model accuracy, resource occupancy of the model, and model calculation rate. In some optional implementations of the embodiments of the present invention, the model performance parameter may be model accuracy.

[0082] If model accuracy is used, model accuracy may include but is not limited to at least one of accuracy, precision, recall, F1 score, receiver operating characteristic (ROC) curve and area under the curve (AUC) value, mean square error (MSE), root mean square error (RMSE), mean absolute error (MAE), log loss (Log Loss), precision (AP) and mean average precision (mAP), and coefficient of determination (R²). These model performance parameters can be collectively referred to as model accuracy. In general, model accuracy is a key indicator for measuring model performance, which indicates the correctness of the model in data processing (such as classification or prediction).

[0083] Among them, accuracy is mainly used to measure classification tasks. For binary classification problems, accuracy can be simply expressed as the number of correctly classified samples (Number of Correct Predictions) divided by the total number of samples (Number of Correct Predictions).

[0084] Precision is used to measure the proportion of samples that are actually positive among the samples predicted by the model to be positive. It is calculated by dividing the number of samples that are actually positive (True Positives) by the number of samples predicted by the model to be positive. The latter includes the sum of the number of samples that are actually positive (True Positives) and the number of samples that are incorrectly predicted to be positive (False Positives).

[0085] Recall is used to measure the proportion of samples that are correctly predicted as positive by the model among all samples that are actually positive. It is calculated by dividing the number of samples that are correctly predicted as positive by the model (True Positives) by the number of samples that are actually positive. The latter is the number of samples that are correctly predicted as positive by the model (True Positives) and the number of samples that are incorrectly predicted as negative by the model (False Negatives).

[0086] The F1 score is used to comprehensively consider the performance of precision and recall. It can be expressed as the harmonic mean of precision and recall, and is calculated as 2×precision×recall / (precision+recall).

[0087] The ROC curve is a graphical tool for evaluating the performance of classification models. It intuitively displays the classification performance of the model by depicting the relationship between the true positive rate (TPR) and the false positive rate (FPR) of the model under different classification thresholds. When evaluating model performance, the area under the ROC curve (AUC) can be used as a comprehensive indicator to directly compare the advantages and disadvantages of multiple models through a single value. The AUC value ranges from 0.5 to 1.0, and a larger AUC represents better performance.

[0088] Mean square error (MSE) is mainly used to measure the difference between the model prediction value and the true value in the regression task. The calculation formula is: .in, is the true value, is the predicted value, is the number of samples.

[0089] The root mean square error (RMSE) is the square root of the mean square error and provides the error magnitude in the same units as the original data. The calculation formula is: .

[0090] The mean absolute error (MAE) is the average of the absolute values ​​of the differences between the model's predicted values ​​and the true values. The calculation formula is: .in, is the true value, is the predicted value, is the number of samples.

[0091] Log loss, also known as logistic loss or cross-entropy loss, is a loss function used in machine learning to evaluate the performance of classification models. It is particularly suitable for binary classification problems, but can also be extended to multi-classification problems. Log loss measures the difference between the model's predicted probability and the actual label.

[0092] Precision (AP) and mean Average Precision (mAP) are important indicators for evaluating the performance of classification and target detection models, and are widely used in information retrieval, image classification, and target detection. AP measures the performance of each category. Specifically, AP is the area under the precision-recall curve. When calculating AP, you need to calculate the Precision and Recall values ​​for each category independently, then draw the Precision-Recall curve, and calculate the area under the curve to get the AP value of the category. mAP is the average of the AP values ​​of multiple categories, which comprehensively considers the performance of all categories. In the target detection task, mAP is a key indicator to measure the quality of the model in all categories. The calculation method is to calculate the AP value for each category separately, and then average them to get the final mAP value.

[0093] The coefficient of determination (R²), also known as the coefficient of determination or goodness of fit, is a statistic used in regression analysis to assess the degree of model fit. It represents the proportion of the variance of the dependent variable explained by the model, that is, the square of the correlation between the model's predicted value and the actual value. The value of R² ranges between 0 and 1. The closer the value is to 1, the better the model fits, that is, the higher the proportion of variance explained by the model.

[0094] In the embodiment of the present invention, one or more model accuracies may be used as model performance parameters, and the combination of different types of model accuracies may be designed according to the type of model calculation task.

[0095] In an embodiment of the present invention, the model data throughput parameter is used to represent the ability of the tested model to process input data, and is an evaluation indicator for the overall model computing system under test. In some optional implementations of the embodiments of the present invention, an effective throughput evaluation parameter may be defined to quantify the ability of the tested model computing system to effectively process input data. In some optional implementations of the embodiments of the present invention, effective processing of input data may be defined as a situation where the processing result of the tested model on the input data is a correct result, such as a situation where the predicted result is a correct result in a predictive calculation.

[0096] Then, determining the modular computing efficiency of the model computing system under test according to the model performance parameters, the model data throughput parameters, and the computing performance parameters in S103 may include: determining the effective throughput evaluation parameters of the model computing system under test according to the model performance parameters and the model data throughput parameters; and determining the modular computing efficiency according to the effective throughput evaluation parameters and the computing performance parameters. In some optional implementations of the embodiments of the present invention, it may be set that the larger the effective throughput corresponding to the effective throughput evaluation parameter, the greater the modular computing efficiency.

[0097] In the embodiment of the present invention, the calculation performance parameter is used to quantitatively represent the performance of the software and hardware framework composed of the tested software framework and the tested device.

[0098] In some optional implementations of the embodiments of the present invention, the step of determining the computing performance parameters may include: determining the computing performance parameters according to the measured computing performance parameters of the model computing system under test performing the model computing task and the theoretical peak performance parameters of the computing system of the tested device. Then in the embodiments of the present invention, the computing efficiency may be defined to represent the hardware performance utilization rate of the tested device when the tested software framework performs the model computing task according to the tested model and data set, and the computing efficiency may be the ratio of the measured computing performance parameters to the theoretical peak performance parameters of the computing system, or calculated according to a function of the ratio.

[0099] According to the above introduction, after the model computing system under test completes the model computing task, the actual efficiency of the model under test under the device under test and the software framework under test can be reflected by determining the model computing efficiency of the model under test, thereby realizing the comprehensive performance evaluation of the model under test itself, the software framework under test and the device under test.

[0100] In S103, after the model calculation task is completed, the model performance parameters, model data throughput parameters and computing performance parameters can be determined respectively according to the execution process or execution results of the model calculation task. A callback function can be deployed on the tested device to enable the tested device to return the relevant results of the model calculation task in order to calculate the model performance parameters, model data throughput parameters and computing performance parameters.

[0101] In S104, the model computing efficiency of the tested model computing system is calculated based on the model performance parameters, the model data throughput parameters and the computing performance parameters.

[0102] Then, the modular calculation efficiency can be directly used as the test result of the tested model calculation system, or the modular calculation efficiency can be used as one of the indicators for calculating the test result.

[0103] The modular calculation efficiency testing method provided by the embodiment of the present invention has the beneficial effect that after controlling the tested software framework to call the tested model to perform the model calculation task according to the data set on the tested device, the model performance parameters, model data throughput parameters and computing performance parameters are determined according to the model calculation task, and the modular calculation efficiency of the tested model calculation system is determined according to the model performance parameters, the model data throughput parameters and the computing performance parameters, so as to quantitatively evaluate the effect of computing power improvement on model performance improvement, thereby realizing the quantitative evaluation of the model calculation performance of the tested model calculation system including the tested model, the tested software framework and the tested device. The modular calculation efficiency and the testing method provided by the present invention are used as an evaluation scheme suitable for various model calculation tasks, which enhances the comparability indicators of different models running on different devices, and provides a reference basis for the improvement and perfection of models and model calculation systems, and then the accuracy of different models and multivariate computing power performance can be quantitatively evaluated, thereby ensuring the effective traction of model capability improvement and application cost reduction.

[0104] Based on the above embodiment, the embodiment of the present invention further introduces the calculation steps of the model parameter efficiency.

[0105] In the above embodiment, it is introduced that the parameter calculated according to the model performance parameter and the model data throughput parameter can be defined as the effective throughput evaluation parameter, and the definition of the effective throughput evaluation parameter can be to reflect the situation that the processing result of the input data by the tested model is a correct result. That is, in S103, the modular calculation efficiency of the tested model calculation system is determined according to the model performance parameter, the model data throughput parameter and the calculation performance parameter, which can include: determining the effective throughput evaluation parameter of the tested model calculation system according to the model performance parameter and the model data throughput parameter; determining the modular calculation efficiency according to the effective throughput evaluation parameter and the calculation performance parameter. And the larger the effective throughput corresponding to the effective throughput evaluation parameter, the greater the modular calculation efficiency.

[0106] Before calculating the effective throughput evaluation parameters, it is necessary to determine the model performance parameters and model data throughput parameters.

[0107] Since the model cannot be put into practical application when the performance is too low, it has no use value, and it is meaningless to calculate its analog calculation efficiency. In an embodiment of the present invention, a benchmark model performance parameter range can be defined to measure the minimum standard of the model performance of the tested model. In an embodiment of the present invention, determining the analog calculation efficiency of the tested model calculation system according to the model performance parameters, model data throughput parameters and calculation performance parameters in S104 may include: if the model performance parameters are within the benchmark model performance parameter range, then determining the analog calculation efficiency of the tested model calculation system according to the model performance parameters, model data throughput parameters and calculation performance parameters; if the model performance parameters are not within the benchmark model performance parameter range, then after setting the model performance parameters to 0, determining the analog calculation efficiency of the tested model calculation system according to the model performance parameters, model data throughput parameters and calculation performance parameters.

[0108] If the model performance parameter is the model accuracy, and the benchmark model performance parameter is the benchmark accuracy, then in S104, the modular efficiency of the model computing system under test is determined according to the model performance parameter, the model data throughput parameter, and the computing performance parameter, including: if the model performance parameter is within the benchmark model performance parameter range, the modular efficiency is determined according to the model performance parameter, the model data throughput parameter, and the computing performance parameter; if the model performance parameter is not within the benchmark model performance parameter range, the model performance parameter is set to 0, and the modular efficiency is determined according to the model performance parameter, the model data throughput parameter, and the computing performance parameter. This process can be expressed as: ; Among them, the measured model accuracy is the measured accuracy of the tested model in the model calculation task, and the benchmark accuracy is the minimum allowable accuracy of the tested model.

[0109] In the embodiment of the present invention, different benchmark model performance parameter ranges may be set for different tested models, different model calculation tasks or different data sets.

[0110] In some optional implementations of the embodiments of the present invention, the effective throughput evaluation parameter can be used to quantify the "amount" of the correct result of the input data processed by the tested model. Then the model data throughput parameter may include the unit model data calculation amount; the unit model data calculation amount is the calculation amount required for the tested model to process the unit input data; the better the model performance corresponding to the model performance parameter, the larger the effective throughput evaluation parameter; the effective throughput evaluation parameter is negatively correlated with the unit model data calculation amount. As described in the above embodiments of the present invention, the model performance parameter can adopt the model accuracy. In the tested model computing system, the larger the amount of input data that can be correctly processed by the tested model, the larger the effective throughput evaluation parameter and the greater the model computing efficiency.

[0111] In an embodiment of the present invention, the unit model data calculation amount is a measured value, that is, the measured unit model data calculation amount. The type of input data of the tested model is determined according to the type of the tested model or the type of the data set to determine the unit of the unit model data calculation amount. For example, if the tested model is a language model, the unit model data calculation amount can be the amount of calculation required for the tested model to process each input token. If the tested model is an image processing model, the unit model data calculation amount can be the amount of calculation required for the tested model to process each input image.

[0112] In some other optional implementations of the embodiments of the present invention, the effective throughput evaluation parameter can be used to quantify the "rate" at which the processing result of the input data by the tested model is the correct result. Then the model data throughput parameter may include throughput rate; throughput rate is the rate at which the tested model processes input data per unit time; the better the model performance corresponding to the model performance parameter, the larger the effective throughput evaluation parameter; the effective throughput evaluation parameter is positively correlated with the throughput rate. Similarly, taking the model performance parameter as an example, in the tested model computing system, the greater the throughput rate at which the tested model can correctly process the input data, the larger the effective throughput evaluation parameter and the greater the model computing efficiency.

[0113] In an embodiment of the present invention, the throughput rate is a measured value, that is, the amount of input data processed by the tested model per unit time. The type of input data of the tested model is determined according to the type of the tested model or the type of data set to determine the unit of the throughput rate. For example, if the tested model is a language model, the throughput rate can be the number of input tokens processed by the tested model per unit time, and the unit of the throughput rate can be tokens / s (that is, the number of input tokens processed per second). If the tested model is an image processing model, the throughput rate can be the number of input images processed by the tested model per unit time, and the unit of the throughput rate can be images / s (that is, the number of input images processed per second).

[0114] Based on the above embodiments, the embodiments of the present invention further introduce the calculation steps of calculating the performance parameters.

[0115] In the above embodiment, the computing performance parameters can be determined based on the measured computing performance parameters of the model computing system under test executing the model computing task and the theoretical peak performance parameters of the computing system of the tested device.

[0116] In the implementation method of determining the computing performance parameters according to the measured computing performance parameters of the model computing system under test to perform the model computing task and the theoretical peak performance parameters of the computing system of the tested device, the ratio of the measured computing performance parameters to the theoretical peak performance parameters of the computing system can be used as the computing performance parameters. The computing efficiency can be defined to represent the hardware performance utilization rate of the tested device when the tested software framework executes the model computing task according to the tested model and the data set, then the computing efficiency can be set to be positively correlated with the measured computing performance parameters of the model computing system under test to perform the model computing task, and the computing efficiency can be negatively correlated with the theoretical peak performance parameters of the computing system of the tested device; the computing performance corresponding to the computing performance parameter is positively correlated with the computing efficiency. In some optional implementation methods of the embodiments of the present invention, the computing efficiency can be the ratio of the measured computing performance parameters to the theoretical peak performance parameters of the computing system, or calculated according to the function of the ratio. In the embodiment of the present invention, the step of determining the measured computing performance parameters can include: obtaining the measured total computing amount of the model computing system under test to perform the model computing task and the measured total time consumed by the model computing system under test to perform the model computing task; determining the measured computing performance parameters according to the measured total computing amount and the measured total time consumed.

[0117] In the embodiment of the present invention, for the tested device, the computing system theoretical peak performance parameter is used to quantify the theoretical peak computing power of the tested device, and the measured computing performance parameter is used to quantify the computing power actually invested by the tested device when executing the model computing task. For the tested model, the measured computing performance parameter is used to quantify the amount of computing per unit time in the process of the tested software framework calling the tested model to execute the model computing task.

[0118] In the embodiment of the present invention, the computing system theoretical peak performance parameter may adopt the parameter provided by the manufacturer of the tested device, which is usually a theoretical value calculated according to the hardware configuration of the tested device (such as the frequency and number of cores of the central processing unit). The measured computing performance parameter is determined according to the execution process or execution result of the model computing task, and is a parameter actually tested.

[0119] In some optional implementations of the embodiments of the present invention, the measured computing performance parameter may be the number of floating-point operations per second of the tested device executing the model computing task, and the computing system theoretical peak performance parameter may be the theoretical peak number of floating-point operations per second of the tested device. Among them, the number of floating-point operations per second (Floating Point Operations per Second, FLOPS) refers to the number of floating-point operations (addition, subtraction, multiplication, division, etc.) that a system or device can perform in a unit of time. In some other optional implementations of the embodiments of the present invention, the measured computing performance parameter may also be the number of tera floating point operations per second (TFLOPS) of the tested device executing the model computing task, and the computing system theoretical peak performance parameter may also be the theoretical peak number of tera floating point operations per second of the tested device.

[0120] Through actual testing, it is found that the data type actually used in the process of executing model calculation tasks affects the model performance parameters and also affects the calculation performance parameters. Therefore, the data type needs to be considered when determining the theoretical peak performance parameters and the measured calculation performance parameters of the calculation system.

[0121] In some optional implementations of the embodiments of the present invention, the step of determining the measured computing performance parameters and the step of determining the theoretical peak performance parameters of the computing system may include: obtaining the theoretical peak performance parameters of the computing system corresponding to the first data type of the tested device; if the actual data types in the model calculation task are all the first data type, the computing performance parameters corresponding to the actual data types in the model calculation task are used as the measured computing performance parameters; if there are actual data types that are not the first data type in the model calculation task, the computing performance parameters corresponding to the actual data types in the model calculation task are equivalent to the computing performance parameters corresponding to the first data type, and the measured computing performance parameters are obtained. That is to say, no matter what data type is actually used in the model calculation task, it is regarded as the first data type to determine the measured computing performance parameters, and then the computing performance parameters are determined in combination with the theoretical peak performance parameters of the computing system. This can greatly simplify the steps of determining the model calculation efficiency, thereby quickly obtaining test results.

[0122] Among them, the first data type can be selected from the theoretical peak performance parameters of the computing system of various data types provided by the manufacturer of the tested device. In practical applications, the first data type can be one of, but not limited to, integer, floating-point, character, string, and array. If the first data type is a floating-point type, it can be one of single-precision floating point, double-precision floating point, extended-precision floating point, quadruple-precision floating point, and half-precision floating point.

[0123] In some optional implementations of the embodiments of the present invention, the first data type may be a single-precision floating point type.

[0124] In some other optional implementations of the embodiments of the present invention, the step of determining the measured computing performance parameters and the step of determining the theoretical peak performance parameters of the computing system also include: obtaining the theoretical peak performance parameters of the computing system corresponding to the second data type of the tested device; if the actual data types in the model calculation task are all the second data type, then the computing performance parameters corresponding to the actual data types in the model calculation task are used as the measured computing performance parameters; if there are actual data types that are not the second data type in the model calculation task, then the computing performance parameters corresponding to the actual data types in the model calculation task are converted into computing performance parameters corresponding to the second data type to obtain the measured computing performance parameters.

[0125] Among them, the second data type can be selected from the theoretical peak performance parameters of the computing system of each data type provided by the manufacturer of the tested device. In practical applications, the first data type can be one of, but not limited to, integer, floating-point, character, string, and array. If the second data type is a floating-point type, it can be one of single-precision floating point, double-precision floating point, extended-precision floating point, quadruple-precision floating point, and half-precision floating point.

[0126] In some optional implementations of the embodiments of the present invention, the second data type may be a single-precision floating point type.

[0127] Based on the above embodiments, an embodiment of the present invention provides a method for determining modular efficiency suitable for practical applications.

[0128] In the embodiment of the present invention, the model computing efficiency of the tested model computing system is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, which can be calculated by the following formula: ; Among them, model accuracy is the model performance parameter; the unit model data calculation amount is the calculation amount required for the tested model to process unit input data; and the calculation efficiency is the calculation performance parameter.

[0129] In the embodiment of the present invention, when the model under test is a language model, the model computing efficiency of the model computing system under test is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, which can be calculated by the following formula: ; Among them, model accuracy is the model performance parameter; the measured calculation amount per input tag is the model data throughput parameter, which means the calculation amount required for the tested model to process each input tag; the calculation efficiency is the calculation performance parameter.

[0130] In the embodiment of the present invention, the model computing efficiency of the tested model computing system is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, and can also be calculated by the following formula: ; Among them, model accuracy is the model performance parameter; the unit model data calculation amount is the calculation amount required for the tested model to process unit input data; the ratio of the measured calculation performance parameters of the tested model calculation system to perform the model calculation task to the theoretical peak performance parameters of the calculation system of the tested device represents the calculation efficiency, and the calculation efficiency is the calculation performance parameter.

[0131] In the embodiment of the present invention, the model computing efficiency of the tested model computing system is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, and can also be calculated by the following formula: ; Among them, model accuracy is the model performance parameter; the unit model data calculation amount is the calculation amount required for the tested model to process unit input data; the ratio of the measured total calculation amount to the measured total time consumption of the model calculation task is the measured calculation performance parameter of the tested model calculation system to execute the model calculation task; the ratio of the measured calculation performance parameter to the theoretical peak performance parameter of the calculation system of the tested device represents the calculation efficiency, and the calculation efficiency is the calculation performance parameter.

[0132] In the embodiment of the present invention, the model computing efficiency of the tested model computing system is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, and can also be calculated by the following formula: ; Among them, model accuracy is the model performance parameter; the unit model data calculation amount is the calculation amount required for the tested model to process unit input data; the ratio of the product of the unit model data calculation amount and the total number of input data to the measured total time consumption of the model calculation task is the measured calculation performance parameter of the tested model calculation system to execute the model calculation task; the ratio of the measured calculation performance parameter to the theoretical peak performance parameter of the calculation system of the tested device represents the calculation efficiency, and the calculation efficiency is the calculation performance parameter.

[0133] In the embodiment of the present invention, the model computing efficiency of the tested model computing system is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, and can also be calculated by the following formula: ; Among them, the model accuracy is the model performance parameter; the ratio of the total number of input data to the total measured time consumption is the model data throughput parameter; and the theoretical peak performance parameter of the computing system is the computing performance parameter.

[0134] In the embodiment of the present invention, the model computing efficiency of the tested model computing system is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, and can also be calculated by the following formula: ; Among them, model accuracy is the model performance parameter; throughput rate is the model data throughput parameter, which means the rate at which the tested model processes input data per unit time; and the theoretical peak performance parameter of the computing system is the computing performance parameter.

[0135] It can be seen from the calculation formula of the above-mentioned analog calculation efficiency provided by the embodiment of the present invention that the analog calculation efficiency expressed by the model data throughput parameters and computing performance parameters in different expressions can be equivalent. For example, when the model performance parameter adopts the model accuracy, the computing performance parameter adopts the computing efficiency, and the model data throughput parameter adopts the unit model data calculation amount (that is, the effective throughput evaluation parameter introduced in the above embodiment can be used to quantify the "amount" at which the processing result of the tested model for the input data is the correct result), or the computing performance parameter adopts the computing system theoretical peak performance parameter, and the model data throughput parameter adopts the throughput rate (that is, the effective throughput evaluation parameter introduced in the above embodiment can be used to quantify the "rate" at which the processing result of the tested model for the input data is the correct result). The analog calculation efficiency finally calculated is equivalent.

[0136] In practical applications, the different methods provided by the embodiments of the present invention are used to determine the model calculation efficiency, which affects the type of test parameters required to be obtained in the model calculation task in S102. To determine the model calculation efficiency, it is necessary to test the model accuracy, calculation efficiency and unit model data calculation amount from the model calculation task; if To determine the model computing efficiency, it is necessary to test the model accuracy, throughput and theoretical peak performance parameters of the computing system from the model computing tasks.

[0137] In some optional implementations of the embodiments of the present invention, the measured computing performance parameter may be the number of floating-point operations per second (FLOPS) or the number of trillion floating-point operations per second (TFLOPS) of the model computing system under test that performs model computing tasks, that is, statistics on the actual amount of floating-point operations such as addition, subtraction, multiplication, and division performed by the tested model in model computing (inference computing or training computing) per unit time.

[0138] The theoretical peak performance parameters of the computing system can adopt the theoretical values ​​calculated according to the hardware characteristics provided by the manufacturer of the tested device, and the units can adopt the single-precision floating-point theoretical peak performance parameters, that is, the number of floating-point operations per second (FLOPS) or the number of trillion floating-point operations per second (TFLOPS).

[0139] The measured total time is the measured time of the model calculation task. The time starts from the start of the model calculation task (inference calculation or training calculation) and stops when the model calculation task ends. The unit can be seconds (s).

[0140] The throughput rate is the rate at which the tested model processes input data per unit time. When the tested model is a language model, the throughput rate can be the number of input tokens processed by the tested model per unit time. In this case, the unit of the throughput rate can be tokens / s (i.e., the number of input tokens processed per second). When the tested model is an image processing model, the throughput rate can be the number of input images processed by the tested model per unit time. In this case, the unit of the throughput rate can be images / s (i.e., the number of input images processed per second). When the tested model is other types of models and the data set is other different data sets, this principle can be used to determine the unit of the throughput.

[0141] In actual applications, it is observed that in addition to the tested device and the tested software framework, the factors affecting the model performance include at least the model parameter quantity, the type of application scenario, the type of data set, and the data type used for model calculation. That is to say, under the combination of the above different factors, the same tested model may also have different model performance, which further affects the final measured model calculation efficiency. In an embodiment of the present invention, the model calculation efficiency of the tested model calculation system can be more comprehensively evaluated by determining the comprehensive model calculation efficiency of the tested model calculation system.

[0142] In some optional implementations of the embodiments of the present invention, determining the modular computing efficiency of the tested model computing system according to the model performance parameters, the model data throughput parameters and the computing performance parameters in S104 to obtain the test result of the tested model computing system may include: determining the comprehensive modular computing efficiency of the tested model computing system according to the modular computing efficiency of the tested model computing system in multiple test scenarios; taking the comprehensive modular computing efficiency as the test result; wherein the test scenario corresponds to the data set one by one. That is, for multiple test scenarios, a data set can be selected for each test scenario to perform the model computing task, and then the modular computing efficiency of each test scenario can be combined to obtain the comprehensive modular computing efficiency.

[0143] In some other optional implementations of the embodiments of the present invention, the modular computing efficiency of the tested model computing system is determined according to the model performance parameters, the model data throughput parameters and the computing performance parameters in S104 to obtain the test result of the tested model computing system, and it can also include: determining the comprehensive modular computing efficiency of the tested model computing system according to the modular computing efficiency of the tested model computing system in multiple test scenarios; taking the comprehensive modular computing efficiency as the test result; wherein one test scenario corresponds to one or more data sets. That is to say, for multiple test scenarios, one or more data sets can be selected for each test scenario to perform the model computing task, and for the test scenario using multiple data sets, the modular computing efficiency of each data set is first integrated to obtain the modular computing efficiency corresponding to the test scenario, and then the modular computing efficiency of each test scenario is integrated to obtain the comprehensive modular computing efficiency.

[0144] Taking the language model as an example, the test scenario can be set as follows: Test scenario 1: The model calculation task is a problem-solving task, and the data sets used may include but are not limited to GSM8K, Math, and Math23K.

[0145] Test scenario 2: The model calculation task is a code generation task, and the datasets used may include but are not limited to Human Eval and MBPP.

[0146] Test scenario 3: The model calculation task is a large-scale multi-task language understanding task, and the dataset used may include but is not limited to MMLU.

[0147] Test scenario 4: The model calculation task is a reading comprehension task, and the dataset used may include but is not limited to ARC-C.

[0148] Then the step of determining the analog calculation efficiency corresponding to a single test scenario may include: if the test scenario corresponds to one data set, then the analog calculation efficiency corresponding to the data set is used as the analog calculation efficiency corresponding to the test scenario; if the test scenario corresponds to multiple data sets, then the analog calculation efficiency corresponding to the test scenario is determined according to the analog calculation efficiencies corresponding to each data set corresponding to the test scenario. In some optional implementations of the embodiments of the present invention, determining the analog calculation efficiency corresponding to the test scenario according to the analog calculation efficiencies corresponding to each data set corresponding to the test scenario may include: taking the arithmetic mean of the analog calculation efficiencies corresponding to each data set as the analog calculation efficiency corresponding to the test scenario. In some other optional implementations of the embodiments of the present invention, determining the analog calculation efficiency corresponding to the test scenario according to the analog calculation efficiencies corresponding to each data set corresponding to the test scenario may also include: taking the geometric mean of the analog calculation efficiencies corresponding to each data set as the analog calculation efficiency corresponding to the test scenario.

[0149] In some further optional implementations of the embodiments of the present invention, the model performance parameters, the model data throughput parameters and the computing performance parameters can also be calculated comprehensively. Then, in S104, the modular computing efficiency corresponding to the test scenario is determined according to the modular computing efficiency corresponding to each data set corresponding to the test scenario, and it can also include: determining the single-scenario model performance parameters corresponding to the test scenario according to the model performance parameters corresponding to each data set corresponding to the test scenario; determining the single-scenario model data throughput parameters corresponding to the test scenario according to the model data throughput parameters corresponding to each data set corresponding to the test scenario; determining the single-scenario computing performance parameters corresponding to the test scenario according to the computing performance parameters corresponding to each data set corresponding to the test scenario; determining the modular computing efficiency corresponding to the test scenario according to the single-scenario model performance parameters, the single-scenario model data throughput parameters and the single-scenario computing performance parameters. That is, by first calculating the single-scenario model performance parameters, the single-scenario model data throughput parameters and the single-scenario computing performance parameters respectively, different comprehensive calculation methods for the parameters corresponding to the data sets can be used for these three types of parameters, so as to better reflect the influence of different types of parameters on the modular computing efficiency calculation.

[0150] In some optional implementations of the embodiments of the present invention, determining the single-scenario model performance parameters corresponding to the test scenario according to the model performance parameters corresponding to each data set corresponding to the test scenario may include: taking the arithmetic mean of the model performance parameters corresponding to each data set as the single-scenario model performance parameter. Determining the single-scenario model data throughput parameters corresponding to the test scenario according to the model data throughput parameters corresponding to each data set corresponding to the test scenario may include: taking the arithmetic mean of the model data throughput parameters corresponding to each data set as the single-scenario model data throughput parameter. Determining the single-scenario computing performance parameters corresponding to the test scenario according to the computing performance parameters corresponding to each data set corresponding to the test scenario may include: taking the arithmetic mean of the computing performance parameters corresponding to each data set as the single-scenario computing performance parameter.

[0151] In other optional implementations of the embodiments of the present invention, determining the single-scenario model performance parameters corresponding to the test scenario based on the model performance parameters corresponding to each data set corresponding to the test scenario may include: taking the geometric mean of the model performance parameters corresponding to each data set as the single-scenario model performance parameter. Determining the single-scenario model data throughput parameters corresponding to the test scenario based on the model data throughput parameters corresponding to each data set corresponding to the test scenario may include: taking the geometric mean of the model data throughput parameters corresponding to each data set as the single-scenario model data throughput parameter. Determining the single-scenario computing performance parameters corresponding to the test scenario based on the computing performance parameters corresponding to each data set corresponding to the test scenario may include: taking the geometric mean of the computing performance parameters corresponding to each data set as the single-scenario computing performance parameter.

[0152] In an embodiment of the present invention, determining the comprehensive modular computing efficiency of the model computing system under test according to the modular computing efficiency of the model computing system under test in multiple test scenarios may include: taking the arithmetic mean of the modular computing efficiencies corresponding to the multiple test scenarios as the comprehensive modular computing efficiency. The comprehensive modular computing efficiency may be calculated by the following formula: ; in, represents the comprehensive modular efficiency, Indicates that the model computing system under test is The modular computing efficiency under each test scenario is Indicates that the model computing system under test is In the embodiment of the present invention, the first weight coefficients corresponding to the various test scenarios may be equal or unequal.

[0153] In an embodiment of the present invention, determining the comprehensive modular computing efficiency of the tested model computing system according to the modular computing efficiency of the tested model computing system in multiple test scenarios may also include: taking the geometric mean of the modular computing efficiencies corresponding to the multiple test scenarios as the comprehensive modular computing efficiency. The comprehensive modular computing efficiency can be calculated by the following formula: ; in, represents the comprehensive modular efficiency, Indicates that the model computing system under test is The modular computing efficiency under each test scenario is represents cumulative multiplication calculation, Indicates that the model computing system under test is In the embodiment of the present invention, the second weight coefficients corresponding to the various test scenarios may be equal or unequal.

[0154] In the embodiment of the present invention, two determination methods, namely, the arithmetic mean and the geometric mean, are proposed for combining multiple modular calculation efficiencies to obtain modular calculation efficiency. In practical applications, the geometric mean is more suitable for comparing data of different scales or units than the arithmetic mean. In some optional implementations of the embodiment of the present invention, if the difference between at least two modular calculation efficiencies among the multiple modular calculation efficiencies corresponding to the tested model calculation system is greater than a preset threshold, the geometric mean between the modular calculation efficiencies is used as the comprehensive modular calculation efficiency to weaken the situation due to test errors or the performance of the tested model calculation system itself being too good or too poor in certain test scenarios or data sets, thereby obtaining a more comprehensive comprehensive modular calculation efficiency.

[0155] On the basis of the above-mentioned embodiment, the embodiment of the present invention further describes the steps of controlling the tested model computing system to perform the model computing task.

[0156] In an embodiment of the present invention, controlling the tested device to run the tested software framework to call the tested model and data set to perform model calculation tasks in S102 may include: loading the tested software framework on the tested device and performing initialization of the tested software framework; starting the tested software framework to call the tested model and data set to perform model calculation tasks.

[0157] After the deployment of the model computing system under test (System Under Test, SUT) is completed, the whole is provided in the form of a dynamic link library (DLL). In addition, the required interfaces and corresponding implementation classes are required. When the test script is started, it will dynamically load the library file (Library, Lib library) of the model computing system under test. The Lib library provides an access interface. If the software framework under test uses C language or C++ language, the Lib library provides a C interface (CreateSUTInstance) to provide external services. The test script obtains the class instance of the model computing system under test by calling this interface. It should be noted that the instance must be derived from the base class (BaseSUT). After that, the test script will start the reasoning test by calling the member function of the instance. After the test is completed, the results will be returned to the test framework, which will save these results and perform subsequent calculations of the model computing efficiency. Finally, the test script can integrate all information and generate a detailed test report.

[0158] In an embodiment of the present invention, loading the tested software framework on the tested device and performing initialization of the tested software framework may include: loading the library file of the tested model computing system on the tested device; obtaining the type of interface function of the tested model computing system through the calling interface of the library file; and initializing the interface function according to the type of the interface function.

[0159] The calling interface can be a C-style interface (CreateSUTInstance) that returns a pointer to an instance of ConcreteSUT (which instantiates the computing system of the model under test). After obtaining this pointer, the test script will cast it to a pointer to the BaseSUT base class for subsequent operations and use. This design allows the test framework to be decoupled from specific implementation details, enhancing the versatility and flexibility of the framework. The calling interface can be defined as: "interface". void* CreateSUTInstance().

[0160] Therefore, in the embodiment of the present invention, the model computing system under test must implement a custom ConcreteSUT class, which inherits from the BaseSUT base class and overrides all abstract methods defined in the base class. Such a design ensures that the model computing system under test can provide necessary functions and behaviors according to the requirements of the test framework. In this way, the ConcreteSUT class encapsulates the specific implementation details while ensuring compatibility with the test script and consistency of the interface.

[0161] In an embodiment of the present invention, the types of interface functions may include, but are not limited to: initialization function (Initialize), completion function (Finalize), data loading function (LoadData), data unloading function (UnLoadData), model calculation control function, and test parameter return function. The initialization function is an interface function that performs initialization according to the test configuration information; the completion function is an interface function for the task performed by the tested model calculation system after completing the model calculation task; the data loading function is an interface function for loading model calculation data; the data unloading function is an interface function for unloading model calculation data; the model calculation control function is an interface function for executing the model calculation task; and the test parameter return function is an interface function for outputting the test parameters required for calculating at least one of the model performance parameters, the model data throughput parameters, and the calculation performance parameters.

[0162] Among them, the model calculation control function is determined according to the type of model calculation task, and the type description can refer to the introduction of the above embodiment. Taking the model calculation task as a model inference task as an example, the model calculation control function can include an inference control function (Inferencet) and a data result callback function (Response). The inference control function (Inferencet) is used to control the tested model calculation system to traverse each data in the test data set for inference calculation. The data result callback function (Response) is used to control the tested model calculation system to call the callback function of the test script in a specified format after completing the inference calculation of each data. The test script saves the test data results and records the test completion time after receiving the information that the inference is completed.

[0163] The test parameter return function is determined according to the selected calculation method of the modular efficiency. The calculation method of the modular efficiency can refer to the introduction of the above embodiment. Taking the determination of analog computing efficiency as an example, the test parameter return function may include a throughput return function (GetThroughput) and a system theoretical peak return function (GetSystemFLOPS), which are used to control the computing system response test script of the model under test to provide its throughput data during runtime and the computing system response test script of the model under test to return the computing system theoretical peak performance parameters of the tested device to meet the needs of analog computing efficiency calculation.

[0164] The interface function may also include an uninitialization function (Uninitialize), which is used by the tested model computing system to release all occupied resources after the test is completed, to ensure that the system can be restored to the state before the test.

[0165] In an embodiment of the present invention, starting the tested software framework to call the tested model to perform a model calculation task may include: starting the tested software framework to perform the model calculation task; obtaining a first result returned by the tested software framework calling a data loading function to load model calculation data and then calling a test parameter return function; obtaining a second result returned by the tested software framework calling a model calculation control function to perform the model calculation task and then calling a test parameter return function; waiting for a third result output after the tested software framework completes the model calculation task; obtaining a fourth result returned by the tested software framework calling a data unloading function to unload the model calculation data and then calling the test parameter return function; and calculating the model accuracy of the tested model according to the execution result of the model calculation task, including: calculating the model accuracy according to at least one of the first result, the second result, the third result, and the fourth result.

[0166] Figure 2 A timing diagram of a modular efficiency testing method provided in an embodiment of the present invention.

[0167] like Figure 2 As shown, in some optional implementations of the embodiments of the present invention, after the model computing system under test is deployed, the steps of executing the model computing efficiency test method by running the test script by the test device may include the following S201~S218. In the implementation of the timing diagram, the test script is deployed in the test device, the library file of the model computing system under test and the model computing system instance under test are deployed in the device under test, and the executed steps are regarded as the steps executed by the corresponding hardware.

[0168] S201: Start. Start the test equipment to start the test of the model calculation efficiency of the model calculation system under test.

[0169] S202: Loading library files. The test device loads the library files of the model computing system under test.

[0170] S203: Obtaining interface functions: The testing device obtains the interface functions of the model computing system under test.

[0171] S204: Generate an instance of the model computing system under test. Generate an instance of the model computing system under test according to the interface function of the model computing system under test.

[0172] S205: Loading a new instance of the model computing system under test. The library file of the model computing system under test loads the new instance of the model computing system under test.

[0173] S206: Return the creation result. The model computing system instance under test returns the creation result to the library file of the model computing system under test.

[0174] S207: Return the creation result. The library file of the model computing system under test returns the creation result to the test script.

[0175] S208: Calling an initialization function. The test device calls the initialization function of the model computing system under test to perform initialization of the model computing system under test.

[0176] S209: Loading data: The model computing system instance under test calls a data loading function to load data required by the model computing task.

[0177] S210: Loading the model. The model computing system instance under test loads the model parameters of the model under test.

[0178] S211: Other operations. The model computing system instance under test performs other initialization operations required for the model computing task.

[0179] S212: Returning the initialization result. The model computing system instance under test returns the initialization result.

[0180] S213: Start the test. The test script controls the model computing system instance under test to start executing the model computing task.

[0181] For each input data, the following S214~S215 are the steps included in one model calculation in the iterative calculation of the model calculation task.

[0182] S214: Model calculation. The model calculation system instance under test performs model calculation according to the input data.

[0183] S215: Send calculation results. The model calculation system instance under test responds to the test script and returns the model calculation results.

[0184] S216: Obtaining theoretical peak performance parameters of the computing system. The test script sends a request to obtain theoretical peak performance parameters of the computing system to the library file of the tested model computing system.

[0185] S217: Returning the theoretical peak performance parameters of the computing system. The library file of the tested model computing system returns the theoretical peak performance parameters of the computing system to the test script.

[0186] S218: Obtaining throughput rate. The test script sends a request for obtaining throughput rate to the library file of the model computing system under test.

[0187] S219: Returning the throughput rate: The library file of the model computing system under test returns the throughput rate to the test script.

[0188] S220: Calculate model performance parameters. The test script calculates model performance parameters, such as model accuracy, based on the results returned by the tested model calculation system during the execution of the model calculation task.

[0189] S221: Release resources. The test script controls the library file of the tested model computing system to call the deinitialization function to release resources.

[0190] S222: Determine the modular computing efficiency. The test script determines the modular computing efficiency of the model computing system under test according to the model performance parameters, the model data throughput parameters and the computing performance parameters.

[0191] S223: Generate a test report. The test script generates a test report of the model computing system under test according to the test results such as the modular computing efficiency and the comprehensive modular computing efficiency.

[0192] The model calculation efficiency testing method provided in the embodiment of the present invention not only proposes evaluation indicators of model calculation efficiency, but also provides a method for testing model calculation efficiency, thereby providing a unified evaluation standard applicable to various types of model calculation systems, which can help model users choose more efficient models, and at the same time assist model developers to optimize models, software frameworks, computing devices, etc., so as to promote the green development of artificial intelligence, promote efficient use of resources, and enhance model quality.

[0193] It should be noted that in the embodiments of the modular efficiency test method of the present invention, some of the steps or features can be ignored or not executed. The hardware or software functional modules divided for the convenience of description are not the only implementation form of the modular efficiency test method provided in the embodiment of the present invention.

[0194] The above describes in detail the various embodiments corresponding to the modular efficiency testing method. On this basis, the present invention also discloses a modular efficiency testing device, equipment, non-volatile storage medium and computer program product corresponding to the above method.

[0195] Figure 3 A schematic diagram of the structure of a modular efficiency testing system provided in an embodiment of the present invention.

[0196] like Figure 3 As shown, the modular efficiency testing system provided by the embodiment of the present invention may include a testing device 301 and a tested device 302 .

[0197] The tested device 302 is used to run the tested software framework to perform model calculation tasks according to the tested model and data set.

[0198] The test device 301 is used to obtain the data set, control the tested device 302 to run the tested software framework to call the tested model and the data set to perform the model calculation task; determine the model performance parameters, model data throughput parameters and calculation performance parameters according to the model calculation task; determine the model calculation efficiency of the tested model calculation system according to the model performance parameters, model data throughput parameters and calculation performance parameters to obtain the test results of the tested model calculation system;

[0199] The model computing system under test includes a model under test, a software framework under test and a device under test 302 .

[0200] In a specific implementation, the test device 301 sends a test control command to the device under test 302 to control the device under test 302 to perform the model test task and return the model calculation result to the test device 301. The steps of the device under test performing the model calculation task and the step of the test device 301 performing the test of the model calculation efficiency of the model calculation system under test can refer to the introduction of any of the above method embodiments of the present invention.

[0201] The modular efficiency testing device provided by the embodiment of the present invention may include: An acquisition unit, used for acquiring a data set; A control unit, used to control the tested device to run the tested software framework to call the tested model and data set to perform model calculation tasks; A determination unit, used to determine the model performance parameters, the model data throughput parameters and the computing performance parameters according to the model computing task; determine the model computing efficiency of the tested model computing system according to the model performance parameters, the model data throughput parameters and the computing performance parameters to obtain the test results of the tested model computing system; The model computing system under test includes the model under test, the software framework under test and the device under test.

[0202] It should be noted that in each implementation of the test device for modular efficiency provided by the embodiment of the present invention, the division of units is only a logical functional division, and other division methods can be adopted. The connection method between different units can adopt electrical, mechanical or other connection methods. The separated units can be located in the same physical location or distributed on multiple network nodes. Each unit can be implemented in the form of hardware or in the form of a software functional unit. That is, part or all of the units provided in the embodiment of the present invention can be selected according to actual needs and the corresponding connection method or integration method can be adopted to achieve the purpose of the embodiment of the present invention.

[0203] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0204] Figure 4 A schematic diagram of the structure of a test device for modular efficiency provided in an embodiment of the present invention.

[0205] like Figure 4 As shown, the test device for modular efficiency provided by an embodiment of the present invention includes: a memory 410, used to store a computer program 411; a processor 420, used to execute the computer program 411, and when the computer program 411 is executed by the processor 420, the steps of the modular efficiency test method provided in any of the above embodiments are implemented.

[0206] Among them, the processor 420 may include one or more processing cores, such as a 3-core processor, an 8-core processor, etc. The processor 420 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 420 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 420 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 420 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0207] The memory 410 may include one or more non-volatile storage media, which may be non-transitory. The memory 410 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 410 is at least used to store the following computer program 411, wherein the computer program 411, after being loaded and executed by the processor 420, can implement the relevant steps in the modular efficiency test method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 410 may also include an operating system 412 and data 413, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 412 may be Windows or other types of operating systems. Data 413 may include, but is not limited to, the data involved in the above method.

[0208] In some embodiments, the simulation efficiency testing device may further include a display screen 430 , a power supply 440 , a communication interface 450 , an input / output interface 460 , a sensor 470 , and a communication bus 480 .

[0209] Those skilled in the art will understand that Figure 4 The structure shown in the figure does not constitute a limitation on the test equipment for modular efficiency, and may include more or less components than those shown in the figure.

[0210] The test device for modular arithmetic efficiency provided in an embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the steps of the modular arithmetic efficiency test method provided in the above embodiment, and the effect is the same as above.

[0211] An embodiment of the present invention provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the modular efficiency testing method provided in any one of the above embodiments can be implemented.

[0212] The non-volatile storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0213] For the introduction of the non-volatile storage medium provided in the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the analog efficiency testing method provided in the embodiment of the present invention, and the present invention will not elaborate on it here.

[0214] An embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the modular efficiency testing method provided in any one of the above embodiments.

[0215] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the above-mentioned method embodiment, and the effect thereof is the same as the modular efficiency testing method provided by the embodiment of the present invention, and the present invention will not elaborate on it here.

[0216] The above is a detailed introduction to a method, device, equipment and non-volatile storage medium for testing modular efficiency provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other. For the devices, equipment, non-volatile storage media and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0217] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A modular efficiency testing method, characterized in that: include: Get the dataset; Controlling the tested device to run the tested software framework to call the tested model and the data set to perform model calculation tasks; Determining model performance parameters, model data throughput parameters and computing performance parameters according to the model computing task; Determine the model computing efficiency of the tested model computing system according to the model performance parameter, the model data throughput parameter and the computing performance parameter to obtain a test result of the tested model computing system; The model under test computing system includes the model under test, the software framework under test and the device under test.

2. The modular efficiency testing method according to claim 1, characterized in that: Determining the model computing efficiency of the tested model computing system according to the model performance parameter, the model data throughput parameter and the computing performance parameter includes: Determine an effective throughput evaluation parameter of the tested model computing system according to the model performance parameter and the model data throughput parameter; The modular calculation efficiency is determined according to the effective throughput evaluation parameter and the calculation performance parameter.

3. The modular efficiency testing method according to claim 2, characterized in that: The larger the effective throughput corresponding to the effective throughput evaluation parameter is, the greater the modular calculation efficiency is.

4. The modular efficiency testing method according to claim 3, characterized in that: The model data throughput parameter includes a unit model data calculation amount; The unit model data calculation amount is the calculation amount required for the tested model to process unit input data; The better the model performance corresponding to the model performance parameter, the larger the effective throughput evaluation parameter; the effective throughput evaluation parameter is negatively correlated with the unit model data calculation amount.

5. The modular efficiency testing method according to claim 4, characterized in that: The model under test is a language model, and the unit model data calculation amount is the calculation amount required for the model under test to process each input tag.

6. The modular efficiency testing method according to claim 4, characterized in that: The model under test is an image processing model, and the unit model data calculation amount is the calculation amount required for the model under test to process each input image.

7. The modular efficiency testing method according to claim 3, characterized in that: The model data throughput parameter includes a throughput rate; The throughput rate is the rate at which the tested model processes input data per unit time; The better the model performance corresponding to the model performance parameter is, the larger the effective throughput evaluation parameter is; and the effective throughput evaluation parameter is positively correlated with the throughput rate.

8. The modular efficiency testing method according to claim 7, characterized in that: The model under test is a language model, and the throughput rate is the number of input tokens processed by the model under test per unit time.

9. The modular efficiency testing method according to claim 7, characterized in that: The model under test is an image processing model, and the throughput rate is the number of input images processed by the model under test per unit time.

10. The modular efficiency testing method according to claim 1, characterized in that: Determining the model computing efficiency of the tested model computing system according to the model performance parameter, the model data throughput parameter and the computing performance parameter includes: If the model performance parameter is within the range of the benchmark model performance parameter, determining the model calculation efficiency according to the model performance parameter, the model data throughput parameter and the calculation performance parameter; If the model performance parameter is not within the range of the benchmark model performance parameter, the model performance parameter is set to 0, and the modular efficiency is determined based on the model performance parameter, the model data throughput parameter and the computing performance parameter.

11. The method for testing modular efficiency according to claim 1, characterized in that: The model performance parameter is the model accuracy of the tested model, and the model accuracy is determined by the following formula: ； The model measured accuracy is the measured accuracy of the tested model in the model calculation task, and the benchmark accuracy is the minimum allowable accuracy of the tested model.

12. The modular efficiency testing method according to claim 1, characterized in that: The computing performance parameters include computing efficiency; The computing efficiency is positively correlated with the measured computing performance parameters of the model computing system under test for executing the model computing task, and the computing efficiency is negatively correlated with the theoretical peak performance parameters of the computing system of the device under test; The computing performance corresponding to the computing performance parameter is positively correlated with the computing efficiency.

13. The modular efficiency testing method according to claim 12, characterized in that: The step of determining the measured and calculated performance parameters comprises: Obtaining the total measured amount of calculations performed by the tested model computing system in executing the model computing task and the total measured time consumed by the tested model computing system in executing the model computing task; The measured calculation performance parameter is determined according to the measured total calculation amount and the measured total time consumption.

14. The modular efficiency testing method according to claim 12, characterized in that: The measured computing performance parameter is the number of floating-point operations per second of the tested device executing the model computing task; The computing system theoretical peak performance parameter is the theoretical peak number of floating-point operations per second of the device under test.

15. The method for testing modular efficiency according to claim 12, characterized in that: The step of determining the measured computing performance parameters and the step of determining the theoretical peak performance parameters of the computing system include: Obtaining a theoretical peak performance parameter of the computing system corresponding to a first data type of the device under test; If the actual data types in the model calculation tasks are all the first data types, the calculation performance parameters corresponding to the actual data types in the model calculation tasks are used as the measured calculation performance parameters; If the model calculation task contains an actual data type that is not the first data type, the calculation performance parameters corresponding to the actual data type in the model calculation task are equated with the calculation performance parameters corresponding to the first data type to obtain the measured calculation performance parameters.

16. The modular efficiency testing method according to claim 12, characterized in that: The step of determining the measured computing performance parameters and the step of determining the theoretical peak performance parameters of the computing system include: Obtaining a theoretical peak performance parameter of the computing system corresponding to a second data type of the device under test; If the actual data types in the model calculation task are all the second data type, the calculation performance parameters corresponding to the actual data types in the model calculation task are used as the measured calculation performance parameters; If the actual data type that is not the second data type exists in the model calculation task, the calculation performance parameters corresponding to the actual data type in the model calculation task are converted into the calculation performance parameters corresponding to the second data type to obtain the measured calculation performance parameters.

17. The modular efficiency testing method according to claim 1, characterized in that: The model computing efficiency of the model computing system under test is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, and is calculated by the following formula: ； Among them, the model accuracy is the model performance parameter; the unit model data calculation amount is the calculation amount required for the tested model to process unit input data; and the calculation efficiency is the calculation performance parameter.

18. The modular efficiency testing method according to claim 1, characterized in that: The model computing efficiency of the model computing system under test is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, and is calculated by the following formula: ； Among them, the model accuracy is the model performance parameter; the measured calculation amount per input tag is the model data throughput parameter, which represents the calculation amount required for the tested model to process each input tag; and the calculation efficiency is the calculation performance parameter.

19. The modular efficiency testing method according to claim 1, characterized in that: The model computing efficiency of the model computing system under test is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, and is calculated by the following formula: ； Among them, the model accuracy is the model performance parameter; the unit model data calculation amount is the calculation amount required for the tested model to process unit input data; the ratio of the measured computing performance parameters of the tested model computing system to execute the model computing task to the theoretical peak performance parameters of the computing system of the tested device represents the computing efficiency, and the computing efficiency is the computing performance parameter.

20. The modular efficiency testing method according to claim 1, characterized in that: The model computing efficiency of the model computing system under test is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, and is calculated by the following formula: ； Among them, the model accuracy is the model performance parameter; the unit model data calculation amount is the calculation amount required for the tested model to process unit input data; the ratio of the measured total calculation amount to the measured total time consumption of the model calculation task is the measured calculation performance parameter of the tested model calculation system to perform the model calculation task; the ratio of the measured calculation performance parameter to the theoretical peak performance parameter of the calculation system of the tested device represents the calculation efficiency, and the calculation efficiency is the calculation performance parameter.

21. The modular efficiency testing method according to claim 1, characterized in that: The model computing efficiency of the model computing system under test is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, and is calculated by the following formula: ； Among them, the model accuracy is the model performance parameter; the unit model data calculation amount is the calculation amount required for the tested model to process unit input data; the ratio of the product of the unit model data calculation amount and the total number of input data to the measured total time consumed by the model calculation task is the measured calculation performance parameter of the tested model calculation system to perform the model calculation task; the ratio of the measured calculation performance parameter to the theoretical peak performance parameter of the calculation system of the tested device represents the calculation efficiency, and the calculation efficiency is the calculation performance parameter.

22. The modular efficiency testing method according to claim 1, characterized in that: The model computing efficiency of the model computing system under test is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, and is calculated by the following formula: ； Among them, the model accuracy is the model performance parameter; the ratio of the total number of input data to the measured total time consumption is the model data throughput parameter; and the theoretical peak performance parameter of the computing system is the computing performance parameter.

23. The modular efficiency testing method according to claim 1, characterized in that: The model computing efficiency of the model computing system under test is determined according to the model performance parameter, the model data throughput parameter and the computing performance parameter, and is calculated by the following formula: ； Among them, the model accuracy is the model performance parameter; the throughput rate is the model data throughput parameter, which represents the rate at which the tested model processes input data per unit time; and the computing system theoretical peak performance parameter is the computing performance parameter.

24. The modular efficiency testing method according to claim 1, characterized in that: Determining the model computing efficiency of the tested model computing system according to the model performance parameter, the model data throughput parameter and the computing performance parameter to obtain a test result of the tested model computing system includes: Determine the comprehensive modular computing efficiency of the tested model computing system according to the modular computing efficiency of the tested model computing system in multiple test scenarios; Taking the comprehensive simulation efficiency as the test result; The test scenarios correspond to the data sets one by one.

25. The modular efficiency testing method according to claim 1, characterized in that: Determining the model computing efficiency of the tested model computing system according to the model performance parameter, the model data throughput parameter and the computing performance parameter to obtain a test result of the tested model computing system includes: Determine the comprehensive modular computing efficiency of the tested model computing system according to the modular computing efficiency of the tested model computing system in multiple test scenarios; Taking the comprehensive simulation efficiency as the test result; Among them, one of the test scenarios corresponds to one or more of the data sets.

26. The modular efficiency testing method according to claim 25, characterized in that: The steps of determining the modular efficiency corresponding to a single test scenario include: If the test scenario corresponds to a data set, the analog calculation efficiency corresponding to the data set is used as the analog calculation efficiency corresponding to the test scenario; If the test scenario corresponds to multiple data sets, the analog computing efficiency corresponding to the test scenario is determined according to the analog computing efficiencies corresponding to each of the data sets corresponding to the test scenario.

27. The modular efficiency testing method according to claim 26, characterized in that: Determining the modular calculation efficiency corresponding to the test scenario according to the modular calculation efficiency corresponding to each of the data sets corresponding to the test scenario includes: The arithmetic mean of the analog calculation efficiencies corresponding to the data sets is taken as the analog calculation efficiency corresponding to the test scenario.

28. The modular efficiency testing method according to claim 26, characterized in that: Determining the modular calculation efficiency corresponding to the test scenario according to the modular calculation efficiency corresponding to each of the data sets corresponding to the test scenario includes: The geometric mean of the analog calculation efficiency corresponding to each of the data sets is used as the analog calculation efficiency corresponding to the test scenario.

29. The modular efficiency testing method according to claim 25, characterized in that: The steps of determining the modular efficiency corresponding to a single test scenario include: If the test scenario corresponds to a data set, the analog calculation efficiency corresponding to the data set is used as the analog calculation efficiency corresponding to the test scenario; If the test scenario corresponds to multiple data sets, the single-scenario model performance parameters corresponding to the test scenario are determined according to the model performance parameters corresponding to each data set corresponding to the test scenario; the single-scenario model data throughput parameters corresponding to the test scenario are determined according to the model data throughput parameters corresponding to each data set corresponding to the test scenario; the single-scenario computing performance parameters corresponding to the test scenario are determined according to the computing performance parameters corresponding to each data set corresponding to the test scenario; The model calculation efficiency corresponding to the test scenario is determined according to the single-scenario model performance parameter, the single-scenario model data throughput parameter and the single-scenario calculation performance parameter.

30. The modular efficiency testing method according to claim 29, characterized in that: Determining the single-scenario model performance parameter corresponding to the test scenario according to the model performance parameters corresponding to each data set corresponding to the test scenario, including: taking the arithmetic mean of the model performance parameters corresponding to each data set as the single-scenario model performance parameter; Determining the single-scenario model data throughput parameter corresponding to the test scenario according to the model data throughput parameters corresponding to each data set corresponding to the test scenario, including: taking the arithmetic mean of the model data throughput parameters corresponding to each data set as the single-scenario model data throughput parameter; Determining the single-scenario computing performance parameter corresponding to the test scenario according to the computing performance parameters corresponding to each data set corresponding to the test scenario includes: taking the arithmetic mean of the computing performance parameters corresponding to each data set as the single-scenario computing performance parameter.

31. The modular efficiency testing method according to claim 29, characterized in that: Determining the single-scenario model performance parameter corresponding to the test scenario according to the model performance parameters corresponding to each data set corresponding to the test scenario includes: taking the geometric mean of the model performance parameters corresponding to each data set as the single-scenario model performance parameter; Determining the single-scenario model data throughput parameter corresponding to the test scenario according to the model data throughput parameters corresponding to each data set corresponding to the test scenario, including: taking the geometric mean of the model data throughput parameters corresponding to each data set as the single-scenario model data throughput parameter; Determining the single-scenario computing performance parameter corresponding to the test scenario according to the computing performance parameters corresponding to each data set corresponding to the test scenario includes: taking the geometric mean of the computing performance parameters corresponding to each data set as the single-scenario computing performance parameter.

32. The modular efficiency testing method according to claim 24 or 25, characterized in that: Determining the comprehensive modular computing efficiency of the tested model computing system according to the modular computing efficiency of the tested model computing system in multiple test scenarios includes: The arithmetic mean of the analog calculation efficiencies corresponding to the multiple test scenarios is taken as the comprehensive analog calculation efficiency.

33. The modular efficiency testing method according to claim 32, characterized in that: The arithmetic mean of the modular efficiency corresponding to the multiple test scenarios is used as the comprehensive modular efficiency, which is calculated by the following formula: ； in, represents the comprehensive modular efficiency, Indicates that the model computing system under test is in The modular computing efficiency under the test scenario is Indicates that the model computing system under test is in A first weight coefficient under the test scenario.

34. The modular efficiency testing method according to claim 24 or 25, characterized in that: Determining the comprehensive modular computing efficiency of the tested model computing system according to the modular computing efficiency of the tested model computing system in multiple test scenarios includes: The geometric mean of the analog calculation efficiencies corresponding to the multiple test scenarios is used as the comprehensive analog calculation efficiency.

35. The modular efficiency testing method according to claim 34, characterized in that: The geometric mean of the analog calculation efficiency corresponding to the multiple test scenarios is used as the comprehensive analog calculation efficiency, which is calculated by the following formula: ； in, represents the comprehensive modular efficiency, Indicates that the model computing system under test is in The module calculation efficiency under the test scenario is represents cumulative multiplication calculation, Indicates that the model computing system under test is in A second weight coefficient under the test scenario.

36. The modular efficiency testing method according to claim 1, characterized in that: Controlling the tested device to run the tested software framework to call the tested model and the data set to perform the model calculation task, including: Loading the tested software framework on the tested device and initializing the tested software framework; The tested software framework is started to call the tested model and the data set to execute the model calculation task.

37. The modular efficiency testing method according to claim 36, characterized in that: Loading the tested software framework on the tested device and initializing the tested software framework includes: Loading the library file of the model computing system under test on the device under test; Acquire the type of the interface function of the tested model computing system through the calling interface of the library file; Initialize the interface function according to the type of the interface function.

38. The modular efficiency testing method according to claim 37, characterized in that: The types of the interface functions include at least: initialization function, completion function, data loading function, data unloading function, model calculation control function, and test parameter return function; The initialization function is the interface function that performs initialization according to the test configuration information; the completion function is the interface function of the task performed by the tested model computing system after completing the model computing task; the data loading function is the interface function for loading model computing data; the data unloading function is the interface function for unloading the model computing data; the model computing control function is the interface function for executing the model computing task; the test parameter return function is the output for calculating the test parameters required for at least one of the model performance parameters, the model data throughput parameters and the computing performance parameters.

39. The modular efficiency testing method according to claim 38, characterized in that: Starting the tested software framework to call the tested model to execute the model calculation task includes: Starting the tested software framework to execute the model calculation task; Obtaining a first result returned by calling the test parameter return function after the tested software framework calls the data loading function to load the model calculation data; Obtaining a second result of calling the test parameter return function after the tested software framework calls the model calculation control function to execute the model calculation task; Waiting for the tested software framework to output a third result after completing the model calculation task; Obtaining a fourth result returned by calling the test parameter return function after the tested software framework calls the data unloading function to unload the model calculation data; The model accuracy of the tested model is calculated according to the execution result of the model calculation task, including: The model accuracy is calculated according to at least one of the first result, the second result, the third result, and the fourth result.

40. A test system for modular efficiency, characterized in that: Includes test equipment and equipment under test; Wherein, the device under test is used to run the software framework under test to perform model calculation tasks according to the model under test and the data set; The test device is used to obtain the data set, control the tested device to run the tested software framework to call the tested model and the data set to perform the model calculation task; determine the model performance parameters, model data throughput parameters and calculation performance parameters according to the model calculation task; determine the model calculation efficiency of the tested model calculation system according to the model performance parameters, the model data throughput parameters and the calculation performance parameters to obtain the test results of the tested model calculation system; The model under test computing system includes the model under test, the software framework under test and the device under test.

41. A test device for modular efficiency, characterized in that: include: An acquisition unit, used for acquiring a data set; A control unit, used to control the tested device to run the tested software framework to call the tested model and the data set to perform model calculation tasks; A determination unit, configured to determine a model performance parameter, a model data throughput parameter, and a computing performance parameter according to the model computing task; Determine the model computing efficiency of the tested model computing system according to the model performance parameter, the model data throughput parameter and the computing performance parameter to obtain a test result of the tested model computing system; The model under test computing system includes the model under test, the software framework under test and the device under test.

42. A test device for modular efficiency, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the modular efficiency testing method as described in any one of claims 1 to 39 are implemented.

43. A non-volatile storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the modular efficiency testing method as described in any one of claims 1 to 39 are implemented.

44. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the modular efficiency testing method as described in any one of claims 1 to 39 are implemented.

Images