Modulation efficiency test method, system, device, equipment, medium and product

Abstract

The invention relates to the technical field of artificial intelligence, and particularly discloses a modular computation efficiency testing method, system, device and equipment, a medium and a product. Determining model performance parameters and calculation performance parameters according to the model calculation task, and determining the model calculation efficiency of a tested model calculation system according to the model performance parameters, the calculation performance parameters and model parameter quantity evaluation parameters of the tested model. Therefore, quantitative evaluation of the model calculation performance of the tested model calculation system comprising the tested model, the tested software framework and the tested equipment is realized. As an evaluation scheme suitable for various model calculation tasks, the comparability indexes of different models running on different devices are enhanced, and a reference basis is provided for improvement and perfection of the model and a model calculation system.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, system, device, equipment, medium and product for testing analog computing efficiency. Background Art

[0002] With the rapid development of artificial intelligence technology, artificial intelligence models are showing trends such as large data processing volume and high computational complexity. The computing power requirements of model computing tasks for hardware devices are rapidly increasing, and a large amount of resources are consumed. At present, the evaluation indicators for model computing performance are usually performance evaluation indicators for the model itself, such as model accuracy. The relevant technology lacks evaluation indicators for the model computing system composed of software and hardware used to perform model calculations, as well as performance testing methods for the model computing system.

[0003] How to test and quantitatively evaluate the model computing performance of a model computing system is a technical problem that needs to be solved by those skilled in the art. Summary of the invention

[0004] The purpose of the present invention is to provide a model calculation efficiency testing method, system, device, equipment, medium and product for testing and evaluating the model calculation performance of the model calculation system.

[0005] In order to solve the above technical problems, the present invention provides a modular efficiency testing method, comprising: Get the dataset; Controlling the tested device to run the tested software framework to call the tested model and the data set to perform model calculation tasks; Determining model performance parameters and calculation performance parameters according to the model calculation task; Determine the model calculation efficiency of the tested model calculation system according to the model performance parameter, the calculation performance parameter and the model parameter quantity evaluation parameter of the tested model to obtain the test result of the tested model calculation system; The model under test computing system includes the model under test, the software framework under test and the device under test.

[0006] On the one hand, determining the model calculation efficiency of the tested model calculation system according to the model performance parameter, the calculation performance parameter and the model parameter quantity evaluation parameter of the tested model includes: Determining the model parameter efficiency of the tested model computing system according to the model performance parameter and the model parameter quantity evaluation parameter; The modular efficiency is determined based on the model parameter efficiency and the computing performance parameter.

[0007] On the other hand, the greater the model parameter efficiency, the greater the modular calculation efficiency.

[0008] On the other hand, the model parameter efficiency of the tested model calculation system is determined according to the model performance parameter and the model parameter quantity evaluation parameter, and is calculated by the following formula: Model calculation efficiency = model parameter efficiency × calculation efficiency; The calculation efficiency is the calculation performance parameter.

[0009] On the other hand, determining the model parameter efficiency of the tested model computing system according to the model performance parameter and the model parameter quantity evaluation parameter includes: Normalizing the model performance parameters and the model parameter quantity evaluation parameters respectively; The model parameter efficiency is determined based on the normalized model performance parameter, the normalized model parameter quantity evaluation parameter and the calculation performance parameter.

[0010] On the other hand, the model performance parameters are normalized, including: The normalized model performance parameters are determined according to the measured model performance parameters in the model calculation task and the reference model performance parameters of the tested model.

[0011] On the other hand, the measured model performance parameter is the model accuracy of the tested model, and the reference model performance parameter is the reference accuracy of the tested model.

[0012] On the other hand, normalizing the model parameter evaluation parameter includes: Determining the normalized model parameter evaluation parameter according to the activation parameter of the tested model and the reference parameter of the tested model; The activation parameter amount is the parameter amount in which the tested model participates in the calculation in the model calculation task.

[0013] On the other hand, the model parameter efficiency of the tested model calculation system is determined according to the model performance parameter and the model parameter quantity evaluation parameter, and is calculated by the following formula: ; Among them, the model accuracy is the measured model performance parameter in the model calculation task, the reference accuracy is the reference model performance parameter of the tested model, and the activation parameter amount is the parameter amount of the tested model involved in the calculation in the model calculation task.

[0014] On the other hand, determining the model calculation efficiency of the tested model calculation system according to the model performance parameter, the calculation performance parameter and the model parameter quantity evaluation parameter of the tested model includes: If the model performance parameter is within the range of the benchmark model performance parameter, determining the model calculation efficiency according to the model performance parameter, the calculation performance parameter and the model parameter quantity evaluation parameter; If the model performance parameter is not within the range of the benchmark model performance parameter, the model performance parameter is set to 0, and the simulation efficiency is determined based on the model performance parameter, the calculation performance parameter and the model parameter quantity evaluation parameter.

[0015] On the other hand, the model performance parameter is the model accuracy of the tested model, and the model accuracy is determined by the following formula: ; The model measured accuracy is the measured accuracy of the tested model in the model calculation task, and the benchmark accuracy is the minimum allowable accuracy of the tested model.

[0016] On the other hand, the step of determining the calculation performance parameter includes: The computing performance parameters are determined based on the measured computing performance parameters of the model computing system under test executing the model computing task and the theoretical peak performance parameters of the computing system of the device under test.

[0017] On the other hand, the computing performance parameter is determined according to the measured computing performance parameter of the model computing system under test executing the model computing task and the theoretical peak performance parameter of the computing system of the device under test, and is calculated by the following formula: ; The computing efficiency is the computing performance parameter.

[0018] On the other hand, the measured computing performance parameter is the number of floating-point operations per second of the tested device executing the model computing task; The computing system theoretical peak performance parameter is the theoretical peak number of floating-point operations per second of the device under test.

[0019] On the other hand, the step of determining the measured computing performance parameter and the step of determining the theoretical peak performance parameter of the computing system include: Obtaining a theoretical peak performance parameter of the computing system corresponding to a first data type of the device under test; If the actual data types in the model calculation task are all the first data type, the calculation performance parameters corresponding to the actual data types in the model calculation task are used as the measured calculation performance parameters; If the model calculation task contains an actual data type that is not the first data type, the calculation performance parameters corresponding to the actual data type in the model calculation task are equated with the calculation performance parameters corresponding to the first data type to obtain the measured calculation performance parameters.

[0020] On the other hand, the first data type is a single-precision floating point type.

[0021] On the other hand, the step of determining the measured computing performance parameter and the step of determining the theoretical peak performance parameter of the computing system include: Obtaining a theoretical peak performance parameter of the computing system corresponding to a second data type of the device under test; If the actual data types in the model calculation task are all the second data type, the calculation performance parameters corresponding to the actual data types in the model calculation task are used as the measured calculation performance parameters; If the actual data type that is not the second data type exists in the model calculation task, the calculation performance parameters corresponding to the actual data type in the model calculation task are converted into the calculation performance parameters corresponding to the second data type to obtain the measured calculation performance parameters.

[0022] On the other hand, determining the model calculation efficiency of the tested model calculation system according to the model performance parameter, the calculation performance parameter and the model parameter quantity evaluation parameter of the tested model to obtain the test result of the tested model calculation system includes: Determine the comprehensive modular computing efficiency of the tested model computing system according to the modular computing efficiency of the tested model computing system in multiple test scenarios; Taking the comprehensive simulation efficiency as the test result; The test scenarios correspond to the data sets one by one.

[0023] On the other hand, determining the model computing efficiency of the tested model computing system according to the model performance parameter, the model data throughput parameter and the computing performance parameter to obtain a test result of the tested model computing system includes: Determine the comprehensive modular computing efficiency of the tested model computing system according to the modular computing efficiency of the tested model computing system in multiple test scenarios; Taking the comprehensive simulation efficiency as the test result; Among them, one of the test scenarios corresponds to one or more of the data sets.

[0024] On the other hand, the step of determining the modular efficiency corresponding to a single test scenario includes: If the test scenario corresponds to a data set, the analog calculation efficiency corresponding to the data set is used as the analog calculation efficiency corresponding to the test scenario; If the test scenario corresponds to multiple data sets, the analog computing efficiency corresponding to the test scenario is determined according to the analog computing efficiencies corresponding to each of the data sets corresponding to the test scenario.

[0025] On the other hand, determining the analog calculation efficiency corresponding to the test scenario according to the analog calculation efficiency corresponding to each of the data sets corresponding to the test scenario includes: The arithmetic mean of the analog calculation efficiencies corresponding to the data sets is taken as the analog calculation efficiency corresponding to the test scenario.

[0026] On the other hand, determining the analog calculation efficiency corresponding to the test scenario according to the analog calculation efficiency corresponding to each of the data sets corresponding to the test scenario includes: The geometric mean of the analog calculation efficiency corresponding to each of the data sets is used as the analog calculation efficiency corresponding to the test scenario.

[0027] On the other hand, determining the analog calculation efficiency corresponding to the test scenario according to the analog calculation efficiency corresponding to each of the data sets corresponding to the test scenario includes: Determine the single-scenario model performance parameter corresponding to the test scenario according to the model performance parameters corresponding to each data set corresponding to the test scenario; Determine the single-scenario computing performance parameter corresponding to the test scenario according to the computing performance parameters corresponding to each data set corresponding to the test scenario; The model calculation efficiency corresponding to the test scenario is determined according to the single-scenario model performance parameter, the single-scenario calculation performance parameter and the model parameter quantity evaluation parameter.

[0028] On the other hand, determining the single-scenario model performance parameter corresponding to the test scenario according to the model performance parameters corresponding to each data set corresponding to the test scenario includes: taking the arithmetic mean of the model performance parameters corresponding to each data set as the single-scenario model performance parameter; Determining the single-scenario computing performance parameter corresponding to the test scenario according to the computing performance parameters corresponding to each data set corresponding to the test scenario includes: taking the arithmetic mean of the computing performance parameters corresponding to each data set as the single-scenario computing performance parameter.

[0029] On the other hand, determining the single-scenario model performance parameter corresponding to the test scenario according to the model performance parameters corresponding to each data set corresponding to the test scenario includes: taking the geometric mean of the model performance parameters corresponding to each data set as the single-scenario model performance parameter; Determining the single-scenario computing performance parameter corresponding to the test scenario according to the computing performance parameters corresponding to each data set corresponding to the test scenario includes: taking the geometric mean of the computing performance parameters corresponding to each data set as the single-scenario computing performance parameter.

[0030] On the other hand, according to the modular computing efficiency of the tested model computing system in multiple test scenarios, determining the comprehensive modular computing efficiency of the tested model computing system includes: The arithmetic mean of the analog calculation efficiencies corresponding to the multiple test scenarios is taken as the comprehensive analog calculation efficiency.

[0031] On the other hand, the arithmetic mean of the modular efficiency corresponding to the multiple test scenarios is used as the comprehensive modular efficiency, which is calculated by the following formula: ; in, represents the comprehensive modular efficiency, Indicates that the model computing system under test is in The modular computing efficiency under the test scenario is Indicates that the model computing system under test is in A first weight coefficient under the test scenario.

[0032] On the other hand, according to the modular computing efficiency of the tested model computing system in multiple test scenarios, determining the comprehensive modular computing efficiency of the tested model computing system includes: The geometric mean of the analog calculation efficiencies corresponding to the multiple test scenarios is used as the comprehensive analog calculation efficiency.

[0033] On the other hand, the geometric mean of the analog calculation efficiencies corresponding to the multiple test scenarios is used as the comprehensive analog calculation efficiency, which is calculated by the following formula: ; in, represents the comprehensive modular efficiency, Indicates that the model computing system under test is in The module calculation efficiency under the test scenario, represents cumulative multiplication calculation, Indicates that the model computing system under test is in A second weight coefficient under the test scenario.

[0034] On the other hand, controlling the tested device to run the tested software framework to call the tested model and the data set to perform the model calculation task includes: Loading the tested software framework on the tested device and initializing the tested software framework; The tested software framework is started to call the tested model and the data set to execute the model calculation task.

[0035] On the other hand, loading the tested software framework on the tested device and initializing the tested software framework includes: Loading the library file of the model computing system under test on the device under test; Acquire the type of the interface function of the tested model computing system through the calling interface of the library file; Initialize the interface function according to the type of the interface function.

[0036] On the other hand, the types of the interface functions include at least: initialization function, completion function, data loading function, data unloading function, model calculation control function; The initialization function is the interface function that performs initialization according to the test configuration information; the completion function is the interface function of the task performed by the tested model calculation system after completing the model calculation task; the data loading function is the interface function that loads the model calculation data; the data unloading function is the interface function that unloads the model calculation data; and the model calculation control function is the interface function that executes the model calculation task.

[0037] On the other hand, starting the tested software framework to call the tested model to execute the model calculation task includes: Starting the tested software framework to execute the model calculation task; Obtaining a first result returned after the tested software framework calls the data loading function to load the model calculation data; Obtaining a second result of the tested software framework calling the model calculation control function to execute the model calculation task; Waiting for the tested software framework to output a third result after completing the model calculation task; Obtaining a fourth result returned by the tested software framework calling the data unloading function to unload the model calculation data; The model accuracy of the tested model is calculated according to the execution result of the model calculation task, including: The model accuracy is calculated according to at least one of the first result, the second result, the third result, and the fourth result.

[0038] In order to solve the above technical problems, the present invention also provides a modular efficiency testing system, including a testing device and a tested device; Wherein, the device under test is used to run the software framework under test to perform model calculation tasks according to the model under test and the data set; The test device is used to obtain the data set, control the tested device to run the tested software framework to call the tested model and the data set to perform the model calculation task; determine the model performance parameters and the calculation performance parameters according to the model calculation task; determine the model calculation efficiency of the tested model calculation system according to the model performance parameters, the calculation performance parameters and the model parameter quantity evaluation parameters of the tested model, so as to obtain the test result of the tested model calculation system; The model under test computing system includes the model under test, the software framework under test and the device under test.

[0039] In order to solve the above technical problems, the present invention also provides a modular efficiency testing device, comprising: An acquisition unit, used for acquiring a data set; A control unit, used to control the tested device to run the tested software framework to call the tested model and the data set to perform model calculation tasks; A determination unit, configured to determine a model performance parameter and a calculation performance parameter according to the model calculation task; and determine a model calculation efficiency of the model calculation system under test according to the model performance parameter, the calculation performance parameter and a model parameter quantity evaluation parameter of the model under test, so as to obtain a test result of the model calculation system under test; The model under test computing system includes the model under test, the software framework under test and the device under test.

[0040] In order to solve the above technical problems, the present invention also provides a modular efficiency testing device, comprising: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the modular efficiency testing method as described in any one of the above items are implemented.

[0041] In order to solve the above technical problems, the present invention also provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the modular efficiency testing method as described in any one of the above items are implemented.

[0042] In order to solve the above technical problems, the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the modular efficiency testing method as described in any one of the above items.

[0043] The modular calculation efficiency testing method provided by the present invention has the beneficial effect of determining model performance parameters and calculation performance parameters according to the model calculation task after controlling the tested software framework to call the tested model to perform the model calculation task according to the data set on the tested device, so as to quantify the performance of the tested model and the performance of the software and hardware architecture respectively, and determining the modular calculation efficiency of the tested model calculation system according to the model performance parameters, calculation performance parameters and the model parameter quantity evaluation parameters of the tested model to obtain the test result of the tested model calculation system, thereby realizing the quantitative evaluation of the model calculation performance of the tested model calculation system including the tested model, the tested software framework and the tested device. The modular calculation efficiency and the testing method provided by the present invention are used as an evaluation scheme suitable for various model calculation tasks, which enhances the comparability index of different models running on different devices, provides a reference basis for the improvement and perfection of models and model calculation systems, and then can quantitatively evaluate the accuracy and multivariate computing power performance of different models, thereby ensuring the improvement of effective traction model capabilities and the reduction of application costs.

[0044] The analog arithmetic efficiency testing system, analog arithmetic efficiency testing device, analog arithmetic efficiency testing equipment, non-volatile storage medium and computer program product provided by the present invention have the above-mentioned beneficial effects, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A flowchart of a modular calculation efficiency testing method provided by an embodiment of the present invention; Figure 2 A timing diagram of a modular efficiency testing method provided by an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a modular efficiency testing system provided by an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a simulation efficiency testing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The core of the present invention is to provide a model calculation efficiency testing method, system, device, equipment, medium and product for testing and evaluating the model calculation performance of the model calculation system.

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] To facilitate understanding of the technical solution provided by the embodiment of the present invention, some key terms used in the embodiment of the present invention are first explained here.

[0050] Computing power: It is used to measure the ability of computing systems (such as computers, servers, data centers, etc.) to process information and perform calculations. The most commonly used unit is floating point operations per second (FLOPS).

[0051] Model Computational Efficiency: A solution for evaluating the efficiency of large models. It is used to reflect the comprehensive efficiency of the tested model under the hardware system and software framework. The evaluation objects are the model itself, the tested software framework, and the tested AI bare metal server (computing power base). It is a comprehensive evaluation of the above three types of performance factors.

[0052] Floating point operations per second (FLOPS) is an indicator to measure the processing power of a computer system or computing device. It refers to the number of floating point operations (addition, subtraction, multiplication, division, etc.) that a system or device can perform per unit time.

[0053] Measured computing performance (in TFLOPS): The actual amount of floating-point operations such as addition, subtraction, multiplication, and division performed by the statistical model per unit time during inference and training.

[0054] The measured total computing power (in TFLOPS) is the total computing power of floating-point operations such as addition, subtraction, multiplication and division during the training or inference process.

[0055] Theoretical peak performance of the computing system (in TFLOPS): a theoretical value calculated based on the main frequency, number of cores and other characteristics of the chip hardware itself, provided by the corresponding chip manufacturer; the single-precision floating-point theoretical peak performance can be used as the basis.

[0056] Figure 1 A flowchart of a modular efficiency testing method provided in an embodiment of the present invention.

[0057] like Figure 1As shown, the modular efficiency testing method provided by the embodiment of the present invention includes: S101: Obtain a data set; S102: Control the tested device to run the tested software framework to call the tested model and data set to perform model calculation tasks; S103: Determine model performance parameters and calculation performance parameters according to the model calculation task; S104: determining the model calculation efficiency of the tested model calculation system according to the model performance parameter, the calculation performance parameter and the model parameter quantity evaluation parameter of the tested model, so as to obtain the test result of the tested model calculation system; The model computing system under test includes the model under test, the software framework under test and the device under test.

[0058] In view of the lack of evaluation schemes for the accuracy and efficiency of model computing power in related technologies, the embodiment of the present invention proposes an evaluation index for the model computing efficiency of the model computing system. The model computing efficiency provided by the embodiment of the present invention can be used as a unified measurement scale for model computing systems using different models, different software frameworks, and different devices, enhance the comparability of model computing performed by different devices, and provide directional guidance for the improvement of models, software frameworks, and hardware.

[0059] After determining the tested model, tested software framework and tested device included in the tested model computing system, the model computing efficiency testing method provided by the embodiment of the present invention can be executed.

[0060] The modular efficiency testing method provided in the embodiment of the present invention can be implemented in software by designing and deploying test scripts, and can be executed based on the tested device or another testing device in hardware.

[0061] When performing tests on the model computing system under test, the test environment requirements may include: The ambient temperature is (25±5)℃, the relative humidity is 45%~75%, and the atmospheric pressure is 86kPa~106kPa; For the equipment under test with a nominal power less than 1.5kW, the test power supply should be AC ​​(220±1%) V; for the equipment under test with a nominal power greater than 1.5kW, the test power supply should be AC ​​(220±4%) V; For the equipment under test with a nominal power less than 1.5kW, the total harmonic distortion of the test power supply shall not exceed 2%; for the equipment under test with a nominal power greater than 1.5kW, the total harmonic distortion of the test power supply shall not exceed 5%; The power input frequency of all tested equipment should be (50±1.0%) Hz; The test environment should avoid interference factors such as strong magnetic fields and strong vibrations.

[0062] Requirements for the device under test may include: The device under test must include the central processing unit (CPU), memory, hard disk, and operating system. The equipment under test should use a standard power supply. During the test, all power supplies should be connected to an AC power source. The tested device should set all software options to default state. Power management technology and / or power saving features can only be tested with the tested server enabled by default. The device under test should be tested using the operating system declared by the manufacturer.

[0063] It should be noted that the tested model in the embodiment of the present invention can be any type of model, which can be divided from the perspectives of function, structure and application field, including but not limited to supervised learning models (classification models, regression models, etc.), unsupervised learning models (clustering models, dimensionality reduction models, association rule learning models, etc.), reinforcement learning models (strategy learning models, value learning models, strategy optimization models, etc.), semi-supervised learning models (models trained with labeled and unlabeled data, such as self-training models, etc.), integrated learning models (combining multiple models to improve performance, such as Bagging, Boosting, Stacking, etc.), deep learning models (convolutional neural networks, recurrent neural networks, long short-term memory networks, gated recurrent units, Transformer models, etc.), generative models (generative adversarial networks, variational autoencoders, etc.), explanatory models (decision trees, rule engines, etc.), probability models (Bayesian networks, hidden Markov models, etc.), meta-learning models, etc.

[0064] The tested software framework in the embodiment of the present invention is software used to call the tested model to perform model calculation tasks according to a data set.

[0065] In the tested model computing system of the embodiment of the present invention, the number of tested devices can be one or more. When the number of tested devices is multiple, different tested devices can be devices of the same type or heterogeneous computing devices. In some optional implementations of the embodiment of the present invention, the tested device can be a server, which can be an artificial intelligence bare metal server (computing power base).

[0066] In the embodiment of the present invention, the model calculation task may be a model training task or a model reasoning task.

[0067] Among them, the model training task is to adjust the parameters of the model through the optimization algorithm so that it can accurately learn knowledge from the data set, so as to accurately process the unknown data. In an embodiment of the present invention, when the number of devices under test is multiple, the model training task can be a synchronous training task or an asynchronous training task. Among them, the synchronous training task is that all devices under test read the model parameters at the same time and use the same model parameters for calculation. After each iteration, all devices under test will update the model parameters synchronously. The asynchronous training task is that different devices under test read the model parameters and calculate independently, without waiting for other devices under test to complete, and the update of the model parameters will be asynchronously shared with other devices under test.

[0068] Among them, the model reasoning task is to use the tested model (a trained model) to process unknown data. In an embodiment of the present invention, when there are multiple devices under test, the model reasoning task can be a synchronous reasoning (SynchronousInference) task or an asynchronous reasoning (Asynchronous Inference) task. Among them, the synchronous reasoning task is that the tested device obtains the next copy of the data to be processed for corresponding processing after each model reasoning is completed. In synchronous reasoning, at any given time point, only one reasoning request is executed, and other reasoning requests must wait for the current reasoning request to be completed before they can start. Synchronous reasoning is usually deployed in a linear blocking manner. If there is a reasoning task being executed in the current process, other reasoning tasks need to wait for the reasoning task to be completed before they can be executed. Asynchronous reasoning tasks allow reasoning tasks and other thread tasks to be executed in parallel, thereby improving resource utilization. In asynchronous reasoning, a piece of data to be processed can be handed over to the model for reasoning, and the program immediately obtains the next piece of data to be processed for corresponding preprocessing, and then waits for the last reasoning calculation to be completed before handing the data to be processed to the model for reasoning.

[0069] In an embodiment of the present invention, the required data set is determined according to at least one of the tested model, the tested software framework, the tested device and the model calculation task. In practical applications, the same model may have the ability to perform multiple application tasks. For example, a language model can be used to perform problem-solving tasks, code generation tasks, language understanding tasks, etc., and different application tasks correspond to different data sets. Then in S101, the acquired data set may be a data set corresponding to the application task to which the tested model is applicable. In addition, according to the different data types used in the model calculation task and the different types of data to be processed, an application task may correspond to one or more data sets. For example, an image processing task may correspond to a cat image data set and a dog image data set. An application task may correspond to data sets of different data types, such as a data set of single-precision floating-point data and a data set of integer data.

[0070] In S102, the test script as described above is applied to control the device under test to run the software framework under test to call the model under test and the data set to perform the model calculation task.

[0071] After the model computing system under test completes the model computing task, the model computing efficiency of the model computing system under test is calculated.

[0072] The evaluation index of modular computing efficiency proposed in the embodiment of the present invention is intended to evaluate the performance of the model computing system from two perspectives: the model itself and the software and hardware framework. As a comparability index that can measure various combinations of model computing systems, it is evaluated from the perspective of the computing power, accuracy and efficiency of the model. The meaning of this index can be further expressed as: the faster the execution rate of the model computing task, the higher the modular computing efficiency; the fewer resources occupied by the model computing task, the higher the modular computing efficiency; the higher the utilization rate of limited resources by the model computing task, the higher the modular computing efficiency. To this end, in the embodiment of the present invention, the modular computing efficiency of the model computing system under test is determined based on the model performance parameters of the model computing system under test, the model parameter quantity evaluation parameters of the model computing system under test, and the computing performance parameters of the model computing system under test.

[0073] In the embodiment of the present invention, the model performance parameter is used to quantitatively represent the performance of the tested model. The performance of the tested model can be represented from the perspectives of the accuracy of the results, the reliability of the results, the stability of the model, etc. In practical applications, the type of model performance parameter is selected according to the requirements of the model calculation task and the characteristics of the data to be processed by the tested model.

[0074] In some optional implementations of the embodiments of the present invention, the model performance parameter may adopt model accuracy. Model accuracy may adopt but is not limited to at least one of accuracy, precision, recall, F1 score, receiver operating characteristic (ROC) curve and area under the curve (AUC) value, mean square error (MSE), root mean square error (RMSE), mean absolute error (MAE), logarithmic loss (Log Loss), accuracy (AP) and mean average precision (mAP), and determination coefficient (R²). In general, model accuracy is a key indicator for measuring model performance, which indicates the correctness of the model in data processing (such as classification or prediction).

[0075] Among them, accuracy is mainly used to measure classification tasks. For binary classification problems, accuracy can be simply expressed as the number of correctly classified samples (Number of Correct Predictions) divided by the total number of samples (Number of Correct Predictions).

[0076] Precision is used to measure the proportion of samples that are actually positive among the samples predicted by the model to be positive. It is calculated by dividing the number of samples that are actually positive (True Positives) by the number of samples predicted by the model to be positive. The latter includes the sum of the number of samples that are actually positive (True Positives) and the number of samples that are incorrectly predicted to be positive (False Positives).

[0077] Recall is used to measure the proportion of samples that are correctly predicted as positive by the model among all samples that are actually positive. It is calculated by dividing the number of samples that are correctly predicted as positive by the model (True Positives) by the number of samples that are actually positive. The latter is the number of samples that are correctly predicted as positive by the model (True Positives) and the number of samples that are incorrectly predicted as negative by the model (False Negatives).

[0078] The F1 score is used to comprehensively consider the performance of precision and recall. It can be expressed as the harmonic mean of precision and recall, and is calculated as 2×precision×recall / (precision+recall).

[0079] The ROC curve is a graphical tool for evaluating the performance of classification models. It intuitively displays the classification performance of the model by depicting the relationship between the true positive rate (TPR) and the false positive rate (FPR) of the model under different classification thresholds. When evaluating model performance, the area under the ROC curve (AUC) can be used as a comprehensive indicator to directly compare the advantages and disadvantages of multiple models through a single value. The AUC value ranges from 0.5 to 1.0, and a larger AUC represents better performance.

[0080] Mean square error (MSE) is mainly used to measure the difference between the model prediction value and the true value in the regression task. The calculation formula is: .in, is the true value, is the predicted value, is the number of samples.

[0081] The root mean square error (RMSE) is the square root of the mean square error and provides the error magnitude in the same units as the original data. The calculation formula is: .

[0082] The mean absolute error (MAE) is the average of the absolute values ​​of the differences between the model's predicted values ​​and the true values. The calculation formula is: .in, is the true value, is the predicted value, is the number of samples.

[0083] Log loss, also known as logistic loss or cross-entropy loss, is a loss function used in machine learning to evaluate the performance of classification models. It is particularly suitable for binary classification problems, but can also be extended to multi-classification problems. Log loss measures the difference between the model's predicted probability and the actual label.

[0084] Precision (AP) and mean Average Precision (mAP) are important indicators for evaluating the performance of classification and target detection models, and are widely used in information retrieval, image classification, and target detection. AP measures the performance of each category. Specifically, AP is the area under the precision-recall curve. When calculating AP, you need to calculate the Precision and Recall values ​​for each category independently, then draw the Precision-Recall curve, and calculate the area under the curve to get the AP value of the category. mAP is the average of the AP values ​​of multiple categories, which comprehensively considers the performance of all categories. In the target detection task, mAP is a key indicator to measure the quality of the model in all categories. The calculation method is to calculate the AP value for each category separately, and then average them to get the final mAP value.

[0085] The coefficient of determination (R²), also known as the coefficient of determination or goodness of fit, is a statistic used in regression analysis to assess the degree of model fit. It represents the proportion of the variance of the dependent variable explained by the model, that is, the square of the correlation between the model's predicted value and the actual value. The value of R² ranges between 0 and 1. The closer the value is to 1, the better the model fits, that is, the higher the proportion of variance explained by the model.

[0086] In the embodiment of the present invention, one or more model accuracies may be used as model performance parameters, and the combination of different types of model accuracies may be designed according to the type of model calculation task.

[0087] In an embodiment of the present invention, the model parameter quantity evaluation parameter can be used to evaluate the parameter quantity of the tested model. For the same model, different model parameters may be used in the actual model calculation. In an embodiment of the present invention, the model parameter quantity evaluation parameter can be the activation parameter quantity of the tested model or calculated according to the function of the activation parameter quantity. The activation parameter quantity is the parameter quantity that the tested model participates in the calculation in the model calculation task.

[0088] In summary, the model performance parameters and the model parameter quantity evaluation parameters are used to evaluate the tested model itself. In the embodiment of the present invention, the evaluation parameters calculated according to the model performance parameters and the model parameter quantity evaluation parameters can be defined as the model parameter efficiency. In the embodiment of the present invention, the model parameter efficiency is determined by the model itself and is used to reflect the contribution of the parameter quantity to the model performance in the corresponding scenario. The fewer the parameter quantity and the better the model performance, the higher the model parameter efficiency.

[0089] In the embodiment of the present invention, determining the modular calculation efficiency of the model calculation system under test according to the model performance parameters, the calculation performance parameters and the model parameter quantity evaluation parameters of the model under test in S104 may include: determining the model parameter efficiency of the model calculation system under test according to the model performance parameters and the model parameter quantity evaluation parameters; determining the modular calculation efficiency according to the model parameter efficiency and the calculation performance parameters. It may be set that the greater the model parameter efficiency, the greater the modular calculation efficiency.

[0090] In an embodiment of the present invention, the computing performance parameters are used to evaluate the computing performance of the device under test, and are used to quantify the performance of the device under test when running the tested software framework to perform model computing tasks according to the tested model and data set.

[0091] In some optional implementations of the embodiments of the present invention, the step of determining the computing performance parameters may include: determining the computing performance parameters according to the measured computing performance parameters of the model computing system under test performing the model computing task and the theoretical peak performance parameters of the computing system of the tested device. Then in the embodiments of the present invention, the computing efficiency may be defined to represent the hardware performance utilization rate of the tested device when the tested software framework performs the model computing task according to the tested model and data set, and the computing efficiency may be the ratio of the measured computing performance parameters to the theoretical peak performance parameters of the computing system, or calculated according to a function of the ratio.

[0092] In the embodiment of the present invention, the higher the model performance of the tested model itself, the fewer the model parameters (that is, less space is occupied and less calculation is required), and the higher the hardware performance utilization of the tested device in the tested software framework, the higher the analog calculation efficiency of the tested model computing system. Therefore, the analog calculation efficiency can reflect the overall comprehensive efficiency of the tested model in the software and hardware framework.

[0093] In some optional implementations of the embodiments of the present invention, the model parameter efficiency of the tested model calculation system is determined according to the model performance parameter and the model parameter quantity evaluation parameter in S104, and is calculated by the following formula: Model calculation efficiency = model parameter efficiency × calculation efficiency; The computational efficiency is a computational performance parameter.

[0094] Based on this, a calculation method for model performance parameters, a calculation method for model parameter quantity evaluation parameters, a calculation method for calculation performance parameters, a calculation method for obtaining model parameter efficiency based on model performance parameters and model parameter quantity evaluation parameters, and a calculation method based on model parameter efficiency and calculation performance parameters can be designed. The specific calculation method will be introduced in the following embodiments of the present invention.

[0095] According to the above introduction, after the model computing system under test completes the model computing task, the actual efficiency of the model under test under the device under test and the software framework under test can be reflected by determining the model computing efficiency of the model under test, thereby realizing the comprehensive performance evaluation of the model under test itself, the software framework under test and the device under test.

[0096] In S103, after the model calculation task is completed, the model performance parameters and the calculation performance parameters can be determined according to the execution process or execution result of the model calculation task.

[0097] In S104, the model parameter quantity evaluation parameter is determined according to the participation of the model parameters of the tested model in the model calculation task, and the model calculation efficiency of the tested model calculation system is determined according to the model performance parameter, the calculation performance parameter and the model parameter quantity evaluation parameter of the tested model.

[0098] Then, the modular calculation efficiency can be directly used as the test result of the tested model calculation system, or the modular calculation efficiency can be used as one of the indicators for calculating the test result.

[0099] The modular calculation efficiency testing method provided by the embodiment of the present invention determines the model performance parameters and the calculation performance parameters according to the model calculation task after controlling the tested software framework to call the tested model to perform the model calculation task according to the data set on the tested device, so as to quantify the performance of the tested model and the performance of the software and hardware architecture respectively, and determines the modular calculation efficiency of the tested model calculation system according to the model performance parameters, the calculation performance parameters and the model parameter quantity evaluation parameters of the tested model to obtain the test result of the tested model calculation system, thereby realizing the quantitative evaluation of the model calculation performance of the tested model calculation system including the tested model, the tested software framework and the tested device. The modular calculation efficiency and the testing method provided by the embodiment of the present invention are used as an evaluation scheme suitable for various model calculation tasks, which enhances the comparability indicators of different models running on different devices, provides a reference basis for the improvement and perfection of the model and the model calculation system, and then can quantitatively evaluate the accuracy and multivariate computing power performance of different models, thereby ensuring the effective traction model capability improvement and application cost reduction.

[0100] Based on the above embodiment, the embodiment of the present invention further introduces the calculation steps of the model parameter efficiency.

[0101] In the above embodiment, it is introduced that the evaluation parameter calculated according to the model performance parameter and the model parameter quantity evaluation parameter can be defined as the model parameter efficiency. The definition of the model parameter efficiency can reflect the contribution of the parameter quantity to the model performance in the corresponding scenario. The smaller the parameter quantity and the better the model performance, the higher the model parameter efficiency. Based on this, in the embodiment of the present invention, the model parameter efficiency can be expressed as the ratio of the model performance parameter to the model parameter quantity evaluation parameter.

[0102] In order to eliminate the influence of units, in an embodiment of the present invention, determining the model parameter efficiency of the tested model computing system based on the model performance parameters and the model parameter quantity evaluation parameters may include: normalizing the model performance parameters and the model parameter quantity evaluation parameters, respectively, and determining the model parameter efficiency based on the normalized model performance parameters, the normalized model parameter quantity evaluation parameters and the computing performance parameters.

[0103] Among them, normalizing the model performance parameters may include: determining the normalized model performance parameters according to the measured model performance parameters in the model calculation task and the reference model performance parameters of the tested model. If the reference model performance parameters and the measured model performance parameters are model performance parameters of the same type, the measured model performance parameters may be the model accuracy of the tested model, and the reference model performance parameters may be the reference accuracy of the tested model. For example, if the measured model performance parameters adopt accuracy, the reference model performance parameters also adopt accuracy.

[0104] The reference model performance parameters are used to eliminate the influence of the units of the measured model performance parameters. The same reference model performance parameters can be used for a specific tested model (with the same model parameters), or for tested models that perform similar model calculation tasks.

[0105] The normalization of the model parameter evaluation parameter may include: determining the normalized model parameter evaluation parameter according to the activation parameter of the tested model and the reference parameter of the tested model; wherein the activation parameter is the parameter involved in the calculation of the tested model in the model calculation task. The reference parameter is used to eliminate the influence of the unit of the activation parameter, and the total parameter of the tested model may be used.

[0106] In the embodiment of the present invention, the model parameter efficiency of the tested model computing system is determined according to the model performance parameter and the model parameter quantity evaluation parameter, which can be calculated by the following formula: ; Among them, the model accuracy is the measured model performance parameter in the model calculation task, the reference accuracy is the reference model performance parameter of the tested model, and the activation parameter quantity is the parameter quantity of the tested model involved in the calculation in the model calculation task.

[0107] On the basis of the above embodiment, since the model cannot be put into practical application when the performance is too low, it has no use value, and it is meaningless to calculate its analog calculation efficiency. In the embodiment of the present invention, a reference model performance parameter range can be defined to measure the minimum standard of the model performance of the tested model. In the embodiment of the present invention, S104 determines the analog calculation efficiency of the tested model calculation system according to the model performance parameters, the calculation performance parameters and the model parameter quantity evaluation parameters of the tested model, which may include: if the model performance parameters are within the reference model performance parameter range, the analog calculation efficiency is determined according to the model performance parameters, the calculation performance parameters and the model parameter quantity evaluation parameters; if the model performance parameters are not within the reference model performance parameter range, the model performance parameters are set to 0, and the analog calculation efficiency is determined according to the model performance parameters, the calculation performance parameters and the model parameter quantity evaluation parameters.

[0108] In a specific implementation, if the model performance parameter involved is positively correlated with the model performance, that is, the better the model performance, the larger the model performance parameter, then the baseline model performance parameter range can be represented by the baseline model performance parameter, then S104 determines the analog calculation efficiency of the tested model calculation system according to the model performance parameter, the calculation performance parameter and the model parameter quantity evaluation parameter of the tested model, which may include: if the model performance parameter is greater than or equal to the baseline model performance parameter, then determining the analog calculation efficiency according to the model performance parameter, the calculation performance parameter and the model parameter quantity evaluation parameter; if the model performance parameter is less than the baseline model performance parameter, then setting the model performance parameter to 0, and then determining the analog calculation efficiency according to the model performance parameter, the calculation performance parameter and the model parameter quantity evaluation parameter.

[0109] If the model performance parameter is the model accuracy, and the benchmark model performance parameter is the benchmark accuracy, then S104 determines the simulation efficiency of the tested model computing system based on the model performance parameters, the calculation performance parameters and the model parameter quantity evaluation parameters of the tested model, which may include: if the model measured accuracy is greater than or equal to the benchmark accuracy, then the model measured accuracy is used as the model accuracy, and the simulation efficiency is determined based on the model accuracy, the calculation performance parameters and the model parameter quantity evaluation parameters; if the model measured accuracy is less than the benchmark accuracy, then 0 is used as the model accuracy, and the simulation efficiency is determined based on the model accuracy, the calculation performance parameters and the model parameter quantity evaluation parameters.

[0110] When the model performance parameter adopts the model accuracy of the tested model, the model accuracy can be determined by the following formula: ; Among them, the measured model accuracy is the measured accuracy of the tested model in the model calculation task, and the benchmark accuracy is the minimum allowable accuracy of the tested model.

[0111] In the embodiment of the present invention, different benchmark model performance parameter ranges may be set for different tested models, different model calculation tasks or different data sets.

[0112] Based on the above embodiments, the embodiments of the present invention further introduce the calculation steps of calculating the performance parameters.

[0113] In the above-mentioned embodiment, it is introduced that the computing performance parameters can be determined according to the measured computing performance parameters of the model computing system under test executing the model computing task and the theoretical peak performance parameters of the computing system of the device under test.

[0114] In an implementation method of determining the computing performance parameters according to the measured computing performance parameters of the model computing system under test executing the model computing task and the theoretical peak performance parameters of the computing system of the tested device, the ratio of the measured computing performance parameters to the theoretical peak performance parameters of the computing system can be used as the computing performance parameters. The computing efficiency can be defined to represent the hardware performance utilization rate of the tested device when the tested software framework executes the model computing task according to the tested model and data set, and the computing efficiency can be the ratio of the measured computing performance parameters to the theoretical peak performance parameters of the computing system, or calculated according to a function of the ratio.

[0115] In the embodiment of the present invention, the computing performance parameter is determined according to the measured computing performance parameter of the model computing system under test executing the model computing task and the theoretical peak performance parameter of the computing system of the tested device, which can be calculated by the following formula: .

[0116] In the embodiment of the present invention, for the tested device, the computing system theoretical peak performance parameter is used to quantify the theoretical peak computing power of the tested device, and the measured computing performance parameter is used to quantify the computing power actually invested by the tested device when executing the model computing task. For the tested model, the measured computing performance parameter is used to quantify the amount of computing per unit time in the process of the tested software framework calling the tested model to execute the model computing task.

[0117] In the embodiment of the present invention, the computing system theoretical peak performance parameter may adopt the parameter provided by the manufacturer of the tested device, which is usually a theoretical value calculated according to the hardware configuration of the tested device (such as the frequency and number of cores of the central processing unit). The measured computing performance parameter is determined according to the execution process or execution result of the model computing task, and is a parameter actually tested.

[0118] In some optional implementations of the embodiments of the present invention, the measured computing performance parameter may be the number of floating-point operations per second of the tested device executing the model computing task, and the computing system theoretical peak performance parameter may be the theoretical peak number of floating-point operations per second of the tested device. Among them, the number of floating-point operations per second (Floating Point Operations per Second, FLOPS) refers to the number of floating-point operations (addition, subtraction, multiplication, division, etc.) that a system or device can perform in a unit of time. In some other optional implementations of the embodiments of the present invention, the measured computing performance parameter may also be the number of tera floating point operations per second (Tera Floating Point Operations Per Second, TFLOPS) of the tested device executing the model computing task, and the computing system theoretical peak performance parameter may also be the theoretical peak number of tera floating point operations per second of the tested device.

[0119] Through actual testing, it is found that the data type actually used in the process of executing model calculation tasks affects the model performance parameters and also affects the calculation performance parameters. Therefore, the data type needs to be considered when determining the theoretical peak performance parameters and the measured calculation performance parameters of the calculation system.

[0120] In some optional implementations of the embodiments of the present invention, the step of determining the measured computing performance parameters and the step of determining the theoretical peak performance parameters of the computing system include: obtaining the theoretical peak performance parameters of the computing system corresponding to the first data type of the tested device; if the actual data types in the model calculation task are all the first data type, the computing performance parameters corresponding to the actual data types in the model calculation task are used as the measured computing performance parameters; if there are actual data types that are not the first data type in the model calculation task, the computing performance parameters corresponding to the actual data types in the model calculation task are equivalent to the computing performance parameters corresponding to the first data type, and the measured computing performance parameters are obtained. That is to say, no matter what data type is actually used in the model calculation task, it is regarded as the first data type to determine the measured computing performance parameters, and then the computing performance parameters are determined in combination with the theoretical peak performance parameters of the computing system. This can greatly simplify the steps of determining the model calculation efficiency, thereby quickly obtaining test results.

[0121] Among them, the first data type can be selected from the theoretical peak performance parameters of the computing system of various data types provided by the manufacturer of the tested device. In practical applications, the first data type can be one of, but not limited to, integer, floating-point, character, string, and array. If the first data type is a floating-point type, it can be one of single-precision floating point, double-precision floating point, extended-precision floating point, quadruple-precision floating point, and half-precision floating point.

[0122] In some optional implementations of the embodiments of the present invention, the first data type may be a single-precision floating point type.

[0123] In some other optional implementations of the embodiments of the present invention, the step of determining the measured computing performance parameters and the step of determining the theoretical peak performance parameters of the computing system may also include: obtaining the theoretical peak performance parameters of the computing system corresponding to the second data type of the tested device; if the actual data types in the model computing task are all the second data type, the computing performance parameters corresponding to the actual data types in the model computing task are used as the measured computing performance parameters; if there are actual data types that are not the second data type in the model computing task, the computing performance parameters corresponding to the actual data types in the model computing task are converted into computing performance parameters corresponding to the second data type to obtain the measured computing performance parameters. In practical applications, the computing power corresponding to different data types can be obtained through testing, that is, there is a conversion relationship that can be determined between the computing performance parameters corresponding to different data types. Therefore, when determining the computing performance parameters of the tested model computing system, the computing performance parameters corresponding to the actual data types used in the model computing task can be converted into computing performance parameters corresponding to the second data type of the selected computing system theoretical peak performance parameters, and then the measured computing performance parameters are determined. In this way, a more refined calculation of computing performance parameters can be achieved, and then a more accurate calculation of modular efficiency can be achieved.

[0124] Among them, the second data type can be selected from the theoretical peak performance parameters of the computing system of each data type provided by the manufacturer of the tested device. In practical applications, the first data type can be one of, but not limited to, integer, floating-point, character, string, and array. If the second data type is a floating-point type, it can be one of single-precision floating point, double-precision floating point, extended-precision floating point, quadruple-precision floating point, and half-precision floating point.

[0125] In some optional implementations of the embodiments of the present invention, the second data type may be a single-precision floating point type.

[0126] Based on the above embodiments, an embodiment of the present invention provides a method for determining modular efficiency suitable for practical applications.

[0127] After the model calculation system under control completes the model calculation task, the model calculation efficiency can be calculated according to the following formula: Model calculation efficiency = model parameter efficiency × calculation efficiency. (1)

[0128] Among them, the model parameter efficiency and calculation efficiency can refer to the description of the above embodiment of the present invention. In the embodiment of the present invention, the model parameter efficiency can be proportional to the model accuracy and inversely proportional to the activation parameter amount. In order to eliminate the influence of the unit, the model parameter efficiency is respectively compared with the reference accuracy and the reference parameter amount, and the model parameter efficiency can be calculated by the following formula: . (2)

[0129] Models can be divided into dense models and sparse models according to the proportion of activation parameters in the reference parameters. If the model being tested is a dense model, the activation parameters are the total parameters of the model being tested.

[0130] The computational efficiency can be calculated by the following formula: . (3)

[0131] Among them, the measured computing performance parameters can be the Tera Floating Point Operations Per Second (TFLOPS) of the tested device executing the model computing task, and the computing system theoretical peak performance parameters can be the theoretical peak of the Tera Floating Point Operations Per Second of the tested device.

[0132] The model accuracy can be determined by the following formula: . (4)

[0133] That is to say, the model measured accuracy that is less than the benchmark accuracy is set to zero.

[0134] In actual applications, it is observed that in addition to the tested device and the tested software framework, factors affecting model performance also include the number of model parameters, the type of application scenario, the type of data set, and the type of data used in model calculation. That is to say, under the combination of the above different factors, the same tested model may also have different model performance, which further affects the final measured model calculation efficiency. In an embodiment of the present invention, the model calculation efficiency of the tested model calculation system can be more comprehensively evaluated by determining the comprehensive model calculation efficiency of the tested model calculation system.

[0135] In some optional implementations of the embodiments of the present invention, determining the model calculation efficiency of the model calculation system under test according to the model performance parameters, the calculation performance parameters and the model parameter quantity evaluation parameters of the model under test in S104 to obtain the test result of the model calculation system under test may include: determining the comprehensive model calculation efficiency of the model calculation system under test according to the model calculation efficiency of the model calculation system under test in multiple test scenarios; taking the comprehensive model calculation efficiency as the test result; wherein the test scenario corresponds to the data set one by one. That is, for multiple test scenarios, a data set can be selected for each test scenario to perform the model calculation task, and then the comprehensive model calculation efficiency can be obtained by combining the model calculation efficiencies of each test scenario.

[0136] In some other optional implementations of the embodiments of the present invention, in S104, the analog calculation efficiency of the model calculation system under test is determined according to the model performance parameters, the calculation performance parameters and the model parameter quantity evaluation parameters of the tested model to obtain the test result of the model calculation system under test, and it can also include: according to the analog calculation efficiency of the model calculation system under test in multiple test scenarios, the comprehensive analog calculation efficiency of the model calculation system under test is determined; the comprehensive analog calculation efficiency is used as the test result; wherein one test scenario corresponds to one or more data sets. That is to say, for multiple test scenarios, one or more data sets can be selected for each test scenario to perform the model calculation task, and for the test scenario using multiple data sets, the analog calculation efficiency of each data set is first integrated to obtain the analog calculation efficiency corresponding to the test scenario, and then the analog calculation efficiency of each test scenario is integrated to obtain the comprehensive analog calculation efficiency.

[0137] Taking the language model as an example, the test scenario can be set as follows: Test scenario 1: The model calculation task is a problem-solving task, and the data sets used may include but are not limited to GSM8K, Math, and Math23K.

[0138] Test scenario 2: The model calculation task is a code generation task, and the datasets used may include but are not limited to Human Eval and MBPP.

[0139] Test scenario 3: The model calculation task is a large-scale multi-task language understanding task, and the dataset used may include but is not limited to MMLU.

[0140] Test scenario 4: The model calculation task is a reading comprehension task, and the dataset used may include but is not limited to ARC-C.

[0141] Then the step of determining the analog calculation efficiency corresponding to a single test scenario may include: if the test scenario corresponds to one data set, then the analog calculation efficiency corresponding to the data set is used as the analog calculation efficiency corresponding to the test scenario; if the test scenario corresponds to multiple data sets, then the analog calculation efficiency corresponding to the test scenario is determined according to the analog calculation efficiencies corresponding to each data set corresponding to the test scenario. In some optional implementations of the embodiments of the present invention, determining the analog calculation efficiency corresponding to the test scenario according to the analog calculation efficiencies corresponding to each data set corresponding to the test scenario may include: taking the arithmetic mean of the analog calculation efficiencies corresponding to each data set as the analog calculation efficiency corresponding to the test scenario. In some other optional implementations of the embodiments of the present invention, determining the analog calculation efficiency corresponding to the test scenario according to the analog calculation efficiencies corresponding to each data set corresponding to the test scenario may also include: taking the geometric mean of the analog calculation efficiencies corresponding to each data set as the analog calculation efficiency corresponding to the test scenario.

[0142] In some other optional implementations of the embodiments of the present invention, the model performance parameters and the calculation performance parameters can also be calculated comprehensively. Then, in S104, the simulation efficiency of the model calculation system under test is determined according to the model performance parameters, the calculation performance parameters and the model parameter quantity evaluation parameters of the tested model to obtain the test results of the model calculation system under test, which can also include: determining the single-scene model performance parameters corresponding to the test scene according to the model performance parameters corresponding to each data set corresponding to the test scene; determining the single-scene calculation performance parameters corresponding to the test scene according to the calculation performance parameters corresponding to each data set corresponding to the test scene; determining the simulation efficiency corresponding to the test scene according to the single-scene model performance parameters, the single-scene calculation performance parameters and the model parameter quantity evaluation parameters. That is to say, by first calculating the single-scene model performance parameters and the single-scene calculation performance parameters respectively, different comprehensive calculation methods for the parameters corresponding to the data sets can be used for these two types of parameters, so as to better reflect the influence of different types of parameters on the calculation of simulation efficiency.

[0143] In some optional implementations of the embodiments of the present invention, the single-scenario model performance parameters corresponding to the test scenario are determined according to the model performance parameters corresponding to each data set corresponding to the test scenario, including: taking the arithmetic mean of the model performance parameters corresponding to each data set as the single-scenario model performance parameter; the single-scenario computing performance parameters corresponding to the test scenario are determined according to the computing performance parameters corresponding to each data set corresponding to the test scenario, including: taking the arithmetic mean of the computing performance parameters corresponding to each data set as the single-scenario computing performance parameter.

[0144] In some other optional implementations of the embodiments of the present invention, the single-scene model performance parameters corresponding to the test scenario are determined according to the model performance parameters corresponding to each data set corresponding to the test scenario, including: taking the geometric mean of the model performance parameters corresponding to each data set as the single-scene model performance parameter; the single-scene computing performance parameters corresponding to the test scenario are determined according to the computing performance parameters corresponding to each data set corresponding to the test scenario, including: taking the geometric mean of the computing performance parameters corresponding to each data set as the single-scene computing performance parameter.

[0145] In an embodiment of the present invention, determining the comprehensive modular computing efficiency of the model computing system under test according to the modular computing efficiency of the model computing system under test in multiple test scenarios may include: taking the arithmetic mean of the modular computing efficiencies corresponding to multiple test scenarios as the comprehensive modular computing efficiency. Then taking the arithmetic mean of the modular computing efficiencies corresponding to multiple test scenarios as the comprehensive modular computing efficiency can be calculated by the following formula: ; in, represents the comprehensive modular efficiency, Indicates that the model computing system under test is The modular computing efficiency under each test scenario is Indicates that the model computing system under test is In the embodiment of the present invention, the first weight coefficients corresponding to the various test scenarios may be equal or unequal.

[0146] In an embodiment of the present invention, determining the comprehensive analog computing efficiency of the model computing system under test according to the analog computing efficiency of the model computing system under test in multiple test scenarios may also include: taking the geometric mean of the analog computing efficiencies corresponding to the multiple test scenarios as the comprehensive analog computing efficiency. The geometric mean of the analog computing efficiencies corresponding to the multiple test scenarios is taken as the comprehensive analog computing efficiency, which can be calculated by the following formula: ; in, represents the comprehensive modular efficiency, Indicates that the model computing system under test is The modular computing efficiency under each test scenario is represents cumulative multiplication calculation, Indicates that the model computing system under test is In the embodiment of the present invention, the second weight coefficients corresponding to the various test scenarios may be equal or unequal.

[0147] In the embodiment of the present invention, two determination methods, namely, the arithmetic mean and the geometric mean, are proposed for combining multiple modular calculation efficiencies to obtain modular calculation efficiency. In practical applications, the geometric mean is more suitable for comparing data of different scales or units than the arithmetic mean. In some optional implementations of the embodiment of the present invention, if the difference between at least two modular calculation efficiencies among the multiple modular calculation efficiencies corresponding to the tested model calculation system is greater than a preset threshold, the geometric mean between the modular calculation efficiencies is used as the comprehensive modular calculation efficiency to weaken the situation due to test errors or the performance of the tested model calculation system itself being too good or too poor in certain test scenarios or data sets, thereby obtaining a more comprehensive comprehensive modular calculation efficiency.

[0148] In the embodiment of the present invention, in addition to the comprehensive analog calculation efficiency, the analog calculation efficiency corresponding to each test scenario can also be output as a test result.

[0149] The modular computing efficiency testing method provided by the embodiment of the present invention can achieve a more comprehensive evaluation of the modular computing efficiency of the model computing system being tested.

[0150] On the basis of the above-mentioned embodiment, the embodiment of the present invention further describes the steps of controlling the tested model computing system to perform the model computing task.

[0151] In an embodiment of the present invention, controlling the tested device to run the tested software framework to call the tested model and data set to perform model calculation tasks in S102 may include: loading the tested software framework on the tested device and performing initialization of the tested software framework; starting the tested software framework to call the tested model and data set to perform model calculation tasks.

[0152] Among them, loading the tested software framework on the tested device and performing initialization of the tested software framework can include: loading the library file of the tested model computing system on the tested device; obtaining the type of interface function of the tested model computing system through the calling interface of the library file; and initializing the interface function according to the type of the interface function.

[0153] After the deployment of the model computing system under test is completed, the whole is provided in the form of a dynamic library, and the required structure and implementation of the specified interface function are provided. After the test equipment starts running the test script, the library file (Library, Lib library) of the model computing system under test is dynamically loaded. The Lib library provides an access interface. If the software framework under test uses C language or C++ language, the Lib library provides a C interface. This interface can be defined as "GetFuncs", which means "calling interface function". This interface can be further defined as:

[0154] void GetFuncs(SUTInterface*interface).

[0155] Among them, SUTInterface is a structure that needs to be implemented by the software framework under test and filled with pointers to the interface functions required by the embodiment of the present invention. The model calculation system under test needs to implement these interface functions to perform model calculation tasks and cooperate in determining the model calculation efficiency.

[0156] In an embodiment of the present invention, the types of required interface functions may include but are not limited to: initialization function (Initialize), completion function (Finalize), data loading function (LoadData), data unloading function (UnLoadData), and model calculation control function; the initialization function is an interface function that performs initialization according to the test configuration information; the completion function is an interface function of the task performed by the tested model calculation system after completing the model calculation task; the data loading function is an interface function for loading model calculation data; the data unloading function is an interface function for unloading model calculation data; and the model calculation control function is an interface function for executing the model calculation task.

[0157] Among them, the model calculation control function is determined according to the type of model calculation task, and the type description can refer to the introduction of the above embodiment. Taking the model calculation task as a model reasoning task as an example, the model calculation control function can include a synchronous reasoning control function (Predict) and an asynchronous reasoning control function (PredictAsync), wherein the synchronous reasoning control function performs synchronous reasoning based on the input data, fills in the corresponding information, and calls the callback function (func) to return the reasoning result. The asynchronous reasoning control function performs asynchronous reasoning based on the input data, fills in the corresponding information, and calls the callback function (func) to return the reasoning result.

[0158] The interface function may also include a system theoretical peak return function (GetSystemFLOPS), which is used for the model computing system under test to respond to the test script to return the computing system theoretical peak performance parameters of the device under test. This interface function may not be set, that is, the test device may obtain the computing system theoretical peak performance parameters of the device under test by other means and store them locally for use in determining the model computing efficiency.

[0159] In some optional implementations of the embodiments of the present invention, the structure SUTInterface may be filled with the following pointers: struct SUTInterface { RetStatus (*Initialize)(char* config_path); RetStatus (*Finalize)(); RetStatus (*LoadData)(const int[] ids, size_t nLength); RetStatus (*UnLoadData)(const int[] ids, size_t nLength); RetStatus (*Predict)(QueryMetaData[] metaData, Response* func); RetStatus (*PredictAsync)(QueryMetaData[] metaData, Response* func); RetStatus (*WaitforComplete)(); size_t (*GetSystemFLOPS)(); }.

[0160] Among them, RetStatus represents the acquisition status, char represents a character, and config_path represents the configuration path. Initialize represents the initialization function, which is used by the tested software framework to call and pass in the test configuration file path of this test, and the tested model computing system parses the configuration information and makes other initialization preparations. Finalize represents the completion function, which is used by the tested model computing system to release resources and other operations after the test is completed. LoadData represents the data loading function, which is used by the tested model computing system to load the data corresponding to the identifier according to the passed identifier (ids). Const represents a constant, and int[] represents an integer. size_t represents the data type of the size or length of an object. nLength is the variable name used to store the length value of a specific object. UnLoadData represents the data unloading function, which is used by the tested model computing system to unload the data of the specified identifier (ids). Predict represents the synchronous reasoning control function, which is used by the tested model computing system to perform synchronous reasoning according to the passed data, fill in the corresponding information, etc., and call the callback function func to return the reasoning result. QueryMetaData[] represents querying metadata, metaData represents metadata, and Response* func represents the response function. PredictAsync represents an asynchronous reasoning control function, which is used for the tested model computing system to perform asynchronous reasoning according to the input data, fill in the corresponding information, etc., and call the callback function func to return the reasoning result. WaitforComplete represents waiting for completion. GetSystemFLOPS represents a system theoretical peak return function, which is used for the tested model computing system to return the computing system theoretical peak performance parameters of the tested device after receiving the system theoretical peak acquisition request input by the test device.

[0161] After completing the initialization of the tested model calculation system, starting the tested software framework to call the tested model to perform the model calculation task, which may include: starting the tested software framework to perform the model calculation task; obtaining the first result returned after the tested software framework calls the data loading function to load the model calculation data; obtaining the second result of the tested software framework calling the model calculation control function to perform the model calculation task; waiting for the third result output after the tested software framework completes the model calculation task; obtaining the fourth result returned by the tested software framework calling the data unloading function to unload the model calculation data; and calculating the model accuracy of the tested model according to the execution result of the model calculation task, including: calculating the model accuracy according to at least one of the first result, the second result, the third result, and the fourth result.

[0162] Figure 2 A timing diagram of a modular efficiency testing method provided in an embodiment of the present invention.

[0163] like Figure 2 As shown, in some optional implementations of the embodiments of the present invention, after the model computing system to be tested is deployed, the steps of executing the model computing efficiency testing method by running the test script by the test device may include the following S201~S218.

[0164] S201: Start. Start the test equipment to start the test of the model calculation efficiency of the model calculation system under test.

[0165] S202: Loading library files. The test device loads the library files of the model computing system under test.

[0166] S203: Obtaining interface functions: The testing device obtains the interface functions of the model computing system under test.

[0167] S204: Return interface function. The device under test returns the interface function.

[0168] S205: Calling an initialization function. The test device calls the initialization function of the model computing system under test to perform initialization of the model computing system under test.

[0169] S206: Return initialization result. The device under test returns the initialization result.

[0170] The following S207 to S214 are steps included in one iteration of the iterative calculation of the model calculation task.

[0171] S207: Calling a data loading function: The test device calls the data loading function of the model computing system under test to load data corresponding to the input identifier (ids).

[0172] S208: Returning data loading results. The device under test returns the data loading results to the testing device.

[0173] The following S209~S210 are the steps performed for each input data (that is, each sample in the model training task and each piece of data to be processed in the model inference task) in one iteration.

[0174] S209: Calling a model calculation control function. The test device calls the model calculation control function of the model calculation system under test to perform model calculation for the current input data.

[0175] S210: Save calculation results. The test device saves the model calculation results of the tested model calculation system for the current input data.

[0176] S211: Waiting for completion. The test device waits for the model computing system under test to complete the model computing task, and sends a status acquisition request to the model computing system under test to determine the execution progress of the model computing task.

[0177] S212: Return the result of the model calculation task. The model calculation system under test returns the result after completing the model calculation task.

[0178] S213: Calling a data unloading function: The test device calls the data unloading function of the tested model computing system to unload data of the specified identifier (ids).

[0179] S214: Returning data unloading result: The device under test returns the data unloading result to the testing device.

[0180] S215: Calling a system theoretical peak value return function. The test device calls a system theoretical peak value return function of the tested model computing system to obtain the computing system theoretical peak performance parameters of the tested device.

[0181] S216: Return the theoretical peak performance parameters of the computing system. The tested device returns the theoretical peak performance parameters of the computing system to the testing device.

[0182] S217: Calculate the modular efficiency. The testing device calculates the modular efficiency of the model computing system under test according to the method introduced in any of the above embodiments.

[0183] S218: Generate a test report. The test equipment summarizes the test results of the tested model computing system, which may include but is not limited to the modular computing efficiency and the comprehensive modular computing efficiency, and generates a corresponding test report.

[0184] The model calculation efficiency testing method provided in the embodiment of the present invention not only proposes evaluation indicators of model calculation efficiency, but also provides a method for testing model calculation efficiency, thereby providing a unified evaluation standard applicable to various types of model calculation systems, which can help model users choose more efficient models, and at the same time assist model developers to optimize models, software frameworks, computing devices, etc., so as to promote the green development of artificial intelligence, promote efficient use of resources, and enhance model quality.

[0185] It should be noted that in the embodiments of the modular efficiency test method of the present invention, some of the steps or features can be ignored or not executed. The hardware or software functional modules divided for the convenience of description are not the only implementation form of the modular efficiency test method provided in the embodiment of the present invention.

[0186] The above describes in detail the various embodiments corresponding to the analog efficiency testing method. On this basis, the present invention also discloses an analog efficiency testing system, device, equipment, non-volatile storage medium and computer program product corresponding to the above method.

[0187] Figure 3 A schematic diagram of the structure of a modular efficiency testing system provided in an embodiment of the present invention.

[0188] like Figure 3 As shown, the modular efficiency testing system provided by the embodiment of the present invention may include a testing device 301 and a tested device 302 .

[0189] The tested device 302 is used to run the tested software framework to perform model calculation tasks according to the tested model and data set.

[0190] The test device 301 is used to obtain a data set, control the tested device 302 to run the tested software framework to call the tested model and the data set to perform the model calculation task; determine the model performance parameters and the calculation performance parameters according to the model calculation task; determine the model calculation efficiency of the tested model calculation system according to the model performance parameters, the calculation performance parameters and the model parameter quantity evaluation parameters of the tested model to obtain the test results of the tested model calculation system.

[0191] The model computing system under test includes a model under test, a software framework under test and a device under test 302 .

[0192] In a specific implementation, the test device 301 sends a test control command to the device under test 302 to control the device under test 302 to perform the model test task and return the model calculation result to the test device 301. The steps of the device under test 302 performing the model calculation task and the steps of the test device 301 performing the model calculation efficiency test on the model calculation system under test can refer to the introduction of any of the above method embodiments of the present invention.

[0193] The modular efficiency testing device provided by the embodiment of the present invention may include: An acquisition unit, used for acquiring a data set; A control unit, used to control the tested device to run the tested software framework to call the tested model and data set to perform model calculation tasks; A determination unit is used to determine the model performance parameters and the calculation performance parameters according to the model calculation task; determine the model calculation efficiency of the tested model calculation system according to the model performance parameters, the calculation performance parameters and the model parameter quantity evaluation parameters of the tested model, so as to obtain the test result of the tested model calculation system; The model computing system under test includes the model under test, the software framework under test and the device under test.

[0194] It should be noted that in each implementation of the analog efficiency test device provided in the embodiment of the present invention, the division of units is only a logical functional division, and other division methods can be adopted. The connection method between different units can adopt electrical, mechanical or other connection methods. The separated units can be located in the same physical location or distributed on multiple network nodes. Each unit can be implemented in the form of hardware or in the form of a software functional unit. That is, part or all of the units provided in the embodiment of the present invention can be selected according to actual needs and the corresponding connection method or integration method can be adopted to achieve the purpose of the embodiment of the present invention.

[0195] Since the embodiments of the apparatus part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the apparatus part, which will not be repeated here.

[0196] Figure 4 A schematic diagram of the structure of a simulation efficiency testing device provided in an embodiment of the present invention.

[0197] like Figure 4 As shown, the analog efficiency testing device provided in an embodiment of the present invention includes: a memory 410, used to store a computer program 411; a processor 420, used to execute the computer program 411, and when the computer program 411 is executed by the processor 420, the steps of the analog efficiency testing method provided in any of the above embodiments are implemented.

[0198] Among them, the processor 420 may include one or more processing cores, such as a 3-core processor, an 8-core processor, etc. The processor 420 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 420 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 420 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 420 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0199] The memory 410 may include one or more non-volatile storage media, which may be non-transitory. The memory 410 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 410 is at least used to store the following computer program 411, wherein the computer program 411, after being loaded and executed by the processor 420, can implement the relevant steps in the modular efficiency test method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 410 may also include an operating system 412 and data 413, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 412 may be Windows or other types of operating systems. Data 413 may include, but is not limited to, the data involved in the above method.

[0200] In some embodiments, the simulation efficiency testing device may further include a display screen 430 , a power supply 440 , a communication interface 450 , an input / output interface 460 , a sensor 470 , and a communication bus 480 .

[0201] Those skilled in the art will understand that Figure 4 The structure shown in the figure does not constitute a limitation on the analog efficiency testing device, and may include more or less components than those shown in the figure.

[0202] The analog arithmetic efficiency testing device provided in an embodiment of the present invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the steps of the analog arithmetic efficiency testing method provided in the above embodiment, and the effect is the same as above.

[0203] An embodiment of the present invention provides a non-volatile storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the modular efficiency testing method provided in any one of the above embodiments can be implemented.

[0204] The non-volatile storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0205] For the introduction of the non-volatile storage medium provided in the embodiment of the present invention, please refer to the above method embodiment, and the effect thereof is the same as the analog efficiency testing method provided in the embodiment of the present invention, and the present invention will not elaborate on it here.

[0206] An embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the modular efficiency testing method provided in any one of the above embodiments.

[0207] For an introduction to the computer program product provided by the embodiment of the present invention, please refer to the above-mentioned method embodiment, and the effect thereof is the same as the modular efficiency testing method provided by the embodiment of the present invention, and the present invention will not elaborate on it here.

[0208] The above is a detailed introduction to the analog efficiency test method, system, device, equipment, medium and product provided by the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the embodiments can be referenced to each other. For the systems, devices, equipment, non-volatile storage media and computer program products disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the present invention.

[0209] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A modular efficiency testing method, characterized in that: include: Get the dataset; Controlling the tested device to run the tested software framework to call the tested model and the data set to perform model calculation tasks; Determining model performance parameters and calculation performance parameters according to the model calculation task; Determine the model calculation efficiency of the tested model calculation system according to the model performance parameter, the calculation performance parameter and the model parameter quantity evaluation parameter of the tested model to obtain the test result of the tested model calculation system; The model under test computing system includes the model under test, the software framework under test and the device under test.

2. The modular efficiency testing method according to claim 1, characterized in that: Determining the model calculation efficiency of the tested model calculation system according to the model performance parameter, the calculation performance parameter and the model parameter quantity evaluation parameter of the tested model includes: Determining the model parameter efficiency of the tested model computing system according to the model performance parameter and the model parameter quantity evaluation parameter; The modular efficiency is determined based on the model parameter efficiency and the computing performance parameter.

3. The modular efficiency testing method according to claim 2, characterized in that: The greater the model parameter efficiency, the greater the modular calculation efficiency.

4. The modular efficiency testing method according to claim 2, characterized in that: The model parameter efficiency of the tested model calculation system is determined according to the model performance parameter and the model parameter quantity evaluation parameter, and is calculated by the following formula: Model calculation efficiency = model parameter efficiency × calculation efficiency; The calculation efficiency is the calculation performance parameter.

5. The modular efficiency testing method according to claim 2, characterized in that: Determining the model parameter efficiency of the tested model computing system according to the model performance parameter and the model parameter quantity evaluation parameter includes: Normalizing the model performance parameters and the model parameter quantity evaluation parameters respectively; The model parameter efficiency is determined based on the normalized model performance parameter, the normalized model parameter quantity evaluation parameter and the calculation performance parameter.

6. The modular efficiency testing method according to claim 5, characterized in that: The model performance parameters are normalized, including: The normalized model performance parameters are determined according to the measured model performance parameters in the model calculation task and the reference model performance parameters of the tested model.

7. The modular efficiency testing method according to claim 6, characterized in that: The measured model performance parameter is the model accuracy of the tested model, and the reference model performance parameter is the reference accuracy of the tested model.

8. The modular efficiency testing method according to claim 5, characterized in that: The model parameter evaluation parameter is normalized, including: Determining the normalized model parameter evaluation parameter according to the activation parameter of the tested model and the reference parameter of the tested model; The activation parameter amount is the parameter amount in which the tested model participates in the calculation in the model calculation task.

9. The modular efficiency testing method according to claim 2, characterized in that: The model parameter efficiency of the tested model calculation system is determined according to the model performance parameter and the model parameter quantity evaluation parameter, and is calculated by the following formula: ； Among them, the model accuracy is the measured model performance parameter in the model calculation task, the reference accuracy is the reference model performance parameter of the tested model, and the activation parameter amount is the parameter amount of the tested model involved in the calculation in the model calculation task.

10. The modular efficiency testing method according to claim 1, characterized in that: Determining the model calculation efficiency of the tested model calculation system according to the model performance parameter, the calculation performance parameter and the model parameter quantity evaluation parameter of the tested model includes: If the model performance parameter is within the range of the benchmark model performance parameter, determining the model calculation efficiency according to the model performance parameter, the calculation performance parameter and the model parameter quantity evaluation parameter; If the model performance parameter is not within the range of the benchmark model performance parameter, the model performance parameter is set to 0, and the analog calculation efficiency is determined based on the model performance parameter, the calculation performance parameter and the model parameter quantity evaluation parameter.

11. The method for testing modular efficiency according to claim 1, characterized in that: The model performance parameter is the model accuracy of the tested model, and the model accuracy is determined by the following formula: ； The model measured accuracy is the measured accuracy of the tested model in the model calculation task, and the benchmark accuracy is the minimum allowable accuracy of the tested model.

12. The modular efficiency testing method according to claim 1, characterized in that: The step of determining the calculation performance parameter comprises: The computing performance parameters are determined based on the measured computing performance parameters of the model computing system under test executing the model computing task and the theoretical peak performance parameters of the computing system of the device under test.

13. The modular efficiency testing method according to claim 12, characterized in that: The computing performance parameter is determined according to the measured computing performance parameter of the model computing system under test executing the model computing task and the theoretical peak performance parameter of the computing system of the device under test, and is calculated by the following formula: ； The computing efficiency is the computing performance parameter.

14. The modular efficiency testing method according to claim 12, characterized in that: The measured computing performance parameter is the number of floating-point operations per second of the tested device executing the model computing task; The computing system theoretical peak performance parameter is the theoretical peak number of floating-point operations per second of the device under test.

15. The modular efficiency testing method according to claim 12, characterized in that: The step of determining the measured computing performance parameters and the step of determining the theoretical peak performance parameters of the computing system include: Obtaining a theoretical peak performance parameter of the computing system corresponding to a first data type of the device under test; If the actual data types in the model calculation task are all the first data type, the calculation performance parameters corresponding to the actual data types in the model calculation task are used as the measured calculation performance parameters; If the model calculation task contains an actual data type that is not the first data type, the calculation performance parameters corresponding to the actual data type in the model calculation task are equated with the calculation performance parameters corresponding to the first data type to obtain the measured calculation performance parameters.

16. The modular efficiency testing method according to claim 15, characterized in that: The first data type is a single-precision floating point type.

17. The modular efficiency testing method according to claim 12, characterized in that: The step of determining the measured computing performance parameters and the step of determining the theoretical peak performance parameters of the computing system include: Obtaining a theoretical peak performance parameter of the computing system corresponding to a second data type of the device under test; If the actual data types in the model calculation task are all the second data type, the calculation performance parameters corresponding to the actual data types in the model calculation task are used as the measured calculation performance parameters; If the actual data type that is not the second data type exists in the model calculation task, the calculation performance parameters corresponding to the actual data type in the model calculation task are converted into the calculation performance parameters corresponding to the second data type to obtain the measured calculation performance parameters.

18. The modular efficiency testing method according to claim 1, characterized in that: Determining the model calculation efficiency of the tested model calculation system according to the model performance parameter, the calculation performance parameter and the model parameter quantity evaluation parameter of the tested model to obtain the test result of the tested model calculation system, including: Determine the comprehensive modular computing efficiency of the tested model computing system according to the modular computing efficiency of the tested model computing system in multiple test scenarios; Taking the comprehensive simulation efficiency as the test result; The test scenarios correspond to the data sets one by one.

19. The modular efficiency testing method according to claim 1, characterized in that: Determining the model computing efficiency of the tested model computing system according to the model performance parameter, the model data throughput parameter and the computing performance parameter to obtain a test result of the tested model computing system includes: Determine the comprehensive modular computing efficiency of the tested model computing system according to the modular computing efficiency of the tested model computing system in multiple test scenarios; Taking the comprehensive simulation efficiency as the test result; Among them, one of the test scenarios corresponds to one or more of the data sets.

20. The modular efficiency testing method according to claim 19, characterized in that: The steps of determining the modular efficiency corresponding to a single test scenario include: If the test scenario corresponds to a data set, the analog calculation efficiency corresponding to the data set is used as the analog calculation efficiency corresponding to the test scenario; If the test scenario corresponds to multiple data sets, the analog computing efficiency corresponding to the test scenario is determined according to the analog computing efficiencies corresponding to each of the data sets corresponding to the test scenario.

21. The modular efficiency testing method according to claim 20, characterized in that: Determining the modular calculation efficiency corresponding to the test scenario according to the modular calculation efficiency corresponding to each of the data sets corresponding to the test scenario includes: The arithmetic mean of the analog calculation efficiencies corresponding to the data sets is taken as the analog calculation efficiency corresponding to the test scenario.

22. The modular efficiency testing method according to claim 20, characterized in that: Determining the modular calculation efficiency corresponding to the test scenario according to the modular calculation efficiency corresponding to each of the data sets corresponding to the test scenario includes: The geometric mean of the analog calculation efficiency corresponding to each of the data sets is used as the analog calculation efficiency corresponding to the test scenario.

23. The modular efficiency testing method according to claim 20, characterized in that: Determining the modular calculation efficiency corresponding to the test scenario according to the modular calculation efficiency corresponding to each of the data sets corresponding to the test scenario includes: Determine the single-scenario model performance parameter corresponding to the test scenario according to the model performance parameters corresponding to each data set corresponding to the test scenario; Determine the single-scenario computing performance parameter corresponding to the test scenario according to the computing performance parameters corresponding to each data set corresponding to the test scenario; The model calculation efficiency corresponding to the test scenario is determined according to the single-scenario model performance parameter, the single-scenario calculation performance parameter and the model parameter quantity evaluation parameter.

24. The modular efficiency testing method according to claim 23, characterized in that: Determining the single-scenario model performance parameter corresponding to the test scenario according to the model performance parameters corresponding to each data set corresponding to the test scenario, including: taking the arithmetic mean of the model performance parameters corresponding to each data set as the single-scenario model performance parameter; Determining the single-scenario computing performance parameter corresponding to the test scenario according to the computing performance parameters corresponding to each data set corresponding to the test scenario includes: taking the arithmetic mean of the computing performance parameters corresponding to each data set as the single-scenario computing performance parameter.

25. The modular efficiency testing method according to claim 23, characterized in that: Determining the single-scenario model performance parameter corresponding to the test scenario according to the model performance parameters corresponding to each data set corresponding to the test scenario includes: taking the geometric mean of the model performance parameters corresponding to each data set as the single-scenario model performance parameter; Determining the single-scenario computing performance parameter corresponding to the test scenario according to the computing performance parameters corresponding to each data set corresponding to the test scenario includes: taking the geometric mean of the computing performance parameters corresponding to each data set as the single-scenario computing performance parameter.

26. The modular efficiency testing method according to claim 18 or 19, characterized in that: Determining the comprehensive modular computing efficiency of the tested model computing system according to the modular computing efficiency of the tested model computing system in multiple test scenarios includes: The arithmetic mean of the analog calculation efficiencies corresponding to the multiple test scenarios is taken as the comprehensive analog calculation efficiency.

27. The modular efficiency testing method according to claim 26, characterized in that: The arithmetic mean of the modular efficiency corresponding to the multiple test scenarios is used as the comprehensive modular efficiency, which is calculated by the following formula: ； in, represents the comprehensive modular efficiency, Indicates that the model computing system under test is in The modular computing efficiency under the test scenario is Indicates that the model computing system under test is in A first weight coefficient under the test scenario.

28. The modular efficiency testing method according to claim 18 or 19, characterized in that: Determining the comprehensive modular computing efficiency of the tested model computing system according to the modular computing efficiency of the tested model computing system in multiple test scenarios includes: The geometric mean of the analog calculation efficiencies corresponding to the multiple test scenarios is used as the comprehensive analog calculation efficiency.

29. The modular efficiency testing method according to claim 28, characterized in that: The geometric mean of the analog calculation efficiencies corresponding to the multiple test scenarios is used as the comprehensive analog calculation efficiency, which is calculated using the following formula: ； in, represents the comprehensive modular efficiency, Indicates that the model computing system under test is in The module calculation efficiency under the test scenario, represents cumulative multiplication calculation, Indicates that the model computing system under test is in A second weight coefficient under the test scenario.

30. The modular efficiency testing method according to claim 1, characterized in that: Controlling the tested device to run the tested software framework to call the tested model and the data set to perform the model calculation task, including: Loading the tested software framework on the tested device and initializing the tested software framework; The tested software framework is started to call the tested model and the data set to execute the model calculation task.

31. The modular efficiency testing method according to claim 30, characterized in that: Loading the tested software framework on the tested device and initializing the tested software framework includes: Loading the library file of the model computing system under test on the device under test; Acquire the type of the interface function of the tested model computing system through the calling interface of the library file; Initialize the interface function according to the type of the interface function.

32. The modular efficiency testing method according to claim 31, characterized in that: The types of the interface functions include at least: initialization function, completion function, data loading function, data unloading function, and model calculation control function; The initialization function is the interface function that performs initialization according to the test configuration information; the completion function is the interface function of the task performed by the tested model calculation system after completing the model calculation task; the data loading function is the interface function that loads the model calculation data; the data unloading function is the interface function that unloads the model calculation data; and the model calculation control function is the interface function that executes the model calculation task.

33. The modular efficiency testing method according to claim 32, characterized in that: Starting the tested software framework to call the tested model to execute the model calculation task includes: Starting the tested software framework to execute the model calculation task; Obtaining a first result returned after the tested software framework calls the data loading function to load the model calculation data; Obtaining a second result of the tested software framework calling the model calculation control function to execute the model calculation task; Waiting for the tested software framework to output a third result after completing the model calculation task; Obtaining a fourth result returned by the tested software framework calling the data unloading function to unload the model calculation data; The model accuracy of the tested model is calculated according to the execution result of the model calculation task, including: The model accuracy is calculated according to at least one of the first result, the second result, the third result, and the fourth result.

34. A modular efficiency testing system, characterized in that: Includes test equipment and equipment under test; Wherein, the device under test is used to run the software framework under test to perform model calculation tasks according to the model under test and the data set; The test device is used to obtain the data set, control the tested device to run the tested software framework to call the tested model and the data set to perform the model calculation task; determine the model performance parameters and the calculation performance parameters according to the model calculation task; determine the model calculation efficiency of the tested model calculation system according to the model performance parameters, the calculation performance parameters and the model parameter quantity evaluation parameters of the tested model, so as to obtain the test result of the tested model calculation system; The model under test computing system includes the model under test, the software framework under test and the device under test.

35. A modular efficiency testing device, characterized in that: include: An acquisition unit, used for acquiring a data set; A control unit, used to control the tested device to run the tested software framework to call the tested model and the data set to perform model calculation tasks; A determination unit, configured to determine a model performance parameter and a calculation performance parameter according to the model calculation task; Determine the model calculation efficiency of the tested model calculation system according to the model performance parameter, the calculation performance parameter and the model parameter quantity evaluation parameter of the tested model to obtain the test result of the tested model calculation system; The model under test computing system includes the model under test, the software framework under test and the device under test.

36. A modular efficiency testing device, characterized in that: include: Memory for storing computer programs; A processor is used to execute the computer program, and when the computer program is executed by the processor, the steps of the modular efficiency testing method as described in any one of claims 1 to 33 are implemented.

37. A non-volatile storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the modular efficiency testing method as described in any one of claims 1 to 33 are implemented.

38. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the modular efficiency testing method as described in any one of claims 1 to 33 are implemented.

Images