Method and device for evaluating performance of AI accelerator card based on domestic platform and medium

Abstract

The invention discloses an AI accelerator card performance evaluation method and device based on a domestic platform and a medium, and belongs to the technical field of artificial intelligence hardware evaluation. The technical problems that in the prior art, the AI acceleration card testing process is complex, testing environment building is tedious, and testing models and parameters are not uniform, so that the performance of all acceleration cards cannot be effectively and transversely compared are solved. The adopted technical scheme comprises the steps that system initialization and information collection are conducted; basic information of the AI accelerator card is obtained through AI accelerator card interface parameters or user input, and the basic information is stored in an AI accelerator card performance database; the collected information comprises an AI accelerator card working state, health information, power consumption, a memory occupancy rate, an accelerator card occupancy rate, temperature and power consumption; setting a test mode and parameters; executing an automatic test; data collection and statistical analysis; processing errors and feeding back; and generating a test report.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence hardware evaluation, and specifically to a method, device and medium for evaluating the performance of an AI accelerator card based on a domestic platform. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, AI applications have an increasing demand for computing resources, especially in the fields of deep learning, computer vision, natural language processing, etc. AI accelerator cards, as hardware devices designed specifically for accelerating AI computing, have a direct impact on the operating efficiency and effectiveness of AI applications.

[0003] However, there are still many challenges in the current AI accelerator card evaluation field:

[0004] ① Complexity of the test environment: AI accelerator cards from different manufacturers and models often have different hardware architectures, interface standards, and optimization algorithms, which requires a lot of customization when setting up a test environment for them. This highly customized test environment not only increases the difficulty and cost of testing, but also easily introduces human errors, affecting the accuracy of test results.

[0005] ② Inconsistent testing standards: There is currently a lack of unified AI accelerator card testing standards and specifications on the market. Different testing organizations or researchers may use different test models, data sets, parameter settings, etc., which makes it difficult to directly compare and evaluate test results. This inconsistency makes it difficult for users to make scientific and reasonable decisions when choosing AI accelerator cards.

[0006] ③ Low test efficiency: Traditional manual testing methods require testers to manually build the test environment, configure test parameters, execute test procedures, etc. The entire testing process is cumbersome and time-consuming. In addition, with the continuous improvement of the performance of AI accelerator cards, the requirements for test equipment are also getting higher and higher, further exacerbating the tension of test resources.

[0007] ④ One-sided performance evaluation: Existing AI accelerator card evaluations often focus on testing a single indicator (such as computing power, memory bandwidth, etc.), while ignoring the evaluation of overall performance (such as power consumption, stability, compatibility, etc.). This one-sided evaluation method cannot fully reflect the true performance level of AI accelerator cards, nor can it meet users' needs for diversified application scenarios. Summary of the invention

[0008] The technical task of the present invention is to provide a method, device and medium for evaluating the performance of an AI accelerator card based on a domestic platform, so as to solve the problems in the prior art such as the complex testing process of the AI ​​accelerator card, the cumbersome test environment construction, and the inconsistency of the test model and parameters, which make it impossible to effectively compare the performance of each accelerator card horizontally.

[0009] The technical task of the present invention is achieved in the following way: a performance evaluation method of an AI accelerator card based on a domestic platform, the method is as follows:

[0010] System initialization and information collection: Insert the AI ​​accelerator card to be evaluated into the evaluation platform, power on the evaluation platform and load the evaluation system. The evaluation system provides users with single-card test and multi-card test options. When testing a single card, the evaluation system will start the corresponding number of container software (such as Docker) according to the number of connected AI accelerator cards; when testing multiple cards, start the corresponding number of container software according to the test group, load the predefined test environment and configuration script in the container, automatically complete the configuration and installation of the driver and SDK software stack and interface parameters provided by the user, accurately identify and connect to the AI ​​accelerator card to be evaluated, and ensure that the AI ​​accelerator card is in normal status;

[0011] Obtain basic information of the AI ​​accelerator card through AI accelerator card interface parameters or user input, and store it in the AI ​​accelerator card performance database; the basic information of the AI ​​accelerator card includes model, computing power, memory size, power consumption, and temperature protection threshold information;

[0012] Collected information: including AI accelerator card working status, health information, power consumption, memory occupancy, accelerator card occupancy, temperature and power consumption;

[0013] Test mode and parameter setting: Provides a graphical user interface (GUI) or command line interface (CLI) to allow users to select the test mode (training / inference) and set the corresponding test parameters. At the same time, the evaluation platform automatically adjusts the test script and configuration file according to the user input to prepare the test environment.

[0014] Perform automated testing;

[0015] Data collection and statistical analysis: Collect and record the real-time status information of the AI ​​accelerator card during all tests, store it in the AI ​​accelerator card performance database, and perform statistical analysis on the test data, calculate the minimum, maximum, and average statistics of the test indicator results, and evaluate the performance stability of the AI ​​accelerator card; then analyze the power consumption data, calculate the energy efficiency ratio, and evaluate the energy efficiency performance of the AI ​​accelerator card;

[0016] Error handling and feedback: During the test process, the test status and test results are monitored in real time. Once an error or abnormal situation (such as test interruption, data abnormality, temperature alarm, etc.) is found, the test is stopped immediately, and a clear error prompt is given. The key information of the test process (such as error code, abnormal occurrence time, etc.) is recorded and an error report is generated for users to review and follow-up investigation;

[0017] Test report generation: Automatically generate detailed test reports based on test data and analysis results.

[0018] Preferably, the evaluation platform includes a processor, memory, hard disk, graphics card, network card, power supply, power meter and temperature sensor.

[0019] Better yet, the evaluation platform also has an AI module interface and a standard PCIE interface to facilitate the connection of the AI ​​module and the standard AI acceleration card.

[0020] Preferably, the configuration script also has the following functions:

[0021] ①The evaluation platform identifies the AI ​​accelerator card, including the lspci command;

[0022] ② Compare the AI ​​accelerator card type with the driver, software stack, and interface parameters provided by the user to see if they match:

[0023] If the match is successful, it will be automatically installed;

[0024] If the match is unsuccessful or the installation fails, feedback information will be provided;

[0025] ③ Generate an information collection script based on the interface parameters to start information collection.

[0026] Preferably, the AI ​​accelerator card performance database includes basic information such as AI accelerator card model, computing power, memory size, maximum power consumption, maximum temperature and temperature protection threshold, as well as entry information of final test results and power consumption.

[0027] Preferably, the test parameters include deep learning framework selection, model selection, weight selection, data set selection, accuracy selection, test rounds and duration, model complexity and data processing volume.

[0028] Preferably, automated testing is performed as follows:

[0029] Execute the test script to automatically deploy the deep learning framework, test model, and data set to the container where the AI ​​accelerator card is located;

[0030] In the selected test mode, start the test program to evaluate the performance of the AI ​​accelerator card, monitor the utilization rate (including memory usage rate) and key indicators such as power consumption and temperature of the AI ​​accelerator card in real time, and monitor the working status and abnormal alarm information in real time;

[0031] Record the test data, and gradually adjust the test pressure according to the preset test plan (such as increasing the complexity of the model, increasing the data processing volume, etc.), and repeat the test process to comprehensively evaluate the performance of the AI ​​accelerator card under different loads.

[0032] Preferably, the report includes test environment information, test parameter settings, test result summary, and performance evaluation charts;

[0033] The test report is saved in the form of electronic document, and the export function is provided to facilitate users to review and share.

[0034] An electronic device comprising: a memory and at least one processor;

[0035] Wherein, the memory stores a computer program;

[0036] The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the performance evaluation method of the AI ​​accelerator card based on the domestic platform as described above.

[0037] A computer-readable storage medium having a computer program stored therein, wherein the computer program can be executed by a processor to implement the above-mentioned method for evaluating the performance of an AI accelerator card based on a domestic platform.

[0038] The method, device and medium for evaluating the performance of an AI accelerator card based on a domestic platform of the present invention have the following advantages:

[0039] (1) The present invention uses automated and intelligent technical means to achieve comprehensive, efficient, and standardized evaluation of AI accelerator card performance, thereby overcoming the defects of the AI ​​accelerator card testing process being complicated, the test environment being cumbersome to build, and the inability to effectively compare the performance of various accelerator cards due to inconsistent test models and parameters, and promoting the advancement of AI hardware evaluation technology;

[0040] (ii) The present invention realizes comprehensive, rapid and accurate evaluation of the performance of AI accelerator cards through an automated and standardized testing process, which can not only reduce testing costs and improve testing efficiency, but also ensure the objectivity and fairness of test results, and provide strong technical support for the selection and application of AI accelerator cards;

[0041] (III) The present invention covers the whole process from test environment preparation, test mode selection, test execution to result statistics and analysis. Through preset scripts and programs, the test process is automated, which significantly reduces the labor intensity of testers and improves test efficiency.

[0042] (IV) In order to fully reflect the actual performance level of the AI ​​accelerator card, the present invention adopts a multi-dimensional performance evaluation method. In addition to traditional performance indicators such as computing power and memory bandwidth, it also introduces evaluation dimensions such as power consumption, stability, and compatibility. By comprehensively considering various performance indicators, it provides users with more comprehensive and objective evaluation results; among them, the multi-dimensionality is reflected in the AI ​​accelerator card performance database composed of AI accelerator card parameters and test results, and the dimensions include computing power dimension, power consumption dimension, computing power / W, memory capacity, maximum batch size, etc., which are sorted;

[0043] (V) During the test process, the present invention has a built-in intelligent error handling and feedback mechanism. When encountering test failures, data anomalies, etc., the mechanism can automatically identify and give clear error prompts to help testers quickly locate the cause of the problem and take corresponding solutions; at the same time, the mechanism can also record key information during the test process to provide strong support for subsequent troubleshooting and performance optimization; Among them, intelligent error handling refers to providing error information and guidance based on the error type, such as compile errors, which can prompt the cause of the error and the solution; when data corruption occurs, such as a picture in the data set is damaged, the damaged picture can be directly located, and the user can be provided with an automatic replacement option;

[0044] (VI) In order to meet the diverse needs of different users, the present invention fully considers scalability and flexibility in its design. Users can customize test modes, test models, parameter settings, etc. according to their own needs to achieve personalized evaluation; in addition, the present invention also supports the evaluation of multiple types of AI accelerator cards, and has strong universality and compatibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The present invention is further described below in conjunction with the accompanying drawings.

[0046] Attached Figure 1 This is a flowchart of the performance evaluation method of AI accelerator cards based on domestic platforms. DETAILED DESCRIPTION

[0047] The performance evaluation method, device and medium of the AI ​​accelerator card based on a domestic platform of the present invention are described in detail below with reference to the drawings and specific embodiments of the specification.

[0048] Embodiment 1:

[0049] As attached Figure 1 As shown, this embodiment provides a method for evaluating the performance of an AI accelerator card based on a domestic platform, and the method is specifically as follows:

[0050] S1. System initialization and information collection: Insert the AI ​​accelerator card to be evaluated into the evaluation platform, power on the evaluation platform and load the evaluation system. The evaluation system provides users with single-card test and multi-card test options. When testing a single card, the evaluation system will start the container software (such as Docker, etc.) according to the number of connected AI accelerator cards; when testing multiple cards, start the corresponding number of container software according to the test group, load the predefined test environment and configuration script in the container, automatically complete the configuration and installation of the driver and SDK software stack and interface parameters provided by the user, accurately identify and connect to the AI ​​accelerator card to be evaluated, and ensure that the AI ​​accelerator card is in normal status;

[0051] S2. Obtain basic information of the AI ​​accelerator card through AI accelerator card interface parameters or user input, and store it in the AI ​​accelerator card performance database; the basic information of the AI ​​accelerator card includes model, computing power, memory size, power consumption, and temperature protection threshold information;

[0052] S3, collect information: including AI accelerator card working status, health information, power consumption, memory occupancy, accelerator card occupancy, temperature and power consumption;

[0053] S4. Test mode and parameter setting: Provide a graphical user interface (GUI) or command line interface (CLI) to allow users to select the test mode (training / inference) and set the corresponding test parameters. At the same time, the evaluation platform automatically adjusts the test script and configuration file according to the user input to prepare the test environment.

[0054] S5. Perform automated testing;

[0055] S6. Data collection and statistical analysis: Collect and record the real-time status information, test results, and power consumption data of the AI ​​accelerator card during all tests, store them in the AI ​​accelerator card performance database, and perform statistical analysis on the test data, calculate the minimum, maximum, and average statistics of the test indicator results, and evaluate the performance stability of the AI ​​accelerator card; then analyze the power consumption data, calculate the energy efficiency ratio, and evaluate the energy efficiency performance of the AI ​​accelerator card;

[0056] S7. Error handling and feedback: During the test process, the test status and test results are monitored in real time. Once an error or abnormal situation (such as test interruption, data abnormality, temperature alarm, etc.) is found, the test is stopped immediately, and a clear error prompt is given. The key information of the test process (such as error code, abnormality occurrence time, etc.) is recorded to generate an error report for users to review and follow-up investigation;

[0057] S8. Test report generation: Automatically generate a detailed test report based on the test data and analysis results.

[0058] The evaluation platform in step S1 of this embodiment includes a processor, a memory, a hard disk, a graphics card, a network card, a power supply, a power meter and a temperature sensor.

[0059] The evaluation platform in step S1 of this embodiment is also provided with an AI module interface and a standard PCIE interface to facilitate the connection between the AI ​​module and the standard AI acceleration card.

[0060] The configuration script in step S1 of this embodiment also has the following functions:

[0061] ①The evaluation platform identifies the AI ​​accelerator card, including the lspci command;

[0062] ② Compare the AI ​​accelerator card type with the driver, software stack, and interface parameters provided by the user to see if they match:

[0063] If the match is successful, it will be automatically installed;

[0064] If the match is unsuccessful or the installation fails, feedback information will be provided;

[0065] ③ Generate an information collection script based on the interface parameters to start information collection.

[0066] The AI ​​accelerator card performance database in step S2 of this embodiment includes basic information such as the AI ​​accelerator card model, computing power, memory size, maximum power consumption, maximum temperature and temperature protection threshold, as well as input information of the final test results and power consumption.

[0067] The test parameters in step S4 of this embodiment include deep learning framework selection, model selection, weight selection, data set selection, accuracy selection, test rounds and duration, model complexity and data processing volume.

[0068] The specific execution of the automated test in step S5 of this embodiment is as follows:

[0069] S501. Execute the test script to automatically deploy the deep learning framework, test model, and data set to the container where the AI ​​accelerator card is located;

[0070] S502. In the selected test mode, start the test program to evaluate the performance of the AI ​​accelerator card, monitor the utilization rate (including memory usage rate) and key indicators such as power consumption and temperature of the AI ​​accelerator card in real time, and monitor the working status and abnormal alarm information in real time;

[0071] S503. Record the test data, and gradually adjust the test pressure (such as increasing the model complexity, increasing the data processing volume, etc.) according to the preset test plan, and repeat the test process to comprehensively evaluate the performance of the AI ​​accelerator card under different loads.

[0072] The report content in step S8 of this embodiment includes test environment information, test parameter settings, test result summary and performance evaluation charts;

[0073] The test report is saved in the form of electronic document, and the export function is provided to facilitate users to review and share.

[0074] Embodiment 2:

[0075] This embodiment also provides an electronic device, including: a memory and a processor;

[0076] Wherein, the memory stores computer-executable instructions;

[0077] The processor executes the computer execution instructions stored in the memory, so that the processor executes the performance evaluation method of the AI ​​accelerator card based on the domestic platform in any embodiment of the present invention.

[0078] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor may be a microprocessor or any conventional processor, etc.

[0079] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state storage devices.

[0080] Embodiment 3:

[0081] This embodiment also provides a computer-readable storage medium, which stores a plurality of instructions, which are loaded by a processor to enable the processor to execute the performance evaluation method of an AI accelerator card based on a domestic platform in any embodiment of the present invention. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code that implements the functions of any of the above embodiments is stored, and a computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage medium.

[0082] In this case, the program code itself read from the storage medium can realize the function of any one of the above-mentioned embodiments, and thus the program code and the storage medium storing the program code constitute a part of the present invention.

[0083] The storage medium embodiments for providing the program code include a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card, and a ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0084] In addition, it should be clear that the functions of any of the above embodiments can be implemented not only by executing the program code read by the computer, but also by enabling an operating system operating on the computer to complete part or all of the actual operations based on instructions from the program code.

[0085] In addition, it can be understood that the program code read from the storage medium is written to a memory provided in an expansion board inserted into the computer or written to a memory provided in an expansion unit connected to the computer, and then based on the instructions of the program code, a CPU installed on the expansion board or the expansion unit is enabled to perform part or all of the actual operations, thereby realizing the functions of any of the above-mentioned embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A performance evaluation method for AI accelerator cards based on a domestic platform, characterized in that: The method is as follows: System initialization and information collection: Insert the AI ​​accelerator card to be evaluated into the evaluation platform, power on the evaluation platform and load the evaluation system. The evaluation system provides users with single-card test and multi-card test options. When testing a single card, the evaluation system will start the corresponding number of container software according to the number of connected AI accelerator cards; when testing multiple cards, the evaluation system will start the corresponding number of container software according to the test group, load the predefined test environment and configuration script in the container, automatically complete the configuration and installation of the driver and SDK software stack and interface parameters provided by the user, accurately identify and connect to the AI ​​accelerator card to be evaluated, and ensure that the AI ​​accelerator card is in normal status; Obtain basic information of the AI ​​accelerator card through AI accelerator card interface parameters or user input, and store it in the AI ​​accelerator card performance database; the basic information of the AI ​​accelerator card includes model, computing power, memory size, power consumption, and temperature protection threshold information; Collected information: including AI accelerator card working status, health information, power consumption, memory occupancy, accelerator card occupancy, temperature and power consumption; Test mode and parameter setting: Provide a graphical user interface or command line interface to allow users to select the test mode and set the corresponding test parameters; at the same time, the evaluation platform automatically adjusts the test script and configuration file according to the user input to prepare the test environment; Perform automated testing; Data collection and statistical analysis: Collect and record the test results and power consumption data in all test steps, store them in the AI ​​accelerator card performance database, and perform statistical analysis on the test data to calculate the minimum, maximum, and average statistics of the test indicator results to evaluate the performance stability of the AI ​​accelerator card; then analyze the power consumption data, calculate the energy efficiency ratio, and evaluate the energy efficiency performance of the AI ​​accelerator card; Error handling and feedback: During the test process, the test status and test results are monitored in real time. Once an error or abnormality is found, the test is stopped immediately, and a clear error prompt is given. The key information during the test is recorded and an error report is generated for users to review and follow up. Test report generation: Automatically generate detailed test reports based on test data and analysis results.

2. The method for evaluating the performance of an AI accelerator card based on a domestic platform according to claim 1 is characterized in that: The evaluation platform includes processor, memory, hard disk, graphics card, network card, power supply, power meter and temperature sensor.

3. The method for evaluating the performance of an AI accelerator card based on a domestic platform according to claim 1 or 2, characterized in that: The evaluation platform also has an AI module interface and a standard PCIE interface to facilitate the connection of the AI ​​module and the standard AI acceleration card.

4. The method for evaluating the performance of an AI accelerator card based on a domestic platform according to claim 3 is characterized in that: The configuration script also has the following functions: ①The evaluation platform identifies the AI ​​accelerator card, including the lspci command; ② Compare the AI ​​accelerator card type with the driver, software stack, and interface parameters provided by the user to see if they match: If the match is successful, it will be automatically installed; If the match is unsuccessful or the installation fails, feedback information will be provided; ③ Generate an information collection script based on the interface parameters to start information collection.

5. The method for evaluating the performance of an AI accelerator card based on a domestic platform according to claim 4 is characterized in that: The AI ​​accelerator card performance database includes basic information such as AI accelerator card model, computing power, memory size, maximum power consumption, maximum temperature and temperature protection threshold, as well as entry information of final test results and power consumption.

6. The method for evaluating the performance of an AI accelerator card based on a domestic platform according to claim 5 is characterized in that: The test parameters include deep learning framework selection, model selection, weight selection, data set selection, accuracy selection, test rounds and duration, model complexity and data processing volume.

7. The method for evaluating the performance of an AI accelerator card based on a domestic platform according to claim 6 is characterized in that: The automated tests are performed as follows: Execute the test script to automatically deploy the deep learning framework, test model, and data set to the container where the AI ​​accelerator card is located; In the selected test mode, start the test program to evaluate the performance of the AI ​​accelerator card, monitor the key indicators of the AI ​​accelerator card's utilization, power consumption, and temperature in real time, and monitor the working status and abnormal alarm information in real time; Record the test data, gradually adjust the test pressure according to the preset test plan, and repeat the test process to comprehensively evaluate the performance of the AI ​​accelerator card under different loads.

8. The method for evaluating the performance of an AI accelerator card based on a domestic platform according to claim 7 is characterized in that: The report includes test environment information, test parameter settings, test result summary and performance evaluation charts; The test report is saved in electronic document form and provides export function.

9. An electronic device, characterized in that: include: memory and at least one processor; Wherein, the memory stores a computer program; The at least one processor executes the computer program stored in the memory, so that the at least one processor executes the performance evaluation method of the AI ​​accelerator card based on the domestic platform as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by a processor to implement the performance evaluation method of an AI accelerator card based on a domestic platform as described in any one of claims 1 to 8.

Images