Accelerator card test method, device and equipment and storage medium
By automatically detecting the accelerator card model and installing drivers, real-time monitoring and diagnostic functions are provided and stress tests are performed, the problem of instability of the accelerator card in the existing technology is solved, ensuring the stability and performance of the server when paired with the accelerator card.
Patent Information
- Application Number
- CN202411120539.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-11-08
AI Technical Summary
The lack of complete driver installation, diagnostic and stress testing solutions for accelerator cards of various brands in the prior art, resulting in unstable acceleration cards when paired with servers.
By automatically detecting the accelerator card model and installing the corresponding driver, it provides real-time monitoring and diagnostic functions, performs stress tests, obtains operating status and substrate management controller system management logs, and stores test results to determine error information.
Ensure that the acceleration card operates stably under high load, provide real-time monitoring and diagnostic functions, and ensure the stability and performance of the server when shipped with the acceleration card.
Smart Images

Figure CN118916255A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of server diagnosis, and in particular to an acceleration card testing method, device, equipment and storage medium. Background Art
[0002] GPU (Graphics Processing Unit) is an electronic circuit specially used for processing images and videos. It was originally designed to accelerate graphics rendering. However, as its powerful parallel computing ability is widely recognized, its application has been expanded to many other fields. However, most GPUs are currently produced by individual manufacturers. However, more and more accelerator card manufacturers are gradually emerging, and its application is becoming more and more extensive. However, in the field of server diagnosis, there is still a lack of a complete driver installation, diagnosis and stress testing solution for various brands of accelerator cards to ensure the reliable operation of these accelerator cards.
[0003] It can be seen that how to perform offline driver installation and diagnostic stress testing on acceleration cards of various brands to ensure the stability of the server to be tested when it is shipped with the acceleration card is a problem that personnel in the server diagnosis field currently need to solve. Summary of the invention
[0004] The purpose of the embodiments of the present invention is to provide an accelerator card testing method, device, equipment and storage medium, which can automatically detect the model of the accelerator card and install the corresponding driver, provide real-time monitoring and diagnosis functions, and perform stress testing to solve the problem of unstable accelerator cards when the server to be tested is shipped with accelerator cards.
[0005] In order to solve the above technical problems, in a first aspect, an embodiment of the present invention provides an accelerator card testing method, comprising:
[0006] Determine a test system corresponding to the accelerator card according to the accelerator card model of the accelerator card in the server to be tested, and obtain a driver corresponding to the accelerator card in a preset driver library under a preset directory through the test system;
[0007] Using the driver to drive the accelerator card in the server to be tested, and running a corresponding graphics processor management tool on the server to be tested according to the accelerator card model, so as to use the graphics processor management tool to read the running status information of the graphics processor in the accelerator card, and determine whether the current running status of the accelerator card is normal according to the preset running status parameters and the running status information;
[0008] If the current running state of the accelerator card is normal, a stress test is performed on the accelerator card to obtain a corresponding stress test result log, and a baseboard management controller system management log of the accelerator card is obtained, so as to obtain a stress test result of the accelerator card based on the stress test result log and the baseboard management controller system management log and store the result in a preset data table;
[0009] The error information of the acceleration card is determined according to the stress test result in the preset data table.
[0010] In some embodiments, obtaining, by the test system, a driver program corresponding to the accelerator card in a preset driver library under a preset directory includes:
[0011] Entering the test system through the pre-startup execution environment, and determining whether the test system supports the installation operation of the driver;
[0012] If the test system does not support the driver installation operation, the system information of the current test system is stored in the preset data table, and a new test system is switched according to the accelerator card model to obtain the driver corresponding to the accelerator card from the preset driver library in the preset directory using the new test system;
[0013] If the test system supports the installation operation of the driver, the driver corresponding to the acceleration card is directly obtained from the preset driver library through the current test system.
[0014] In some embodiments, before obtaining the driver corresponding to the acceleration card in the preset driver library under the preset directory through the test system, the method further includes:
[0015] Determine whether the preset directory contains the preset driver library;
[0016] If the preset directory does not contain the preset driver library, a corresponding first error message is generated, and the error message is stored in the preset data table;
[0017] If the preset directory contains the preset driver library, the driver corresponding to the acceleration card is determined according to the acceleration card model, and the driver is stored in the preset driver library under the preset directory.
[0018] In some embodiments, storing the driver in the preset driver library under the preset directory includes:
[0019] If the installation mode of the driver corresponding to the accelerator card is offline installation, the offline installation package of the driver is directly downloaded to the preset driver library under the preset directory, and the corresponding driver installation command is determined according to the test system to execute the driver installation command to install the offline installation package in the preset driver library to the server to be tested;
[0020] If the installation method of the driver corresponding to the acceleration card is online installation, run the online installation script to install the driver to the preset hard disk system, generate the corresponding target image based on the preset hard disk system through the preset image generation command and store it in the preset driver library under the preset directory, so as to can the target image in the preset driver library to the server to be tested.
[0021] In some embodiments, judging whether the current operating state of the accelerator card is normal according to the preset operating state parameter and the operating state information includes:
[0022] Determine the number of accelerator cards in operation of the accelerator cards in the test system by using the operation status information, and compare the number of accelerator cards in operation with the actual number of accelerator cards currently in the test system;
[0023] If the number of the accelerator cards in operation is not equal to the actual number of the accelerator cards, generating a first alarm message and storing it in the preset data table;
[0024] Determine the current operating state parameters of the accelerator card in the current test system by using the operating state information, and compare the preset operating state parameters with the current operating state parameters, so as to judge whether the current operating state of the accelerator card is normal according to the comparison result;
[0025] If there is an abnormality in the current running state of the acceleration card, a second alarm message is generated and stored in the preset data table.
[0026] In some embodiments, the step of performing a stress test on the accelerator card to obtain a corresponding stress test result log includes:
[0027] Creating a target matrix to be operated according to a preset matrix creation rule, and copying the target matrix to the accelerator card memory of the accelerator card, so as to use the accelerator card to perform matrix operations on the target matrix in the accelerator card memory according to a matrix operation function in a preset matrix operation library, and recording the operation time of the target matrix, and determining the number of floating-point operations of the accelerator card within a preset unit time according to the operation time, so as to determine the data operation performance of the accelerator card according to the number of floating-point operations;
[0028] Testing the video memory bandwidth in the accelerator card by using a preset bandwidth test tool to determine the data operation speed of the accelerator card according to a preset test command, and determining the data operation performance of the accelerator card according to the data operation speed;
[0029] Creating several types of accelerator card running tasks, and using the accelerator card to run the accelerator card running tasks simultaneously through the server to be tested, so as to collect the memory utilization and real-time temperature of the accelerator card when the accelerator card running tasks are running, and determining the task running performance of the accelerator card according to the memory utilization and the real-time temperature;
[0030] Adjusting the ambient temperature of the environment in which the accelerator card is located, and continuously running the accelerator card running task, judging whether the accelerator card triggers a preset high temperature warning when the accelerator card running task is running, so as to obtain a corresponding judgment result, and determining the corresponding high temperature running performance of the accelerator card when running under the preset high temperature condition according to the judgment result;
[0031] The stress test result log of the acceleration card is generated based on the data calculation performance, the data operation performance, the task running performance and the high temperature running performance.
[0032] In some embodiments, obtaining the stress test result of the accelerator card based on the stress test result log and the baseboard management controller system management log includes:
[0033] Determine whether there are abnormalities in voltage data, temperature data and preset accelerator card test parameters when the accelerator card is subjected to stress testing based on the baseboard management controller system management log;
[0034] If there is an abnormality, a third alarm message is generated and stored in the preset data table, and the process of stress testing the accelerator card is stopped;
[0035] If there is no abnormality, whether the current stress test of the acceleration card passes is determined based on the data calculation performance, the data operation performance, the task running performance and the high temperature running performance in the stress test result log; if it fails, corresponding fault information is generated based on the stress test result log and stored in the preset data table.
[0036] In a second aspect, the present invention discloses an accelerator card testing device, comprising:
[0037] A driver acquisition module is used to determine a test system corresponding to the accelerator card according to the accelerator card model of the accelerator card in the server to be tested, and to acquire a driver program corresponding to the accelerator card in a preset driver library under a preset directory through the test system;
[0038] a state judgment module, used to drive the accelerator card in the server to be tested by using the driver, and run a corresponding graphics processor management tool on the server to be tested according to the accelerator card model, so as to read the running state information of the graphics processor in the accelerator card by using the graphics processor management tool, and judge whether the current running state of the accelerator card is normal according to preset running state parameters and the running state information;
[0039] a stress test module, configured to, if the current operating state of the accelerator card is normal, perform a stress test on the accelerator card to obtain a corresponding stress test result log, and obtain a baseboard management controller system management log of the accelerator card, so as to obtain a stress test result of the accelerator card based on the stress test result log and the baseboard management controller system management log and store the result in a preset data table;
[0040] An information determination module is used to determine the error information of the acceleration card according to the stress test result in the preset data table.
[0041] In a third aspect, the present invention discloses an accelerator card testing device, comprising:
[0042] Memory for storing computer programs;
[0043] A processor is used to execute the computer program to implement the steps of the aforementioned acceleration card testing method.
[0044] In a fourth aspect, the present invention discloses a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned acceleration card testing method are implemented.
[0045] It can be seen that the present application can determine the test system corresponding to the accelerator card according to the accelerator card model of the accelerator card in the server to be tested, and obtain the driver corresponding to the accelerator card in the preset driver library under the preset directory through the test system, and then use the driver to drive the accelerator card in the server to be tested, and run the corresponding graphics processor management tool on the server to be tested according to the accelerator card model, so as to use the graphics processor management tool to read the operating status information of the graphics processor in the accelerator card, and judge whether the current operating status of the accelerator card is normal according to the preset operating status parameters and operating status information; if the current operating status of the accelerator card is normal, then perform a stress test on the accelerator card to obtain the corresponding stress test result log, and obtain the baseboard management controller system management log of the accelerator card, so as to obtain the stress test result of the accelerator card based on the stress test result log and the baseboard management controller system management log and store it in the preset data table, so as to determine the error information of the accelerator card according to the stress test result in the preset data table.
[0046] Beneficial effect: It can be seen from the above technical solution that the present application provides a method for driver installation and diagnostic stress testing of an accelerator card, which can identify the model of the accelerator card and extract and install the corresponding driver from the driver library. After running, it provides real-time monitoring of the performance indicators of the accelerator card, and after performing a series of stress tests, log analysis of the stress test results is performed to obtain the fault information of the accelerator card. In this way, the model of the accelerator card can be automatically detected and the corresponding driver can be installed to provide real-time monitoring and diagnostic functions, and stress tests can be performed to ensure the stability and performance of the accelerator card under high load, which can ensure the stability of the server to be tested when it is shipped with the accelerator card. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0048] Figure 1 A flow chart of an accelerator card testing method provided by an embodiment of the present invention;
[0049] Figure 2 A flowchart of an accelerator card driver installation provided by an embodiment of the present invention;
[0050] Figure 3 A flow chart of the feedback of the stress test result of an accelerator card provided by an embodiment of the present invention;
[0051] Figure 4 A schematic diagram of an accelerator card testing system provided by an embodiment of the present invention;
[0052] Figure 5 A specific flow chart of an accelerator card testing method provided by an embodiment of the present invention;
[0053] Figure 6 A schematic diagram of an accelerator card diagnostic result provided by an embodiment of the present invention;
[0054] Figure 7 A flow chart of an accelerator card stress test provided by an embodiment of the present invention;
[0055] Figure 8 A schematic diagram of an accelerator card stress test result provided by an embodiment of the present invention;
[0056] Fig. 9 A schematic diagram of the structure of an accelerator card testing device provided by an embodiment of the present invention;
[0057] Fig.10 A structural diagram of an electronic device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0058] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0059] The terms "including" and "having" in the specification of the present invention and the above-mentioned drawings, as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.
[0060] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0061] Next, an acceleration card testing method provided by an embodiment of the present invention is described in detail. Figure 1 An accelerator card testing method provided by an embodiment of the present invention includes:
[0062] Step S11: determining a test system corresponding to the accelerator card according to the accelerator card model of the accelerator card in the server to be tested, and obtaining a driver corresponding to the accelerator card in a preset driver library under a preset directory through the test system.
[0063] In this embodiment, Figure 2 As shown, in order to ensure product and data security, the general production line environment will not be connected to the external network, and the test is performed by booting into the system through PXE (Preboot eXecution Environment). In addition, the server is not shipped with the system, so that customers can install the required operating system according to their own needs after the product arrives. However, this creates trouble for installing the accelerator card driver during production line testing. Therefore, when obtaining the driver corresponding to the accelerator card in the preset driver library under the preset directory through the test system, after entering the test system through the pre-boot execution environment, it can be determined whether the test system supports the driver installation operation. If the test system does not support the driver installation operation, the system information of the current test system is stored in the preset data table, that is, Figure 2In the pass / failed table of the database shown, the new test system is switched to according to the accelerator card model to use the new test system to obtain the driver corresponding to the accelerator card from the preset driver library under the preset directory; if the test system supports the installation operation of the driver, the driver corresponding to the accelerator card is directly obtained from the preset driver library through the current test system. That is to say, in this embodiment, the accelerator card model needs to be identified first, and then the server to be tested is powered on and booted into the common production test system (usually the redhat system) through PXE, and the current server to be tested is judged to be equipped with a certain accelerator card. Different accelerator cards have different device numbers. The "lspci |grep -idevice number" command can be used to check which accelerator card is currently used. After identifying the model of the accelerator card, the corresponding test system can be switched. After switching different test systems according to different accelerator cards, if this accelerator card can support the redhat system, the next step is directly continued, otherwise it is switched to the corresponding supported system for subsequent processes to realize the function of automatically identifying the accelerator card model, and judge the operating system supported by the current model, switch to the corresponding system, and then obtain the driver library and automatic detection tool in the specific directory of the main server, which can realize the automatic driver installation of various models of accelerator cards during production testing.
[0064] It should be pointed out that, in this embodiment, before obtaining the driver corresponding to the accelerator card in the preset driver library under the preset directory through the test system, it is also necessary to determine whether the preset directory contains the preset driver library. In a specific embodiment, if the preset directory does not contain the preset driver library, a corresponding first error message is generated, and the error message is stored in the preset data table. In another specific embodiment, if the preset directory contains the preset driver library, the driver corresponding to the accelerator card is determined according to the accelerator card model, and the driver is stored in the preset driver library under the preset directory. Specifically, when the driver is stored in the preset driver library under the preset directory, if the installation method of the driver corresponding to the accelerator card is offline installation, the offline installation package of the driver is directly downloaded to the preset driver library under the preset directory, and the corresponding driver installation command is determined according to the test system, so as to execute the driver installation command to install the offline installation package in the preset driver library to the server to be tested; if the installation method of the driver corresponding to the accelerator card is online installation, the online installation script is run to install the driver to the preset hard disk system, and the corresponding target image is generated based on the preset hard disk system by the preset image generation command and stored in the preset driver library under the preset directory, so as to can the target image in the preset driver library to the server to be tested.
[0065] It should be pointed out that, in this embodiment, when the test system corresponding to the accelerator card is determined according to the accelerator card model of the accelerator card in the server to be tested, and the driver corresponding to the accelerator card in the preset driver library under the preset directory is obtained through the test system to drive the accelerator card for testing, if the current server to be tested can be equipped with several accelerator cards of different models, the accelerator card driver of each accelerator card can be installed in the server to be tested according to the test system corresponding to each accelerator card, so as to perform stress testing on multiple types of accelerator cards at the same time, thereby improving the test efficiency of the accelerator card stress test.
[0066] Based on the above installation process, in this embodiment, the actual installation of the corresponding driver of each type of accelerator card is taken into account. There are two main ways to adapt the accelerator card installation driver: In a specific embodiment, for a simple and easy-to-adapt accelerator card, it can be installed by downloading the installation package offline. Specifically, the offline installation package can be installed by using different installation commands corresponding to different systems. For example, if it is a redhat system, it can be installed by rpm -ivh xxx.rpm; if it is an ubuntu system, it can be installed by dpkg -i xxx.deb. After the required offline installation package is installed, run the xxx-smi tool (i.e., the graphics processor management tool) to print out the GPU's device, Node, Temp, Power and other information. In another specific embodiment, for an accelerator card with complex functions, there are various dependencies in the offline installation package. It may take a lot of effort and time to adapt to the production line environment. Therefore, in the development stage of the server to be tested, the online installation script can be run through the hard disk system to install the accelerator card driver. After the installation is successful, the driver has been installed in the hard disk system. Then, the hard disk system is converted into an iso image through the command mkisofs -o / path / to / save / image.iso / path / to / mounted / image or into an img image through the command dd if= / dev / sdX of= / path / to / save / image.img bs=4M status=progress. Finally, it is canned into the server to be tested for testing. After completing the above installation process, all offline installation packages and / or iso images and img images are placed in a specific directory of the main server to facilitate acquisition during the test of the server to be tested. In this way, after starting the automatic driver installation during the test, switch to the corresponding supported system of the accelerator card, and extract the corresponding driver from the driver library in the specific directory of the main server according to different accelerator cards (the driver includes the above-mentioned offline installation package and image) and download it to the server to be tested, and then run . / install.sh to automatically install the driver. In this way, in this embodiment, the accelerator card model can be automatically identified, the operating system supported by the current model can be determined, the corresponding system can be switched to, and then the driver library and automatic detection tool can be obtained from the specific directory of the main server. Finally, the driver library is extracted and . / install.sh is run to install the corresponding driver, which can realize the automatic driver installation of the accelerator card during production testing, and also facilitate the update and iteration of the corresponding driver when the accelerator card version is updated later.
[0067] Step S12: using the driver to drive the accelerator card in the server to be tested, and running a corresponding graphics processor management tool on the server to be tested according to the accelerator card model, so as to use the graphics processor management tool to read the running status information of the graphics processor in the accelerator card, and determine whether the current running status of the accelerator card is normal according to preset running status parameters and the running status information.
[0068] This embodiment is mainly used for preliminary testing of the accelerator card. Specifically, the accelerator card in the server to be tested can be driven by a driver, and a corresponding graphics processor management tool (xxx-smi tool) can be run on the server to be tested according to the accelerator card model, so as to use the graphics processor management tool to read the running status information of the graphics processor in the accelerator card, and determine whether the current running status of the accelerator card is normal according to the preset running status parameters and the running status information.
[0069] Step S13: If the current operating status of the accelerator card is normal, perform a stress test on the accelerator card to obtain a corresponding stress test result log, and obtain a baseboard management controller system management log of the accelerator card, so as to obtain the stress test result of the accelerator card based on the stress test result log and the baseboard management controller system management log and store it in a preset data table.
[0070] In this embodiment, if the current operating state of the accelerator card is normal, the accelerator card is stress tested to obtain the corresponding stress test result log, and the baseboard management controller system management log (BMC SEL log, Baseboard Management Controller System Event Log) of the accelerator card is obtained, so as to obtain the stress test result of the accelerator card based on the stress test result log and the baseboard management controller system management log and store it in a preset data table. It can be understood that the preset data table is the database pass / failed table in the above steps, and the preset data table mentioned later means the database pass / failed table. By storing various types of data generated during the accelerator card stress test process, especially the data when the accelerator card driver fails to install and a fault occurs during the stress test, in the above data table, it is convenient to generate the corresponding error information later.
[0071] Step S14: Determine error information of the acceleration card according to the stress test result in the preset data table.
[0072] In this embodiment, Figure 3As shown, first, each type of data in the database table can be filtered, and then the error information obtained after filtering is stored in the failed table in the pass / failed table, and other information is stored in the pass table, and other acceleration cards are diagnosed, and the error data is fed back to the corresponding production line manager through a preset feedback method, and the production line manager checks the fault and performs corresponding replacement processing. It is understandable that the above feedback method includes but is not limited to information feedback methods such as email feedback and SMS feedback.
[0073] In summary, this embodiment provides a Figure 4 The driver installation and diagnostic system of the accelerator card shown can automatically identify the accelerator card model through the automated driver installation module, determine the operating system supported by the current model, and switch to the corresponding system, while automatically obtaining the driver library and the automatic detection tool, and finally extract and run it in the driver library, and then use the diagnostic module to run the xxx-smi tool to read the relevant information of the accelerator card, run a simple test to determine the status of the accelerator card, and then use the stress test module to execute a series of stress test cases, analyze the accelerator card stress test process to obtain the corresponding log, and better perform accelerator card fault analysis and performance analysis, so as to use the reminder template to read the error information, and send the error information to the relevant production line management personnel in the form of an email to remind the production line personnel to handle the error information in a timely manner to avoid unnecessary impact. In this way, by automatically detecting the model of the accelerator card and installing the corresponding driver, real-time monitoring and diagnostic functions are provided, and stress tests are performed to ensure the stability and performance of the accelerator card under high load.
[0074] Based on the previous embodiment, it can be seen that the present invention can automatically detect the model of the accelerator card and install the corresponding driver, provide real-time monitoring and diagnosis functions, and execute the corresponding stress test process. Next, the process of the accelerator card stress test will be described in detail in this embodiment. Figure 5 As shown, an embodiment of the present invention discloses a specific accelerator card testing method, the method comprising:
[0075] Step S21, using a driver to drive the accelerator card in the server to be tested, and running a corresponding graphics processor management tool on the server to be tested according to the accelerator card model, so as to use the graphics processor management tool to read the running status information of the graphics processor in the accelerator card, and determine whether the current running status of the accelerator card is normal according to preset running status parameters and the running status information.
[0076] In this embodiment, when judging whether the current running state of the accelerator card is normal according to the preset running state parameters and the running state information, the running state information can be used to determine the number of running accelerator cards in the test system, and the running number of accelerator cards can be compared with the actual number of accelerator cards in the current test system; if the running number of accelerator cards is not equal to the actual number of accelerator cards, a first alarm message is generated and stored in a preset data table. If the running number of accelerator cards is equal to the actual number of accelerator cards, the running state information is used to determine the current running state parameters of the accelerator cards in the current test system, and the preset running state parameters are compared with the current running state parameters, so as to judge whether the current running state of the accelerator card is normal according to the comparison result obtained, and when there is an abnormality in the current running state of the accelerator card, a second alarm message is generated and stored in the preset data table.
[0077] In this embodiment, after the automatic driver installation module installs the driver, the xxx-smi tool can be run to read the relevant information of the GPU. After that, the number of accelerator cards can be simply determined. If it is inconsistent with the actual number, a production line alarm is generated. Otherwise, a simple test is run to collect the operation data of the accelerator card to determine whether there is an abnormality in the current operation status of the accelerator card. In a specific embodiment, Figure 6 As shown, the operating status of the accelerator card and the GPU occupancy rate can be determined. When high temperature or other abnormalities occur, they are recorded in the database error table and an alarm is generated.
[0078] Step S22: If the current operating status of the accelerator card is normal, perform a stress test on the accelerator card to obtain a corresponding stress test result log, and obtain a baseboard management controller system management log of the accelerator card, so as to obtain the stress test result of the accelerator card based on the stress test result log and the baseboard management controller system management log and store it in a preset data table.
[0079] When the accelerator card is stress-tested in this embodiment, the target matrix to be operated can be first created according to the preset matrix creation rules, and the target matrix can be copied to the accelerator card memory of the accelerator card, so that the accelerator card can perform matrix operations on the target matrix in the accelerator card memory according to the matrix operation function in the preset matrix operation library, and the operation time of the target matrix can be recorded, and the number of floating-point operations of the accelerator card in the preset unit time can be determined according to the operation time, so as to determine the data operation performance of the accelerator card according to the number of floating-point operations. Then, the video memory bandwidth in the accelerator card is tested by the preset broadband test tool, so as to determine the data operation speed of the accelerator card according to the preset test command, and the data operation performance of the accelerator card can be determined according to the data operation speed. It is also necessary to create several types of acceleration card running tasks, and use the acceleration card to run the acceleration card running tasks simultaneously through the server to be tested, so as to collect the memory utilization and real-time temperature of the acceleration card when the acceleration card running tasks are running, and determine the task running performance of the acceleration card based on the memory utilization and real-time temperature; then adjust the ambient temperature of the environment where the acceleration card is located, and continue to run the acceleration card running tasks to determine whether the acceleration card triggers a preset high temperature warning when the acceleration card running tasks are running, so as to obtain the corresponding judgment result, and determine the corresponding high temperature running performance of the acceleration card when running under the preset high temperature conditions based on the judgment result. After passing the above stress test, the stress test result log of the acceleration card can be generated based on the data calculation performance, data operation performance, task running performance and high temperature running performance.
[0080] Based on the above stress testing process, if Figure 7 As shown in the figure, you can perform stress testing on the accelerator card, and continuously collect the temperature change information, power consumption change information, and performance information of the accelerator card during the stress test. After the stress test is completed, a corresponding stress test result log is generated, named result.log, and the temperature change information, power consumption change information, and performance information of the accelerator card are recorded in the logs, named temp_result.log, power_result.log, and FPS_result.log respectively. After that, you can directly locate the accelerator card problem based on result.log. In addition, you can also combine the BMC SEL log and system log generated by ipmitool sel list to determine the accelerator card failure.
[0081] In a specific embodiment, the stress test is mainly performed from the following perspectives, mainly including benchmark testing, storage and bandwidth testing, simulation testing, and failure and recovery testing.
[0082] First, run the benchmark test, mainly through the matrix multiplication test to evaluate the performance of the device when processing large-scale linear algebra operations and floating-point computing capabilities. The implementation method is to create two large matrices (such as 1000x1000 or larger), copy them to the accelerator card memory, use the matrix multiplication function of the CUBLAS library, and record the operation time, and then calculate the number of floating-point operations per second through FLOPS=2*N^3 / time, where N represents the dimension of the matrix and time represents the operation time of the matrix multiplication function. The performance of the accelerator card is judged by the operation time. Generally, the operation time of 1024x1024 size is between a few milliseconds and more than ten milliseconds; the size of 4096x4096 is between tens of milliseconds and hundreds of milliseconds; the size of 8192x8192 is between hundreds of milliseconds and a few seconds. The higher the number of floating-point operations, the faster the accelerator card can complete more calculations, that is, the better the data operation performance of the accelerator card.
[0083] Next, run storage and bandwidth tests, mainly using test tools to test video memory bandwidth. Use bandwidthtest to test read, write, and copy speeds (. / bandwidthTest --device=x, where x is the number of an accelerator card).
[0084] Then, we run the load simulation test, mainly by simulating mixed workloads, with the server running multiple different types of tasks at the same time, to observe the performance of the accelerator card when processing multiple tasks at the same time. At the same time, we conduct high load tests on the accelerator card and memory stress tests to observe the utilization of the accelerator card and memory and whether the accelerator card generates high temperature.
[0085] Finally, run the fault and recovery test, mainly through the high temperature environment test, run the high load task of the accelerator card in a heated environment, observe the temperature control and protection mechanism of the accelerator card, and observe whether the accelerator card will trigger the high temperature warning and then pass the signal to the CPLD (Complex Programmable Logic Device) for shutdown operation.
[0086] After the current stress test is completed, you can get the following Figure 8The stress test results shown can be used for detailed analysis. First, based on the baseboard management controller system management log, determine whether there are any abnormalities in the voltage data, temperature data, and preset acceleration card test parameters during the stress test of the accelerator card; if there are any abnormalities, a third alarm message is generated and stored in the preset data table, and the process of stress testing the accelerator card is stopped; if there are no abnormalities, based on the data calculation performance, data operation performance, task running performance, and high temperature running performance in the stress test result log, determine whether the current stress test of the accelerator card has passed. If it has not passed, the corresponding fault information is generated based on the stress test result log and stored in the preset data table. Figure 7 As shown, when analyzing the stress test results, first check the stress test results to check whether the result.log log is fail. If fail, report an error and stop. If no error is reported, check the BMC SEL log to determine whether the stress test program causes the voltage and temperature of the server to be tested to exceed the threshold. If yes, report an error and stop; and continue to determine whether the SEL log contains faults related to the accelerator card (such as high temperature of the accelerator card, bandwidth reduction of the pcie link where the accelerator card is located, etc.). If yes, report an error and stop. Then analyze the collected temp_result.log, power_result.log, and FPS_result.log logs, and evaluate the temperature curve of the accelerator card under high load through temp_result.log to determine whether there is overheating; evaluate the power consumption of the accelerator card under high load through power_result.log to determine whether it exceeds the recommended range; check the changes in frame rate, video memory frequency, etc. in FPS_result.log to determine whether the performance of the accelerator card is stable. Finally, if the above logs are correct, it means that the accelerator card stress test has passed, indicating that the stability is good under long-term high-load operation, otherwise it means that the accelerator card has a fault. It is also understandable that before performing the above stress test, the current accelerator card model can also be identified through lspci and whether the corresponding accelerator card driver currently exists. The above lspci is a command line tool used to list the detailed information of all PCI (Peripheral Component Interconnect) buses and devices in the system. The displayed information includes device ID, vendor name, device name, etc. It is usually used to diagnose hardware problems, check which devices are connected to which PCI buses, and determine the driver module required for a specific device.
[0087] Through the above technical solution, in this embodiment, the xxx-smi tool can be run to read the relevant information of the accelerator card, and a simple test can be run to collect the operation data of the accelerator card to determine the operation status of the accelerator card. At the same time, real-time monitoring of various performance indicators of the accelerator card is provided, and then a series of stress test cases are executed for the accelerator card to collect the temperature change information, power consumption change information, and performance information of the accelerator card during the stress test process, and the accelerator card stress test process log, OS (Operating System, operating system) log and BMC SEL log are gradually analyzed, and then the accelerator card stress test process log, BMC SEL log and temperature change, power consumption change and performance information log are gradually analyzed, so as to better perform accelerator card fault analysis and performance analysis, and obtain error information in time after the stress test program reports an error, and regularly feedback the error information to the production line management personnel, and handle it in time.
[0088] Fig. 9 A schematic diagram of the structure of an accelerator card testing device provided by an embodiment of the present invention includes:
[0089] A driver acquisition module 11 is used to determine a test system corresponding to the accelerator card according to the accelerator card model of the accelerator card in the server to be tested, and to acquire a driver corresponding to the accelerator card in a preset driver library under a preset directory through the test system;
[0090] A status judgment module 12 is used to drive the accelerator card in the server to be tested by using the driver, and run a corresponding graphics processor management tool on the server to be tested according to the accelerator card model, so as to use the graphics processor management tool to read the running status information of the graphics processor in the accelerator card, and judge whether the current running status of the accelerator card is normal according to the preset running status parameters and the running status information;
[0091] The stress test module 13 is used to perform a stress test on the accelerator card to obtain a corresponding stress test result log if the current running state of the accelerator card is normal, and obtain a baseboard management controller system management log of the accelerator card, so as to obtain a stress test result of the accelerator card based on the stress test result log and the baseboard management controller system management log and store it in a preset data table;
[0092] The information determination module 14 is used to determine the error information of the acceleration card according to the stress test result in the preset data table.
[0093] Fig. 9 The description of the features in the corresponding embodiments can be found in Figure 1 The relevant descriptions of the corresponding embodiments will not be repeated here one by one.
[0094] Through the above technical scheme, this embodiment first determines the test system corresponding to the accelerator card according to the accelerator card model of the accelerator card in the server to be tested, and obtains the driver corresponding to the accelerator card in the preset driver library under the preset directory through the test system, and then uses the driver to drive the accelerator card in the server to be tested, and runs the corresponding graphics processor management tool on the server to be tested according to the accelerator card model, so as to use the graphics processor management tool to read the running status information of the graphics processor in the accelerator card, and judge whether the current running status of the accelerator card is normal according to the preset running status parameters and running status information; if the current running status of the accelerator card is normal, then the accelerator card is subjected to a stress test to obtain the corresponding stress test result log, and the baseboard management controller system management log of the accelerator card is obtained, so as to obtain the stress test result of the accelerator card based on the stress test result log and the baseboard management controller system management log and store it in the preset data table, so as to determine the error information of the accelerator card according to the stress test result in the preset data table. In this way, in this embodiment, the model of the accelerator card can be identified and the corresponding driver can be extracted and installed from the driver library to run, and the performance indicators of the accelerator card can be provided for real-time monitoring, and after a series of stress tests are performed, the log analysis of the stress test result is performed to obtain the fault information of the accelerator card. In this way, the model of the accelerator card can be automatically detected and the corresponding driver can be installed to provide real-time monitoring and diagnostic functions, and stress testing can be performed to ensure the stability and performance of the accelerator card under high load, thus ensuring the stability of the server under test when it is shipped with the accelerator card.
[0095] Fig.10 A structural diagram of an electronic device corresponding to an accelerator card testing method provided by an embodiment of the present invention, such as Fig.10 As shown, the electronic device comprises: a memory 20 for storing a computer program;
[0096] The processor 21 is used to implement the steps of the acceleration card testing method in the above embodiment when executing a computer program.
[0097] The electronic device provided in this embodiment may include but is not limited to a smart phone, a tablet computer, a laptop computer or a desktop computer.
[0098] Among them, the processor 21 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 21 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 21 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.
[0099] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein, after the computer program is loaded and executed by the processor 21, it can implement the relevant steps of the acceleration card testing method disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 202 may include Windows, Unix, Linux, etc. Data 203 may include but is not limited to data transmitted from an external device and received by the electronic device, etc.
[0100] In some embodiments, the electronic device may further include a display screen 22 , an input / output interface 23 , a communication interface 24 , a power supply 25 , and a communication bus 26 .
[0101] Those skilled in the art will understand that Fig.10 The structure shown in the figure does not constitute a limitation on the acceleration card testing method, and may include more or fewer components than those shown in the figure.
[0102] It is understandable that if the acceleration card testing method in the above embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the current technology or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium to execute all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk and other media that can store program codes.
[0103] Based on this, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned acceleration card testing method are implemented.
[0104] The above is a detailed introduction to an accelerator card testing method provided by an embodiment of the present invention. The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.
[0105] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0106] The above is a detailed introduction to the scheme provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, the present invention can also be improved and modified in a number of ways, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.
Claims
1. A method for testing an accelerator card, characterized in that: include: Determine a test system corresponding to the accelerator card according to the accelerator card model of the accelerator card in the server to be tested, and obtain a driver corresponding to the accelerator card in a preset driver library under a preset directory through the test system; Using the driver to drive the accelerator card in the server to be tested, and running a corresponding graphics processor management tool on the server to be tested according to the accelerator card model, so as to use the graphics processor management tool to read the running status information of the graphics processor in the accelerator card, and determine whether the current running status of the accelerator card is normal according to the preset running status parameters and the running status information; If the current running state of the accelerator card is normal, a stress test is performed on the accelerator card to obtain a corresponding stress test result log, and a baseboard management controller system management log of the accelerator card is obtained, so as to obtain a stress test result of the accelerator card based on the stress test result log and the baseboard management controller system management log and store the result in a preset data table; The error information of the acceleration card is determined according to the stress test result in the preset data table.
2. The accelerator card testing method according to claim 1, characterized in that: The step of obtaining the driver corresponding to the accelerator card in a preset driver library under a preset directory through the test system includes: Entering the test system through the pre-startup execution environment, and determining whether the test system supports the installation operation of the driver; If the test system does not support the driver installation operation, the system information of the current test system is stored in the preset data table, and a new test system is switched according to the accelerator card model to obtain the driver corresponding to the accelerator card from the preset driver library in the preset directory using the new test system; If the test system supports the installation operation of the driver, the driver corresponding to the acceleration card is directly obtained from the preset driver library through the current test system.
3. The accelerator card testing method according to claim 1, characterized in that: Before obtaining the driver corresponding to the accelerator card in the preset driver library under the preset directory through the test system, the method further includes: Determine whether the preset directory contains the preset driver library; If the preset directory does not contain the preset driver library, a corresponding first error message is generated, and the error message is stored in the preset data table; If the preset directory contains the preset driver library, the driver corresponding to the acceleration card is determined according to the acceleration card model, and the driver is stored in the preset driver library under the preset directory.
4. The accelerator card testing method according to claim 3, characterized in that: The step of storing the driver in the preset driver library under the preset directory includes: If the installation mode of the driver corresponding to the accelerator card is offline installation, the offline installation package of the driver is directly downloaded to the preset driver library under the preset directory, and the corresponding driver installation command is determined according to the test system to execute the driver installation command to install the offline installation package in the preset driver library to the server to be tested; If the installation method of the driver corresponding to the acceleration card is online installation, run the online installation script to install the driver to the preset hard disk system, generate the corresponding target image based on the preset hard disk system through the preset image generation command and store it in the preset driver library under the preset directory, so as to can the target image in the preset driver library to the server to be tested.
5. The accelerator card testing method according to claim 1, characterized in that: The determining whether the current operating state of the accelerator card is normal according to the preset operating state parameter and the operating state information includes: Determine the number of accelerator cards in operation of the accelerator cards in the test system by using the operation status information, and compare the number of accelerator cards in operation with the actual number of accelerator cards currently in the test system; If the number of the accelerator cards in operation is not equal to the actual number of the accelerator cards, generating a first alarm message and storing it in the preset data table; Determine the current operating state parameters of the accelerator card in the current test system by using the operating state information, and compare the preset operating state parameters with the current operating state parameters, so as to judge whether the current operating state of the accelerator card is normal according to the comparison result; If there is an abnormality in the current running state of the acceleration card, a second alarm message is generated and stored in the preset data table.
6. The accelerator card testing method according to any one of claims 1 to 5, characterized in that: The stress test on the accelerator card to obtain a corresponding stress test result log includes: Creating a target matrix to be operated according to a preset matrix creation rule, and copying the target matrix to the accelerator card memory of the accelerator card, so as to use the accelerator card to perform matrix operations on the target matrix in the accelerator card memory according to a matrix operation function in a preset matrix operation library, and recording the operation time of the target matrix, and determining the number of floating-point operations of the accelerator card within a preset unit time according to the operation time, so as to determine the data operation performance of the accelerator card according to the number of floating-point operations; Testing the video memory bandwidth in the accelerator card by using a preset bandwidth test tool to determine the data operation speed of the accelerator card according to a preset test command, and determining the data operation performance of the accelerator card according to the data operation speed; Creating several types of accelerator card running tasks, and using the accelerator card to run the accelerator card running tasks simultaneously through the server to be tested, so as to collect the memory utilization and real-time temperature of the accelerator card when the accelerator card running tasks are running, and determining the task running performance of the accelerator card according to the memory utilization and the real-time temperature; Adjusting the ambient temperature of the environment in which the accelerator card is located, and continuously running the accelerator card running task, judging whether the accelerator card triggers a preset high temperature warning when the accelerator card running task is running, so as to obtain a corresponding judgment result, and determining the corresponding high temperature running performance of the accelerator card when running under the preset high temperature condition according to the judgment result; The stress test result log of the acceleration card is generated based on the data calculation performance, the data operation performance, the task running performance and the high temperature running performance.
7. The accelerator card testing method according to claim 6, characterized in that: The obtaining the stress test result of the accelerator card based on the stress test result log and the baseboard management controller system management log includes: Based on the baseboard management controller system management log, it is determined whether there are abnormalities in the voltage data, temperature data and preset acceleration card test parameters when the acceleration card is subjected to stress testing; If there is an abnormality, a third alarm message is generated and stored in the preset data table, and the process of stress testing the accelerator card is stopped; If there is no abnormality, whether the current stress test of the acceleration card passes is determined based on the data calculation performance, the data operation performance, the task running performance and the high temperature running performance in the stress test result log; if it fails, corresponding fault information is generated based on the stress test result log and stored in the preset data table.
8. An accelerator card testing device, characterized in that: include: A driver acquisition module is used to determine a test system corresponding to the accelerator card according to the accelerator card model of the accelerator card in the server to be tested, and to acquire a driver program corresponding to the accelerator card in a preset driver library under a preset directory through the test system; a state judgment module, used to drive the accelerator card in the server to be tested by using the driver, and run a corresponding graphics processor management tool on the server to be tested according to the accelerator card model, so as to read the running state information of the graphics processor in the accelerator card by using the graphics processor management tool, and judge whether the current running state of the accelerator card is normal according to preset running state parameters and the running state information; a stress test module, configured to, if the current operating state of the accelerator card is normal, perform a stress test on the accelerator card to obtain a corresponding stress test result log, and obtain a baseboard management controller system management log of the accelerator card, so as to obtain a stress test result of the accelerator card based on the stress test result log and the baseboard management controller system management log and store the result in a preset data table; An information determination module is used to determine the error information of the acceleration card according to the stress test result in the preset data table.
9. An accelerator card testing device, characterized in that: include: Memory for storing computer programs; A processor, configured to execute the computer program to implement the steps of the acceleration card testing method as claimed in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the acceleration card testing method according to any one of claims 1 to 7 are implemented.
Citation Information
Cited By
Method for acquiring acceleration card performance, acceleration card performance test equipment and product
CN120295883A
Link state detection method and device, electronic equipment and storage medium
CN120434154A
Method, system and computer program product for diagnosing hardware state of accelerator card
CN121858372A
