A method for testing training and inference performance of an artificial intelligence accelerator card product

Abstract

The application discloses a kind of training and inference performance test methods of artificial intelligence acceleration card product, comprising the following steps: (1) preparing test environment, providing uniform benchmark hardware detection platform, guaranteeing platform configuration same, ensuring that the baseline is consistent with test mounting;(2) coordinating the resources of the artificial intelligence product to be tested, completing the artificial intelligence acceleration card hardware preparation, driver software preparation, model adaptation work, etc., completing the installation package and test environment deployment required for test preparation;(3) carry out testing according to test scheme, respectively basic performance test, inference single machine single card test and training single machine four card test;(4) according to test results, scoring, inference and training are scored separately, and the specific scoring method can be formulated according to the actual situation.The more intuitive and more comparative evaluation results can be formed according to test data.

Description

Technical Field

[0001] This invention belongs to the field of testing, specifically relating to a testing method for the training and inference performance of AI accelerator cards, particularly for high-computing-power cloud scenarios, to test the adaptability and performance of AI accelerator cards to mainstream frameworks and general models. Background Technology

[0002] With the continuous development and evolution of artificial intelligence (AI) technology, the requirements for the functionality and performance of AI-related businesses and applications are constantly increasing. Research on AI accelerator processors has become a hot topic in both academia and business, with many commercial and academic institutions launching their own dedicated AI accelerators. From the perspective of the main functional classification of existing accelerators, they can be mainly divided into two categories: Training and Inference.

[0003] Currently, the hardware computing power of various AI accelerator cards is evaluated independently by each manufacturer, resulting in inconsistent standards for evaluating AI chip computing power, the data types used in testing, and accuracy constraints. Furthermore, the evaluation tools, data types, datasets, and evaluation methods employed by each manufacturer differ. This leads to a lack of cross-comparison of test results, making it difficult to intuitively compare the merits of different products.

[0004] Therefore, to conduct fair, effective, and targeted evaluations of various AI products mounted on servers, it is necessary to design and establish a more universal, fairer, and more reliable evaluation method. This method should test and evaluate high-performance dedicated AI accelerator cards for inference and training from aspects such as basic hardware computing power performance, performance in applications with models, and power consumption. By combining the relevant technical routes and business status of AI in the power industry, the method should test the adaptability of AI accelerator cards to mainstream power models and general models, providing a reference for the localization of AI basic hardware in the power industry.

[0005] Currently, the hardware computing power of various AI accelerator cards is evaluated independently by each manufacturer. The standards for evaluating computing power, the types of data used in the tests, and the data precision are all inconsistent. Furthermore, the evaluation methods, tools, and test datasets employed by each manufacturer differ. This results in a lack of cross-comparison of test results, making it difficult to intuitively compare the merits of different products. Summary of the Invention

[0006] Purpose of the invention: The technical problem to be solved by the present invention is to provide a more universal performance testing method for artificial intelligence accelerator cards, which addresses the lack of uniformity in existing evaluation standards and the inadequacy of existing evaluation methods for basic artificial intelligence hardware. This method can generate more intuitive and comparative evaluation results based on test data.

[0007] To address the aforementioned technical problems, this invention provides a method for testing the training and inference performance of an artificial intelligence accelerator card product, comprising the following steps:

[0008] (1) Prepare the test environment and provide a unified benchmark hardware testing platform to ensure that the platform configuration is the same and that the test baseline is consistent;

[0009] (2) Coordinate the resources of the AI ​​products to be tested, complete the hardware preparation of the AI ​​accelerator card, the driver software preparation, and the model adaptation, and complete the installation package and test environment deployment required for test preparation;

[0010] (3) Conduct tests according to the test plan, namely basic performance test, inference single machine single card test and training single machine four card test;

[0011] (4) Scoring is conducted based on the test results. Reasoning and training are scored separately. Specific scoring methods can be formulated according to the actual situation.

[0012] The basic performance tests include matrix multiplication, peak computing power testing, and bandwidth testing.

[0013] The matrix multiplication operation was tested using a single machine and a single graphics card, examining the performance of matrix multiplication for different data types.

[0014] The peak computing power test: using a single machine and a single card, the matrix multiplication performance with the highest chip utilization is used as the peak computing power performance;

[0015] The bandwidth test measures the maximum throughput rate of DDR data read or write operations on the AI ​​accelerator card.

[0016] Furthermore, the matrix multiplication operation specifically involves: inputting the matrix size [M, N, K], performing calculations using INT8, FP16, and INT16 data types respectively, recording the calculation results and computation time, and obtaining the matrix multiplication performance. In the matrix multiplication operation, the number of operations is m*n*k*2.0 (where m is the number of rows in the left matrix, n is the number of columns in the right matrix, and k is the cumulative dimension). Since the actual computation time (elapsed_time) is in microseconds (µs), to maintain consistency in the calculation units, it needs to be divided by (1000*1024). The specific calculation method for computing power is as follows:

[0017] TF LOPS = (m × n × k × 2.0) / (elapsed_time × 1000 × 1024)

[0018] Furthermore, the peak computing power test uses the ratio of single-machine, single-card matrix multiplication performance to the nominal computing power of the tested accelerator card as the chip utilization rate. Multiple tests are conducted, and the matrix multiplication performance with the highest chip utilization rate is calculated as the peak computing power performance. The chip utilization rate is calculated as follows:

[0019] USE RATIO=(m × n × k × 2.0) / (elapsed_time × 1000 × 1024) / nominalvalue

[0020] Furthermore, the bandwidth test involves having multiple cores on the accelerator card transmit a certain amount of data (data_size) multiple times to test the minimum hardware time (hardware_time), thereby obtaining the maximum bandwidth of the entire accelerator card. The bandwidth calculation formula is as follows: bandwidth (GB / s) = data_size / hardware_time.

[0021] Furthermore, the inference single-machine single-card test includes the following test contents:

[0022] 1) Open source general model - image classification: Three tests were performed. In each test, the test samples were tested using the ResNet50 model under different data types and batch sizes. The results showed the Top1 accuracy, Top5 accuracy, sample processing performance per second, and energy efficiency.

[0023] The Top1 / Top5 accuracy is an evaluation metric for classification networks, representing the accuracy of the first / fifth ranked class in matching the actual results. It is obtained from the log output after executing the ResNet image classification model task.

[0024] The sample performance data processed per second is the maximum number of input instances that can be processed per unit of time. It can be obtained directly from the log output. Sample performance per second = (number of batches * batch size) / total time (in seconds).

[0025] The energy efficiency ratio is calculated using the sample performance data processed per second: Energy efficiency ratio = Sample performance data processed per second / Power consumption.

[0026] 2) Open source general model - object detection: Three tests were performed. In each test, the mAP accuracy, sample processing performance data per second, and energy efficiency ratio of the test sample using the YOLOv3 model under different data types and data batch sizes were measured.

[0027] 3) Open source general model - NLP: Perform three tests. In each test, the F1-Score accuracy, sample processing performance data per second, and energy efficiency ratio of the test sample using the BERT-BASE model under different data types and data batch sizes are measured.

[0028] Furthermore, the training single-machine four-card test includes the following test contents:

[0029] 1) Open source general model - image classification: The training time of the ResNet50 model to achieve a top-1 accuracy of ≥75% under single-machine four-card, data type FP32, fixed data quantity and hyperparameters was tested three times.

[0030] 2) Open source general model - object detection: The training time of the YOLOv5S model to achieve mAP@0.5 ≥53% accuracy under single-machine four-card, data type FP32, fixed data quantity and hyperparameters was tested three times.

[0031] 3) Open-source general-purpose model - NLP: The training time for the BERT-BASE model under the PyTorch framework, with a single machine, four GPUs, FP32 data type, fixed data volume, and hyperparameters, to achieve F1 accuracy ≥ 86.9% was tested three times. Beneficial effects: The designed testing and evaluation methods for the training and inference performance of AI accelerator card products allow for a fairer and more targeted evaluation of the basic hardware computing power, inference and training performance with model applications, power consumption, and interconnect bandwidth of AI accelerator card products mounted on servers in data centers. Simultaneously, considering the technical roadmap and current business situation in the power industry, the test of the AI ​​accelerator card's adaptability to mainstream power models and general-purpose models provides a reference for the localization of related AI basic hardware in the power industry. More intuitive and comparative evaluation results can be generated based on the test data. Attached Figure Description

[0032] Figure 1 This is the test flowchart described in this invention;

[0033] Figure 2 This is a schematic diagram of the scoring and evaluation index model according to an embodiment of the present invention. Detailed Implementation

[0034] The implementation process of the present invention will be described in detail below with reference to the embodiments and the accompanying drawings.

[0035] Figure 1 The following is a flowchart of the testing process described in this invention:

[0036] Step 1: Prepare the test environment. All servers equipped with accelerator cards should be of the same model and configuration.

[0037] Step 2: Prepare the necessary driver software and other preparatory work for the accelerator card.

[0038] Step 3: In order to test different application scenarios, basic computing power tests and three typical deep learning scenarios were conducted: image classification, object detection, and natural language processing.

[0039] Step 4: Matrix multiplication. Test the performance of matrix multiplication for different data types (INT8, FP16, INT16, etc.). Input matrix size [M, N, K], and perform calculations using INT8, FP16, and INT16 respectively. Record the calculation results and calculation time t (µs). Calculate the average hardware utilization efficiency based on the calculation time. In matrix multiplication, the number of operations is m*n*k*2.0 (where m is the number of rows in the left matrix, n is the number of columns in the right matrix, and k is the accumulation dimension). Since the actual calculation time (elapsed_time) is in µs, it needs to be divided by (1000*1024) to maintain the consistency of the calculation units. The specific calculation method for computing power is as follows:

[0040] TF LOPS = (m × n × k × 2.0) / (elapsed_time × 1000 × 1024)

[0041] Matrix size [m, n, k]: [1024, 1024, 1024], [2048, 2048, 2048].

[0042] Step 5: Peak computing power test. By setting different data types (INT8, FP16, INT16, etc.), select the matrix multiplication scales M, N, and K that maximize chip utilization, run matrix multiplication operations, and record the chip utilization. The ratio of the single-machine, single-card matrix multiplication performance to the nominal computing power of the tested accelerator card is used as the chip utilization rate. Multiple tests are conducted, and the matrix multiplication performance that maximizes chip utilization is calculated as the peak computing power performance. The chip utilization rate is calculated as follows:

[0043] USE RATIO=(m × n × k × 2.0) / (elapsed_time × 1000 × 1024) / nominalvalue

[0044] Step 6: Bandwidth Test: Select the Bandwidth tool from the general testing tool Benchmark and test the maximum communication bandwidth by setting different parameter combinations. Test the minimum hardware time (in microseconds) by having multiple cores on the accelerator card repeatedly transmit a certain amount of data (data_size). The maximum bandwidth of the entire accelerator card is then obtained. The bandwidth calculation formula is as follows: bandwidth (GB / s) = data_size / hardware_time.

[0045] Step 7: For the inference function acceleration card in the image classification scenario, the ImageNet2012 dataset and the ResNet50V1.1 model under the Tensorflow framework are adopted, with the accuracy constraints of top1>74.2% and top5>91.2%. Single machine and single card are used. The model is used for inference at accuracy INT8, FP16 and FP32 and different batch sizes (1 / 4 / 8 / 16 / 32 / 64). The accuracy of top1 and top5 inference, the sample performance data processed per second and power consumption are recorded.

[0046] Step 8: For the inference function acceleration stuck in the object detection scenario, the COCO2017 dataset and the YOLOv3 model under the Tensorflow framework were adopted, with the accuracy constraint mAP@0.5>53%. The model was inferred under INT8, FP16 and FP32 and different batch sizes (1 / 4 / 8 / 16 / 32 / 64), and the accuracy of the top 1 and top 5 inference, the sample performance data processed per second and the power consumption were recorded.

[0047] Step 9: For inference acceleration stuck in natural language processing scenarios, the SquADv1.1 dataset and the BERT-BASE model under the Tensorflow framework are adopted, with an accuracy constraint of F1>86.9%. The model is used for inference under INT8, FP16 and FP32 and different batch sizes (1 / 4 / 8 / 16 / 32 / 64), and the accuracy of the top 1 and top 5 inference, the sample performance data processed per second and the power consumption are recorded.

[0048] Step 10: For training the function acceleration card in the image classification scenario, the ImageNet2012 dataset and the ResNet50 model under the PyTorch framework are adopted. The accuracy constraint is top1>75%. The model starts from 0 with accuracy FP32. It is trained using a single machine with four cards. The accuracy of top1 and top5, the sample processing performance data per second, and the power consumption are recorded.

[0049] Step 11: For the training function acceleration card in the object detection scenario, the COCO2017 dataset and the YOLOv5 model under the PyTorch framework are adopted. The accuracy constraint is mAP@0.5>53%. The model starts from 0 with an accuracy of FP32. It is trained using a single machine with four cards. The accuracy of mAP@0.5, the sample processing performance data per second, and the power consumption are recorded.

[0050] Step 12: For training the function acceleration card in the natural language processing scenario, the SquADv1.1 dataset and the bert-base pre-trained model under the PyTorch framework are adopted. The accuracy constraint is F1>86.9%. The model starts from 0 with an accuracy of FP32. It is trained using a single machine with four cards. The accuracy of F1 and excat_match, the sample processing performance data per second, and the power consumption are recorded.

[0051] Figure 2 The scoring rules described in this invention are as follows:

[0052] Step 13: Basic performance scores are as follows:

[0053] 1) Matrix multiplication operation: INT16 / FP16 / INT8: 3 points for utilization rate > 50%; 2 points for utilization rate 20% - 50%; no points for utilization rate < 20%.

[0054] 2) Peak computing power test: INT16 / FP16 / INT8: Utilization ≥70%, get 3 points; Utilization 60-70%, get 2 points; 50%-60%, get 1 point; Utilization <50%, get 0 points.

[0055] 3) Communication bandwidth test: Ranking and scoring are based on the measured maximum throughput rate.

[0056] Step 14: The inference card scoring in the image classification scenario is as follows:

[0057] 1) ResNet50 model function: The accuracy of top1 ≥ 74.2% and top5 >= 91.2% is acceptable. Otherwise, no points will be awarded for the performance of the subsequent models. 1 point is added for every 0.2% of the top1 accuracy. No points are added for less than 0.2%. The maximum score is 10 points.

[0058] 2) ResNet50 model performance: Based on the functional requirements of the ResNet50 model, the sample performance data processed per second: scored according to the ranking, energy efficiency ratio: energy efficiency ratio scored according to the ranking.

[0059] Step 15: The inference card scoring in the object detection scene is as follows:

[0060] 1) YOLOv3 model functionality: An accuracy of mAP@0.5 ≥ 53.3% is required to pass; otherwise, no points will be awarded for subsequent model performance. 1 point will be added for every 0.3% increase in accuracy, and no points will be added for less than 0.3%, with a maximum of 10 points.

[0061] 2) YOLOv3 model performance: Based on the functional requirements of the YOLOv3 model, the sample performance data processed per second: scored according to ranking, energy efficiency ratio: energy efficiency ratio scored according to ranking.

[0062] Step 16: The inference card scoring in the natural language processing scenario is as follows:

[0063] 1) BERT-BASE model function: The accuracy reaches F1≥87% and exact_match≥79.7% to pass. Otherwise, no points will be awarded for the model performance. 1 point will be added for each 0.3% difference in accuracy between F1 and accuracy. No points will be added for less than 0.3%. The maximum score is 10 points.

[0064] 2) BERT-BASE model performance: Based on the functional requirements of the BERT-BASE model, the sample performance data processed per second: scored according to the ranking, energy efficiency ratio: energy efficiency ratio scored according to the ranking.

[0065] Step 17: The training card scores for the image classification scenario are as follows:

[0066] 1) ResNet50 model functionality: Top-1 accuracy ≥ 75% is acceptable; otherwise, no points will be awarded for subsequent model performance. 1 point is added for every 0.3% increase in accuracy, and no points are added for less than 0.3%, with a maximum of 10 points.

[0067] 2) ResNet50 model performance: Based on the functional requirements of the ResNet50 model, training time and power consumption are scored according to the ranking.

[0068] Step 18: The scoring of the training card in the object detection scene is as follows:

[0069] 1) Yolov5 model function: accuracy mAP@0.5 ≥53% is acceptable, otherwise no points will be awarded for the model performance. 1 point will be added for every 0.3% increase in accuracy, and no points will be added for less than 0.3%, with a maximum of 10 points.

[0070] 2) YOLOv5 model performance: Based on the functional requirements of the YOLOv5 model, training time and power consumption are scored according to the ranking.

[0071] Step 19: The training card scores in the natural language processing scenario are as follows:

[0072] 1) Bert-base model function: Accuracy F1 > 86.9% is acceptable, otherwise no points will be awarded for subsequent model performance. 1 point is added for every 0.2% increase in accuracy, and no points are added for less than 0.2%, with a maximum of 10 points.

[0073] 2) Bert-base model performance: Based on the functional requirements of the Bert-base model, training time and power consumption are scored according to the ranking.

[0074] Currently, the hardware computing power of various AI accelerator cards is evaluated independently by each manufacturer. The standards for evaluating computing power, the types of data used in the tests, and the data precision are all inconsistent. Furthermore, the evaluation methods, tools, and test datasets employed by each manufacturer differ. This results in a lack of cross-comparison of test results, making it difficult to intuitively compare the merits of different products.

[0075] Therefore, this method employs a unified hardware evaluation platform, a unified basic hardware computing power testing method, and a unified dataset. By leveling the baseline, unifying the methods, and conducting comparative analysis in the same direction, it achieves the testing of core hardware fundamental to artificial intelligence through a universal, unified, and fair evaluation approach. Through the designed evaluation index weights, a numerical value reflecting performance can be calculated based on the test results, making comparisons between similar systems more intuitive.

[0076] Considering target detection and recognition scenarios, which prioritize real-time efficiency and require timely data feedback, using the core computing power of AI accelerator cards as an evaluation metric is more instructive. However, for learning and training scenarios, which often employ multi-card, multi-machine collaborative approaches, the requirements for data efficiency and accuracy, training time, and energy consumption are higher. Therefore, comparing performance under a fixed framework and model is more instructive. Consequently, tests and evaluations conducted using this method better reflect the comparability of different products.

[0077] The designed testing and evaluation methods for the training and inference performance of AI accelerator cards enable a fairer and more targeted assessment of the basic hardware computing power, inference and training performance with model applications, power consumption, and interconnect bandwidth of AI accelerator cards deployed on servers in data centers. Furthermore, considering the technological roadmap and current business situation in the power industry, the testing of the compatibility of AI accelerator cards with mainstream power models and general-purpose models provides a reference for the localization of related AI infrastructure hardware in the power industry.

[0078] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for testing the training and inference performance of an artificial intelligence accelerator card product, characterized in that, Includes the following steps: (1) Prepare the test environment and provide a unified benchmark hardware testing platform to ensure that the platform configuration is the same and that the test baseline is consistent; (2) Coordinate the resources of the AI ​​products to be tested, complete the hardware preparation of the AI ​​accelerator card, the driver software preparation, and the model adaptation, and complete the installation package and test environment deployment required for test preparation; (3) Conduct tests according to the test plan, namely basic performance test, inference single machine single card test and training single machine four card test; (4) Scoring is conducted based on the test results, with reasoning and training scored separately. Specific scoring methods can be formulated according to the actual situation. The basic performance tests include matrix multiplication, peak computing power testing, and bandwidth testing. The matrix multiplication operation was tested using a single machine and a single graphics card, examining the performance of matrix multiplication for different data types. The peak computing power test: using a single machine and a single card, the matrix multiplication performance with the highest chip utilization is used as the peak computing power performance; The bandwidth test: tests the maximum throughput rate of DDR data read or write on the AI ​​accelerator card; The inference single-machine single-card test includes the following content: 1) Open source general model - image classification: Three tests were performed. In each test, the test samples were tested using the ResNet50 model under different data types and batch sizes. The results showed the Top1 accuracy, Top5 accuracy, sample processing performance per second, and energy efficiency. 2) Open source general model - object detection: Three tests were performed. In each test, the mAP accuracy, sample processing performance data per second, and energy efficiency ratio of the test sample using the YOLOv3 model under different data types and data batch sizes were measured. 3) Open source general model - NLP: Perform three tests. In each test, the F1-Score accuracy, sample processing performance data per second, and energy efficiency ratio of the test sample using the BERT-BASE model under different data types and data batch sizes are evaluated. The training single-machine four-card test includes the following content: 1) Open source general model - image classification: The training time of the ResNet50 model to achieve a top-1 accuracy of ≥75% under single-machine four-card, data type FP32, fixed data quantity and hyperparameters was tested three times. 2) Open source general model - object detection: The training time of the YOLOv5S model to achieve mAP@0.5 ≥53% accuracy under single-machine four-card, data type FP32, fixed data quantity and hyperparameters was tested three times. 3) Open source general model - NLP: The training time of the BERT-BASE model under the PyTorch framework with a single machine and four cards, data type FP32, fixed data quantity and hyperparameters to achieve F1 ≥ 86.9% accuracy was measured in three tests.

2. The method for testing the training and inference performance of an artificial intelligence accelerator card product according to claim 1, characterized in that, The matrix multiplication operation is specifically as follows: input matrix size [M, N, K], perform calculations using INT8, FP16 and INT16 data types respectively, record the calculation results and calculation time, and obtain the matrix multiplication performance.

3. The method for testing the training and inference performance of an artificial intelligence accelerator card product according to claim 1, characterized in that, The peak computing power test uses the ratio of the single-machine, single-card matrix multiplication performance to the nominal computing power of the accelerator card under test as the chip utilization rate. The matrix multiplication performance with the highest chip utilization rate is calculated through multiple tests and taken as the peak computing power performance.

4. The method for testing the training and inference performance of an artificial intelligence accelerator card product according to claim 1, characterized in that, The bandwidth test involves having multiple cores on the accelerator card transmit a certain amount of data (data_size) multiple times to test the minimum hardware time (hardware_time), thereby obtaining the maximum bandwidth of the entire accelerator card. The bandwidth calculation formula is as follows: bandwidth (GB / s) = data_size / hardware_time.

Images