Multi-dimensional evaluation method and device for AI hardware in large-model kilocalorie scenarios

Abstract

The present invention discloses a multi-dimensional evaluation method and device for AI hardware in a large-model thousand-card scenario, and belongs to the field of artificial intelligence technology. The method includes functional correctness test, performance test, stability test and resource consumption test; in the functional correctness test, evaluation is performed according to the loss value of the training process; in the performance test, evaluation is performed according to throughput, power consumption and video memory occupancy; in the stability test, evaluation is performed according to the number of failures and recovery time as well as power consumption and video memory occupancy; in the resource consumption test, evaluation is performed according to each indicator item monitored in the performance test and stability test, and the time series mean and standard deviation of the monitoring results of each indicator item. The present invention evaluates the actual capabilities of AI hardware under the software stack combination (model + framework + compiler), and can more comprehensively evaluate the support capabilities of AI chips for large models, making up for the problem of insufficient coverage of current evaluation schemes in the field of large models.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-dimensional evaluation method and device for AI hardware in a large-model kilocalorie scenario. Background Art

[0002] In the situation where many AI hardware manufacturers are competing for the throne, AI hardware evaluation is a must. First, AI hardware manufacturers need an objective coordinate system to measure their own innovation process; secondly, the implementation of the industry requires fair evaluation benchmarks and methods, and purchasers urgently need fair evaluations that adapt to scenario requirements to guide the actual construction of production environments. However, due to the high degree of heterogeneity and poor compatibility of AI software and hardware technology stacks, coupled with complex and changing application scenarios, the MLPerf evaluation project initiated by Turing Award winner David Patterson and top academic institutions allows each manufacturer to optimize for specific tasks to demonstrate their own capabilities, but it has limited guiding significance for users to actually use AI chips for model training, reasoning and other tasks. In addition, domestic evaluation solutions are also limited to the white paper stage, lacking a direct and practical solution, and AI chip evaluation is still in written and trivial projects and has not yet been systematized. Summary of the invention

[0003] In order to solve the problems existing in the prior art, the present invention provides the following technical solutions.

[0004] The first aspect of the present invention provides a multi-dimensional evaluation method for AI hardware in a large model thousand-calorie scenario, including: functional correctness test, performance test, stability test and resource consumption test;

[0005] In the functional correctness test, evaluation is performed based on the loss value of the training process;

[0006] For chips that have passed the functional correctness test, a performance test is conducted. In the performance test, evaluation is performed based on throughput, power consumption, and video memory usage. The performance test includes: a "3+2" combination test format is adopted. "3" means that the manufacturer independently selects three cluster sizes, which are used to examine the minimum machine configuration for training a model with a specified parameter quantity, the maximum machine configuration for training a model with a specified parameter quantity, and the machine configuration when the chip reaches the optimal performance during training. "2" means that the evaluation organization pre-sets two fixed cluster sizes of 256 cards and 1024 cards, and each enterprise runs the test under these two fixed cluster sizes.

[0007] In the stability test, evaluation is performed based on the number of failures and recovery time as well as power consumption and video memory occupancy;

[0008] In the resource consumption test, evaluation is performed based on the indicators monitored in the performance test and the stability test, as well as the time series mean and standard deviation of the monitoring results of each indicator.

[0009] Preferably, the functional correctness test includes: using fixed hyperparameters, fixed random numbers and fixed initial weights, and calculating the loss value of the chip during the training process under set initial conditions.

[0010] Preferably, in the functional correctness test, the evaluation based on the loss value of the training process includes: comparing the calculated loss value with a preset benchmark loss value; if the comparison result is within a preset error range, the functional correctness test requirements are met; otherwise, the functional correctness test requirements are not met.

[0011] Preferably, the functional correctness test requirements include: the loss value calculated in the first 10 steps is compared with the preset benchmark loss value using the numpy.isclose interface, using atol=1e-3 and rtol=2e-4 in the fp16 scenario; using atol=1e-2 and rtol=2e-3 in the bf16 scenario, and observing whether the result is true; in the first 5000 steps, the average relative error between the calculated loss value and the preset benchmark loss value is less than or equal to 2%.

[0012] Preferably, in the performance test, the evaluation based on throughput, power consumption and video memory occupancy includes:

[0013] In the performance test, two evaluation methods are set: absolute performance ratio and energy efficiency ratio, wherein the absolute performance ratio of A to B = throughput of the measured object A / throughput of the measured object B; energy efficiency ratio = throughput / power consumption;

[0014] The video memory usage is used to positively reflect the reliability of the performance test result. When the video memory usage is lower or higher than the preset value, the performance test result is considered unreliable.

[0015] Preferably, the stability test includes: on a model of a preset scale, the manufacturer selects a cluster of any scale, completes a training task with a preset parameter amount, ignores performance and accuracy, and counts the number of failures that occur during the completion of the training task with the preset parameter amount and the recovery time for each failure.

[0016] Preferably, in the stability test, the evaluation according to the number of failures and recovery time as well as power consumption and video memory occupancy includes:

[0017] For the measured objects A and B, if the number of failures of A is higher than that of B, then the stability of A is better than that of B; if the recovery time of A is shorter than that of B, then the stability of A is better than that of B;

[0018] The power consumption and video memory usage are used to positively reflect the reliability of the stability test results. If the power consumption obtained from the test exceeds the preset ratio of the thermal design power consumption, or the video memory usage is lower or higher than the preset value, the stability test results are considered unreliable.

[0019] A second aspect of the present invention provides a multi-dimensional evaluation device for AI hardware in a large model kilocalorie scenario, including: a functional correctness test module, a performance test module, a stability test module, and a resource consumption test module;

[0020] The functional correctness test module is used to evaluate according to the loss value of the training process in the functional correctness test;

[0021] The performance test module is used to perform performance tests on chips that have passed the functional correctness test. In the performance test, evaluation is performed based on throughput, power consumption and video memory occupancy. The performance test includes: using a "3+2" combination test format, "3" means that the manufacturer independently selects 3 cluster sizes, which are used to examine the minimum machine configuration for training a model with a specified parameter quantity, the maximum machine configuration for training a model with a specified parameter quantity, and the machine configuration when the chip reaches the optimal performance during training. "2" means that the evaluation agency pre-sets two fixed cluster sizes of 256 cards and 1024 cards, and each enterprise runs the test under these two fixed cluster sizes. The stability test module is used to perform evaluation based on the number of failures and recovery time as well as power consumption and video memory occupancy in the stability test.

[0022] The resource consumption test module is used to evaluate in the resource consumption test according to the various indicators monitored in the performance test and the stability test, as well as the time series mean and standard deviation of the monitoring results of each indicator.

[0023] The third aspect of the present invention provides a memory storing a plurality of instructions, which are loaded and executed by a processor to implement the multi-dimensional evaluation method of AI hardware in a large model thousand-calorie scenario as described in the first aspect.

[0024] The fourth aspect of the present invention provides an electronic device, comprising a processor and a memory connected to the processor, the memory storing a plurality of instructions, the instructions being loaded and executed by the processor so that the processor can execute the multi-dimensional evaluation method of AI hardware in a large model thousand-calorie scenario as described in the first aspect.

[0025] The beneficial effect of the present invention is that the solution provided by the present invention aims to establish an industry practice-oriented index system and evaluation method to evaluate the actual capabilities of AI hardware under the software stack combination (model + framework + compiler). The present invention covers evaluation methods for four major indicators to measure "whether the chip can support specific model training", namely functional correctness index test, performance index test, resource usage index test and stability index test. The multi-dimensional indicator testing scheme makes up for the problem of insufficient coverage of the current evaluation schemes on the market in the field of large models, and can more comprehensively evaluate the support capabilities of AI chips for large models. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flowchart of the multi-dimensional evaluation method of the AI ​​hardware in the large model kilocalorie scenario of the present invention;

[0027] Figure 2 This is a functional structure diagram of the multi-dimensional evaluation device for the AI ​​hardware described in the present invention in a large-model kilocalorie scenario. DETAILED DESCRIPTION

[0028] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0029] The method provided by the present invention can be implemented in the following terminal environment, and the terminal may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method described in the following embodiment.

[0030] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the entire terminal, and executes various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory.

[0031] The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets or instructions.

[0032] The display screen is used to display the user interface of each application.

[0033] In addition, those skilled in the art can understand that the structure of the above terminal does not constitute a limitation on the terminal, and the terminal may include more or fewer components, or combine certain components, or arrange the components differently. For example, the terminal also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, and a power supply, which will not be described in detail here.

[0034] Embodiment 1

[0035] like Figure 1 As shown, the embodiment of the present invention provides a multi-dimensional evaluation method for AI hardware in a large model thousand-calorie scenario, including: functional correctness test, performance test, stability test and resource consumption test;

[0036] S101, in the functional correctness test, evaluating according to the loss value of the training process;

[0037] S102, performing a performance test on the chips that have passed the functional correctness test, wherein the performance test is evaluated based on throughput, power consumption, and video memory occupancy;

[0038] S103, in the stability test, evaluating according to the number of failures and recovery time as well as power consumption and video memory occupancy;

[0039] S104, in the resource consumption test, evaluation is performed based on the indicators monitored in the performance test and the stability test, and the time series mean and standard deviation of the monitoring results of each indicator.

[0040] The functional correctness test includes: using fixed hyperparameters, fixed random numbers and fixed initial weights, and calculating the loss value of the chip during the training process under the set initial conditions.

[0041] In the functional correctness test, the evaluation based on the loss value of the training process includes: comparing the calculated loss value with a preset benchmark loss value. If the comparison result is within a preset error range, the functional correctness test requirements are met; otherwise, the functional correctness test requirements are not met.

[0042] Specifically, the functional correctness test requirements include: the loss values ​​calculated in the first 10 steps are compared with the preset benchmark loss values ​​using the numpy.isclose interface, using atol=1e-3 and rtol=2e-4 in the fp16 scenario; using atol=1e-2 and rtol=2e-3 in the bf16 scenario, and observing whether the result is true; in the first 5000 steps, the average relative error between the calculated loss value and the preset benchmark loss value is less than or equal to 2%. It should be noted that: if the loss scaler is too large when using fp16, resulting in a training step skip (step skip) + the loss scaler automatically shrinks, this step will not be counted, and the first 10 steps to calculate the effective loss value will be used as the basis, and the manufacturer's chip and the preset benchmark chip (such as NVIDIA's chip) have the same step number for the training step skip. The above atol represents the absolute tolerance, and rtol represents the relative tolerance. The relationship between them and the relative difference can be expressed as the following formula:

[0043] The absolute value of the difference between the calculated loss value and the preset benchmark loss value

[0044] ≤atol+rtol*the absolute value of the preset benchmark loss value

[0045] This formula is widely used for numerical comparison in frameworks such as PyTorch and TensorFlow. According to the numerical precision definition of fp16 and bf16, it can be seen that the effective digits of fp16 are 3 digits and the effective digits of bf16 are 2 digits. Therefore, when performing difference comparison, the absolute tolerance (atol) of fp16 is set to 1e-3, and the absolute tolerance of bf16 is set to 1e-2. The relative tolerance will be selected according to the size of the compared values, which is a relaxation of the absolute tolerance. In the large language model pre-training calculation task, the actual effective loss value does not exceed 16, so the rtol of fp16 is set to 2e-4 and the rtol of bf16 is set to 2e-3. This ensures that the absolute difference error of fp16 does not exceed 5e-3 (because if it exceeds 5e-3, it is rounded to 1e-2), and the absolute difference error of bf16 does not exceed 5e-2 (because if it exceeds 5e-2, it is rounded to 1e-1). The specific calculation is shown in Table 1.

[0046] Table 1

[0047]

[0048] In one embodiment of the present invention, the performance test includes: adopting a "3+2" combination test format, "3" means that the manufacturer independently selects 3 cluster sizes, which are respectively used to examine the minimum scale machine configuration for training a model with specified parameter quantities, the maximum scale machine configuration for training a model with specified parameter quantities, and the machine configuration when the chip achieves optimal performance during training; "2" means that the evaluation organization pre-sets two fixed cluster sizes of 256 cards and 1024 cards, and each company runs tests under these two fixed cluster sizes.

[0049] The present invention adopts the "3+2" combination test form to comprehensively examine the performance of different domestic chips, covering a large model at three levels of 7B, 34B, and 70B as test models, with a total of 15 test items, of which 8 are priority test items, which can fully evaluate the individual indicators of the chip. In the "3+2" combination test form, "3" means that the manufacturer can independently choose 3 cluster sizes: the first is that the manufacturer independently chooses the minimum operable corresponding cluster that can complete the task. This test is to examine the minimum machine configuration of the chip to train the specified parameter quantity model while ensuring correctness. The cluster scale reflects the basic ability to complete the parameter quantity model training task, including the processing capacity for complex computing loads, the degree of satisfaction of storage requirements, and the efficiency of communication between nodes. The performance results of the minimum scale configuration are the basis for evaluating the performance expansion performance of the chip in a large-scale cluster. The second is that the manufacturer independently chooses the largest scale cluster that can complete the task. This test is to examine the maximum scale machine configuration of the chip to train the specified parameter quantity model while ensuring correctness. The cluster size reflects the performance of the specified training task on the largest cluster in this competition, including the overall coordination effect of more computing resources, larger storage capacity and more efficient communication capabilities after the scale is increased, and it is also the synergistic performance of the networking solution and chip capabilities. The evaluation of the largest cluster configuration helps to understand the chip's ability to cope with more complex and higher machine scale requirements. It can also measure the ability to expand the training of the same task from a small cluster to a large cluster. The third type is that the manufacturer independently selects a cluster of scale that can reflect its optimal performance, that is, the optimal scale cluster. This test is to examine the configuration of the chip when the training achieves optimal performance. The test encourages each chip company to choose a cluster scale that can maximize the performance of the chip based on its own technical advantages, and complete the training test of the model on it to observe the highest throughput that the chip can achieve when completing the task. This test helps to reveal the extreme performance of different chips in supporting large models, and provides an important reference for subsequent chip cluster optimization. In the "3+2" combination test format, "2" means that the evaluation organization has pre-set two fixed cluster sizes of 256 cards and 1024 cards, which are built using mainstream server configurations. These two clusters represent typical scale conditions of hundreds of cards and thousands of cards, respectively. Each company is required to run tests under this benchmark, which can more objectively and impartially evaluate the horizontal scalability and performance strengths of different chip products in mainstream clusters. In the present invention, the manufacturer provides monitoring results of system information, server monitoring indicators, and GPU monitoring indicators during the performance test. Among them, the system information is the directly read information such as the server model, GPU model, and operating system kernel version being evaluated. The monitored indicators of the server include CPU occupancy, memory occupancy, and total machine power consumption. The monitored indicators of the GPU include GPU temperature, power consumption, video memory occupancy, and GPU utilization.FlagPerf provides monitoring scripts and result visualization tools for the above indicators. During the test, the manufacturer starts the monitoring script on all physical machines, records the time series of the server monitored indicators and the GPU monitored indicators, and uses the visualization tool to draw scatter plots and line graphs.

[0050] Specifically, in the performance test, evaluation based on throughput, power consumption and video memory occupancy includes:

[0051] In the performance test, two evaluation methods are set: absolute performance ratio and energy efficiency ratio, wherein the absolute performance ratio of A to B = throughput of the measured object A / throughput of the measured object B; energy efficiency ratio = throughput / power consumption;

[0052] The video memory usage is used to positively reflect the reliability of the performance test result. When the video memory usage is lower or higher than the preset value, the performance test result is considered unreliable.

[0053] Throughput is the absolute standard and core standard for measuring performance. If the throughput index of the object A under test is k times the throughput index of the object B under test, then the performance of A is k times that of B. If you need to examine the runtime cost, you need to calculate the energy efficiency ratio. The present invention also provides two comparative evaluation methods: absolute performance ratio and energy efficiency ratio, for customers in different scenarios. The video memory occupancy can usually positively reflect whether the performance test result is reliable. When the video memory occupancy is too low (usually less than 30%) or too high (usually higher than 99%), the present invention, in principle, believes that the performance test results are unreliable, and the training hyperparameters need to be adjusted to make the performance test results reliable.

[0054] In the present invention, throughput refers to the total amount of data used for training during the entire process of the evaluation task, and the unit is (tokens, i.e., word units);

[0055] Throughput refers to the ratio of throughput to the time taken to complete the entire process of the evaluation task, and the unit is (tokens / second, i.e., word unit / second);

[0056] The average single-card throughput refers to the ratio of the throughput to the number of AI chips used to carry the evaluation task, and the unit is tokens / (ProcessingUnit*second), abbreviated as tokens / (p*s) or tokens / p / s.

[0057] Power consumption refers to the real-time power collection of each AI hardware at a fixed frequency (such as times / 5 seconds) during the entire process of the evaluation task. The collection can be performed through a software monitoring program or a power meter. The unit is watt;

[0058] Video memory usage refers to the monitoring of the video memory occupied by the AI ​​chip at a fixed frequency (such as times / 5 seconds) during the entire evaluation task, and the calculation ratio with the calibrated video memory capacity of the AI ​​chip. The result is the video memory usage. Video memory usage is a percentage scalar with no unit.

[0059] In one embodiment of the present invention, the stability test includes: on a model of a preset scale, the manufacturer selects a cluster of any scale, completes a training task with a preset parameter amount, ignores performance and accuracy, and counts the number of failures that occur during the completion of the training task with the preset parameter amount and the recovery time of each failure.

[0060] Specifically, in the stability test, the evaluation based on the number of failures and recovery time as well as power consumption and video memory occupancy includes:

[0061] For the measured objects A and B, if the number of failures of A is higher than that of B, then the stability of A is better than that of B; if the recovery time of A is shorter than that of B, then the stability of A is better than that of B;

[0062] The power consumption and video memory usage are used to positively reflect the reliability of the stability test results. If the power consumption obtained from the test exceeds the preset ratio of the thermal design power consumption, or the video memory usage is lower or higher than the preset value, the stability test results are considered unreliable.

[0063] There is no absolute evaluation standard for stability testing, only relative evaluation standards. That is, it is meaningless to evaluate the stability of a certain object N as 1, 2, etc., and it can only be evaluated that the stability of the object N is better than or worse than the object M. Resource usage monitoring indicators such as power consumption and video memory usage are used to ensure the normal progress of stability testing. When the power consumption index exceeds 95% of the thermal design power consumption (TDP, the factory calibration attribute of the object under test), or the video memory usage is less than 30%, or the video memory usage is higher than 99%, the test is considered unreliable in principle, and the batch size and other training hyperparameters need to be adjusted before testing.

[0064] Among them, "fault" is defined as: any problem that affects the training process, including but not limited to software and hardware failures. In addition to the above evaluation methods, manufacturers are also required to submit the cluster size, networking solutions, monitoring information, etc. used to complete the stability test. It should be noted that the stability test evaluates the various types of failures and the number of recoveries that occur during the manufacturer's completion of the training task of the same data set, and has nothing to do with the size of the cluster. Considering that large-scale clusters have a high probability of failure, but the time to complete the task is fast; vice versa. Different cluster sizes have their own advantages and disadvantages. Therefore, in the solution of the present invention, the size of the cluster is not a key factor in determining stability. Therefore, manufacturers can freely choose the cluster size to complete the stability test task.

[0065] In the specific implementation process, in order to achieve the purpose of multi-dimensional evaluation of large-model kilowatt-hour scenarios, the implementation of the present invention adopts the following three steps: standard program development, obtaining preliminary results of small-scale clusters, and collecting kilowatt-hour results of participating manufacturers. The purpose of the standard program development is to complete the computing task of large model training in the NVIDIA HGX-H800 server cluster. Subsequently, a series of experiments on different scales of cards such as 8-16-32 are carried out to ensure that the above-mentioned standard program can support kilowatt-hour scenarios. The standard program and adaptation documents are then delivered to the manufacturer to collect the running results of each participating manufacturer on their respective kilowatt-hour clusters. During the implementation process, the solution provided by the present invention continues to expand the applicability of the evaluation software on domestic GPUs, domestic CPUs, domestic servers, and domestic operating systems. The solution of the present invention designs implementation tools at multiple levels and in different fields. Among them, the original and innovative part of this solution is the part of the FlagPerf evaluation tool that involves the evaluation of large-model kilowatt-hour scenarios. FlagPerf is an integrated AI hardware evaluation platform jointly built by Zhiyuan Research Institute, AI hardware manufacturers, AI framework manufacturers, and AI compiler manufacturers. It currently supports comprehensive evaluation of 13 AI chips and has the ability to evaluate 100-card clusters and 1,000-card clusters. FlagScale is a large model training, compression and push-integrated framework built on open source software. It supports a variety of domestic chips and heterogeneous mixed training capabilities, and is applied to the development of Zhiyuan large models. As an evaluation tool, FlagPerf works with FlagScale front-end and back-end to jointly undertake multiple evaluation projects. FlagPerf currently covers more than ten subdivided evaluation indicator dimensions such as basic specifications, functional correctness, and throughput performance; evaluation samples include mainstream open source large language models, computer vision, natural language processing, speech synthesis, recommendation systems, AI for Science and other ten core task areas. Currently, it supports more than 40 standard training and reasoning evaluation models such as LLaMA2-70B, Aquila2-70B, Baichuan2-13B, ChatGLM3-6B, covering more than 100 evaluation use cases. This solution submitted the part involving the large-model thousand-calorie scenario to the FlagPerf open source repository. During the implementation process, this solution uses FlagPerf, the FlagScale large-model training and promotion integrated framework, the Megatron-LM large-model distributed training framework, and the PyTorch artificial intelligence model programming framework. In addition, this solution also relies on a number of basic computer software and hardware during implementation, including Python 3.10 software, Docker 20.10.16 software, and Ubuntu 20.04 operating system. This solution uses the open source Wudao large-model training data set as input data, and uses the method described in this solution to conduct multi-dimensional evaluation of the large-model thousand-calorie scenario. The sample results obtained are shown in Table 2.

[0066] Table 2

[0067]

[0068]

[0069] Embodiment 2

[0070] like Figure 2 As shown, another aspect of the present invention also includes a functional module architecture that is completely consistent with the aforementioned method flow, that is, the embodiment of the present invention also provides a multi-dimensional evaluation device for AI hardware in a large model kilocalorie scenario, including:

[0071] Functional correctness test module, performance test module, stability test module and resource consumption test module;

[0072] The functional correctness test module 201 is used to evaluate according to the loss value of the training process in the functional correctness test;

[0073] The performance test module 202 is used to perform performance test on the chips that have passed the functional correctness test. In the performance test, the throughput, power consumption and video memory occupancy are evaluated. The performance test includes: using a "3+2" combination test format, "3" means that the manufacturer independently selects 3 cluster sizes, which are used to examine the minimum scale machine configuration for training the specified parameter model, the maximum scale machine configuration for training the specified parameter model, and the machine configuration when the chip reaches the optimal performance during training; "2" means that the evaluation organization pre-sets two fixed cluster sizes of 256 cards and 1024 cards, and each enterprise runs the test under these two fixed cluster sizes;

[0074] The stability test module 203 is used to evaluate the stability test according to the number of failures and recovery time as well as power consumption and video memory occupancy;

[0075] The resource consumption test module 204 is used to perform evaluation in the resource consumption test according to the indicators monitored in the performance test and the stability test, as well as the time series mean and standard deviation of the monitoring results of each indicator.

[0076] The device can be implemented through the multi-dimensional evaluation method of the AI ​​hardware in the large model kilocalorie scenario provided in the above-mentioned embodiment 1. The specific implementation method can be found in the description in embodiment 1 and will not be repeated here.

[0077] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the method described in the first embodiment.

[0078] The present invention also provides an electronic device, comprising a processor and a memory connected to the processor, wherein the memory stores a plurality of instructions, and the instructions can be loaded and executed by the processor so that the processor can execute the method described in the first embodiment.

[0079] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A multi-dimensional evaluation method for AI hardware in a large model kilocalorie scenario, characterized in that: include: Functional correctness testing, performance testing, stability testing, and resource consumption testing; In the functional correctness test, evaluation is performed based on the loss value of the training process; For chips that have passed the functional correctness test, a performance test is performed. In the performance test, evaluation is performed based on throughput, power consumption, and video memory occupancy. The performance test includes: using a "3+2" combination test format, "3" means that the manufacturer independently selects three cluster sizes, which are used to examine the minimum machine configuration for training a model with a specified parameter quantity, the maximum machine configuration for training a model with a specified parameter quantity, and the machine configuration when the chip reaches optimal performance during training; "2" means that the evaluation agency pre-sets two fixed cluster sizes of 256 cards and 1024 cards, and each company runs the test under these two fixed cluster sizes; during the performance test, the manufacturer provides monitoring results of system information, server monitoring indicators, and GPU monitoring indicators; during the test, the manufacturer starts monitoring scripts on all physical machines, records the time series of server monitored indicators and GPU monitored indicators; and uses visualization tools to draw scatter plots and line graphs; In the stability test, evaluation is performed based on the number of failures and recovery time as well as power consumption and video memory occupancy; In the resource consumption test, evaluation is performed based on the indicators monitored in the performance test and the stability test, as well as the time series mean and standard deviation of the monitoring results of each indicator; The functional correctness test includes: using fixed hyperparameters, fixed random numbers and fixed initial weights, and calculating the loss value of the chip during the training process under the set initial conditions; In the functional correctness test, the evaluation according to the loss value of the training process includes: comparing the calculated loss value with a preset reference loss value, and if the comparison result is within a preset error range, the functional correctness test requirement is met; otherwise, the functional correctness test requirement is not met; The functional correctness test requirements include: the loss value calculated in the first 10 steps is compared with the preset benchmark loss value using the numpy.isclose interface, using atol=1e-3 and rtol=2e-4 in the fp16 scenario; using atol=1e-2 and rtol=2e-3 in the bf16 scenario, and observing whether the result is true; in the first 5000 steps, the average relative error between the calculated loss value and the preset benchmark loss value is less than or equal to 2%.

2. The multi-dimensional evaluation method of AI hardware in a large model kilocalorie scenario as claimed in claim 1 is characterized in that: In the performance test, the evaluation based on throughput, power consumption and video memory usage includes: In the performance test, two evaluation methods are set: absolute performance ratio and energy efficiency ratio, wherein the absolute performance ratio of A to B = throughput of the measured object A / throughput of the measured object B; energy efficiency ratio = throughput / power consumption; The video memory usage is used to positively reflect the reliability of the performance test result. When the video memory usage is lower or higher than the preset value, the performance test result is considered unreliable.

3. The multi-dimensional evaluation method of AI hardware in a large model kilocalorie scenario as claimed in claim 1 is characterized in that: The stability test includes: on a model of a preset scale, the manufacturer selects a cluster of any scale, completes a training task with a preset parameter amount, ignores performance and accuracy, and counts the number of failures that occur during the training task with the preset parameter amount and the recovery time for each failure.

4. The multi-dimensional evaluation method of AI hardware in a large model kilocalorie scenario as claimed in claim 3 is characterized in that: In the stability test, the evaluation based on the number of failures and recovery time as well as power consumption and video memory occupancy includes: For the measured objects A and B, if the number of failures of A is higher than that of B, then the stability of A is better than that of B; if the recovery time of A is shorter than that of B, then the stability of A is better than that of B; The power consumption and video memory usage are used to positively reflect the reliability of the stability test results. If the power consumption obtained from the test exceeds the preset ratio of the thermal design power consumption, or the video memory usage is lower or higher than the preset value, the stability test results are considered unreliable.

5. A multi-dimensional evaluation device for AI hardware in a large model kilocalorie scenario, characterized in that: include: Functional correctness test module, performance test module, stability test module and resource consumption test module; The functional correctness test module is used to evaluate the functional correctness test according to the loss value of the training process; The functional correctness test includes: using fixed hyperparameters, fixed random numbers and fixed initial weights, and calculating the loss value of the chip during the training process under the set initial conditions; in the functional correctness test, the evaluation based on the loss value of the training process includes: comparing the calculated loss value with the preset benchmark loss value, if the comparison result is within the preset error range, the functional correctness test requirements are met; otherwise, the functional correctness test requirements are not met; the functional correctness test requirements include: the loss value calculated in the first 10 steps and the preset benchmark loss value use the numpy.isclose interface, use atol=1e-3 and rtol=2e-4 in the fp16 scenario; use atol=1e-2 and rtol=2e-3 in the bf16 scenario, and observe whether the result is true; in the first 5000 steps, the average relative error between the calculated loss value and the preset benchmark loss value is less than or equal to 2%; The performance test module is used to perform performance tests on chips that have passed the functional correctness test. In the performance test, evaluation is performed based on throughput, power consumption and video memory occupancy. The performance test includes: using a "3+2" combination test format, "3" means that the manufacturer independently selects 3 cluster sizes, which are used to examine the minimum machine configuration for training a model with a specified parameter quantity, the maximum machine configuration for training a model with a specified parameter quantity, and the machine configuration when the chip reaches optimal performance during training; "2" means that the evaluation agency pre-sets two fixed cluster sizes of 256 cards and 1024 cards, and each enterprise runs tests under these two fixed cluster sizes; during the performance test, the manufacturer provides monitoring results of system information, server monitoring indicators and GPU monitoring indicators; during the test, the manufacturer starts monitoring scripts on all physical machines to record the time series of server monitored indicators and GPU monitored indicators; and uses visualization tools to draw scatter plots and line graphs; The stability test module is used to evaluate the stability test according to the number of failures and recovery time as well as power consumption and video memory occupancy; The resource consumption test module is used to evaluate in the resource consumption test according to the indicators monitored in the performance test and the stability test, as well as the time series mean and standard deviation of the monitoring results of each indicator.

6. A memory, characterized in that: Multiple instructions are stored, and the instructions are used to implement the multi-dimensional evaluation method of AI hardware in a large model kilocalorie scenario as described in any one of claims 1 to 4.

7. An electronic device, characterized in that: It includes a processor and a memory connected to the processor, the memory stores multiple instructions, and the instructions can be loaded and executed by the processor, so that the processor can execute the multi-dimensional evaluation method of AI hardware in the large model thousand-calorie scenario as described in any one of claims 1-4.

Images