Multi-dimensional chip evaluation method, device and electronic equipment

Abstract

The application discloses a multi-dimensional chip evaluation method and device and electronic equipment, and belongs to the technical field of chip evaluation. The method comprises the following steps: managing a system through a main module, wherein the main module comprises a cluster management module, a container management module, a log module, an image management module and a system monitoring module; evaluating a chip to be evaluated through a sub-module to obtain a corresponding multi-dimensional evaluation result, wherein the sub-module comprises a training sub-module, an inference sub-module, a basic specification sub-module and an operator sub-module; and determining a comprehensive evaluation result of the chip to be evaluated based on the multi-dimensional evaluation result. The application has strong adaptability and can comprehensively reflect the performance of the chip and the ecological adaptability thereof.

Description

Technical Field

[0001] This application belongs to the field of chip evaluation technology, and specifically relates to multi-dimensional chip evaluation methods, devices and electronic equipment. Background Technology

[0002] Currently, AI chip evaluation technology has made significant progress in building multi-dimensional indicator systems, supporting multi-scenario tasks and multi-test environments, and ensuring test fairness and result reproducibility. However, current evaluation methods still have shortcomings in expanding indicator systems to cover more complex real-world application scenarios, optimizing task and operator adaptation processes, enhancing the correlation and comprehensiveness of results across multiple test environments, and ensuring the fairness of automated code review. Furthermore, their adaptability to emerging hardware ecosystems and frameworks is limited, resulting in some evaluations failing to fully reflect chip performance and its ecosystem adaptability.

[0003] To address the aforementioned issues, this application proposes a multi-dimensional chip evaluation method, apparatus, and electronic device. Summary of the Invention

[0004] To address the shortcomings of the prior art, this application provides a multi-dimensional chip evaluation method, apparatus, and electronic device, which solves the problem that the existing technology has limited adaptability to chip evaluation, resulting in some evaluations failing to fully reflect chip performance and its ecosystem adaptability.

[0005] The technical effect to be achieved in this application is accomplished through the following solution:

[0006] Firstly, this application provides a multi-dimensional chip evaluation method, the method comprising:

[0007] The system is managed through a main module, which includes a cluster management module, a container management module, a log module, an image management module, and a system monitoring module. The cluster management module provides unified management of the evaluation cluster machines, the container management module provides unified management of the evaluation cluster containers, the log module provides different log display colors for four modes—debugging, message, warning, and error—by integrating different levels of display tools, the image management module operates and maintains Docker images, and the system monitoring module protects processes and continuously collects server performance data.

[0008] The chip under evaluation is evaluated in modules to obtain corresponding multi-dimensional evaluation results. The modules include training module, inference module, basic specification module and operator module.

[0009] Based on the multi-dimensional evaluation results, the comprehensive evaluation result of the chip under evaluation is determined.

[0010] In some embodiments, prior to evaluating the chip to be evaluated module by module, the process includes:

[0011] The original evaluation indicators are determined, which include relevant indicators in four aspects: computing power, storage, interconnection, and energy consumption.

[0012] Obtain the published theoretical values ​​corresponding to each original evaluation index.

[0013] In some embodiments, the chip to be evaluated is evaluated in modules to obtain corresponding multi-dimensional evaluation results, including:

[0014] Determine the target evaluation index corresponding to the chip to be evaluated;

[0015] The chip under test is evaluated by training module, inference module, basic specification module and operator module to obtain multi-dimensional evaluation results of operators corresponding to the target evaluation index.

[0016] In some embodiments, determining the comprehensive evaluation result of the chip under evaluation based on multi-dimensional evaluation results includes:

[0017] Based on the multi-dimensional evaluation results of the operator and the published theoretical values, the comprehensive evaluation results of the chip under evaluation are determined.

[0018] The indicators corresponding to the comprehensive evaluation results include correctness indicators, performance indicators, stability indicators, resource indicators, and important proportion indicators.

[0019] In some embodiments, the correctness metric is associated with the loss value during the training process;

[0020] Performance metrics are related to throughput, power consumption, and video memory usage;

[0021] Stability metrics are related to the number of failures and recovery time, power consumption, and memory usage during training.

[0022] Resource metrics are associated with the performance metrics and the stability metrics;

[0023] The key proportional indicators are associated with the statistics of the accuracy indicators, performance indicators, stability indicators, and resource indicators.

[0024] In some embodiments, the evaluation examples include: a cornerstone model and a classic model; wherein...

[0025] The foundational model includes model code, parameter configuration, interfaces and modules, and event logs; user optimizations for the evaluation samples of the foundational model include: distributed communication, batch data volume, acceleration strategies, parallel strategies, and operator or computation graph optimizations; user optimizations for the evaluation samples of the foundational model do not include: quantization and sparse matrix calculations.

[0026] The classic model uses open-source code repositories, configurations, and datasets to ensure consistency in evaluation configurations.

[0027] In some embodiments, during model training, if the correctness metric of the classic model meets the target correctness metric provided by the open-source code repository, it means that the evaluation of the correctness metric has been passed.

[0028] In some embodiments, the correctness metric is evaluated for the foundational model in one of the following two ways:

[0029] Method 1:

[0030] Under the algorithm alignment rules, a training task with a fixed global batch size is completed on the cluster to be tested, and a baseline loss value is determined. If the relative error of the average single-step loss value of the cornerstone model from step t to step m during training is less than 2% compared with the baseline loss value, it is considered to have passed the evaluation of the correctness index; where t and m are both positive integers, and t is less than m, and m represents the maximum number of running steps.

[0031] Method 2:

[0032] Obtain the loss value of the foundation model in the first 10 steps of the training process. If the loss value of the first 10 steps is less than the baseline loss value in the first stage, continue to judge the loss value corresponding to the first 5000 steps.

[0033] Determine the baseline loss value for the second stage. If the relative error of the average single-step loss value of the foundation model in the first 5000 steps during training is less than 2% compared with the baseline loss value for the second stage, it means that the correctness metric has been evaluated.

[0034] Secondly, this application provides a multi-dimensional chip evaluation device, the device comprising: a main module and sub-modules, the main module comprising a cluster management module, a container management module, a log module, an image management module and a system monitoring module, the sub-modules comprising a training sub-module, an inference sub-module, a basic specification sub-module and an operator sub-module; the device is used to implement the aforementioned multi-dimensional chip evaluation method.

[0035] Thirdly, this application provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned multi-dimensional chip evaluation method.

[0036] The multi-dimensional chip evaluation method, apparatus, and electronic device provided in this application evaluate the chip under test by setting a main module and sub-modules, and obtain corresponding multi-dimensional evaluation results. The chip under test is comprehensively evaluated based on the multi-dimensional evaluation results. The multi-dimensional evaluation results are relatively comprehensive. The evaluation method has strong adaptability and can fully reflect the chip performance and its ecological adaptability. Attached Figure Description

[0037] To more clearly illustrate the embodiments of this application or the existing technical solutions, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of a multi-dimensional chip evaluation method in one embodiment of this application;

[0039] Figure 2 This is a schematic block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0040] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0041] It should be noted that, unless otherwise defined, the technical or scientific terms used in one or more embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in one or more embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0042] In addition to the functional correctness metric of "whether the chip can support specific model training", the evaluation system of this application also includes more dimensions of performance metrics, resource usage metrics, and ecosystem adaptation capability metrics.

[0043] The various non-limiting embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0044] First, refer to Figure 1 This application provides a detailed explanation of the multi-dimensional chip evaluation method.

[0045] This application provides a multi-dimensional chip evaluation method, the method comprising:

[0046] S1: The system is managed through the main module, which includes a cluster management module, a container management module, a log module, an image management module, and a system monitoring module. The cluster management module provides unified management of the evaluation cluster machines, the container management module provides unified management of the evaluation cluster containers, the log module provides different log display colors for four modes—debugging, message, warning, and error—by integrating different levels of display tools, the image management module operates and maintains Docker images, and the system monitoring module protects processes and continuously collects server performance data.

[0047] S2: The chip to be evaluated is evaluated in modules to obtain corresponding multi-dimensional evaluation results. The modules include training module, inference module, basic specification module and operator module.

[0048] S3: Based on the multi-dimensional evaluation results, determine the comprehensive evaluation result of the chip under evaluation.

[0049] The multi-dimensional chip evaluation method of this application evaluates the chip under test by setting up a main module and sub-modules to obtain corresponding multi-dimensional evaluation results. Based on the multi-dimensional evaluation results, the chip under test is comprehensively evaluated. The multi-dimensional evaluation results are relatively comprehensive. This evaluation method has strong adaptability and can fully reflect the chip performance and its ecological adaptability.

[0050] The modules in this application can be divided into four modules: training module, inference module, basic specification module, and operator module. They cover more than 30 classic models, more than 80 training examples, more than 10 basic specification examples, and nearly 60 operator examples in fields such as computer vision, natural language processing, speech, and multimodal computing, supporting the evaluation of the training and inference capabilities of AI hardware.

[0051] For example, the training task of the training module refers to the model training task using a certain number of machine nodes, typically interconnected using interconnection technology. In this training task scenario, PyTorch, TensorFlow, Megatron, and Deepspeed are supported.

[0052] For example, the inference task in the inference module refers to a model inference task performed using a certain number of machine nodes, typically interconnected via interconnect technology. Compared to the training task, the inference task does not change the model parameters, but only completes the forward pass of the model. In the inference task scenario, this device is compatible with inference acceleration engines from multiple chip manufacturers and training framework R&D teams to more flexibly connect the AI ​​hardware and software ecosystem, expanding the boundaries and efficiency of evaluation.

[0053] For example, the basic specification task of the basic specification module refers to using test cases to evaluate a certain characteristic of a certain number of machine nodes, including three types: single card, intra-machine, and inter-machine.

[0054] For example, the operator task in the operator sub-module refers to evaluating the actual performance of the chip using operators written in the Triton language. The operator task includes basic test cases commonly used in computer vision, large language models, and multimodal large models.

[0055] In some embodiments, the training module consists of six sub-modules: cluster configuration sub-module, test parameter setting sub-module, model loading and adaptation sub-module, data loading and preprocessing sub-module, training execution script sub-module, and performance monitoring sub-module. These sub-modules work closely together to meet the needs of different hardware environments and training tasks.

[0056] Through the collaborative work of the various sub-modules mentioned above, the training module provides end-to-end support from configuration to execution, exhibiting high efficiency, flexibility, and adaptability. Whether it's adapting to different hardware resources or addressing needs such as resuming training after interruptions and performance optimization during the training process, this module offers reliable solutions and is a key technical support for large-scale distributed deep learning model training.

[0057] In some embodiments, the inference module consists of six sub-modules: model loading and preprocessing sub-module, input data adaptation sub-module, inference execution sub-module, post-processing and result saving sub-module, inference execution script sub-module, and performance monitoring sub-module. Each sub-module works closely together to meet the needs of different hardware environments and inference tasks.

[0058] Through the collaborative work of the aforementioned sub-modules, the inference module provides end-to-end support from model loading to result generation, demonstrating high efficiency, flexibility, and adaptability. Whether it's adapting to multimodal inputs or optimizing performance in multi-hardware environments, this module offers reliable solutions and is a key technical support for large-scale deep learning model inference.

[0059] In some embodiments, the basic specifications module consists of three main sub-modules: a cluster information configuration sub-module, a two-type task startup method sub-module, and a basic specifications performance monitoring and logging sub-module.

[0060] Through the collaborative work of the aforementioned sub-modules, the basic specifications module provides comprehensive support for computing power, storage, and interconnect testing, exhibiting high adaptability and reliability. This module offers robust technical support for the execution of deep learning tasks, serving as a crucial infrastructure for ensuring efficient task execution.

[0061] In some embodiments, the operator submodule consists of three main submodules: cluster information configuration submodule, operator implementation and adaptation submodule, and operator performance monitoring and logging submodule.

[0062] Through the collaborative work of the various sub-modules described above, the operator module achieves comprehensive evaluation and optimization of the basic computational units in deep learning models, exhibiting high efficiency, flexibility, and adaptability. Whether adapting to different hardware resources or optimizing operator performance, this module provides users with reliable solutions and is a key technical support for improving the performance of deep learning models.

[0063] In some embodiments, prior to evaluating the chip to be evaluated module by module, the process includes:

[0064] The original evaluation indicators are determined, which include relevant indicators in four aspects: computing power, storage, interconnection, and energy consumption.

[0065] Obtain the published theoretical values ​​corresponding to each original evaluation index.

[0066] For example, the published theoretical values ​​may be the theoretical values ​​of various indicators corresponding to the chips determined by each manufacturer.

[0067] Specifically, computing power can include metrics such as FP64, FP32, TF32, FP16, BF16, and INT8; storage can include metrics such as main storage bandwidth and main storage capacity; interconnect can include metrics such as CPU-chip interconnect, intra-server P2P direct connection, intra-server MPI direct connection, cross-server P2P direct connection, and cross-server MPI direct connection; and power consumption can include metrics such as single-chip power consumption and single-server power consumption during computing power-bound operation; single-chip power consumption and single-server power consumption during storage-bound operation; and single-chip power consumption and single-server power consumption during interconnect-bound operation.

[0068] For example, for different chips to be evaluated, evaluation indicators can be selected according to the characteristics of the chip. This can yield comprehensive and accurate evaluation indicators, and thus comprehensive and accurate comprehensive evaluation results of the chips to be evaluated.

[0069] In some embodiments, the chip to be evaluated is evaluated in modules to obtain corresponding multi-dimensional evaluation results, including:

[0070] Determine the target evaluation index corresponding to the chip to be evaluated;

[0071] The chip under test is evaluated by training module, inference module, basic specification module and operator module to obtain multi-dimensional evaluation results of operators corresponding to the target evaluation index.

[0072] For example, the target evaluation index corresponding to the chip under test is determined based on the original evaluation index. Since each chip under test has different characteristics, it is necessary to determine the corresponding target evaluation index based on the characteristics of the chip under test. The target evaluation index is part or all of the original evaluation index.

[0073] For example, the computing power, storage, interconnect and power consumption of the chip under test are evaluated by multiple evaluation items. Each evaluation item can contain 3 result records: multi-dimensional evaluation results of operators (such as PyTorch operators) and published theoretical values ​​(the theoretical values ​​corresponding to the chip products).

[0074] Specifically, the multi-dimensional evaluation results of operators are obtained by implementing hardware-related interfaces and providing their own operating configurations. Each evaluation item will provide adjustment constraints for all its subordinate configurations. Users can freely adjust the relevant configurations within the constraints to achieve evaluation results that are more suitable for their own hardware characteristics. The multi-dimensional evaluation results of operators reflect the actual situation when used by upper-level users.

[0075] Specifically, when publishing theoretical values, relevant product specifications can be referenced, and the theoretical values ​​corresponding to all evaluation items can be filled in. Users can freely choose to publish or keep the theoretical values ​​confidential in this evaluation plan.

[0076] For example, energy consumption is presented through monitoring results. During the operation of each evaluation result record for each of the above evaluation items, this application will sample the energy consumption of the AI ​​chip and the overall server at fixed time intervals to form a time series of energy consumption values. In addition, this application will also sample the silent energy consumption of the AI ​​chip and the overall server over a period of time.

[0077] In some embodiments, determining the comprehensive evaluation result of the chip under evaluation based on multi-dimensional evaluation results includes:

[0078] Based on the multi-dimensional evaluation results of the operator and the published theoretical values, the comprehensive evaluation results of the chip under evaluation are determined.

[0079] The indicators corresponding to the comprehensive evaluation results include correctness indicators, performance indicators, stability indicators, resource indicators, and important proportion indicators.

[0080] In some embodiments, the correctness metric is associated with the loss value during the training process;

[0081] Performance metrics are related to throughput, power consumption, and video memory usage;

[0082] Stability metrics are related to the number of failures and recovery time, power consumption, and memory usage during training.

[0083] Resource metrics are associated with the performance metrics and the stability metrics;

[0084] The key proportional indicators are associated with the statistics of the accuracy indicators, performance indicators, stability indicators, and resource indicators.

[0085] For example, the correctness metric is used to test the correctness of the system's output, usually based on whether the task is completed correctly;

[0086] For example, the main performance evaluation metrics include the following:

[0087] Average single-card throughput refers to the average amount of data or computational operations that each chip can process per unit of time (per second), reflecting the speed or efficiency of a computer chip or processor in performing computational tasks. Generally, a higher value for this metric means that the chip can execute computational tasks or process data faster, achieving better performance in high-performance computing, data center workloads, and artificial intelligence computing scenarios such as scientific computing.

[0088] For large language models, training data is in units of tokens; therefore, the average throughput per card is measured in units of tokens. <tokens p s>.

[0089] Average throughput per unit of computing power refers to the amount of data or computational operations that a chip can process or perform per unit of time (per second) on average, reflecting the efficiency and performance of hardware devices when processing large deep learning models. The higher this metric, the faster the hardware can complete model training tasks.

[0090] FLOPS (floating point operations per second) refers to the number of floating-point operations per second, which can be understood as computing speed and is a metric for measuring hardware performance.

[0091] For large language models, training data is in units of tokens; therefore, the average throughput per unit of computing power is... <tokens tflops s>.

[0092] Model FLOPS Utilization (MFU): This refers to the utilization rate of floating-point operations in large-scale model training. This metric reflects the effective utilization of hardware resources when performing floating-point operations on the model. A higher MFU means that the hardware is fully utilizing its floating-point computing power to perform the model's computational tasks.

[0093] MFU = Model computational cost / Actual computational cost used;

[0094] The computational cost of a model is usually denoted as FLOPS, which refers to the theoretical amount of computation required to complete a training task, i.e., the number of floating-point operations per second. It is usually used to describe the complexity of an algorithm or model.

[0095] Cluster total throughput = throughput / total time;

[0096] Cluster computing throughput = throughput / computing time;

[0097] Total training throughput of the cluster = throughput / training time.

[0098] For example, the monitoring results corresponding to power consumption may include system power consumption and single card power consumption; wherein, average power consumption represents the average power consumption of the system during the test, maximum power consumption represents the maximum power consumption reached by the system during the test, and power consumption standard deviation represents the degree of power consumption fluctuation.

[0099] In some embodiments, the evaluation examples include: a cornerstone model and a classic model; wherein...

[0100] The foundational model includes model code, parameter configuration, interfaces and modules, and event logs; user optimizations for the evaluation samples of the foundational model include: distributed communication, batch data volume, acceleration strategies, parallel strategies, and operator or computation graph optimizations; user optimizations for the evaluation samples of the foundational model do not include: quantization and sparse matrix calculations.

[0101] The classic model uses open-source code repositories, configurations, and datasets to ensure consistency in evaluation configurations.

[0102] In some embodiments, during model training, if the correctness metric of the classic model meets the target correctness metric provided by the open-source code repository, it means that the evaluation of the correctness metric has been passed.

[0103] In some embodiments, the correctness metric is evaluated for the foundational model in one of the following two ways:

[0104] Method 1:

[0105] Under the algorithm alignment rules, a training task with a fixed global batch size is completed on the cluster to be tested, and a baseline loss value is determined. If the relative error of the average single-step loss value of the cornerstone model from step t to step m during training is less than 2% compared with the baseline loss value, it is considered to have passed the evaluation of the correctness index; where t and m are both positive integers, and t is less than m, and m represents the maximum number of running steps.

[0106] For example, the relative error of the average single-step loss value of the foundational model from step t to step m during training is less than 2%, which can be expressed by the following formula:

[0107] ;

[0108] in, This represents the loss value corresponding to each step. This represents the baseline loss value corresponding to each step of the foundation model, where i is a positive integer. There are multiple baseline loss values, which may be the same or different.

[0109] Method 2:

[0110] Obtain the loss value of the foundation model in the first 10 steps of the training process. If the loss value of the first 10 steps is less than the baseline loss value in the first stage, continue to judge the loss value corresponding to the first 5000 steps.

[0111] For example, in the FP16 scenario, atol=1e-3 and rtol=2e-4 can be used; in the BF16 scenario, atol=1e-2 and rtol=2e-3 can be used, where atol represents the absolute tolerance and rtol represents the relative tolerance.

[0112] Determine the baseline loss value for the second stage. If the relative error of the average single-step loss value of the foundation model in the first 5000 steps during training is less than 2% compared with the baseline loss value for the second stage, it means that the correctness metric has been evaluated.

[0113] For example, the relative error of the average single-step loss value of the foundation model in the first 5000 steps during training is less than 2%, which can be expressed by the following formula.

[0114] ;

[0115] in, This represents the loss value corresponding to each step. This represents the second-stage baseline loss value corresponding to each step of the foundation model, where i is a positive integer. There are multiple second-stage baseline loss values, which may be the same or different.

[0116] For example, taking the classic LLaMA2-7B model as an example, the standard evaluation examples provided in this application use Megatron-DeepSpeed ​​as the training optimization library for large-scale deep learning models to improve the performance and scalability of large-scale models. The test dataset uses OpenWebText, and other parameter settings are the same as those published by the researchers of the LLaMA2-70B model. When using this as an evaluation example, users can adapt and optimize it according to their own needs.

[0117] This application includes mainstream open-source cornerstone large language models LLaMA1-7B / 13B and LLaMA2-7B / 70B, mainstream domestic cornerstone language models Aquila2-7 / 34 / 70B, Baichuan2-13B, and ChatGLM3-6B, as well as the multimodal model StableDiffusion. It includes pre-trained, fine-tuned, and inference evaluation examples for large models. During this period, the scope and number of evaluation example models are continuously expanding, especially focusing on open-source models that meet the concerns of users with computing power, including but not limited to covering more domestic cornerstone language models (LLM), cornerstone multimodal models (LMM), and hybrid expert models (MoE).

[0118] In the evaluation scenario of the cornerstone model in this application, two different evaluation rules are set: algorithm alignment rule and performance alignment rule.

[0119] Specifically, the algorithm alignment rule requires users to freely change parameters such as gradient accumulation steps and local batch size, complete a training task with a fixed global batch size on the test cluster, and additionally determine the functional correctness of the results. Evaluation requires that functional correctness verification be completed before the algorithm alignment rule test is considered complete, and only then will subsequent performance evaluations and other evaluation dimensions have practical significance.

[0120] Specifically, the performance alignment rule requires the participants to fix the gradient accumulation steps and freely expand the global batch size according to the size of the cluster under test, without needing to determine functional correctness. For any user evaluating any large model, completing the evaluation of the algorithm alignment rule is a prerequisite for evaluating the performance alignment rule.

[0121] For example, in the testing of the classic model, the model was fully trained and its correctness verified using a common training cluster of 8 GPUs on a single machine. The chip's support for model operation was verified by testing its convergence during model training and its compliance with the target correctness metrics provided by the open-source code repository. In addition, training time of approximately 2 hours was set for both single-machine single-GPU and dual-machine 16-GPU models to confirm that their convergence trends were the same as those of the single-machine 8-GPU model.

[0122] The device in this application integrates dozens of classic models that are frequently used in the research and industrialization of artificial intelligence as evaluation examples for a complete training task.

[0123] Specifically, we will use the ResNet model as an example. This model is a deep learning convolutional neural network (CNN) structure primarily used to solve classification tasks in the visual domain. Since its inception, numerous variant models with different parameter scales have emerged. This application uses the ResNet50 model with 20 million parameters for testing. The test dataset is ImageNet2012, containing over 1 million images across 1000 different categories, primarily used for image classification and object detection tasks. Users can directly pull the code and dataset for evaluation without requiring extensive and complex adjustments or adaptations.

[0124] Examples of the correctness metrics for the classic model evaluation samples supported by this application are shown in the table below:

[0125]

[0126] To comprehensively cover evaluation aspects such as AI chip diversity, scalability, and practical application simulation, this application sets up three test environments: "single machine with single card", "single machine with 8 cards", and "dual machine with 16 cards", as detailed below:

[0127] Single-machine, single-card setups are suitable for small-scale training tasks, testing the chip's performance on small-scale classic model training tasks. Furthermore, the performance data from single-machine, single-card training tasks will serve as a benchmark for calculating speedup ratios in multi-card, multi-machine scenarios.

[0128] The single-machine 8-card setup is suitable for training medium-sized models such as large convolutional neural networks, small or medium-sized multi-head self-attention networks, and tests the chip's performance on medium-sized model pre-training tasks.

[0129] The dual-machine 16-card configuration is suitable for simulating complex multi-machine scenarios under resource-constrained conditions, and for testing the chip's performance on large-scale complex model pre-training tasks.

[0130] The cross-server MPI direct connection computing involved in this application includes the following:

[0131] First, in cross-server MPI direct connection computing, the three commonly used performance metrics are as follows:

[0132] samples / s (algbw): This metric measures the number of samples processed per second, representing the algorithm's bandwidth. This metric reflects the algorithm's computational efficiency.

[0133] busbw: This metric represents bus bandwidth, measuring the data transfer rate on the system bus. This metric is crucial for understanding the communication efficiency between different parts of the system.

[0134] busbw×2: This metric is an extension of busbw, taking into account bidirectional data transmission; it doubles the bus bandwidth to reflect the system's full-duplex capability.

[0135] For example, the reasons for choosing busbw as the primary metric include, but are not limited to:

[0136] 1) Rank-independent comparison: The busbw, calculated using the algorithm bandwidth formula, reflects the speed of inter-GPU communication. This bus bandwidth allows for comparison with the hardware peak bandwidth, regardless of the number of ranks used.

[0137] 2) Supports horizontal comparison of different MPI modes: The busbw metric allows horizontal comparison of MPI operations of different modes, such as all-gather, all-reduce, reduce-scatter, etc.

[0138] For example, the formula derivation includes:

[0139] 1. Definition of Algorithm Bandwidth: Algorithm bandwidth (algbw) is defined as the ratio of the number of samples S processed to the time required. The ratio:

[0140] ;

[0141] Where S represents the number of samples processed. Indicates the required time.

[0142] 2. Optimal time T for AllReduce operation:

[0143] ;

[0144] Where S represents the number of samples processed, n represents the number of ranks, and B represents the bandwidth of each rank.

[0145] 3. Rearrange the formula to solve for bandwidth B. Rearranging the above formula, we can obtain:

[0146]

[0147] Therefore, in order to obtain an AllReduce bandwidth measurement that can be compared with the hardware peak bandwidth, bandwidth B is calculated:

[0148]

[0149] Second, the theoretical computing power (theoretical FLOPS) of a single node involved in the unified calculation formula for metrics such as computing power (FP64, FP32, TF32, FP16, BF16, and INT8) can be expressed as:

[0150]

[0151] Where C represents the number of cores, F represents the frequency per core (Hz), and O represents the number of floating-point operations per cycle.

[0152] The multi-dimensional chip evaluation method proposed in this application can quickly and efficiently adapt and evaluate AI hardware, solving the challenges currently faced by AI hardware, such as poor compatibility, high degree of heterogeneity in technology stacks, and complex and ever-changing application scenarios.

[0153] This application's multi-dimensional chip evaluation method only allows manufacturers to adjust code strongly related to the hardware, ensuring the fairness of the evaluation tool and enabling rapid evaluation of AI hardware using new models, methods, and operators. Furthermore, the evaluation-related code is publicly available and transparent, guaranteeing the impartiality of the evaluation. The modular design of the evaluation program facilitates user adaptation. For chips, this application provides users with an objective, fair, and comprehensive view of chip capabilities, offering professional data reports, supporting out-of-the-box retesting to ensure the reliability of evaluation results, and allowing users to assess the capabilities of AI hardware in different domain scenarios using methods from ten years ago, eliminating the need for adaptation and distributed solution determination processes—it's ready to use out of the box.

[0154] This application's multi-dimensional chip evaluation method continuously expands evaluation capabilities and improves evaluation standards by incorporating user feedback, thereby promoting practical standards for AI hardware development and participating in the development of testing methods for AI chips in areas such as basic models and large models.

[0155] This application also provides a multi-dimensional chip evaluation device, the device comprising: a main module and sub-modules, the main module comprising a cluster management module, a container management module, a log module, an image management module and a system monitoring module, the sub-modules comprising a training sub-module, an inference sub-module, a basic specification sub-module and an operator sub-module; the device is used to implement the multi-dimensional chip evaluation method described in any of the preceding claims.

[0156] The multi-dimensional chip evaluation device provided in this application can achieve all the technical effects of the aforementioned multi-dimensional chip evaluation method, which will not be elaborated here.

[0157] It should be noted that the methods of one or more embodiments of this application can be executed by a single device, such as a computer or server. The methods of this embodiment can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the methods of one or more embodiments of this application, and these multiple devices will interact with each other to complete the multi-dimensional chip evaluation method.

[0158] It should be noted that the above description describes specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0159] Based on the same inventive concept, and corresponding to any of the above embodiments, this application also discloses an electronic device.

[0160] Specifically, Figure 2 This diagram illustrates the hardware structure of an electronic device for multi-dimensional chip evaluation provided in this embodiment. The device may include: a processor 410, a memory 420, an input / output interface 430, a communication interface 440, and a bus 450. The processor 410, memory 420, input / output interface 430, and communication interface 440 are interconnected internally via the bus 450.

[0161] The processor 410 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0162] The memory 420 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 420 can store the operating system and other applications. When the technical solutions provided in the embodiments of this application are implemented by software or firmware, the relevant program code is stored in the memory 420 and is called and executed by the processor 410.

[0163] Input / output interface 430 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0164] The communication interface 440 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (e.g., USB, Ethernet cable, etc.) or wireless means (e.g., mobile network, WIFI, Bluetooth, etc.).

[0165] Bus 450 includes a pathway for transmitting information between various components of the device, such as processor 410, memory 420, input / output interface 430, and communication interface 440.

[0166] It should be noted that although the above-described device only shows the processor 410, memory 420, input / output interface 430, communication interface 440, and bus 450, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this application, and not necessarily all the components shown in the figures.

[0167] The electronic devices described above are used to implement the corresponding multi-dimensional chip evaluation methods in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0168] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, one or more embodiments of this application also provide a computer-readable storage medium storing computer instructions for causing the computer to execute the multi-dimensional chip evaluation method as described in any of the above embodiments.

[0169] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0170] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the multi-dimensional chip evaluation method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0171] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of one or more embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0172] Additionally, to simplify the description and discussion, and to avoid obscuring one or more embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring one or more embodiments of this application, and this also takes into account the fact that the details of implementation of these block diagram apparatuses are highly dependent on the platform on which one or more embodiments of this application will be implemented (i.e., these details should be fully within the understanding of those skilled in the art). While specific details (e.g., circuits) are set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that one or more embodiments of this application may be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0173] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0174] One or more embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of one or more embodiments of this application should be included within the protection scope of this application.< / tokens> < / tokens>

Claims

1. A multi-dimensional chip evaluation method, characterized in that, The method includes: The system is managed through a main module, which includes a cluster management module, a container management module, a log module, an image management module, and a system monitoring module. The cluster management module provides unified management of the evaluation cluster machines, the container management module provides unified management of the evaluation cluster containers, the log module provides different log display colors for four modes—debugging, message, warning, and error—by integrating different levels of display tools, the image management module operates and maintains Docker images, and the system monitoring module protects processes and continuously collects server performance data. The chip under evaluation is evaluated by dividing it into modules to obtain corresponding multi-dimensional evaluation results. The modules include a training module, an inference module, a basic specification module, and an operator module. Based on the multi-dimensional evaluation results, the comprehensive evaluation result of the chip under evaluation is determined; The evaluation examples include: the cornerstone model and the classic model; among them, The foundational model includes model code, parameter configuration, interfaces and modules, and event logs; user optimizations for the evaluation samples of the foundational model include: distributed communication, batch data volume, acceleration strategies, parallel strategies, and operator or computation graph optimizations; user optimizations for the evaluation samples of the foundational model do not include: quantization and sparse matrix calculations. The classic model uses open-source code repositories for its code, configuration, and dataset to ensure consistency in the evaluation configuration. During model training, if the correctness metric of the classic model meets the target correctness metric provided by the open-source code repository, it means that the correctness metric has been evaluated. For the aforementioned foundational model, the correctness metric is evaluated using one of the following two methods: Method 1: Under the algorithm alignment rules, a training task with a fixed global batch size is completed on the cluster to be tested, and a baseline loss value is determined. If the relative error of the average single-step loss value of the cornerstone model from step t to step m during training is less than 2% compared with the baseline loss value, it is considered to have passed the evaluation of the correctness index; where t and m are both positive integers, and t is less than m, and m represents the maximum number of running steps. During training, the relative error of the average single-step loss value of the foundational model from step t to step m is less than 2%, as expressed by the following formula: ； in, This represents the loss value corresponding to each step. This represents the baseline loss value corresponding to each step of the cornerstone model, where i is a positive integer. There are multiple baseline loss values, which may be the same or different. Method 2: Obtain the loss value of the foundation model in the first 10 steps of the training process. If the loss value of the first 10 steps is less than the baseline loss value in the first stage, continue to judge the loss value corresponding to the first 5000 steps. Determine the baseline loss value for the second stage. If the relative error of the average single-step loss value of the foundation model in the first 5000 steps during training is less than 2% compared with the baseline loss value for the second stage, it means that the correctness index has been passed. During training, the relative error of the average single-step loss value of the foundational model in the first 5000 steps is less than 2%, as expressed by the following formula: ； in, This represents the loss value corresponding to each step. This represents the second-stage baseline loss value corresponding to each step of the foundation model, where i is a positive integer. There are multiple second-stage baseline loss values, which may be the same or different.

2. The multi-dimensional chip evaluation method according to claim 1, characterized in that, Before evaluating the chip to be tested in modules, the process includes: The original evaluation indicators are determined, which include relevant indicators in four aspects: computing power, storage, interconnection, and energy consumption. Obtain the published theoretical values ​​corresponding to each original evaluation index.

3. The multi-dimensional chip evaluation method according to claim 2, characterized in that, By evaluating the chips under test in modules, corresponding multi-dimensional evaluation results are obtained, including: Determine the target evaluation index corresponding to the chip to be evaluated; The chip under test is evaluated by training module, inference module, basic specification module and operator module to obtain multi-dimensional evaluation results of operators corresponding to the target evaluation index.

4. The multi-dimensional chip evaluation method according to claim 3, characterized in that, The determination of the comprehensive evaluation result of the chip under evaluation based on multi-dimensional evaluation results includes: Based on the multi-dimensional evaluation results of the operator and the published theoretical values, the comprehensive evaluation results of the chip under evaluation are determined. The indicators corresponding to the comprehensive evaluation results include correctness indicators, performance indicators, stability indicators, resource indicators, and important proportion indicators.

5. The multi-dimensional chip evaluation method according to claim 4, characterized in that, The accuracy metric is correlated with the loss value during the training process; Performance metrics are related to throughput, power consumption, and video memory usage; Stability metrics are related to the number of failures and recovery time, power consumption, and memory usage during training. Resource metrics are associated with the performance metrics and the stability metrics; The key proportional indicators are associated with the statistics of the accuracy indicators, performance indicators, stability indicators, and resource indicators.

6. A multi-dimensional chip evaluation device, characterized in that, The device includes a main module and sub-modules. The main module includes a cluster management module, a container management module, a log module, an image management module, and a system monitoring module. The sub-modules include a training sub-module, an inference sub-module, a basic specification sub-module, and an operator sub-module. The device is used to implement the multi-dimensional chip evaluation method as described in any one of claims 1 to 5.

7. An electronic device, the electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, it implements the multi-dimensional chip evaluation method as described in any one of claims 1 to 5.

Images